Book Summary of The Signal and the Noise — Why So Many Predictions Fail But Some Don’t

“The Signal and the Noise — Why So Many Predictions Fail But Some Don’t” is a book by Nate Silver, the creator of the predictions website 538. In this book Silver explores the common reasons that humans forecast badly and techniques we can employee to make better forecasts. He does this by exploring a diversity of forecasting domains ranging from weather prediction to economic forecasting. This summary will be organized chapter by chapter, and will finally conclude with a summary of my personal major takeaways from the book.

Chapter 1: The Catastrophic Failure of Prediction

Chapter 1 is about the 2008 housing crisis that sent the global economy spiraling. Silver argues that the 2008 housing crisis can be viewed as a failure of prediction. There were four parties involved which each made bad forecasts due to underlying bad assumptions.

• Home owners assumed house prices would always go up, and so it was better to buy now at an unaffordable price than wait two years for an even more unaffordable price.
• Rating agencies and banks assumed that mortgage backed securities were safe because after all, people paid their mortgages… right?
• Economic analysts assumed that even if the housing market crashed it could not cause a global economic melt down because the portion of GDP which housing accounts for is too small. But they did not account for the fact that banks were leveraged on mortgage backed securities at a multiplier rate of 50x!
• Finally, politicians assumed that recovery from the crisis would be V shaped. This was based on evidence from recent recessions in which the economy typically rebounds quickly after a recession.

In each of these cases a party made a bad forecast because the data they considered was too narrow. If homeowners looked at house pricing data before World War 2, they would see that housing prices have not historically always gone up, if rating agencies cracked open the bundled up mortgage backed securities they would see they were full of crap and if economists looked at secondary effects of a housing crash they would have seen the leveraged up banks.

The name of this type of forecasting error is “Out of Sample” error. Basically, it happens when you make a prediction that seems reasonable based on a narrow view of the world, but totally falls apart once you consider a broader set of data. The solution to avoid this type of error is to broaden your perspective on the things you consider.

Chapter 2: Are You Smarter Than a Television Pundit?

Chapter 2 explores just how often television pundits are correct. And the short of it is, they are really not very good at making forecasts. The problem with television pundits is they succeed by making good entertainment and good entertainment does not make for good forecasts. Good forecasts entail high error bars, lots of uncertainty, lots of detail and most of the time good forecasts go along with the consensus of the crowds (not always but most of the time). These traits of good forecasting make for boring television. The pundit that makes far reaching predictions in a bold and decisive way is going to get way more viewership than the one that forecasts what most folks already believe and includes large error bars in their prediction.

In order to illustrate the nature of how pundits are typically bad at forecasting, Silver breaks down forecasters into two categories — the foxes and the hedgehogs. Foxes represent good forecasters and hedgehogs represent bad forecasters. Silver argues that most pundits on televisions are hedgehogs not foxes, and explores the reasons why foxes are better at making predictions —

• Multidisciplinary: Foxes make their forecasts by bringing together ideas from multiple disciplines. Whereas, hedgehogs typically have a single disciplinary lens through which they model the world.
• Adaptable: Foxes change there forecasts as they get new data. Whereas, hedgehogs stick to their beliefs even as the data suggests they should pivot.
• Tolerant of Complexity: Foxes accept that the world is a complex and messy place. As a result foxes do not assume that forecasts can be reduced down to simplistic mental models. Whereas, hedgehogs assume the world is governed by straightforward models and those models can be used to forecast with high accuracy.
• Self-Critical: Foxes are self critical in their forecasts and accept blame when they made a forecasting error. Whereas, hedgehogs will blame the failure of their forecasts on bad luck.
• Cautious: Foxes express their forecasts in terms of probabilities. Whereas, hedgehogs will express their forecasts in terms of definite outcomes.
• Empirical: Foxes rely more on observation than theory. Whereas, hedgehogs rely on reduced down theory to build forecasting models.

If we want to make better predictions we need to be more fox-like and less hedgehog-like. This means building detailed, multidisciplinary models of the world in a self critical and adaptive way. It means seeking out evidence that we are wrong and accepting that world is a messy hard to predict place. It also means thinking probabilistically and accepting that the best forecasts we can generate will still have large error bars (this is a sign of a good prediction not a bad one).

Even if we are not the ones making predictions, we should use the learnings in this chapter to be skeptical about forecasters that are entertaining. Entertainment really does not mix well with the attributes that make for good forecasts. So as a rule of thumb, if you are being entertained by a forecasting model your default position should be one of skepticism.

Chapter 3: All I Care about Is W’s and L’s

Honestly, I am confused by the point of chapter 3. Silver talked about a lot of interesting baseball history but I didn’t really get much out of it. I largely skimmed and skipped through this chapter.

Chapter 4: For Years You’ve Been Telling Us That Rain is Green

Chapter 4, was all about weather prediction and the progress we have made at building better weather forecasts. It turns out that humans are actually pretty good at predicting the weather — despite all the hate the weatherman get. We are good at weather prediction for basically four reasons

1. Computers: Our ability to predict the weather has improved greatly since the 1970s with the advent of computing. Computers are really helpful at predicting the weather because it turns out that weather prediction involves doing the same type of small calculations over and over again. This is something humans are bad at, and computer are very good at.
2. Human Computer Interaction: Despite the fact that computers are helpful at predicting the weather, weather forecasting is still very much a human process. We do not predict the weather by just throwing a bunch of data at a computer and having it spit out a number. Humans are actually involved in building the forecast. It is probably a surprising fact that humans actually have anything to contribute to weather forecasts in the age of big data and computing. But it’s a common theme throughout the book that humans using computers to make forecasts as a team result in better forecasts than computers & big data can produce alone.
3. Simple Laws: Humans basically have worked out how weather functions. The laws that dictate weather are not that complex. This is unlike other domains such as psychology, economics and politics where humans don’t actually have very good models for how those systems function.
4. Feedback: Weather forecasters get a lot of data about if they were right or not and they don’t need to wait very long to get this data. In data rich environments with quick feedback we are able to become better forecasters faster than in more sparse domains.

Chapter 5: Desperately Seeking Signal

Chapter 5 is about our failure to predict earthquakes. The bottom line is that humans cannot predict earthquakes and we have made almost no progress at improving our earthquake forecasting models. Earthquake forecasting occupies an extremely frustrating space — earthquakes are not totally random, but also not predictable and yet they are hugely consensual. The illusiveness of earthquake predictions has frustrated humans for centuries and as a result many people have tried and failed to build models to predict earthquakes.

Earthquake prediction is hard for basically two reasons. The first is that unlike the weather, we don’t really understand the underlying laws governing earthquakes. We have some understanding, but not enough understanding to really form reliable models. The second difficulty with earthquake prediction is that the data is fairly sparse (at least for large earthquakes). We just don’t get enough data to have the same sort of tight feedback loop we have in weather prediction.

As a result result of earthquakes not being totally random yet not predictable and highly impactful — there have been a lot of failed attempts to build earthquake prediction models. These models have all eventually failed. But what is interesting is some of these models look like they preform very well when they are used retroactively to “predict” past earthquakes. But this is just a sleight of hand called overfitting. It turns out you can build models that look very good by overfitting them to the data they are trained on. But overfit models will fall apart on future data. The problem with these models is they are not generalizable, they are complex and overly contrived attempts to fit historic data.

Getting the level of complexity right in your forecasts is a difficult balancing act. On one side, the world is a complex, messy place and that complexity needs to be reflected in your models. On the other side, it’s always possible to make a model overly complex in order to fit historic data. These overfit models won’t do well on future data. So a good model balances generalizability with real complexity without overfitting past data. This is hard.

Chapter 6: How to Drown in Three Feet of Water

Chapter 6 was about how bad economists are at making forecasts. Not all of this is the fault of economists — it’s a hard field to make predictions in. There are several reasons why economic forecasting is hard —

1. Dynamic Feedback Loop: Economic predictions shape behavior. This is not true in all forecasting domains. For example, the weather is not going to change its mind based on what the weatherman predicts. But someone tuning into CNBC very well might adjust their behavior based on the predictions they hear. Systems in which the act of making a prediction effects the results are intrinsically harder to predict.
2. Bad Data: It’s surprisingly hard to get clear economic data. It’s not just that forecasting is hard, just gathering the actual current (or recent past) state of the economic world in a reliable way is hard. Economic data is very noisy.
3. Incentives: Everyone cares about the economy — companies, government and people. With such great concern for the state of the economy incentives creep into forecasting with detrimental effects on the accuracy of the forecast.
4. Error is Not Accepted: Unlike weather prediction, the culture around economic prediction is one of certainty. Typical economic predictions are framed in terms of absolute numbers (e.g. “We predict that unemployment will drop by 5%.” rather than “We predict with 80% likelihood that unemployment will drop between 2% and 9%.”). Uncertainty is an important part of good forecasts. The world is a complex and messy place — therefore, our predictions should capture this uncertainty. For whatever, reason the expression of this uncertainty is not accepted in economic reporting… and this is bad.
5. Underlying Models: Unlike weather prediction, we don’t have that great of models when it comes to the economy. We do have models about how the economy works, but they just aren’t that great. Economists are constantly surprised by new economic trends that buck their existing models. It’s hard to make predictions when the underlying models governing how systems operate aren’t very good.

In short, economic forecasts are really hard for a handful of reasons. Getting a great economic forecast is probably a lost cause, but we can get better economic forecasts by seeking out forecasters that express uncertainty in their models and that can clearly explain the underlying systems they used to arrive at their forecasts. Economic prediction is not a lost cause, but it’s very hard and our default position should be one of skepticism.

Chapter 7: Role Models

Chapter 7 was about forecasting in the domain of epidemiology. It talked about how we can build better models for predicting how diseases will spread. The only net new idea introduced in this chapter was in regards to how hard it is to make predictions about exponential systems. Systems governed by exponential equations are really hard to predict because even small margins of errors in exponent estimation will result in widely inaccurate models.

When it comes to making predictions about exponential systems, we should be extra cautious and humble.

Chapter 8: Less and Less Wrong

Chapter 8 introduced the ideas of Bayes’ theorem as a method for making better predictions. The basic idea of Bayes’ theorem is that we should think about future outcomes in probabilistic terms, and that we should compute those probabilities by joining together our assumptions about the probabilities without any data with the data we have gathered.

Bayes’ theorem asks us to estimate a prior probability, this is the probability of some event happening before we have any data about the system. For example, suppose you are shown a coin that looks like a fair sided coin, and you are asked what is the probability of a head showing up on a single coin toss. Your answer would be 50% and that number represents your prior probability before any evidence is gathered. Now suppose the coin is tossed and a heads shows up. That single event will not move your estimate of the probability. However, if the coin is tossed a million times and every single time a heads shows up, eventually, you will have to adjust your probability to assume that the coin is not fair.

This is an extremely simple idea but a powerful one. It encourages us to think probabilistically about future outcomes and to form these probabilities not just based on data but also on the basis our prior assumptions of how systems operate. This is perhaps the most central theme of the book — predictions should not be made on the basis of raw data crunching alone, they should be based on the union of an underlying understanding of how a system operates and adjustments to the understanding based on data that is collected.

Theoretical models are not enough nor is raw data crunching good enough. We need to marry the two, and Bayes’ is the theorem that governs how they should be related to each other.

Chapter 9: Rage Against the Machines

Chapter 9 talks about the evolution of computers playing chess. A computer playing chess can be viewed as a forecasting problem — the computer is forecasting which move is most likely to result in a win condition being satisfied. In this chapter, Silver argues that computers are good at forecasting in systems that have two attributes — there is a clear win condition and there are a fairly simple set of rules. This makes computers very good at things like chess and weather prediction and not so great at economic forecasting.

Chapter 10: The Poker Bubble

Chapter 10 is all about decision making in poker. Silver illustrates that poker is a very high variance game. It is a skill based game, but only when you consider A LOT of poker hands. As long as we are taking about less than 10,000 hands, luck will likely dominate the outcomes. In terms of long term excepted value, the more skilled poker player will take all the money away from the less skilled poker player, but this only happens over the course of a long enough time to drown out the impact of luck.

The implication of this observation is that professional poker players must get comfortable with how they are making decisions regardless of the outcome. It’s common in poker to play a hand perfectly, but still lose all your money. In poker you can play well and lose it all; and you can play badly and win it all.

Poker players are forced to learn a simple lesson that the rest of us, would benefit from learning — the outcome of a forecasting model is less important than the process by which you arrived at the forecast. If you are happy with the decision making that went into the forecast’s construction but your forecast did not play out, you should still be pleased. On the flip side, if your forecast is trash but it happened to predict the right outcome you should be displeased.

My Final Take Aways

• Raw data crunching for predictions is not good enough, you actually need a fundamental understanding of the system you are trying to predict. This creates the dynamic in which humans plus computers are better at making predictions than either one in isolation for many domains.
• Computers are really good at making predictions in closed loop systems, with simple sets of rules, a clear win conditions and for problems that can be decomposed into many small calculations.
• Good prediction makers are multidisciplinary, make prediction with high error bars, accept the messy uncertainty of the world, adjust their predictions when data changes and are self-critical.
• By being multidisciplinary and seeking to expand the set of information we incorporate into our prediction models we can avoid the problem of making predictions from too narrow of a view point.
• It’s easy to create overfit models that look really good on past data but do badly on future data. Finding this balance is hard.
• It’s really hard to predict systems for which we do not have good mental models for how the underlying system works. This applies to a lot of domains and is surprisingly still very true in economics.
• Exponential systems are very very hard to make predictions about… lets not get overconfident about any predictions in the AI space.
• The thing that matters is the quality of your decision making process and not the actual outcome. The future is defined by a set of possible probabilities and just because you happened to predict the future outcome that materialized does not mean your underlying model was good. The world is a very noisy place, bad models can be right and great models can be wrong. The point is not so much to be right but to focus on building better forecasts so that we can be right marginally more often in the long run.