Review

A great deep dive into the world of forecasting with some practical advice scattered throughout. Using forecasting as a lens to explore team management best practices (psychological safety, diversity, etc) was illuminating. It’s an excellent, evidence-based read, particularly helpful for product managers seeking to enhance their product sense.

Key Takeaways

The 20% that gave me 80% of the value.

• Predictability has its limits, but we shouldn’t dismiss all prediction as futile.
• You can learn to be a superforecaster if you adopt their techniques. Commitment to self-improvement might be the strongest predictor of performance.

• System 1 thinking is designed to jump to conclusions from little evidence. A defining feature of intuitive judgment is its insensitivity to the quality of the evidence on which the judgment is based. It is designed to deliver strong conclusions at lightning speed. If we want to forecast accurately, we need to slow down and engage System 2. Chess champion Magnus Carlsen respects his intuition, but he also does a lot of “double-checking” because he knows that sometimes intuition can let him down and conscious thought can improve his judgment.

• Keeping a track record is the key to assessing forecasters, but also a helpful learning tool for forecasters. If we are serious about measuring and improving forecasts: terms must be precise, timelines must be stated, probabilities must be expressed in numbers and we must have lots of forecasts. Outside prediction tournaments, predictions are rarely apples to apples. So it’s hard to compare forecasters.
• Having many tabulated set of probabilistic forecasts enables us to determine the track record of a forecaster. The Brier score is a way to measure how good your predictions are. It looks at both calibration (how accurate your predictions are overall) and resolution (how specific and decisive your predictions are). A perfect score is 0, which means all your predictions were spot on. If you always predict a 50/50 chance for everything, or if you just guess randomly, your Brier score will be around 0.5. The worst score you can get for a single prediction is 2.0, it happens if you say something is 100% certain to happen, but you’re wrong.
• The best predictions are the ones that are both accurate and decisive. Try to be as accurate and specific as possible.
• How do you compare to benchmarks? Random, assuming no change, other forecasters?
• Hedgehogs hold firm beliefs and use more information to reinforce them, while foxes are pragmatic, versatile, discuss probabilities, and are open to changing their minds. Foxes outperformed hedgehogs in predictions, exhibiting better foresight, calibration, and resolution.
• The Wisdom of Crowds: Aggregating the judgment of many consistently beats the accuracy of the average member of the group. This is true when information is dispersed widely. All the valid information points in one direction, and all the errors cancel themselves out.
• Foxes approach forecasting by doing a kind of aggregation, by seeking out information from many sources and then synthesizing it all into a single conclusion. They benefit from a kind of wisdom of the crowds by integrating different perspectives and the information contained within them.

• Enrico Fermi understood that by breaking down a question, we can better separate the knowable and the unknowable. Doing so brings our guessing process out into the light of day where we can inspect it. The net result is a more accurate estimate.
• Starting a forecast with the a base rate (e.g. the outside view, how common something is within a broader class) will reduce the anchoring effect.
• Thesis → Antithesis → Synthesis. You now need to merge the outside view and the inside view. How does one affect the other. You can train yourself to generate different perspectives. Writing down your judgments, scrutinize your view. Seek evidence that you’re wrong. Beliefs are hypotheses to be tested, not treasures to be guarded.
• Dragonfly forecasting: superforecasters pursue point-counterpoint discussions routinely. Constantly encountering different perspectives, they are actively open-minded.

• Superforecasters tend to be probabilistic thinkers.
• When a question is loaded with irreducible uncertainty be cautious, keep estimates inside the maybe zone between 35% and 65% and moving out tentatively.
• The best forecasters are precise. They sometimes debate differences that most of us see as inconsequential 3% vs 4% or 1% vs 0.5%. Granularity was a predictor of accuracy.

• A common method emerged among Superforecasters:
  • Unpack the question into components.
  • Distinguish between the known and unknown and leave no assumptions unscrutinized.
  • Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena.
  • Then adopt the inside view that plays up the uniqueness of the problem.
  • Explore the similarities and differences between your views and those of others and from the wisdom from crowds.
  • Synthesize all these different views into a single vision as acute as that of a dragonfly.
  • Express your judgment as precisely as you can, using a finely grained scale of probability.
  • Update to reflect the latest available information
• Superforecasters update forecasts more regularly, but they make smaller changes (e.g. 3.5%). Train your brain to think in smaller units of doubt.
• The Bayesian belief updating equation: your new belief should depend on your prior belief (and all the knowledge that informed it) multiplied by the “diagnostic value” of the new information. Bayes’ core insight is to gradually get closer to the truth by updating in proportion to the weight of the evidence.

• Grit is passionate perseverance of long-term goals, even in the face of frustration and failure. Married with a growth mindset, it is a potent force for personal progress. Superforecasters are in perpetual beta, always learning.
  • Superforecasters have a ‘growth mindset’ they believe their abilities are largely the product of effort. Failure is an opportunity to learn – to identify mistakes, spot new alternatives, and try again.
  • We learn new skills by doing. Informed practice will accelerate your progress (knowing what mistakes to look out for and what best practice looks like).
• Typically meteorologists and bridge players don’t suffer from over confidence as they both get clear prompt feedback.
• Put as much effort into postmortems with teammates as you do to initial forecasts.
• Superforecasters are cautious, humble and nondeterministic. They tend to be actively open-minded, intellectually curious, introspective and self critical. They aren’t wedded to ideas. They’re capable of stepping back. They value and synthesise diverse views. They think in small units of doubt, update forecasts thoughtfully and aware of their cognitive biases.

• Group Think: Members of any small cohesive group tend to unconsciously develop a number of shared illusions and related norms that interfere with critical thinking and reality testing. Groups that get along too well don’t question assumptions or confront uncomfortable facts.
• Aggregation can only do its magic when people form judgments independently.
• Precision questioning (from Dennis Matthies and Monica Worline) can help you tactfully dissect the vague claims people often make.
• Do a team pre-mortem: assume a course of action has failed and to explain why. It helps team members feel safe and express doubts.
• Aim for a group of opinionated people who engage one another in pursuit of the truth. Foster a culture of sharing.
• Diversity trumps ability, the aggregation of different perspectives is a potent way to improve judgment. The more diverse the team, the greater the chance that some will possess scraps of information that others don’t.

• The principle of “Auftragstaktik” or “mission command” emphasizes that decision-making power should be decentralized. Commanders should provide the goal but not dictate the methods, allowing those on the ground to adapt quickly to changing circumstances. This strategy blends strategic coherence with decentralized decision making.
  • No plan survives contact with the enemy. Two cases never will be exactly the same.
  • Improvisation is essential.
  • Decisive action is required, so draw a line between deliberation and implementation. Once a decision has been made. Forget uncertainty and complexity. Act!
• Mission Command: Let your people know what you want them to accomplish, but don’t tell them how to achieve those goals.
• Smart people are always tempted by a simple cognitive shortcut: I know the answer, I don’t need to think long and hard about it. Don’t fall for it

• What makes Superforecasters good is what they do: the hard work of research, the careful thought and self-criticism, the gathering and synthesizing of other perspectives, the granular judgments and relentless updating.
• Our training guidelines urge forecasters to mentally tinker with “the question asked” (e.g. explore how answers to a timing question might change if the cutoff date were six months out instead of twelve). Such thought experiments can stress-test the adequacy of your mental model.
• The ‘black swan’ is an event literally inconceivable before it happens. But Taleb also offers a more modest definition of a black swan as a highly improbable consequential event. To the extent that such forecasts can anticipate the consequences of events like 9/11, and these consequences make a black swan what it is, we can forecast black swans.
• Though there are limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems.
• Humility should not obscure the fact that people can, with considerable effort, make accurate forecasts about at least some developments that really do matter.

Ten Commandments for Superforecasters

1. Triage. Focus on questions where work can pay off. Don’t waste time either on easy “clock-like” questions (where simple rules of thumb can get you close) or on impenetrable “cloud-like” questions (where fancy models won’t help). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.
2. Break seemingly intractable problems into tractable sub-problems. Channel the playful but disciplined spirit of Enrico Fermi. Decompose the problem into its knowable and unknowable parts. Flush ignorance into the open. Expose and examine your assumptions. Dare to be wrong by making your best guesses. Better to discover errors quickly than to hide them behind vague verbiage.
3. Strike the right balance between inside and outside views. Nothing is 100% unique. Look for comparison classes even for seemingly unique events. Ask: How often do things of this sort happen in situations of this sort?
4. Strike the right balance between under- and overreacting to new evidence. Belief updating pays off in the long term. Skilful updating requires spotting non-obvious lead indicators about what would have to happen before X could.
5. Look for the clashing causal forces at work in each problem. Acknowledge counter arguments. List in advance, the signs that would nudge you toward the other. Synthesis is an art that requires reconciling irreducibly subjective judgments. Create a nuanced view.
6. Strive to distinguish as many degrees of doubt as the problem permits but no more. Nuance matters. The more degrees of uncertainty you can distinguish, the better a forecaster you are likely to be. In poker you need to know a 55/45 from 45/55.
7. Strike the right balance between under- and overconfidence, between prudence and decisiveness. Long-term accuracy requires getting good scores on both calibration and resolution. Know your track record, and find creative ways to tamp down both types of forecasting errors (misses and false alarms).
8. Look for the errors behind your mistakes but beware of rearview mirror hindsight biases. Don’t try to justify or excuse your failures. Own them! Conduct unflinching postmortems. Ask: Where exactly did I go wrong? Don’t forget to do postmortems on your successes too.
9. Bring out the best in others and let others bring out the best in you. Master perspective taking (understanding the arguments of the other side), precision questioning (helping others to clarify their arguments so they are not misunderstood), and constructive confrontation (learning to disagree without being disagreeable).
10. Master the error-balancing bicycle. Implementing each commandment requires balancing opposing errors. Learning requires doing, with good feedback that leaves no ambiguity about whether you are succeeding.

Don’t treat commandments as commandments. Guidelines are the best we can do in a world where nothing is certain or exactly repeatable. Superforecasting requires constant mindfulness, even when dutifully trying to follow these commandments.