Superforecasting

Summary of Superforecasting

1. 1

   Why the Smartest People in the Room Are Usually Wrong

   

   00:00

   00:00

   Here's something that should shake your confidence: in 2011, a retired government employee from Nebraska with no special credentials started outpredicting CIA analysts, celebrated pundits, and sophisticated prediction markets on questions like whether Russia would annex Ukrainian territory. Bill Flack wasn't lucky - he was part of a revolution in how we think about the future. When over 20,000 volunteers participated in a government-sponsored forecasting tournament, researchers discovered something remarkable: the ability to see the future isn't a mystical gift or the domain of credentialed experts. It's a learnable skill, built on specific thinking habits that anyone can develop. The catch? Most of us - especially the experts we trust most - are doing it completely wrong. In 1956, a renowned medical specialist diagnosed Archie Cochrane with terminal cancer. Cochrane, a physician himself, accepted this death sentence without question and began planning his final months. The specialist was completely wrong. What's terrifying isn't just the misdiagnosis - it's that neither man questioned it. They fell into what psychologists call the certainty trap, where our minds rush to judgment and then defend those judgments against all evidence. This trap has killed countless people throughout history. George Washington's physicians bled, purged, and blistered him to death in 1799, supremely confident in treatments that were actually hastening his demise. For centuries, doctors rarely questioned methods that ranged from useless to lethal. As one medical historian put it, they were "like blind men arguing over the colors of the rainbow."

2. 2

   How Our Minds Betray Us and Why Experts Fail

   Start Listening

   Our brains operate in two modes. System 1 delivers instant, effortless judgments-mental autocomplete. System 2 is conscious, effortful thinking. The problem? We rely heavily on System 1's primitive logic: if it feels true, it must be true. We unconsciously substitute hard questions with easier ones. When Cochrane heard "terminal cancer," his mind answered "Is this the sort of person who should know?" instead of "Do I actually have cancer?" We crave coherent explanations so desperately that we cherry-pick supporting evidence while dismissing contradictions. When researchers assembled 284 credentialed professionals to make roughly 28,000 forecasts, the average expert performed no better than random guessing. But not all experts failed equally. The underperformers organized thinking around Big Ideas and expressed high confidence. The modest outperformers used diverse analytical tools, acknowledged uncertainty, and readily admitted mistakes. Hedgehogs knew "one big thing" and forced every problem into that framework. Foxes drew on multiple perspectives and remained intellectually nimble. This matters because we're surrounded by hedgehogs masquerading as experts. Consider the 2002 National Intelligence Estimate concluding Iraq possessed weapons of mass destruction with absolute certainty. The critical failure wasn't being wrong-it was expressing certainty where profound uncertainty existed.

3. 3

   Breaking Down the Impossible and Merging Multiple Views

   Start Listening

   In 2008, Sandy Sillman was diagnosed with multiple sclerosis. By 2011, weakened and facing early retirement from atmospheric science, he joined a forecasting tournament to keep his mind sharp. Without Middle East expertise, he tackled questions like whether European investigators would find elevated polonium in Yasser Arafat's exhumed body. His breakthrough came from physicist Enrico Fermi's technique: breaking impossible questions into manageable parts. Estimate Chicago's piano tuners by breaking it down-city population, piano-owning households, tuning frequency, time per tuning, annual working hours. Each estimate might be imperfect, but together they produce surprisingly accurate answers. Sandy avoided answering a different question than what's actually asked. Most minds substitute "Did Israel poison Arafat?" for the actual question about what tests would show. Sandy methodically examined whether polonium remained detectable years later, identified contamination pathways, and noted only one of two European teams needed positive results. The next step establishes the "outside view"-how common something is within a broader class. Start with base rates: 62% of American households own pets. This statistical foundation prevents vivid details from misleading us. Only after anchoring to base rates should you explore the "inside view"-specific factors that might push this case above or below average. The magic happens when you merge both perspectives, letting broad patterns inform specific judgments.

4. Chapter 4

   The Precision Paradox and the Art of Updating

   When CIA analysts briefed President Obama on bin Laden's location, their estimates ranged from 30% to 95%. Obama called it "fifty-fifty"-revealing how most people handle uncertainty with three settings: "gonna happen," "not gonna happen," and "maybe." This crude mental dial made evolutionary sense for quick decisions, not nuanced probability. Superforecasters subdivide "maybe" into precise numerical gradations. When average forecasters say "fifty-fifty," they mean "I don't know." When superforecasters estimate 37%, they mean exactly that-not 30%, not 40%. This precision enables continuous updating as information emerges. When the Swiss team investigating Arafat's remains announced a testing delay, Bill Flack recognized what others missed. Understanding polonium's properties, he inferred they'd likely detected it and were ruling out natural decay. He raised his forecast to 65% before others grasped the significance-validated when polonium was found. Distinguishing obvious developments from subtle signals separates good forecasters from superforecasters.

   Chapter 5

   The Growth Mindset and Learning from Failure

   Mary Simpson's PhD in economics didn't prevent her from missing the 2007 financial crisis and watching her retirement savings crater. Rather than accepting defeat, this failure motivated her to improve - exemplifying the "growth mindset," the belief that abilities develop through effort rather than being fixed traits. John Maynard Keynes demonstrated this same mindset as an investor between the World Wars. Despite being nearly wiped out by wrong currency forecasts in 1920 and blindsided by the 1929 crash, he rebounded stronger each time, famously noting: "There is no harm in being sometimes wrong, especially if one is promptly found out." Effective forecasting requires practice with clear, timely feedback. Without it, hindsight bias distorts memory, making us recall predictions as closer to outcomes than they were. Superforecasters combat this through thorough postmortems, readily acknowledging when luck contributed to success. They embrace "I was almost wrong" scenarios that most experts dismiss. This requires "grit" - passionate perseverance toward long-term goals. Anne Kilkenny, an Alaska housewife with no geopolitical background, exemplified this by researching UN refugee data in the Central African Republic, even emailing the agency directly for better information.

   Chapter 6

   The Wisdom of Humble Collaboration

   The Kennedy administration's Bay of Pigs disaster and Cuban missile crisis triumph show how the same team can fail or succeed dramatically. After the Bay of Pigs, Kennedy transformed decision-making by instituting skepticism, designating "intellectual watchdogs," and sometimes absenting himself to encourage freer discussion. When researchers created "superteams" of top forecasters, teams became 23% more accurate than individuals and outperformed prediction markets by 15-30%. Initially hampered by excessive politeness, dynamics improved as members explicitly welcomed criticism and thanked others for feedback. Counterintuitively, teamwork increased individual effort. Successful teams fostered sharing cultures where "givers" contributed tools and analyses-and these givers consistently ranked among top individual forecasters. Teams succeeded by avoiding both groupthink and destructive conflict, creating "psychological safety" for challenging ideas respectfully. The lesson: intellectual humility combined with collaborative truth-seeking produces results exceeding what even talented individuals achieve alone.

   Chapter 7

   Your Move

   Months before Scotland's 2014 independence referendum, superforecasters accurately predicted the wide "no" margin while pundits struggled with dramatic polling shifts. Political scientist Daniel Drezner asked the crucial question: "What does one do with data points like this to adjust one's worldview?" The strongest resistance to evidence-based forecasting comes from valuing winning over accuracy. When Nate Silver's forecasts favored Obama, Democrats praised him and Republicans attacked; when he predicted Republican Senate control, the sides reversed. A century ago, physician Ernest Codman proposed tracking patient outcomes to evaluate hospitals on results rather than reputation. The medical establishment rejected him, costing his position at Massachusetts General Hospital. Yet his ideas became the foundation of evidence-based medicine. This pattern has repeated across fields - policy, charity evaluations, sports analytics - representing a shift from intuition toward analysis. The future belongs to those willing to keep score. Not to gloat when right, but to learn when wrong. The tools of superforecasting - breaking down questions, establishing base rates, thinking probabilistically, updating beliefs, embracing uncertainty, collaborating humbly - aren't just for geopolitics. They're for navigating any complexity where the confident are usually wrong and the humble keep improving. The question isn't whether you can see the future, but whether you're willing to think clearly enough to improve your odds.

   Continue Reading