1장
When Numbers Speak Louder Than Words
Imagine a world where a Princeton economist with a simple weather formula can predict wine quality better than expert tasters, where baseball teams win championships by ignoring scouts' visual assessments in favor of statistical analysis, and where algorithms consistently outperform human experts across countless fields. This isn't science fiction-it's the reality Ian Ayres documents in "Super Crunchers," a book that has fundamentally changed how businesses, governments, and individuals make decisions. Since its 2007 publication, the book has become required reading in MBA programs worldwide and influenced countless organizations to embrace data-driven approaches. Even tech giants like Google and Amazon have cited its principles in developing their algorithmic systems. The book arrived at the perfect cultural moment-just as "Big Data" was entering the mainstream consciousness but before many understood its transformative potential. Ayres, with his unique background as both Yale law professor and econometrician, bridges the gap between technical statistical concepts and practical real-world applications, making this revolution accessible to everyone.
2장
The Revolution of Super Crunching
The battle between intuitive expertise and data analysis is transforming our world. Princeton economist Orley Ashenfelter revolutionized wine prediction with a simple weather-based formula that outperformed traditional tasting methods. Despite initial scorn from wine critics like Robert Parker, who called his approach "Neanderthal" and "absurd," Ashenfelter's predictions proved remarkably accurate. His newsletter correctly identified the exceptional 1989 and 1990 Bordeaux vintages as "wines of the century" long before critics could taste them.
Similarly, Bill James challenged baseball's conventional wisdom by demonstrating that statistical analysis outperforms visual scouting. As Michael Lewis documented in "Moneyball," Oakland A's general manager Billy Beane drafted Jeremy Brown based solely on his exceptional walk rate despite scouts dismissing him as too overweight. Brown eventually reached the majors batting .300, validating the data-over-appearance approach.
These aren't isolated examples but part of a historic shift where intuitive expertise consistently loses to number crunching. Super Crunchers analyze enormous datasets-measured in terabytes and petabytes-to discover correlations between seemingly unrelated variables. Walmart stores 570 terabytes of data while Google processes four petabytes. These aren't just academic exercises but practical tools that impact real-world decisions through size, speed, and scale.
The implications are profound. In medicine, evidence-based approaches clash with intuitive expertise. In chess, Garry Kasparov lost to Deep Blue largely because the computer accessed 700,000 grandmaster games. Super Crunchers aren't just displacing experts; they're changing how decisions are made and uncovering hidden relationships that traditional experts never considered.
This revolution extends beyond corporate boardrooms to everyday life. When my cell phone was stolen, I traced the thief by analyzing call patterns in the data. Law enforcement used similar techniques to identify Michael Jordan's father's killers. On a larger scale, the National Security Agency analyzes trillions of phone records to map terrorist networks, with network analyst Valdis Krebs demonstrating that all nineteen 9/11 hijackers were within two connections of known al-Qaeda operatives.
3장
How Algorithms Are Reshaping Your Life
Every time you shop online, stream entertainment, or search for information, sophisticated algorithms are working behind the scenes to predict your preferences. These "preference engines" have evolved from simple popularity lists like the New York Times' "most emailed articles" to sophisticated systems like Amazon and Netflix that use collaborative filtering to recommend products based on what similar customers enjoyed.
These systems benefit both consumers and retailers-nearly two-thirds of Netflix rentals come from recommendations, and they enable access to the "long tail" of niche products that physical stores can't stock. However, critics worry these personalized filters create "Daily Me" experiences that deprive citizens of common cultural touchpoints.
Dating services represent another frontier of algorithmic matchmaking. Unlike traditional services matching people based on conscious preferences, eHarmony uses regression analysis-a statistical technique pioneered by Francis Galton in 1877-to analyze 29 emotional, social, and cognitive attributes predicting couple compatibility. Their approach favors matching similar personalities, while competitors like Perfectmatch look for complementary traits. The key innovation is using data to uncover hidden compatibility factors beyond conscious preferences.
This hyper-individualized customer segmentation extends to many industries. Harrah's casinos excel at predicting each customer's "pain point"-how much they can lose while still enjoying the experience. Using their Total Rewards card system, they track gambling patterns and combine this with demographic data to calculate when a gambler approaches their threshold. When someone nears their pain point, a "luck ambassador" intervenes with complimentary services to transform the negative experience into a positive one.
Companies like Hertz, Cingular, and Blockbuster can predict individual customer behaviors with remarkable accuracy-sometimes better than customers can predict themselves. This corporate omniscience eerily echoes Psalm 139: "You have searched me and you know me." Rather than prohibiting such analysis, we might require companies to disclose what they know. Avis could inform you that people like you typically return cars with a third of a tank, making prepaid gas overpriced.
Fortunately, entrepreneurs are developing counter-crunching tools to level the playing field. Computer scientist Oren Etzioni created Farecast.com after discovering fellow passengers had paid less for waiting to book. This travel website not only finds the lowest current fare but predicts whether prices will rise or fall, giving consumers strategic buying guidance by analyzing a five-terabyte database containing fifty billion prices.
4장
The Science Behind the Numbers
The power of Super Crunching comes from two fundamental statistical techniques: regression analysis and randomized trials. Regression analysis, pioneered by Francis Galton in 1877, identifies relationships between variables and predicts outcomes based on those relationships. What makes regression remarkable is that it simultaneously produces both a prediction and its precision level. When data is insufficient for accurate predictions, the regression itself reveals this limitation.
Wal-Mart's employment test regression, for instance, predicts an applicant's likely tenure while also calculating the probability of shorter employment. It even measures the precise impact of specific questions-like how non-conformists might work 2.8 months less than others. Unlike traditional experts, regressions transparently report their own reliability.
Randomized trials complement regression analysis by establishing causation rather than just correlation. Capital One has revolutionized customer service through running over 28,000 randomized experiments annually to test products, advertising approaches, and contract terms. By randomly dividing prospects and applying different treatments, they isolate causation in ways historical data mining cannot. Their 1995 experiment with 600,000 prospects revealed that a 4.9% six-month teaser rate outperformed a 7.9% twelve-month rate, directly informing business strategy.
Google similarly uses randomization for AdWords testing, automatically shifting toward ads with higher click-through rates. I used this approach to name this book, discovering that "Super Crunchers" received 63% more clicks than "The End of Intuition" in tests with over 250,000 page views.
Continental Airlines used randomized trials to determine how best to respond to "transportation events" like flight cancellations. Customers who received apology letters spent 8% more on tickets the following year, generating $6 million in additional revenue from just 4,000 customers. When expanded to Continental's top customers, this program yielded $150 million in new revenue. Surprisingly, compensation wasn't necessary-just acknowledging the problem significantly improved customer loyalty.
While randomization still requires human creativity to generate alternatives, it puts intuition to the test. The simplicity of randomized trials makes results easier to explain than complex statistical models; audiences only need to trust that researchers randomized properly. Despite their power, many businesses hesitate to adopt randomized testing, possibly because it requires advance hypothesizing rather than after-the-fact analysis.
5장
When Government Gets Smart with Data
In 1966, MIT economics graduate student Heather Ross pioneered policy randomization by securing a $5 million grant to test the Negative Income Tax (NIT). Her study revealed that while the NIT didn't significantly reduce employment as feared, it unexpectedly increased divorce rates among poor families. The greatest impact of Ross's experiment was methodological-applying medical-style randomization to policy evaluation sparked hundreds of subsequent randomized public policy experiments.
U.S. lawmakers increasingly embrace randomization as a non-partisan approach to separate effective programs from ineffective ones. In 1993, economist Larry Katz convinced Congress to save $2 billion annually by funding job-search assistance for the unemployed. His persuasive argument relied on randomized tests from multiple states showing that workers receiving assistance found jobs about a week earlier without sacrificing pay. For every dollar invested in assistance, the government saved two dollars through reduced unemployment benefits and increased tax receipts.
Randomized experiments have finally fulfilled the promise of states as "laboratories of democracy" by providing proper control groups for policy comparisons. Unlike traditional state experiments where differences between states confound results, randomization within states creates quality data for evidence-based policymaking.
Researchers can also "piggyback" on existing randomized processes in government. Joel Waldfogel brilliantly exploited random assignment of judges to criminal cases to answer whether longer sentences affect recidivism. Since judges within districts see similar cases, differences in sentencing reflect judicial temperament rather than case differences. The research revealed that longer sentences neither increased nor decreased recidivism rates after release, challenging both "hardening" and "rehabilitation" theories.
Randomized testing of social policy has become global, with developing countries often leading the way. The Poverty Action Lab, founded at MIT in 2003, has been instrumental in spreading this methodology worldwide. Their motto, "translating research into action," reflects their commitment to testing development strategies through randomized trials.
Mexico's Progresa program represents the most significant randomized social experiment in development policy. Created by President Ernesto Zedillo in 1997, it provided conditional cash transfers to mothers who kept children in school and obtained healthcare. The randomized study of 24,000 households across 506 villages showed remarkable results: 10% higher school attendance for boys, 20% for girls, 12% reduction in illness, and children nearly a centimeter taller than control groups. The program's success led to its continuation under the subsequent administration, and the conditional cash transfer model has spread to thirty countries worldwide.
6장
Medicine's Evidence Revolution
The struggle over Evidence-Based Medicine (EBM) mirrors the broader conflict around Super Crunching. In 1992, Canadian physicians Gordon Guyatt and David Sackett published a manifesto calling for medical treatments to be based on the best statistical evidence available. While they didn't advocate for statistics to exclusively guide treatment decisions, their push for statistical evidence to play a more prominent role remains controversial.
Statistical testing in medicine dates back to the 1840s when Ignaz Semmelweis documented how hand-washing reduced maternity clinic mortality from 12% to 2%. Despite facing fierce resistance from physicians who refused to believe they caused deaths and complained about wasting time washing hands, Semmelweis was fired and died in a mental hospital after a breakdown.
This tragedy remains relevant today as physician resistance to hand-washing persists. Don Berwick, a pediatrician and president of the Institute for Healthcare Improvement, has become a modern-day Semmelweis. Radicalized by both a 1999 Institute of Medicine report showing 98,000 annual preventable hospital deaths and his wife's poor treatment during her illness, Berwick launched the "100,000 Lives Campaign" in 2004.
The campaign implemented six evidence-based interventions in 3,000 hospitals, including simple measures like elevating hospital beds to prevent ventilator infections and systematic hand-washing to reduce central-line catheter infections. By June 2006, the campaign had exceeded its goal, preventing an estimated 122,342 deaths in just eighteen months-a massive victory for evidence-based medicine.
Despite evidence-based medicine's success, doctors often remain unaware of or deliberately ignore statistical findings. Many still perform annual physical exams with routine pelvic, rectal, and testicular exams for symptomless patients, despite dozens of studies showing these make no difference in survival rates. Numerous "medical myths" persist in practice long after being refuted by strong statistical evidence.
With digital medical records, diagnostic systems like Isabel are evolving beyond simply compiling journal articles. Soon they'll analyze vast databases of patient experiences to provide probability-based diagnoses tailored to specific symptoms and histories. The NextGen partnership creates structured input fields that systematically collect richer data than traditional physician notes. This enables "real-time epidemiology" where patterns emerge from aggregate data that individual doctors might miss.
7장
When Experts Face the Algorithm
In 1954, psychologist Paul Meehl published the groundbreaking "Clinical Versus Statistical Prediction," comparing how well clinical experts predicted outcomes versus simple statistical models across twenty studies. His startling conclusion: none showed experts outperforming statistical equations. Despite being president of the American Psychological Association and developer of the MMPI personality test, Meehl referred to his work as "my disturbing little book."
This became the first salvo in what would become a vast industry of man-versus-machine comparison studies. Researchers have since completed dozens of studies comparing statistical and expert approaches to predicting everything from marital satisfaction to business failures, lie detection, and even sexual orientation-with Super Crunchers consistently demonstrating superior accuracy.
Our brightest experts consistently lose to statistical predictions because human minds suffer from well-documented cognitive failings. We overweight unusual, salient events, cling stubbornly to mistaken beliefs, and discount contradictory evidence. Most critically, we're systematically overconfident-when tested on confidence intervals, 99% of people fail to include correct answers within their stated ranges.
Unlike humans, regression equations have no egos, assign appropriate weights to variables, aren't emotionally attached to prior predictions, and provide their own confidence intervals. Even crude regressions with few variables routinely outpredict human experts, who remain overconfident even when presented with contradictory data.
While experts do make better predictions when provided with statistical results, the combined approach still underperforms pure statistical methods. Rather than having statistics serve expert judgment, experts should serve statistical algorithms. Mark Nissen of the Naval Postgraduate School sees a fundamental shift toward systems "where machines are actually in charge, but they have enough awareness to seek out people to help them when they get stuck."
The evidence increasingly suggests that when human intuition and statistical prediction disagree, we should defer to the algorithm. Even in parole decisions, states have shifted toward Super Crunching risk assessments like the VRAG (Violence Risk Appraisal Guide), which predicts the probability of recidivism. Human discretion, when overriding statistical predictions, tends to make poorer decisions overall.
What's left for humans? In a word, hypothesize. While statistical regressions can test causal relationships and estimate their magnitude, humans remain essential for generating hypotheses about what causes what. With finite data, we can only estimate a finite number of causal effects, making human hunches essential in determining what to test and what to ignore.
8장
Why Super Crunching Is Happening Now
The Super Crunching revolution stems from our digitalized world where information is stored as binary bytes. More data is captured electronically-from email to credit card swipes-creating electronic records of most consumer purchases. Even paper records are being unlocked through scanning technologies, with services like Questia.com offering full text access to over 67,000 books, Amazon's "Search Inside the Book" feature, and Google's ambitious project to scan 30 million books in ten years.
My own research journey illustrates the data explosion. In 1989, I studied car dealership discrimination using just six testers at 90 dealerships, collecting data on paper forms. The small study revealed significant discrimination: white women paid 40% higher markups than white men, black men paid twice as much, and black women paid triple.
Fast-forward to the new millennium, and I've analyzed over three million car sales in class-action litigation against automotive lenders. These studies revealed that African-American borrowers paid nearly $700 in loan markups versus $300 for whites. Such massive studies became possible only because lenders now keep detailed electronic records, and because state driver's license databases with race information could be easily merged with lender data.
Digitalized data has become a commodity, with aggregators like ChoicePoint and Acxiom flourishing. Since 1997, ChoicePoint has acquired over 70 database companies, offering one-stop access to credit reports, motor-vehicle records, police records, and more. Acxiom is even larger, managing 20 billion customer records (850 terabytes) on nearly every U.S. household.
The Super Crunching revolution stems more from technological advances than statistical breakthroughs. The core statistical techniques like randomized trials and regression analysis have existed for decades or even centuries. More critical was the explosion in storage capacity following Kryder's Law, where storage density doubles every two years. Since 1956, information storage density has increased 100-million fold, with 30-40% annual price declines per gigabyte. This makes it economically feasible to capture and store massive datasets.
Neural networks represent an important new statistical technique driving the Super Crunching revolution. Developed to simulate human brain learning processes, neural networks use interconnected mathematical "switches" that receive, evaluate and transmit information. Unlike traditional regression analysis where researchers specify equation forms, neural networks allow the data itself to determine optimal relationships by testing millions of alternative weights.
9장
The Human Cost of Data-Driven Decisions
The battle over Direct Instruction in education exemplifies the broader power struggle between intuition and data. Created by Siegfried "Zig" Engelmann in the 1960s, Direct Instruction requires teachers to follow precise scripts that break learning into digestible concepts, with students answering up to ten questions per minute in unison.
Project Follow Through, a massive $600 million study of 79,000 children over twenty years, found DI outperformed sixteen other teaching methods not just in basic skills but also in higher-order thinking and even self-esteem. More recent studies continue to show DI's superiority, particularly for economically disadvantaged students and minorities.
Yet Direct Instruction faces fierce resistance from teachers who resent being turned into "robots" following scripted lessons. Most educators are taught to value creativity and innovation-the stuff of inspirational teacher movies like Dead Poets Society-not repetitive, scripted instruction. Despite the Bush administration's No Child Left Behind law requiring "scientifically based" educational programs, DI has captured barely 1% of the grade-school market.
This battle represents a broader power struggle as traditional experts-from teachers to loan officers to doctors-are losing both discretion and status as statistical algorithms increasingly make better decisions than human judgment. Bank loan officers, once respected decision-makers, have become "glorified secretaries" who merely input data into centralized systems. Similarly, physicians find patients demanding to see research studies rather than deferring to medical authority.
Super Crunching transforms not just how we work but how we consume. We've moved from a world of expert certainty to one of statistical probabilities-from doctors who definitively say "yes" or "no" to those who cite percentages and confidence intervals. While this precision can undermine public confidence, many Super Crunching applications clearly benefit consumers.
The application of Super Crunching to creative fields like filmmaking raises fears about artistic freedom. When Epagogix's statistical formulas tell screenwriters to add a buddy character or change a plot point, it seems like the death of art. Yet this concern ignores that commercial constraints have long limited artistic freedom in Hollywood. The real problem isn't studio interference but that it's been done badly, based on executives' flawed intuition rather than reliable data.
10장
The Future of Human Judgment
The rise of statistical thinking doesn't mean the end of intuition or expertise, but rather their reinvention. Decision makers will increasingly toggle between intuition and data-based approaches, with each counterbalancing the other's weaknesses. The best data miners use their intuition to question whether statistical analysis makes sense, especially when results diverge widely from expectations.
The future belongs to people who can toggle between intuition and statistics. Justin Wolfers exemplifies this "new way to be smart"-when analyzing college basketball point spreads, he noticed favored teams only covered 47% of the time with large spreads. This statistical anomaly led him to hypothesize about point shaving, which he then tested by examining game patterns in the final minutes.
What makes statistical analysis particularly valuable is that it simultaneously predicts and measures its own accuracy through standard deviations. The 2SD rule (that 95% of observations fall within two standard deviations of the mean) provides an intuitive foundation for understanding statistical significance-a result is significant when it's more than two standard deviations from what would be expected by chance.
Standard deviations give us a vocabulary for discussing variability that most people lack. The standard deviation provides remarkable economy of information-knowing that a diversified stock portfolio has an expected return of 10% with a 20% standard deviation immediately tells us there's a 95% chance the return will fall between -30% and +50%.
Most doctors fail to accurately communicate statistical information about pregnancy. They still use the outdated Naegele formula from 1812 to calculate due dates, despite modern research showing pregnancy is typically eight days longer, with a standard deviation of fifteen days. Though medicine now uses "triple screen" or "quad screen" tests that combine multiple predictors, many doctors still can't properly integrate all available data using Bayesian updating to provide a single, accurate probability estimate.
Understanding statistical tools like the 2SD rule and Bayes' theorem can dramatically improve decision quality. Ben Polak advocates overcoming the widespread "phobia" about statistics, rejecting the notion that statistical analysis is somehow "right-wing" or "illiberal." The stories throughout this book demonstrate that Super Crunching transcends ideology-it empowers initiatives like the Poverty Action Lab to improve the world while maintaining passion and creativity.
The future relationship between intuition, expertise and data analysis was presciently depicted in the 1957 film "Desk Set," where Katharine Hepburn's reference librarian character competed against Spencer Tracy's "EMERAC" computer. Just as we now accept that computers vastly outperform humans at information retrieval, we will increasingly recognize that Super Crunching algorithms are simply better than human experts at determining predictive weights for causal factors.
Super Crunching isn't a substitute for intuition but a complement. While traditional expertise may face challenges, the future belongs to those comfortable in both worlds. Our intuitions, experiences, and statistics should work together to produce better choices, though intuition will still drive many day-to-day decisions where quantitative analysis isn't practical.
