Глава 1
Statistics: The High-Caliber Weapon of the Data Age
When Netflix recommends a movie you end up loving, or when Target sends pregnancy-related coupons to a teenage girl before her father even knows she's pregnant, you're experiencing the power of statistics in action. Charles Wheelan's "Naked Statistics" strips away the intimidating mathematical formulas to reveal how these powerful tools shape our everyday lives. The book has become a staple in university courses nationwide, praised by The Economist as brilliantly "exposing the sexy stuff underneath" complex statistical concepts. Even celebrities like Bill Gates have recommended it as essential reading for understanding our data-driven world. Unlike traditional textbooks, Wheelan's approach uses relatable examples-from baseball averages to casino profits to medical breakthroughs-to demonstrate how statistics helps us process information, whether trivial like sports statistics or profound like measuring income inequality. In a world increasingly governed by algorithms and big data, understanding statistics isn't just academically valuable-it's a survival skill for the 21st century.
Глава 2
The Power of Numbers: Describing Our World
Statistics serves three fundamental purposes: description, inference, and identifying relationships. Descriptive statistics transform overwhelming information into manageable insights-converting Derek Jeter's 9,868 at-bats into a simple .313 career batting average. These numbers help us answer both trivial questions (who was the greatest baseball player?) and profound ones (what's happening to America's middle class?).
When analyzing data, our first instinct is finding the "central tendency." The mean (average) is calculated by summing all values and dividing by the number of observations. However, means are easily distorted by outliers. When Bill Gates walks into a bar with ten $35,000-earners, the average income jumps to $91 million without anyone getting richer. This sensitivity to extreme values makes the median often more useful-the point dividing a distribution in half. Unlike the mean, the median remains $35,000 whether Bill Gates or Warren Buffett joins the group.
Beyond these basic measures, we use standard deviation to capture how dispersed data points are from their mean. Consider two groups with identical 155-pound average weights: airline passengers versus marathon runners. The passengers' weights vary dramatically (from tiny infants to 320-pound football players), while marathoners cluster more tightly. Standard deviation quantifies this difference with a single number.
The normal distribution-that familiar bell-shaped curve-describes many natural phenomena, from heights to popcorn popping patterns. Its elegance lies in mathematical predictability: exactly 68.2% of observations fall within one standard deviation of the mean, 95.4% within two, and 99.7% within three.
For complex phenomena that single statistics can't capture, we create indices combining multiple measures. The UN Human Development Index, for instance, combines income with life expectancy and education metrics. For baseball players, the best descriptive statistics are on-base percentage, slugging percentage, and at bats-by these measures, Babe Ruth stands as the greatest player ever due to his unique hitting and pitching abilities.
For measuring middle-class economic health, economists recommend examining changes in median wages (adjusted for inflation) over decades, which reveals American middle-class workers have been "running in place" for nearly thirty years while the 90th percentile has done much better.
Глава 3
When Statistics Lie: The Art of Deception
Statistics can be technically accurate yet deeply misleading. The crucial distinction between precision and accuracy explains how this happens-precision reflects exactitude of expression ("41.6 miles") while accuracy measures consistency with truth. Precision can mask inaccuracy by creating false certainty, as when McCarthy claimed to have a list of "205" communists in the State Department (the paper was blank).
The "health" of U.S. manufacturing perfectly illustrates statistical ambiguity-manufacturing output grew steadily while employment fell dramatically (six million jobs lost in a decade). Both stories are true: America remains the world's third-largest manufacturing exporter with rising productivity, but requires fewer workers to achieve this output.
The choice between mean and median can dramatically alter perception. The Bush administration touted tax cuts averaging $1,083 for 92 million Americans-technically true but misleading since the median cut was under $100. A small number of wealthy recipients skewed the average upward, while the median better reflected what typical households received.
Comparing unlike quantities creates meaningless statistics. Just as comparing hotel prices in pounds versus euros requires currency conversion, meaningful financial comparisons across time require inflation adjustment. Hollywood studios routinely manipulate box office comparisons by using nominal figures, making recent films appear more successful than classics. When adjusted for inflation, Gone with the Wind tops the list, while Avatar falls to 14th.
Even when we measure performance with appropriate statistics, we must ensure we're measuring what truly matters. School quality evaluations based solely on test scores present dangerously inaccurate pictures because they fail to account for students' different backgrounds and abilities. What matters isn't absolute achievement but the "value-added" by schools and teachers.
Statistical management creates perverse incentives. When administrators are evaluated on graduation rates, they may reclassify dropouts as "transfers." In Houston, schools reported a 1.5% dropout rate when the actual figure was 25-50%. New York's mortality scorecards for cardiologists demonstrate how statistics can backfire tragically. Rather than improving surgical techniques, doctors rationally responded by refusing to operate on the sickest patients who might benefit most from procedures.
Глава 4
The Magic of Correlation: Finding Patterns in Chaos
Correlation measures relationships between variables. Positive correlation means variables change in the same direction (like height and weight); negative correlation means they move in opposite directions (like exercise and weight). The correlation coefficient brilliantly collapses complex relationships into a single number between -1 and 1, with 1 indicating perfect positive correlation, -1 indicating perfect negative correlation, and 0 indicating no relationship.
The coefficient's magic comes from its unit-free nature. We can calculate correlation between height (inches) and weight (pounds) or between number of televisions and SAT scores. This happens by converting measurements to standard deviations from the mean, allowing comparison across different units.
The SAT exists because high school grades are imperfect descriptive statistics. Students with mediocre grades in challenging courses may have more academic potential than those with better grades in easier classes. The College Board created the SAT to "democratize access to college" by providing a standardized measure comparable across all students.
But the crucial question remains: Is it a good measure? The SAT does a reasonably good job of predicting first-year college grades, with a correlation of .56 between SAT scores and first-year college GPA-the same correlation as between high school GPA and college performance. The best predictor is actually a combination of SAT scores and high school GPA, with a correlation of .64.
One critical point: correlation doesn't imply causation. For example, there's likely a positive correlation between SAT scores and number of televisions in a household, but buying more TVs won't improve test scores. Both are probably influenced by parental education and income. Indeed, students with family incomes over $200,000 have mean SAT math scores of 586, compared to 460 for students from families earning $20,000 or less.
Netflix exploits correlation to recommend films. First, I rate movies I've seen. Netflix compares my ratings with those of other customers to identify viewers whose tastes correlate with mine. It then recommends films those like-minded customers rated highly that I haven't seen.
The actual methodology is far more complex. In 2006, Netflix launched a million-dollar contest challenging the public to improve its recommendation system by at least 10%. The winning team, announced in 2009, consisted of statisticians and computer scientists from four countries who developed an extraordinarily sophisticated algorithm.
Despite its complexity, Netflix's system is fundamentally a super fancy version of what people have always done: find someone with similar tastes and ask for recommendations. That's the essence of correlation.
Глава 5
Playing the Odds: Probability in Action
In 1981, Schlitz Brewing spent $1.7 million on a bold marketing campaign: live taste tests during NFL playoff games culminating in a Super Bowl halftime showdown between Schlitz and Michelob. The twist? They used 100 self-proclaimed Michelob drinkers as testers.
This wasn't as risky as it appeared. Schlitz understood that most mainstream beers taste similar enough that blind taste tests are essentially coin flips. If the test is truly random, roughly half of any brand's drinkers will prefer Schlitz in a blind test. The genius was testing only competitors' customers-when half of them inevitably chose Schlitz, it made for compelling advertising: "Half of all Michelob drinkers prefer Schlitz!"
Probability is the study of events and outcomes involving uncertainty. While I can't predict with certainty whether a coin will land heads or tails, I can determine that some outcomes (like getting two heads and two tails in four flips) are more likely than others (like getting four heads).
Our fears often don't align with statistical realities. When a NASA satellite was plummeting to earth in 2011, the probability of any individual being hit was 1 in 21 trillion, though the chance someone somewhere would be hit was 1 in 3,200. Similarly, Freakonomics revealed that backyard swimming pools are 100 times more dangerous to children than guns.
After 9/11, Americans' fear of flying led many to drive instead, resulting in an estimated 344 additional traffic deaths per month in late 2001. Cornell researchers calculated that this fear-induced shift to driving may have caused over 2,000 deaths-indirect casualties of the attacks.
Expected value is perhaps the most useful tool in managerial decision-making. It's calculated by summing all possible outcomes, each weighted by its probability and payoff. For a die-rolling game paying $1-$6 based on the number rolled, the expected value is $3.50.
While you can't actually earn $3.50 on a single roll, this figure tells you whether playing makes financial sense. If the game costs $3, it's worth playing because the expected value ($3.50) exceeds the cost. This concept applies to real-world decisions like football strategy-kicking an extra point (expected value 0.94) versus two-point conversions (expected value 0.74).
The law of large numbers proves that as trials increase, outcomes converge to the expected value. This explains why casinos always profit long-term and why lottery tickets (with expected payouts around $0.56 per $1 dollar ticket) are mathematically terrible investments.
Глава 6
When Probability Goes Wrong: Statistical Pitfalls
Statistics cannot be smarter than the people using them, and sometimes they make smart people do dumb things. The 2008 financial crisis exemplifies this, when Wall Street firms relied on Value at Risk (VaR) models to quantify risk. VaR collapsed complex information into a single number representing maximum potential loss with 99% probability over a specified timeframe. This false precision created dangerous complacency.
Two fatal flaws undermined VaR: First, the models used historical data that didn't account for unprecedented market shifts. Second, they ignored the catastrophic potential of the 1% tail risk-exactly what ultimately destroyed trillions in wealth. As hedge fund manager David Einhorn put it, VaR was "like an air bag that works all the time, except when you have a car accident."
A common probability error is assuming events are independent when they're actually related. The tragic misapplication of this principle occurred in Britain when pediatrician Sir Roy Meadow testified that multiple sudden infant deaths in one family must indicate murder. He calculated the probability as (1/8,500)2, or 1 in 73 million, wrongly assuming these deaths were independent events rather than potentially linked by genetic or environmental factors.
The opposite error occurs when people fail to recognize genuine independence. The "gambler's fallacy" exemplifies this-believing that after a series of black outcomes in roulette, red is "due." In reality, each spin remains independent with unchanged probabilities (16/38 for red). Even after flipping 1,000,000 heads in a row, the probability of tails on the next flip remains 1/2.
We often see patterns where none exist. Cancer clusters in specific areas may seem to suggest environmental causes, but could simply result from random chance. This is similar to lottery winners-the odds of any specific person winning might be 1 in 20 million, but we're not surprised when someone wins because millions of tickets are sold.
The "Sports Illustrated jinx"-where featured athletes subsequently perform worse-exemplifies reversion to the mean. Athletes typically appear on the cover after exceptional performances that deviate from their normal level. Their subsequent "decline" is simply a return to their typical performance.
Probability raises ethical questions about discrimination. Insurance companies traditionally charge different rates based on gender-men pay more for auto insurance because they crash more, while women pay more for annuities because they live longer. In 2012, the European Commission banned gender-based insurance premiums, not denying the statistical correlation but declaring the practice unacceptable.
Глава 7
The Foundation of Statistics: Quality Data
Data are to statistics what a good offensive line is to a star quarterback-they don't get much credit, but without them, you won't ever see success. No amount of fancy analysis can salvage fundamentally flawed data-hence "garbage in, garbage out."
We often need data samples that represent larger populations. The power of statistics comes from the fact that properly drawn samples can be as accurate as surveying an entire population. The simplest approach is a random sample where each observation has an equal chance of being included. Like drawing marbles from an urn-if there are 60% blue marbles and 40% red, a random sample should reflect similar proportions.
Size matters-larger samples reduce random variation, but a larger biased sample is actually worse than a smaller one because it creates false confidence.
Data often need to provide comparison between groups. Is a new medicine better than current treatment? Do charter school students outperform similar public school students? The goal is finding groups similar in all ways except for the "treatment" we're studying. The "gold standard" is randomization-randomly assigning subjects to treatment or control groups.
Sometimes we collect data with no specific purpose, suspecting it might prove valuable later. The Framingham Heart Study exemplifies this approach. Started in 1948 with 5,209 Massachusetts residents, this longitudinal study has tracked participants for decades, gathering data on everything from weight and blood pressure to smoking and diet. The study has produced over 2,000 academic articles, revealing crucial findings we now take for granted: smoking increases heart disease risk, physical activity reduces it, high blood pressure increases stroke risk.
Behind every important study lies good data, and behind every bad study lies garbage. While people often talk about "lying with statistics," many egregious mistakes involve lying with data-the statistical analysis might be fine, but the underlying data is bogus or inappropriate.
Selection bias occurs when samples don't represent the population they're meant to study. The infamous Literary Digest poll of 1936 demonstrates this perfectly. Despite sampling 10 million people, it wrongly predicted Alf Landon would defeat Roosevelt with 57% of the vote. Roosevelt won in a landslide with 60%. Why? The magazine's subscribers and telephone/automobile owners in 1936 were wealthier than average Americans, making them more likely to vote Republican.
Publication bias means positive findings get published while negative results languish in file drawers. Manufacturers of antidepressants like Prozac and Paxil published 94% of positive studies but only 14% of non-positive ones, misleading doctors and patients about effectiveness.
Глава 8
The Statistical Crystal Ball: Making Predictions
The central limit theorem is the "power source" for statistical inference, allowing us to draw sweeping conclusions from relatively small samples. It's the statistical principle that enables us to poll just 1,000 voters to predict an election or test 100 chicken breasts to determine if an entire processing plant is safe.
The central limit theorem's core principle is that a large, properly drawn sample will resemble the population from which it's drawn. Just as you could identify a bus of sausage festival attendees who are too heavy to be marathon runners, the theorem allows us to make powerful inferences from samples.
Sample means form a normal distribution around the population mean, with the distribution becoming tighter as sample size increases. This powerful property allows us to quantify our statistical intuition. The central limit theorem works regardless of the underlying population's distribution shape-whether it's household incomes skewed right or weights of Americans-the sample means will still form a bell curve around the true population mean, provided samples are large enough (generally at least 30).
Standard error measures how tightly sample means cluster around the population mean. Unlike standard deviation (which measures dispersion in the underlying population), standard error measures the dispersion of sample means. The formula is SE = /n, where is the population standard deviation and n is the sample size. This means sample means will cluster more tightly with larger samples and when the underlying population is less dispersed.
Statistical inference uses hypothesis testing to make conclusions from data. The process starts with a null hypothesis (initial assumption) that we try to reject based on evidence. For the Atlanta standardized test cheating scandal, the null hypothesis was that test scores were legitimate and unusual erasure patterns happened by chance. However, some Atlanta classrooms showed wrong-to-right erasures twenty to fifty standard deviations above state norms-a pattern so improbable that an official likened it to 70,000 people over seven feet tall showing up at a football game.
Researchers typically use significance levels to determine when to reject a null hypothesis. The common threshold is 5 percent (.05), meaning we reject the null hypothesis if there's less than a 5% chance of observing our results if the null were true.
Глава 9
Finding Deeper Patterns: Regression Analysis
Regression analysis helps researchers identify meaningful associations in data that cannot be studied through randomized experiments. When examining whether "low job control" truly causes heart disease, researchers must account for confounding factors like education, smoking habits, and income. Simple associations between job type and health outcomes aren't sufficient-regression analysis allows us to isolate the effect of one variable while controlling for others.
Regression analysis finds the "best fit" linear relationship between variables using ordinary least squares (OLS). This approach minimizes the sum of squared residuals-the vertical distances between data points and the regression line. For example, when analyzing height and weight data, regression produces an equation (y = a + bx) where the coefficient b describes the relationship between variables. From the Changing Lives study with 3,537 participants, the regression equation WEIGHT = -135 + (4.5) x HEIGHT IN INCHES shows that each additional inch of height is associated with 4.5 pounds of weight.
Multiple regression reveals complex relationships by estimating the effect of each variable while controlling for others. The Changing Lives study shows food stamp recipients weigh 5.6 pounds more than others, while non-Hispanic blacks weigh about 10 pounds more than others, even after controlling for all other variables.
A real-world application examined gender discrimination among MBA graduates. Initially showing women earning 45% less than men after ten years, regression analysis revealed most of this gap wasn't due to discrimination. By controlling for factors like finance coursework, GPA, work experience, labor force gaps, and hours worked, the unexplained portion shrank from 29% to just 1%.
Despite its power, regression can yield wildly misleading results when misused. The hormone replacement therapy case demonstrates this risk dramatically: statistical associations from observational studies suggested estrogen supplements protected women's health, but clinical trials later revealed they actually increased risks of heart disease, stroke, and cancer-likely causing tens of thousands of premature deaths.
When important variables are left out of regression analysis, their effects get wrongly attributed to included variables. If a study shows golfers have higher rates of heart disease without controlling for age, we might wrongly conclude golf causes health problems. In reality, golfers tend to be older, and age is the true driver of increased disease risk.
Глава 10
Statistics in the Real World: Solving Society's Problems
In our data-rich era, statistical tools can address significant social challenges, unlike earlier times when information was scarce and expensive to analyze. During the Great Depression, policymakers lacked basic economic metrics like GDP and unemployment figures, forcing them to "navigate through a forest without a compass." Today, we're awash in data that can help answer crucial societal questions.
Football faces a crisis as mounting evidence links the sport to permanent neurological damage. Studies show former NFL players suffer from dementia and memory-related diseases at rates 5-19 times higher than national averages. Researchers have documented tau protein buildup in players' brains causing chronic traumatic encephalopathy (CTE), while helmet sensors reveal players routinely experience head impacts equivalent to 25mph car crashes.
CDC data shows autism spectrum disorder diagnoses have nearly doubled in less than a decade, now affecting 1 in 88 American children (and 1 in 18 boys). The first statistical question is whether this represents a true epidemic or simply increased diagnosis of previously unrecognized cases. With lifetime management costs averaging $3.5 million per individual, understanding causes is crucial.
The seemingly logical approach to improving education is rewarding good teachers and schools while removing ineffective ones. Test scores offer objective performance measures, but students' results vary for reasons unrelated to classroom instruction. Value-added assessments attempt to solve this by measuring student progress over time while accounting for demographic factors and prior performance.
Economist Esther Duflo is transforming our knowledge about fighting global poverty by conducting randomized, controlled experiments on interventions. In rural Indian schools plagued by teacher absenteeism, she tested giving teachers attendance bonuses verified by tamper-proof cameras. The result? Absenteeism dropped by half, test scores improved, and more students advanced to the next education level.
Our unprecedented capacity to gather and analyze massive data sets has created new privacy concerns requiring new rules. Retailers like Target employ "predictive analytics" to understand customer behavior in extraordinary detail. Target's statisticians identified twenty-five products that together create a "pregnancy prediction score," allowing them to identify pregnant shoppers even before their families know.
As we wrestle with these issues, we must remember that while statistics is more important than ever, math cannot replace human judgment in determining appropriate data use. The power of statistics comes not just from the calculations themselves, but from the wisdom to know which questions to ask and how to interpret the answers we receive.
