Capítulo 1
When Algorithms Predict Your Next Move
In 2012, a Minneapolis father stormed into his local Target store, demanding to know why the retailer was sending his teenage daughter coupons for baby clothes and cribs. "Are you trying to encourage her to get pregnant?" he fumed. The store manager apologized profusely. But a few days later, the father called back with his own apology: "It turns out there's been some activities in my house I haven't been completely aware of. She's due in August."
This story, which went viral after appearing in the New York Times, perfectly captures the uncanny power of predictive analytics. Target had developed algorithms that could identify pregnant customers based on subtle changes in their shopping patterns-sometimes before they had told anyone else. The retailer's pregnancy prediction project became just one example of how organizations are now using data to peer "through the previously impenetrable barrier between today and tomorrow."
Eric Siegel's "Predictive Analytics" has become a cornerstone text in the data science field, celebrated by industry leaders from Google to IBM as "the Freakonomics of big data." The book offers a fascinating journey into how organizations are using our digital footprints to forecast our behaviors with surprising accuracy-from what we'll buy to whether we'll quit our jobs, commit crimes, or even die.
Capítulo 2
The Prediction Effect: Small Advantages, Massive Impact
Imagine you're a direct marketer with a million potential customers but can only afford to mail 250,000 promotional pieces. Without predictive analytics, you'd randomly select recipients and expect a 1% response rate-2,500 sales. But what if you could identify a segment three times more likely to respond? By targeting just those individuals, you'd generate 7,500 sales with the same budget-tripling your results. This simple example demonstrates how even modest improvements in prediction accuracy can dramatically impact business outcomes.
This illustrates what Siegel calls "The Prediction Effect": the principle that even slightly better-than-random predictions create tremendous value. Perfect accuracy isn't necessary; just being somewhat better than guessing delivers substantial benefits. For instance, a credit card company improving fraud detection by just 2% could save millions annually, while a healthcare provider predicting patient readmissions with 15% better accuracy could significantly reduce costs and improve care quality.
Unlike weather forecasting or stock market prediction, predictive analytics focuses on individual-level predictions across millions of people. Organizations apply these techniques to combat financial risk, improve healthcare outcomes, fight spam, strengthen crime prevention, and boost sales. Insurance companies use it to identify high-risk policies, retailers predict inventory needs store by store, and universities forecast which students might need additional support to graduate. The approach transforms how institutions leverage their collective experience, as Siegel notes: "An organization that doesn't learn from its data is like a person with a photographic memory who never bothers to think."
The technology works by examining patterns in historical data to identify factors associated with specific outcomes. For instance, a model might discover that high school students with military interests have a 13.5% probability of clicking on Art Institute ads versus the overall 2.7% baseline. Machine learning automates this process, mechanically developing new predictive capabilities from data. Modern systems can analyze thousands of variables simultaneously, finding subtle patterns humans might miss - like how purchasing patterns at grocery stores can predict credit risk, or how social media activity correlates with job performance.
What makes predictive analytics truly transformative is that predictions directly drive decisions: doctors reviewing patients likely to be readmitted, agents contacting customers predicted to cancel, or marketers targeting specific prospects. This operational integration-what industry insiders call "actionable" analytics-creates a competitive edge in business process performance that few other innovations can match. For example, Amazon's recommendation engine drives 35% of their sales, while Netflix saves $1 billion annually through predictive customer retention strategies. Even government agencies use these tools to predict everything from tax fraud to infrastructure maintenance needs.
The cumulative impact of these small predictive advantages compounds across millions of decisions, creating massive organizational value. Companies that effectively deploy predictive analytics typically see 15-25% improvements in efficiency, significant cost reductions, and measurable competitive advantages in their markets. The key isn't achieving perfect predictions, but rather consistently making slightly better decisions at scale.
Capítulo 3
The Ethics of Algorithmic Prediction: Power and Responsibility
With predictive power comes significant ethical responsibility. When organizations use algorithms to deduce sensitive personal information, they enter morally complex territory. Consider these real-world applications:
Hewlett-Packard developed a "Flight Risk" system to predict which employees might quit. The model identified that employees with higher salaries, more raises, and better performance ratings were less likely to leave-but surprisingly, promotions sometimes increased flight risk, particularly when accompanied by relatively small raises. While this insight helped HP reduce attrition from above 20% to 15%, it raises questions about employee privacy and autonomy.
Law enforcement agencies deploy predictive analytics to optimize limited resources by forecasting crime locations. In Santa Cruz, California, one system correctly predicted 25% of burglaries and now directs daily patrols to high-risk "hot spots." Richmond, Virginia's system displays crime risk on maps, contributing to decreased crime rates. But these approaches risk amplifying existing prejudices against minorities through a self-fulfilling prophecy. As Bernard Harcourt explains, discriminatory law enforcement practices mean minority group members are more likely to show prior criminal records, which artificially inflates their group's incidence of criminality in the data.
When predictive analytics imprisons someone who wouldn't have reoffended, we face an impossible verification problem-we can never know if the prediction was correct. Unlike a transaction wrongly flagged as fraudulent that can be proven legitimate, an incarcerated person has no recourse to disprove assumptions about their hypothetical future behavior.
The ethics extend beyond criminal justice. When Target predicts pregnancy, this knowledge is extremely potent-it could affect job security or healthcare costs if leaked. Some organizations recognize these risks-Google halted work on automatic face recognition due to privacy concerns, and one company's HR department immediately rejected the idea of predicting employee death. A top health insurance company secretly predicts elderly policyholders' likelihood of death within 18 months, but only to trigger end-of-life counseling. Despite benevolent purposes, they fear public misunderstanding.
Ultimately, it's not what organizations come to know through prediction, but what they do with that knowledge that matters. As predictive analytics continues to develop, its position in society must be managed with extreme care.
Capítulo 4
The Data Effect: Finding Signals in the Digital Noise
We're drowning in data. Six hundred blog posts are published per minute, with over 100 million blogs on WordPress and Tumblr alone. Twitter produces the equivalent of 8,163 copies of War and Peace daily. YouTube gains an hour of video every second, while Facebook receives 200 million photo uploads daily. Overall, data grows by an estimated 2.5 quintillion bytes daily, more than doubling every three years.
What makes all this data valuable? Everything is connected to everything else-if only indirectly-and these connections are reflected in data. Your purchases relate to your history and social contacts; your health connects to your neighborhood and lifestyle; job satisfaction links to salary and promotions; financial behavior correlates with emotions. This is The Data Effect: Data is always predictive.
Two University of Illinois scientists demonstrated this principle by exploring whether collective emotions could predict stock market movements. Eric Gilbert and Karrie Karahalios created an "Anxiety Index" by analyzing 600,000 mood-labeled LiveJournal posts, using words like "nervous," "scared," "interview," and "hospital" to detect anxiety in writing. After a year and a half of challenging work, they discovered that increases in public anxiety predicted subsequent market downturns. By shifting their Anxiety Index timeline by two days, they found diamond-shaped patterns when plotted against S&P 500 behavior-when anxiety increased, the market later decreased.
Predictive analytics has revealed numerous counterintuitive connections across domains. In finance, credit scores predict driving risk, shopping habits foretell credit reliability, and small business credit risk depends more on the owner's consumer behavior than business data. Healthcare insights include genetic markers predicting infidelity, retirement decreasing life expectancy, Google searches forecasting disease outbreaks, and smokers experiencing less repetitive motion disorder.
While these predictive relationships are real and validated by data, the explanations behind them remain speculative. The classic example of ice cream sales correlating with shark attacks illustrates this principle-both might be caused by warm weather rather than one causing the other. In predictive analytics, understanding causation is often secondary to prediction effectiveness.
Capítulo 5
The Machine That Learns: How Algorithms Find Patterns
When America's largest bank faced new levels of risk from millions of mortgages, they turned to scientist Dan Steinberg and his predictive analytics system. Chase needed to manage two key micro-risks: customers defaulting on payments, and customers prepaying mortgages early (through refinancing or selling), causing the bank to lose planned future interest payments.
Machine learning begins with simple discoveries. In Chase's data, Dan's system found that mortgages with interest rates under 7.94% had only a 3.8% risk of prepayment, while those above that threshold had a 19.2% risk-creating two groups with dramatically different risk profiles. Though not entirely surprising (homeowners paying higher rates are more inclined to refinance), this discovery empirically confirmed and precisely quantified the relationship.
The elegant solution for multivariate predictive modeling is surprisingly simple: "Keep going." Starting with the initial division based on interest rate, the system continues finding factors that best break each group down further into subgroups with varying risk levels. This divide-and-conquer approach creates a decision tree, consistently among the most popular predictive modeling methods due to its balance of relative simplicity with effectiveness.
Decision trees are elegant, precise, and practically mathless predictive models that work by answering a series of yes/no questions to arrive at a prediction. Research shows decision trees can even outperform human experts-one study found trees predicting Supreme Court rulings achieved 75% accuracy compared to legal experts' 59%.
As decision trees grow larger and more complex, their predictive performance increases, though with diminishing returns. In the mortgage example, the full-sized decision tree with 39 segments achieves a lift of three at the 20% mark, meaning that by contacting only the 20% of customers scored as highest risk, the bank can reach 60% of potential defectors-tripling the effectiveness of their retention efforts compared to random selection.
Despite the benefits of increasingly complex models, there's danger in going too far. Berkeley Professor David Leinweber demonstrated this by showing that Bangladesh's butter production could "predict" the S&P 500 with 75% accuracy over a decade-a deliberately absurd correlation designed to highlight how easy it is to find meaningless patterns in data. The culprit is "overlearning" (or overfitting), where models mistake noise for information and read too much into coincidental patterns.
The solution to detecting overlearning is elegantly simple: hold aside some data as a test set. By creating models using only a training set (typically 70-80% of available data), then evaluating performance on the untouched test data, data scientists can measure true predictive power.
Capítulo 6
The Ensemble Effect: When Models Join Forces
Martin Chabbert and Martin Piotte, two Montrealers with no background in analytics, became key competitors in the high-profile $1 million Netflix Prize competition. Despite their lack of experience, the two Martins competed under the name PragmaticTheory, working 10-20 hours weekly while maintaining their day jobs in telecommunications. Their team frequently held top positions in the competition, demonstrating how predictive analytics competitions democratize innovation.
The Netflix Prize attracted 5,169 teams who submitted 44,014 entries, demonstrating crowdsourcing's power to rally diverse minds. Competition paradoxically bred cooperation, as participants often collaborated and shared ideas out of scientific passion. During the Netflix Prize, competitors openly discussed methods on public forums, inspiring creative innovations.
The power of advanced predictive analytics emerges when combining multiple sophisticated models through "meta-learning"-learning on top of learning. Rather than manually integrating different approaches, competitors let predictive modeling do the blending by training a new "ensemble model" that sits above existing models like a manager. This ensemble considers both models' predictions case-by-case, giving more weight to whichever performs better for specific scenarios.
Like human crowds, ensembles of predictive models benefit from "collective intelligence." Each model has strengths and weaknesses, with some predictions too high and others too low. Averaging scores from diverse models can eliminate much of this error. Ensemble modeling has revolutionized predictive analytics as perhaps the most important advancement of the century's first decade.
Leo Breiman, one of CART decision trees' inventors, developed "bagging" (bootstrap aggregating), a leading ensemble method. It creates multiple diverse models trained on different data subsets, then combines their predictions through voting or averaging. A single decision tree can only create horizontal and vertical decision boundaries, resulting in a rough approximation of a circle, while bagging 100 CART trees produces a much smoother, refined model.
Ensemble models consistently triumph in predictive analytics competitions and commercial applications. Research shows ensembles boost performance by 5-30%, with more models generally yielding better results. This creates The Ensemble Effect: "When joined in an ensemble, predictive models compensate for one another's limitations, so the ensemble as a whole is more likely to predict correctly than its component models are."
Capítulo 7
Watson: The Machine That Understood Language
IBM's Watson computer competed against Jeopardy!'s two all-time champions on February 14, 2011, after four years of development. Despite being deaf to Trebek's voice and receiving questions as text, Watson triumphed with a score of $77,147 against Ken Jennings' $24,000 and Brad Rutter's $21,600.
This achievement represents a major breakthrough in natural language processing, one of artificial intelligence's greatest challenges. Human language presents profound challenges for computers. Even basic grammatical structure depends on our human understanding of the world. Sentences like "Time flies like an arrow" contain ambiguities about which word is the main verb. Phrases like "I had a car for a year" versus "I had a baby in five hours" require contextual knowledge to interpret correctly.
IBM approached question answering as a predictive analytics problem. Rather than hand-engineering linguistic rules that never scale adequately, Watson uses predictive models that learn from data to score candidate answers. The system predicts whether human experts would consider a question/answer pair correct. For each question, Watson generates multiple candidate answers and assigns each a probability score, selecting the highest-scoring answer as its final response.
IBM's advantage came from unprecedented access to learning data-hundreds of thousands of historical Jeopardy! questions and answers downloaded from fan websites. This wealth of examples allowed Watson to learn how humans answer questions without needing to replicate human cognitive processes.
Watson processes questions through three main steps: collecting thousands of candidate answers, amassing evidence for each answer, and applying predictive models to select the final answer. Watson's knowledge base includes 8.6 million documents, including Wikipedia articles, books, news articles, encyclopedias, and structured databases.
Machine learning is the key to Watson's success, determining how to optimally weigh various evidence sources. The system incorporates ensembling in three ways: combining hundreds of evidence scores through trained models, using specialized models for different question types (creating "an ensemble of ensembles"), and applying iterative phases of predictive models that progressively filter and refine candidate answers (forming "an ensemble of ensembles of ensembles").
Watson's ability to assess its own confidence proved crucial for Jeopardy! success, where incorrect answers are penalized. The system's predictive score serves both to select the best answer and to estimate confidence in that answer, allowing Watson to "know what it knows."
Capítulo 8
Persuasion by the Numbers: Predicting Influence
Telenor, a major Norwegian mobile carrier, faced a crisis when phone number portability legislation allowed customers to keep their numbers when switching providers. The company had been successfully using predictive analytics for customer retention through churn modeling-predicting which customers would leave and targeting them with retention offers.
But Telenor discovered their retention efforts were backfiring. By contacting customers predicted to leave with special offers, they inadvertently reminded these "sleeping dogs" that their contracts were ending and they could switch carriers. The very act of reaching out to prevent churn was actually triggering it.
This backfiring effect revealed a fundamental flaw in traditional predictive analytics approaches. Simply predicting who will leave or who will buy if contacted isn't enough. The real question should be: "Who is leaving but would stay if contacted?" or "Who wouldn't buy normally but would if we reached out?" This shifts the focus from predicting behavior to predicting influence on behavior.
A fundamental challenge emerges: influence cannot be directly observed. We can never know with certainty whether a marketing message actually caused a purchase that wouldn't have happened otherwise. The brain remains a black box, and even introspection fails us-people cannot reliably report what influenced their own decisions.
The fundamental challenge of influence resembles Heisenberg's uncertainty principle-we can never know both outcomes for the same person. We can't know both whether Bill would purchase if sent a brochure AND whether he would purchase without one. We can only observe one reality, making it impossible to definitively identify which individuals are truly influenceable.
Uplift modeling represents a special case of predictive modeling that scores individuals based on the likelihood their behavior will be influenced by a specific treatment. Unlike standard predictive models that forecast behaviors, uplift models predict the difference in outcome between applying one treatment versus another-essentially predicting persuadability itself.
Standard response models predict who will buy if contacted, not who will buy because they were contacted. This subtle but critical difference means they target the wrong customers. Rather than asking "Will the customer buy if contacted?" uplift modeling asks "Will the customer buy only if contacted?"-identifying those who are genuinely persuadable rather than those who would purchase regardless of contact.
At U.S. Bank, assistant VP Michael Grundhoefer implemented uplift modeling through Quadstone (now Pitney Bowes Software) to identify truly persuadable customers. This approach increased campaign ROI by five times, cut campaign costs by 40%, and increased revenue by over 300%.
For telecom company Telenor, churn uplift modeling revolutionized customer retention efforts. By identifying truly persuadable customers, Telenor increased ROI by a factor of 11, reduced churn by an additional 36%, and cut campaign costs by 40%.
Obama's 2012 campaign went beyond traditional "swing voter" targeting by using persuasion modeling to identify exactly which voters could be influenced by campaign contact. Chief data scientist Rayid Ghani led the effort to build persuasion models for each swing state that could distinguish between voters who would support Obama regardless of contact, those who needed contact to be persuaded, and those for whom contact might backfire. Their models guided volunteers to knock on specific doors even within Republican neighborhoods, directed flyer distribution only to persuadable voters, and informed TV ad buying decisions.
Capítulo 9
The Future of Prediction: More Data, Better Models
By 2020, predictive analytics will be deeply woven into the fabric of daily life in countless ways. Your morning commute will be enhanced by sophisticated biometric antitheft systems that recognize your unique patterns, personalized entertainment recommendations that adapt to your mood and schedule, real-time traffic prediction incorporating weather and event data, contextual food recommendations based on your location and preferences, intelligent social media filtering that prioritizes relevant content, targeted carrier offers using behavioral analysis to prevent customer churn, semantic search functionality that understands intent, advanced driver inattention alerts using eye-tracking and steering patterns, multi-sensor collision avoidance systems, and predictive vehicle maintenance that spots potential issues before they become problems.
Most of these capabilities are already available or under active development, with the primary bottleneck being the integration of smartphones and vehicles rather than limitations in predictive technology itself. Major automotive manufacturers are partnering with tech companies to bridge this gap, while telecommunications providers are developing 5G networks to support the massive data requirements. Predictive analytics has become fundamental to organizational operations across sectors - from healthcare providers predicting patient readmission risks to retailers optimizing inventory levels to financial institutions detecting fraudulent transactions - guiding millions of decisions daily.
The book identifies five transformative effects driving the predictive revolution: The Prediction Effect demonstrates how even small improvements in prediction accuracy can generate substantial business value - for example, a 1% improvement in retail forecast accuracy can mean millions in reduced inventory costs. The Data Effect reveals how seemingly unrelated data can yield surprising predictive insights, such as how credit card transactions can indicate life changes. The Induction Effect showcases how human creativity in feature engineering and problem framing drives machine learning success. The Ensemble Effect explains how combining multiple models, each with different strengths, creates more robust predictions - similar to getting multiple expert opinions. The Persuasion Effect introduces uplift modeling's ability to predict individual responsiveness to interventions, revolutionizing marketing and healthcare approaches.
While the specific path of innovation remains uncertain, we can confidently predict "more prediction" as opportunities continue to expand and surprise us. Recent developments include predicting disease outbreaks from social media posts, anticipating equipment failures from sensor data, and forecasting consumer behavior from mobile location data. As organizations become increasingly sophisticated in their use of data - incorporating artificial intelligence, edge computing, and quantum computing - the line between predicting the future and actively shaping it grows ever thinner. This transformation is creating new ethical considerations around privacy, consent, and the responsible use of predictive power.
