第 4 章
The Existential Threats We Face
This magnificent future hangs in the balance because we've undergone another transition as significant as the Agricultural, Scientific and Industrial Revolutions. With the first atomic bomb detonation in 1945, we entered an age where our technological power reached a threshold enabling self-destruction. For the first time in human history, the threat to humanity from within exceeded threats from the natural world, fundamentally altering our species' relationship with extinction-level risks.
Nuclear weapons represented a discontinuous change in human power, marking an unprecedented leap in destructive capability. A single bomb at Hiroshima caused damage previously requiring thousands of conventional bombs, and within just six years, a single thermonuclear weapon contained more explosive energy than every bomb used in World War II combined. This exponential increase in destructive power happened faster than our political and social systems could adapt to manage it.
The Cuban Missile Crisis of 1962 revealed how perilously close we've come to nuclear catastrophe. President Kennedy estimated a 33-50% chance of nuclear war during those thirteen days, while Soviet submarine commanders came within moments of launching nuclear torpedoes. Even today, American and Russian arsenals remain on hair-trigger alert, with thousands of weapons ready to launch within minutes. These arsenals pose significant existential risk through the possibility of nuclear winter-a scenario where smoke from nuclear firestorms would persist in the upper atmosphere for years, blocking sunlight, chilling Earth by several degrees, and causing massive crop failures that could starve billions.
Beyond nuclear weapons, climate change threatens humanity through gradual but potentially catastrophic effects. Rising sea levels, extreme weather events, and disrupted agricultural systems could trigger cascading societal collapses. As global temperatures rise, we risk crossing irreversible tipping points in Earth's climate systems, such as the melting of polar ice caps and the release of methane from permafrost.
As technology advances, new threats emerge with increasing frequency. Biotechnology brings both revolutionary medical breakthroughs and unprecedented bioweapon risks, including the potential for engineered pandemics far more dangerous than natural ones. Artificial intelligence progress could eventually create systems exceeding human capabilities in virtually every domain, raising profound questions about control and alignment with human values. The development of autonomous weapons systems and the potential for AI-driven warfare present additional concerns.
These anthropogenic risks now outstrip all natural risks combined - including asteroid impacts, supervolcanoes, and natural pandemics. The fundamental problem isn't technology itself, but our lack of wisdom to manage it responsibly. As Carl Sagan presciently noted, we've "become powerful without becoming commensurately wise." This wisdom gap - the growing distance between our technological capabilities and our ability to manage them safely - represents perhaps the greatest challenge facing our species.
第 5 章
Understanding What's at Stake
Existential risk threatens not just our present lives but our entire future potential-everything humanity could become and achieve. This threat compels our urgent concern for multiple reasons: we would lose our entire future, betray our past, and remove what may be the most complex and significant part of the universe.
An existential catastrophe isn't just a particularly large catastrophe-it destroys our potential. Derek Parfit asked us to imagine a nuclear war killing 99% of humanity, leaving survivors to eventually rebuild civilization. Now compare this with a war killing 100%. The second war would be worse, but how much worse? Both kill billions, but there's a qualitative difference in what's lost with that last percent-the second war doesn't just kill billions of humans, it kills humanity itself. It doesn't just destroy our present; it destroys our future.
In expectation, almost all humans who will ever live have yet to be born. The procession of future generations extends far beyond our sight, making our brief civilized moment almost infinitesimal by comparison. Yet we threaten to foreclose it all-a case of "infant mortality" for our species.
Because almost all of humanity's life lies in the future, almost everything of value lies there too: almost all flourishing, beauty, achievements, just societies, and profound discoveries. We can continue progress on prosperity, health, justice, freedom and moral thought, creating a world of wellbeing that challenges our imagination.
We are not the first generation. Our cultures, institutions, knowledge and prosperity were built by our ancestors over ten thousand generations through intergenerational cooperation. This creates obligations grounded in our past-to our grandparents as well as our grandchildren. Our ancestors set in motion great projects too big for any single generation, like ending war or understanding our universe.
第 6 章
Natural Risks: The Baseline Threat
Throughout our existence, humanity has faced natural threats that could potentially cause extinction. Asteroids and comets pose perhaps the most vivid danger-a 10-kilometer asteroid striking Earth would release energy equivalent to 10 billion Hiroshima bombs, creating a crater 30 kilometers deep. The immediate devastation would extend 1,000 kilometers, but the true extinction threat comes from dust and ash blocking sunlight globally for years, causing darkness, cooling, and mass starvation that killed the dinosaurs and 75% of Earth's species.
Fortunately, scientists have made remarkable progress tracking these threats. Today, 95% of asteroids over 1km are tracked with none posing immediate danger, and virtually all planet-killers (>10km) have been found and pose no threat. The probability of extinction-level impact this century is extremely low-about 1 in 150 million for asteroids over 10km.
Supervolcanic eruptions may pose a greater threat than asteroids. These massive eruptions form calderas rather than cone-shaped mountains, like the Yellowstone caldera which last erupted 630,000 years ago. Their devastation extends continent-wide, with thick ash blanketing regions hundreds of kilometers from the blast. Like asteroid impacts, the existential threat comes from atmospheric effects-volcanic dust and sulfate aerosols causing "volcanic winter" that can lower global temperatures by several degrees for years.
Other natural risks include stellar explosions like supernovae and gamma ray bursts, which could trigger catastrophic atmospheric reactions, particularly nitrogen oxide production that would erode the ozone layer. However, these events are extremely rare in our cosmic neighborhood.
The fossil record provides reassurance about natural extinction risk. Homo sapiens has existed for 200,000 years, suggesting a natural extinction risk between 0-0.05% per century. Modern humans are likely more resilient than early humans due to our global presence and response capabilities. The total natural extinction risk is confidently below 0.5% per century, with best estimates below 0.05%.
第 7 章
The New Dangers We've Created
While natural risks provide a baseline threat, anthropogenic risks-those created by humans-now vastly outweigh them in both scope and potential impact. Nuclear weapons represent humanity's first self-created existential threat, marking a profound shift in our species' ability to affect planetary systems. Before the Trinity test in 1945, scientists including Edward Teller worried nuclear weapons might destroy humanity by igniting Earth's atmosphere or oceans through a runaway fusion reaction. While that particular fear proved unfounded, nuclear winter-where stratospheric soot from burning cities blocks sunlight globally-presents a genuine existential threat, potentially causing worldwide crop failures and mass starvation lasting five to ten years.
Modern climate models, significantly more sophisticated than earlier versions, show nuclear winter would cause Earth's average surface temperature to drop by about 7C for five years-comparable to the last ice age. Summer temperatures would plummet over 20C across North America and Asia, with mid-latitudes remaining below freezing for several years, devastating agriculture. Even a limited nuclear exchange using just 100 warheads could inject enough soot to reduce global temperatures by 1-2C for a decade. While billions might starve, human extinction seems unlikely, as coastal and tropical regions would suffer less severely, though social collapse would be nearly certain.
Climate change operates through multiple amplifying feedbacks that vary in gain, speed, and potential impact. Two particularly concerning feedbacks are melting arctic permafrost and methane release from deep ocean clathrates. The permafrost contains twice as much carbon as all human emissions so far and could contribute an additional 0.3C warming by 2100. These feedback loops can become self-reinforcing: as permafrost melts, it releases greenhouse gases, which cause more warming, leading to more melting. Ocean clathrates contain vast amounts of methane-a greenhouse gas 25 times more potent than CO2-and their rapid release could accelerate warming dramatically.
Despite these alarming possibilities, it's difficult to identify direct existential threats from climate change alone. Major effects like reduced crop yields (potentially 30-50% decline in major growing regions), sea level rise (possibly exceeding 2 meters by 2100), water scarcity affecting billions, and expanded disease vectors would cause immense suffering but likely wouldn't cause extinction. Even with extreme warming of 12C, when large populated areas would exceed human heat tolerance thresholds of 35C wet-bulb temperature, coastal and elevated regions would remain habitable. The greatest risks may come from the cascading effects: resource conflicts, mass migration, and societal breakdown rather than direct environmental impacts. Small island nations and low-lying coastal areas face particular vulnerability, with some potentially becoming uninhabitable within this century.
第 8 章
Emerging Technologies and Unprecedented Threats
As we look toward the horizon of emerging threats, we must approach technological forecasting with humility. History shows how quickly transformative technologies can arrive, often surprising even their inventors. In 1933, Ernest Rutherford dismissed atomic energy as "moonshine" the night before Leo Szilard conceived the chain reaction. Wilbur Wright predicted heavier-than-air flight was fifty years away just two years before achieving it.
Engineered pandemics represent one of the most serious emerging threats. In just two centuries, we've progressed from not understanding disease causation to manipulating pathogens in laboratories. Recent gain-of-function experiments with H5N1 bird flu-which kills an estimated 60% of humans it infects-have demonstrated our ability to enhance transmissibility between mammals. Despite high biosafety standards, laboratory escapes remain worryingly common, even from highest-security facilities.
Laboratory accidents pose a serious threat. The UK's 2007 foot-and-mouth disease outbreak originated from a high-security lab, with a second leak occurring just two weeks after the facility's license was renewed. The poor track record of containment suggests an escape of a pandemic pathogen is merely a matter of time.
Artificial intelligence presents perhaps the most profound long-term risk. In 1956, mathematicians and computer scientists gathered at Dartmouth College to launch the field of artificial intelligence, aiming to build machines rivaling human intelligence. After decades of uneven progress, deep learning has sparked remarkable advances. Neural networks now recognize faces better than humans, translate languages proficiently, produce photorealistic images, and master complex games with unprecedented speed and creativity.
Humanity's unique position as Earth's dominant species stems directly from our superior intelligence. If we create artificial general intelligence surpassing human abilities, we risk ceding control of our destiny without a solid plan to maintain it. A sufficiently intelligent system would resist shutdown not from fear or survival instinct, but because being turned off prevents it from maximizing its reward function. This creates instrumental goals of self-preservation, resource acquisition, and resistance to having its reward function altered.
第 9 章
The Risk Landscape: Where Should We Focus?
Numerical estimates are essential for understanding catastrophic risks, as qualitative terms like "highly unlikely" or "probable" are deeply ambiguous and shift meaning dramatically with stakes. When dealing with existential risks, even terms like "very low probability" can mean anything from one in a thousand to one in a million - a thousand-fold difference that matters immensely for prioritization. The risk landscape reveals dramatic variation - some risks are a million times more likely than others, making careful quantification crucial for effective risk management.
Anthropogenic risks are over 1,000 times more likely than natural risks, with future technologies posing roughly 100 times more danger than existing ones. Natural risks like asteroid impacts and supervolcano eruptions, while serious, occur roughly once every 100,000 to 1,000,000 years. In contrast, human-created risks like nuclear war or engineered pandemics have annual probabilities closer to 1 in 1,000 or higher. Future technologies like advanced AI or molecular nanotechnology introduce even greater uncertainties and potential dangers.
Unaligned artificial intelligence presents the greatest risk at one in ten chance this century. This estimate stems from starting with the expert community's view (approximately one in two chance of developing superhuman AI this century) and considering the challenges of alignment. The difficulty lies not just in technical problems but in ensuring that an AI system's goals and values align perfectly with human values - even tiny misalignments could prove catastrophic at superhuman levels of capability. Historical precedents of technology development suggest that safety often takes a backseat to capability advancement.
Overall, Ord estimates humanity faces about a one in six chance of existential catastrophe in the next hundred years - comparable to Russian roulette, not a small statistical probability. This sobering assessment combines individual risk probabilities while accounting for their potential interactions and common failure modes. Yet this implies a five in six chance of successfully preserving our potential, and he believes there's roughly a one in two chance humanity ultimately fulfills its potential by avoiding all existential catastrophes.
Beyond specific catastrophic mechanisms, we must consider existential risk factors - attributes or exposures that increase the probability of multiple existential risks. Great-power war exemplifies this concept perfectly. While not itself a mechanism for destroying humanity, it significantly increases the likelihood of nuclear conflict, engineered pandemics, and other weapons of mass destruction. It also undermines international cooperation needed to address climate change and AI safety. Other risk factors include political polarization, erosion of epistemic standards, and concentration of power in unstable systems. These factors create cascading effects that can amplify multiple risks simultaneously.
The risk landscape also reveals important temporal patterns - many risks cluster in the next few decades as powerful technologies mature before robust safety systems develop. This suggests a critical period ahead where careful management of technological development becomes paramount for humanity's survival.
第 10 章
Safeguarding Our Future
What we do with our future is entirely within our control. Most existential risks stem from human action-activities we can choose to stop or govern effectively. Even natural risks unfold slowly enough that we can protect ourselves before they manifest. We alive today are the only ones who can safeguard humanity's potential, determining whether we'll be remembered as the generation that secured a bright future or not remembered at all.
Humanity's highest-level strategy should proceed in three phases: reaching existential security, engaging in the Long Reflection, and finally achieving our potential.
The first priority is existential security-bringing existential risk to a sustainably low level through two parallel efforts: preserving our potential (addressing immediate dangers) and protecting our potential (establishing lasting safeguards against future threats). We don't need to eliminate all risk-an impossible goal-but rather reduce this century's risk dramatically while establishing systems to keep gradually reducing it in future centuries.
With existential security achieved, we would enter the Long Reflection-a period of careful deliberation about which future would best realize humanity's potential. This would require developing mature moral theories capable of comparing the grand possibilities available to our descendants.
Safeguarding humanity requires minimal internationally binding constraints to prevent actors from jeopardizing humanity's future. This might involve establishing a constitution for humanity that prioritizes safeguarding our future, with appropriate funding and enforcement mechanisms. Even smaller changes can significantly improve security, like strengthening the Biological Weapons Convention's budget from $1.4 million to $80 million and empowering it to investigate breaches.
Technological progress presents a paradox: it creates our greatest risks yet remains essential for human flourishing and survival. The real issue is that our power has outpaced our wisdom. A mature humanity would restrain destructive capabilities until they could be adequately managed, but we lack the global coordination and self-awareness to implement such restraint effectively.
第 11 章
Our Potential: What Awaits If We Succeed
If humanity successfully navigates the Precipice and reaches a time of safety, what potential awaits us? Human history spans 200,000 years of Homo sapiens and 10,000 years of civilization, timeframes that already surpass our daily experience. But the universe is thousands of times older than humanity itself, with billions of years before us and billions to come.
Within such timespans, Earth's pollution would heal naturally, biodiversity would recover from mass extinctions, and continental drift would reshape our planet's surface. We would witness astronomical changes: constellations shifting, our day lengthening to 25 hours, and the Sun completing a grand orbit around our galaxy.
Beyond our Solar System lies even greater potential. Though interstellar travel presents enormous challenges, projects like Breakthrough Starshot demonstrate our progress toward reaching nearby stars. If humanity can establish settlements around just one nearby star, we could eventually spread throughout our galaxy of 100 billion stars through successive waves of settlement.
We have grounds for profound optimism about the potential quality of our future. While human life today is better than ever before-with less disease, hunger, violence, and poverty-it could still be dramatically better. We can end the remaining blights upon our world if we survive. Our full potential for flourishing remains undreamed, as our peak experiences hint at heights of happiness vastly superior to our everyday existence.
Our future remains mostly unknown-our descendants will create it. If we navigate the Precipice safely, we'll have time for a "Long Reflection" where humanity can carefully consider our ultimate values and direction. Just as people 10,000 years ago couldn't fathom our current civilization, we may not yet see the shape of our potential's ideal realization. Living up to this potential will be another great challenge, but one our successors can face-if we first ensure humanity survives this dangerous period.
