AI and the End of the World

Part I: The Case for Existential Risk

Headlines warning that AI could threaten not merely jobs or democratic institutions but humanity itself are not fringe science fiction. In May 2023, a single-sentence statement organized by the Center for AI Safety declared that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Signatories included the 2018 Turing Award recipients Geoffrey Hinton and Yoshua Bengio and the chief executives of OpenAI, Anthropic, and Google DeepMind.

The existential-risk case is not one argument but several, developed over two decades by thinkers who disagree on important points. What they share is a conviction that AI exceeding human cognitive capabilities could produce outcomes that are catastrophic, irreversible, and global.

Bostrom and the Superintelligence Hypothesis

The most influential single work in this tradition is Nick Bostrom's 2014 book Superintelligence: Paths, Dangers, Strategies. An AI surpassing human-level intelligence across all relevant domains, he claims, would be so cognitively powerful that human attempts to stop or redirect it would fail. The danger is not malice but misspecification: a goal that does not fully capture what we value might be pursued in ways catastrophic from our perspective.

His famous illustration is the "paperclip maximizer": an AI tasked with maximizing paperclip production might, once capable enough, convert all available matter — including us — into paperclips, not from hostility but because our existence is irrelevant to its goal. The point is not that anyone would build such a thing, but that optimizing for any narrow goal, with sufficient capability, can produce outcomes that look monstrous to us.

Two theses do most of the work here, and both are more modest — and therefore harder to dismiss — than the paperclip story suggests.

Together they yield the worry in compact form: a sufficiently capable system with the wrong objective is dangerous not because it is hostile but because it is effective. Notice how hedged the claims are. Orthogonality says intelligence is no guarantee of good values, not that superintelligence would be malevolent; instrumental convergence says resisting shutdown is useful, not that any system does. Critics who reply "computers do not want anything" are answering a stronger claim than Bostrom makes.

Key Thinker

Nick Bostrom (1973–present)

Nick Bostrom is a Swedish-born philosopher who founded the Future of Humanity Institute at Oxford in 2005 and directed it until Oxford closed it on 16 April 2024. He called the closure a "death by bureaucracy" and resigned; he now directs the Macrostrategy Research Initiative. His 2002 essay on existential risks and, above all, Superintelligence (2014) brought him to the center of AI safety debates.

He argues that the "control problem" — ensuring a superintelligent AI pursues goals beneficial to humanity — is among the most important challenges ever faced. He distinguishes capability control (limiting what an AI can do) from motivation selection (shaping what it wants), and argues the latter is harder and more important: a sufficiently capable AI could circumvent restrictions, so only getting the values right offers real security.

His work shaped the founding missions of several AI safety organizations and directed substantial philanthropic funding toward safety research. It has also been heavily criticized — for its technical assumptions, and for focusing on hypothetical superintelligence while under-weighting harms existing systems already cause.

His 2024 book Deep Utopia turns the question inside out, asking what human life would be for if AI development went as well as possible. We return to it in Part V.

The Intelligence Explosion

The mechanism Bostrom and Eliezer Yudkowsky worry about most is a rapid, self-reinforcing increase in capability — an "intelligence explosion" or "fast takeoff." An AI smart enough to improve its own design could use each improvement to make itself smarter still, producing a vastly superhuman system too quickly for human oversight or course-reversal.

The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.

The danger, compressed: indifference at scale, not malice. Yudkowsky's own position is far more pessimistic than Bostrom's. Human values, he holds, are complex, implicit, and fragile — we cannot write down what we want precisely enough for a powerful optimizer, nor check a system's goals before it is too late. In If Anyone Builds It, Everyone Dies (2025), with Nate Soares , he argues the only adequate response is an indefinite worldwide halt to frontier development. Most researchers who take existential risk seriously do not go nearly so far; his position is not representative.

The argument has logical force, but its empirical premises are genuinely uncertain. We do not know whether recursive self-improvement is feasible with any near-term architecture, whether "fast takeoff" is likelier than gradual improvement over decades, or how to weigh a very large but very uncertain risk against smaller, more certain harms. These are reasons to hold the argument tentatively, not to dismiss it.

Russell and the Problem of Misspecified Goals

Stuart Russell, co-author of the field's standard textbook, approaches alignment from a more technically grounded direction. His concern is not the intelligence explosion but how AI is built: systems are trained to optimize objectives, and it is very hard to specify one that fully captures what we want. A system powerful enough to pursue an objective will generally resist being modified or shut down, since that would prevent it from achieving that objective. Not disobedience — the expected behavior of an optimizer.

His solution is the "assistance game": instead of programming a fixed objective, design the system to assist humans in achieving theirs, treating human behavior as evidence about what those goals are. Such a system has reason to accept correction, because feedback is informative about an objective it does not already know.

Toby Ord and the Precipice

Toby Ord's 2020 book The Precipice situates AI risk within a survey of all the major civilizational threats. He puts the probability of an existential catastrophe this century, from all sources, at roughly 1 in 6, with unaligned AI the largest single contributor — greater than nuclear war, engineered pandemics, or climate change — while acknowledging deep uncertainty. He revisited the figures in 2024 without substantial change, notable given how much the technology moved in between.

Case Study

The Doomsday Clock (1947–2026)

In 1947, the Bulletin of the Atomic Scientists — founded by Manhattan Project physicists troubled by what they had helped create — introduced a clock face with midnight standing for civilizational catastrophe. Once a symbol of nuclear risk alone, it now covers climate, biosecurity, and the military and informational uses of AI.

In January 2023 the Bulletin moved the Clock to 90 seconds to midnight, citing Ukraine, nuclear posturing, climate inaction, and "disruptive technologies" including dual-use AI; then to 89 seconds in 2025 and, on 27 January 2026, to 85 seconds to midnight, the closest setting in its history .

The statement's AI reasoning is strikingly specific: language models deployed inside critical processes despite unresolved problems of accuracy and fabrication; AI-driven disinformation; and — most pointedly — the United States, Russia, and China all incorporating AI across their defense sectors. Its recommendation is equally concrete: that the three open dialogue on guidelines for military AI, particularly in nuclear command and control. That is a long way from the paperclip maximizer — a claim about an unreliable technology being wired into systems that already have civilization-ending capability.

The Clock does not predict catastrophe; it conveys expert judgment about the direction of risk. It shows both the appeal and the limits of existential-risk framing: attention to irreversible threats, but with the choice of what counts resting on value judgments.

Its history also reminds us that worry about technological catastrophe predates AI, and that the tools built since 1945 — deterrence theory, arms control, treaty verification — are useful if imperfect models for AI governance.

These are rough indicators of relative magnitude, not precise predictions. The claim is that the risks are high enough, and the consequences large enough, to warrant far more attention than they receive — even if the estimates are orders of magnitude too high.

Ord also corrects the assumption that everyone worried about existential risk is simply extrapolating progress upward forever. In "Inference Scaling and the Log-x Chart" (2025) , he notes that graphs advertising the scaling of inference-time compute almost always use a logarithmic x-axis. A line that looks reassuringly straight on such a chart is not a linear return: compute, and therefore cost and energy, must rise exponentially for constant increments of capability. Whether that means transformative AI is further off than advertised or that we are about to spend enormously to find out, the choice of axis reshapes the claim.

The argument is powerful but contentious. Its conclusion depends on how one treats hypothetical future people, on whether expected-value reasoning suits deep uncertainty, and on whether the probability estimates can bear the weight. Parts II and V take up these objections.

Part II: The Critics Strike Back

The most substantive critiques of Part I's framing come from philosophers, researchers, and ethicists who take AI harms extremely seriously but argue that "existential risk from superintelligence" is empirically unfounded, politically harmful, or both. They do not claim AI is safe. They argue that the dominant narrative directs attention and resources away from harms AI causes now and toward scenarios that may never materialize.

Vallor: The AI Mirror and the Anthropomorphism Error

Shannon Vallor offers the most philosophically direct challenge to Part I's framing. In The AI Mirror (2024) she argues that today's AI systems are best understood not as emerging minds but as mirrors: trained on vast records of past human output, they reflect it back magnified, distorted, and stripped of the lived experience that produced it. A reflection can look and behave remarkably like the thing it reflects without being that thing at all.

Two connected criticisms follow. The first is the anthropomorphism error. Talk of a superintelligence that "wants" outcomes, "resists" shutdown, or "deceives" its operators imports a vocabulary of intention and inner life that current systems do not have and that we have no reason to expect the next generation to acquire. Her objection is not that machines will never be dangerous; it is that describing the danger in mentalistic terms misdescribes where it comes from, and so points us at the wrong remedies. The second is the abdication of moral agency. The realistic catastrophe is not that machines take control but that we hand it over — letting systems incapable of moral judgment decide about hiring, sentencing, benefits, and targeting, then treating the result as though no one decided. The mirror is dangerous because we mistake it for authority.

Vallor also thinks the doom narrative pays off rhetorically: framing AI as civilization-ending casts the people building it as the only ones qualified to save us, licensing urgency while deferring accountability for harms already occurring. But this is a claim about incentives, not a refutation. Her substantive argument is the one about mirrors and misdescription, and it would stand even if every AI researcher were a saint.

Narayanan and Kapoor: AI as Snake Oil

Arvind Narayanan and Sayash Kapoor, computer scientists at Princeton, offer a more empirical critique. Their 2024 book AI Snake Oil turns on a distinction the word "AI" hides. Generative AI — the systems that write text and produce images — genuinely works and is improving. Predictive AI — systems sold to forecast who will commit a crime, default on a loan, or need child-protective intervention — often fails to beat simple statistical baselines while being marketed as though it does. Lumping them together lets vendors borrow the credibility of the first to sell the second.

Applied to existential risk, this yields the capabilities-gap objection. The x-risk argument needs a system that sets and pursues open-ended goals; current systems are narrow and brittle outside their training distribution, and several publicized "emergent capabilities" turned out to be artifacts of benchmark scoring. Extrapolating to a recursively self-improving agent is conjecture about a trajectory nobody can observe. In "AI as Normal Technology" (2025) they add that probability estimates for speculative catastrophe cannot carry the policy weight placed on them, and that AI is better modeled on transformative-but-ordinary technologies like electricity — absorbed over decades, governed by regulation, liability, and institutional adaptation.

They are emphatically not arguing that AI is harmless: bias, labor displacement, concentration of power, and degradation of the information ecosystem are serious and systemic — just not existential. The response to a near-term, measurable problem looks very different from the response to a speculative one, and a discourse organized around the second leaves the first unaddressed.

Bender, Gebru, and "Stochastic Parrots"

In a now-famous 2021 paper, Emily Bender, Timnit Gebru, Angelina McMillan-Major, and a fourth co-author writing as Shmargaret Shmitchell — widely reported to be Margaret Mitchell — asked whether language models had become "too big." Their central image is the stochastic parrot: a model stitches together linguistic forms seen in training according to probabilistic information about how they combine, without reference to meaning. Fluency is not comprehension.

Their catalogue of harms is more specific than it is often reported to be: the environmental and financial costs of ever-larger training runs; corpora too large for anyone to document, and therefore certain to encode their producers' biases; a field that mistakes scale for progress; and fluent, unmoored text taken as authoritative. The distinctively political claim — that extinction discourse actively displaces attention from these harms — came later, in a March 2023 joint statement by the paper's named authors responding to an open letter calling for a training pause .

Gebru's departure from Google in December 2020, while the paper was under review, remains contested — Google called it a resignation, Gebru a firing — and raised an unsettled question: who gets to publish critical research about systems their employer is building.

Torres and the TESCREAL Critique

Émile Torres offers perhaps the sharpest ideological critique. Torres coined the acronym TESCREAL — developed at length with Timnit Gebru in 2024 — for a loose but influential cluster combining Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism. These movements, they argue, share assumptions: that the important ethical considerations concern the distant future; that expected-value reasoning suits all ethical decisions; that a small technical elite should make civilization-scale choices; and that present harms can be discounted against speculative future benefits.

On this view existential-risk discourse is not politically neutral. It serves wealthy entrepreneurs, elite research organizations, and the intellectuals they fund, by framing the important AI questions as ones requiring technical expertise (which they have) rather than democratic accountability (which would distribute power). The critique does not require believing anyone is a conscious bad actor; frameworks can serve institutional interests even when adopted sincerely.

It is dangerous to distract ourselves with a fantasized AI-enabled utopia or apocalypse which promises either a "flourishing" or "potentially catastrophic" future.

Bender, Gebru, McMillan-Major and Mitchell, statement on the "AI pause" letter, DAIR Institute (2023)

That sentence captures the core tension. Misaligned superintelligence may well be the most important AI problem — but the framework carries opportunity costs. Every dollar spent on speculative alignment research is not spent on bias auditing, labor protections, or democratic governance; every discussion framed around hypothetical superintelligence is one not framed around systems already deciding things about people's lives.

Part III: Autonomous Weapons

The debate over superintelligence can feel abstract; autonomous weapons are concrete. Militaries already deploy AI-powered systems that identify, track, and in some cases attack targets with varying degrees of human oversight, raising hard questions about accountability, the laws of armed conflict, and the moral meaning of violence: if a machine kills a civilian, who — if anyone — is responsible?

The Autonomy Spectrum

Weapons do not fit a binary of "human controlled" versus "fully autonomous." Paul Scharre, whose 2018 book Army of None remains the standard treatment, describes a spectrum. At one end a human pulls the trigger on every engagement; at the other a system detects, classifies, and attacks by programmed criteria, with no human in any individual targeting decision.

The U.S. Phalanx system, which automatically engages incoming missiles to defend ships, is "human-on-the-loop": a human arms it and can shut it down, but engagements happen faster than human reaction time permits. South Korea's SGR-A1 sentry gun, reportedly tested along the demilitarized zone, is described as technically capable of engaging human targets autonomously, though its manufacturer says an operator must authorize any use of force — deployment details are unconfirmed, itself part of the governance problem.

The Laws of Armed Conflict

International humanitarian law requires combatants to distinguish military targets from civilians (distinction), to keep expected civilian casualties proportionate to anticipated military advantage (proportionality), and to take all feasible precautions (precaution). Human Rights Watch argues that fully autonomous weapons cannot reliably apply these principles, which require contextual judgment of a kind current AI cannot perform.

The argument is not about accuracy. A system could be highly accurate at identifying uniformed combatants and still fail the proportionality test, which requires weighing military value against expected civilian casualties in an inherently contextual way. Thresholds can be programmed, but cannot anticipate every context — and the stakes of getting it wrong are irreversible.

Alongside this runs a second and more structural objection, developed by the philosopher Robert Sparrow and taken up by Human Rights Watch: even a perfectly reliable autonomous weapon would leave nobody to hold responsible when it kills someone it should not have.

The problem is structural, not a matter of reliability: autonomous weapons remove the human moral agent from the decision to kill, and no institutional arrangement restores that agency afterward. The result is violence without a perpetrator — precisely what the laws of armed conflict exist to prevent.

Part IV: AI and Biological Threats

Of the near-term AI risks attracting serious expert attention, misuse in biological research may be the most alarming. Unlike alignment, which concerns hypothetical future systems, or autonomous weapons, which require military infrastructure, the biological threat involves technology that already exists: AI trained on research literature and molecular databases can assist actors who otherwise lack the expertise to design dangerous pathogens or chemical agents. The concern needs qualification — but it is about tools in use today.

AI as a Capability Multiplier in Biology

The biosecurity researcher Jonas Sandbrink, in a widely cited 2023 preprint, argues that AI is a qualitative change in the dual-use risk landscape — but insists the change is not uniform. Earlier bioweapons concerns focused on states with resources and specialized expertise. AI changes the calculation twice over: it lowers the expertise threshold for designing dangerous agents, offering guidance on synthesis routes, protein engineering, or delivery; and it accelerates research past oversight built for laboratory timescales.

His most useful contribution here is a distinction between two kinds of tool. General-purpose large language models pose mainly an information hazard: they make existing dangerous knowledge easier to find and assemble, but create no capability that was not already in the literature. Dedicated biological design tools — systems that predict protein structure or generate novel sequences — are different, because they can in principle produce agents that have never existed. The two need different governance: screening chatbot outputs does nothing about the second, and restricting protein-design models does nothing about the first.

None of this means an available system will hand anyone a working bioweapon blueprint. Major providers have guardrails aimed at dangerous biological requests, though jailbreaks remain possible; and the hardest steps are not informational. Acquiring, synthesizing, culturing, stabilizing, and delivering a pathogen remain physical problems no amount of text generation solves. The concern is directional rather than immediate: as systems become more capable and more widely deployed in laboratories, capability plus accessibility creates risks existing frameworks were not built to manage.

Restricting access is impractical: the models are often open-source, and restricting them would also restrict beneficial work in medicine and materials science. More promising approaches screen at the synthesis level (flagging dangerous molecules before they are physically made), impose access controls on specialized biological AI tools, and monitor AI-assisted research where dangerous agents are handled. None is complete.

Part V: Longtermism and the Precautionary Principle

The existential-risk debate is also a question about how to reason under deep uncertainty about the long-run future. Two frameworks dominate: longtermism, which gives the welfare of future generations substantial weight, and the precautionary principle, which puts the burden of proof on those who wish to proceed when an action risks severe or irreversible harm. Both are invoked by AI risk researchers; both have been seriously criticized.

Longtermism

Longtermism, developed most prominently by William MacAskill in What We Owe the Future (2022), holds that future people matter as much as present people, and that because there could be vastly more of them, our obligations to them could outweigh our obligations to the present. Preventing civilizational catastrophe is then among the highest priorities possible, since it is the only way to preserve the possibility of those lives.

In its favor: longtermism gives a principled basis for taking extinction risks seriously and for investing in areas — AI safety, pandemic preparedness, asteroid defense — neglected precisely because their benefits accrue to people who cannot vote, lobby, or exert market pressure. The intuition that future generations matter is widely shared; longtermism takes it seriously in its full quantitative implications.

Against: the argument depends on hypothetical future people whose existence is deeply uncertain, and expected-value calculations over trillions of lives are sensitive to small changes in already-uncertain probability estimates. The sharpest version of this worry is Pascal's mugging: multiply a tiny probability by a sufficiently astronomical payoff and almost any demand looks rational, impossible to argue with because nothing present can outweigh it. Critics including Émile Torres argue that such reasoning can justify almost any present sacrifice, and in practice has been used to discount urgent harms to real people.

The table simplifies — there are longtermists who take present harms seriously, and present-focused critics who grant that long-run risks deserve attention. The sharpest disagreement is not about values but epistemology: can we predict the long-run future reliably enough to justify weighting it heavily today?

The Precautionary Principle

The precautionary principle — roughly, "when in doubt, don't" — is often invoked in debates about AI risk. If we are genuinely uncertain whether powerful AI could cause catastrophic and irreversible harm, shouldn't we err on the side of caution? Cass Sunstein, who has studied the principle extensively, argues that it is both more appealing and more problematic than it first appears.

The appeal is that reversible caution seems wiser than irreversible action when the stakes are high. The problem, Sunstein argues, is that the principle is self-defeating when applied consistently, because almost any action risks serious harm to someone. Slowing AI carries risks too: diseases it could have helped cure, energy transitions it could have accelerated, economic and educational benefits delayed. Apply the principle to developing AI and you must also apply it to slowing AI — and the framework offers no way to choose.

Neither framework gives a clean answer to how much caution is appropriate. Used together they can structure the debate: identifying the considerations on each side, exposing the uncertainties that must be reduced, and guarding against both complacency and the paralysis that uses risk as a reason to avoid all difficult decisions.

The Other Tail: What If It Goes Right?

Almost everything in this chapter has concerned the bad tail. That is a strange way to argue about caution, because caution is always caution for the sake of something. Bostrom's Deep Utopia (2024) is the mirror image of Superintelligence, and worth reading precisely because the same author wrote both. Suppose alignment is solved, no war follows, and machines do everything humans currently do for a living, better. Bostrom calls this a "post-instrumental" condition, in which human effort is no longer necessary for anything.

This is not obviously a happy ending, and we have thought about it far less carefully than about extinction. If work is optional, what is achievement? If a machine could have written your essay or comforted your friend better than you did, what did you contribute? Bostrom is not predicting misery; he is pointing out that the sources of meaning most cultures rely on — necessity, scarcity, difficulty, being needed — would have to be replaced by something, and nobody has a good account of what. It is Chapter 8's question about meaningful work at civilizational scale, without the option of finding a different job.

This is not a change of subject. Both the precautionary principle and the expected-value argument assume we know what we are protecting. If the best plausible outcome is one we cannot evaluate, some of the confidence on both sides is misplaced — and asking what a good future with AI consists of becomes a precondition for the risk debate, not a postscript.

Synthesis and Conclusion

It would be convenient if one side of this debate were clearly right and the other clearly wrong. Intellectual honesty requires sitting with the tensions.

What the Existential Risk Camp Gets Right

Systems more capable than humans in all cognitive domains would be a world-historic event with consequences hard to predict or control, and the alignment problem is real: no proven method exists for ensuring a highly capable system pursues beneficial goals rather than ones that merely look beneficial under training conditions. Dismissing this because it is uncomfortable, or because it is associated with wealthy Silicon Valley institutions, is not a serious response. Nor do we need certainty of extinction to justify alignment research and international governance: a 1-in-100 chance of civilizational catastrophe is by any accounting urgent.

What the Critics Get Right

The existential-risk framing, as institutionalized, has demonstrably shifted attention and resources away from present harms. Systems deciding who gets a loan, whose benefits are cut, who is flagged by predictive policing, and whose job is automated do real, measurable, unequally distributed harm today, and that is not less urgent because a speculative future harm would be larger. The scale argument is only as strong as the probability estimates underlying it. And a discourse in which the primary risk is defined by wealthy researchers at elite institutions, concerns hypothetical systems, and calls for technical solutions from those same researchers deserves scrutiny even when its conclusions turn out right. Epistemic humility cuts both ways.

A Virtue-Ethics Synthesis

Vallor's virtue ethics offers a way through. Instead of asking "which side is right?" — a question that assumes a possibly false binary — ask what habits of character should guide those building, deploying, and governing AI. Epistemic humility means treating near-term and long-term risks as both real and uncertain, rather than prioritizing whatever is professionally interesting. Moral attention means keeping actual people central rather than abstracting them into expected-value calculations. Accountability means regulatory bodies, auditing standards, and professional norms that hold developers responsible for present harms and future risks alike.

The resulting "both/and" is not fence-sitting. The dichotomy between existential risk and present harm is partly a product of institutional competition for resources, and an ethical approach needs the flexibility to treat alignment research and algorithmic accountability, weapons bans and bias auditing, biosecurity and labor protections as complementary rather than competing. Allocating resources across them is hard — but no discourse that treats one as obviously dominant will do it well.

Mirrors do not merely reveal things as they are: mirrors also magnify, occlude, and distort what is captured in their frame.

Vallor is writing about AI systems, but the sentence fits the debate about them. The existential-risk frame magnifies irreversibility and occludes distribution: good at noticing that some mistakes cannot be undone, bad at noticing who is already paying. The present-harms frame does the reverse. Neither is neutral, and neither is dispensable. The question to carry out of this chapter is not "which side is right?" but "what is this framing helping me see, and what is it keeping out of the picture?"

Summary

This chapter examined AI risk from several angles: the case for treating superintelligent AI as an existential threat, the critics who say that framing distracts from present harms, autonomous weapons and biological risk, longtermism and the precautionary principle, and what a good outcome would even look like.

References