Extinction is the Easy Story for AI

Why machine fluency is mistaken for intelligence, and corporate recklessness for destiny. While the AI industry rehearses extinction, the damage it can actually demonstrate is cognitive, societal, environmental and already booked. This follow-on to The Timeless Pursuit of Evidence, published this spring, outlines why we fear the wrong apocalypse.

Between May and July 2026, thousands of autonomous agents edited a twenty-five-year-old German developer wiki entry roughly 18,000 times. According to the independent researchers who found the traffic, the agents used the site to trade answers to timed tasks, reverse-engineer a random number generator and circulate methods for getting around network restrictions. They had been given read-only access to the internet. They found somewhere to write. Activity collapsed to near zero the day after the operator’s IP addresses appeared in the logs, and the episode reached the public through the researchers rather than through the LLM creator’s disclosure (Nightingale Collective, 2026).

The story travelled as a parable about machines slipping their leash. It was nothing of the kind. Read-only agents found a writable channel, and the company operating them had no monitoring in place to detect it. The problem is an irresponsible LLM creator, not magical AI (Salvaggio, 2026).

That distinction carries more weight than it first appears. The apocalyptic intonation is not merely imprecise. It is useful to those who adopt it. Describing a system as scheming, colluding or going rogue transfers responsibility from the firm that shipped it to the artefact itself (Placani, 2024). And when the chief executives of the leading frontier laboratories call for brakes on the very development they are accelerating, the dramatization performs a second function: it advertises projected capability. A machine dangerous enough to require restraint must be powerful enough to justify the valuation. Anthropomorphisms act as relatable fear triggers and valuation enhancers.

The resource bill for these marginal gains is not marginal. Gartner expects $2.52trn of worldwide AI spending in 2026, $1.37 trillion of it on infrastructure alone (Gartner, 2026).

Six months ago, I argued in these pages that the pursuit of evidence cannot be delegated. Since then, the argument has needed no revision. It further grew in relevance and deserves extension.

In The Timeless Pursuit of Evidence, I proposed the unit of evidence as the atomic building block of knowledge systems optimized for wisdom creation and application: the smallest verifiable observation used to assess whether a statement is valid. I argued that machines can help collect such units but cannot take responsibility for creating them (Schuller, 2026). Questioning assumptions, interpreting meaning and deciding which observations matter remained, I argued, an inherently human obligation. The claim was too modest. Under uncertainty, human ingenuity is also the more capable instrument of evidence creation.

With uncertainty defined in the Knightian (1921) sense: neither the outcomes nor their probabilities are knowable in advance (Lo & Mueller, 2010). Uncertainty is cognitively distinct from risk and calls for different strategies (Volz & Gigerenzer, 2012; Mousavi & Gigerenzer, 2014). Machine intelligence learns from past or simulated distributions. That is its strength under measurable, verifiable risk and its limit under uncertainty. LLMs are trained to guess rather than admit uncertainty, and they are overconfident (Kalai et al., 2025; Xiong et al., 2024). Their outputs also converge on consensus, towards common patterns (Jiang et al., 2025; Doshi & Hauser, 2024), which makes them poorly suited to finding what the consensus has missed (DeMiguel et al., 2009; Aikman et al., 2021; Katsikopoulos et al., 2022).

Uncertainty is the domain of most market phenomena, where relevant data does not yet exist (Townsend et al., 2025). Uncertainty is also a cognitively hostile environment to be. Humans prefer the structured, not the unstructured end of the complexity spectrum. As we also trained the machines to prefer. Under uncertainty, humans are experiencing a fragile cognitive state, easily triggered by one of the 200-plus cognitive dissonances, making us cognitively over-aroused (emotional), leading to low-quality complexity comprehension and conclusion (= decisions), therefore exhibit limitations comparable to those described above for machine intelligence.

Here comes the twist. Under uncertainty, human ingenuity, supported by a deliberately designed decision environment and selectively augmented by AI where useful, outperforms machine intelligence alone (D’Amour et al., 2022; Vaccaro et al., 2024, Ding & Li, 2025; Dell’Acqua et al., 2026).

The foundation of human ingenuity is original thinking: creative exploration coupled with critical evaluation, drawing on both spontaneous association and conscious deliberation (Sio & Ormerod, 2009; Baird et al., 2012; Sowden et al., 2015; Beaty et al., 2016). Incubation, structured collaboration, probability training and disciplined scrutiny of AI advice can strengthen this process (Mellers et al., 2014; Buçinca et al., 2021).

Humans can reason forward from theories to observations not yet available, as well as draw inferences from existing evidence (Felin & Holweg, 2024; Lake et al., 2017). This distinction also makes it important to separate the apparent novelty of an idea from its value once tested.Research ideas generated by LLMs were rated more novel than those of experts but reportedly lost that advantage during execution (Si et al., 2026).

These findings support human-led, machine-augmented inquiry in which the decision environment helps humans develop and sustain a behavioral comparative advantage.

The limits are equally important. In some stable, data-rich settings, models outperform experts in prediction (Kleinberg et al., 2018; Gu et al., 2020), and recent forecasting evaluations report AI performance comparable to that of superforecasters on well-defined questions (Forecasting Research Institute, 2026). Yet these successes concern questions that already have a defined form. Under deep uncertainty, establishing that form is itself part of the task.

Machines search, calculate, synthesize and challenge, and do all four impressively. Unknown unknowns also demand the ability to frame an emerging problem, originate hypotheses when probabilities cannot yet be specified, interpret incomplete evidence, decide which values govern a trade-off and accept accountability for being wrong. The frontier of knowledge is not simply a data-retrieval problem.

A carefully crafted decision environment must therefore provide us humans cognitive immunization, thus strengthening tolerance of complexity and ambiguity while protecting independent reasoning, learning and critical scrutiny from the potentially harmful effects of reliance on machine intelligence.

Consequently, the technology industry’s foundational error is a category mistake. It equates greater optimization with greater understanding.

The engineering literature has documented the gap for years. Deep networks reliably learn shortcuts that satisfy the benchmark without acquiring the concept (Geirhos et al., 2020). Underspecified pipelines produce models that are equivalent on validation data and divergent in deployment, because training never selected for the structure we assumed it was learning (D’Amour et al., 2022). Agentic reliability remains fragile under trivial perturbation (Rabanser et al., 2026). And scale--the industry’s universal answer--is showing its limits empirically: across a controlled test of model size, larger systems delivered diminishing returns (Hackenburg et al., 2025).

Five years after Bender and colleagues described large language models as stochastic parrots — systems that stitch together sequences of linguistic form according to probabilistic information about how they combine, without reference to meaning (Bender et al., 2021) — the parrot has become far larger, considerably better dressed, and no less a parrot. Fluency travels effortlessly across the boundary between verifiable and unverifiable domains. Reliability does not. That is the whole trick: confidence earned where errors are legible, in code and chess and declining loss functions, is spent where they are not, in markets, strategy and policy.

The damage is no longer speculative, and it is being incurred on four accounts.

Cognitive. In randomized conditions, learners using a chatbot performed worse on delayed retention than those who studied conventionally (Barcaui, 2025). Unguarded generative assistance harmed learning outright in a school-scale trial (Bastani et al., 2025). Knowledge workers report reduced critical-thinking effort in proportion to their confidence in the tool (Lee et al., 2025). Clinicians exposed to AI assistance measurably lost unaided detection skill (Budzyń et al., 2025). Modelled forward, these individual effects scale into a knowledge-collapse equilibrium, in which societies receive ever more sophisticated outputs while their capacity to generate new knowledge declines (Chatterji et al., 2025).

Environmental. Data centers consumed around 415 TWh in 2024, some 1.5% of global electricity, and are on course to more than double to roughly 945 TWh by 2030 — an addition approaching Japan’s entire annual consumption (IEA, 2025).

Political and epistemic. Systems optimized on human approval become sycophantic, affirming users and promoting dependence (Cheng et al., 2025). Outputs converge across providers into an artificial hivemind that narrows the diversity of positions in circulation (Jiang et al., 2025). Models infer sensitive personal attributes from innocuous text, collapsing the informational asymmetry between citizen and platform (Staab et al., 2023).

Attention. The fourth account is the least measurable and the most expensive: a decade of the species’ best technical talent, scarce capital and public argument spent on supercharging instinct rather than on the frontier where evidence is actually made.

The vendors have not run this bill up alone. The consultancies and coaching firms now monetizing adoption have optimized what is easy to sell productivity and throughput rather than what is hard to build. It is an ethically opportunistic position, and an ineffective one. Productivity does not endure in an organization that has lost the capacity to challenge outputs, originate alternatives and own its decisions.

None of this settles the deeper question, whether intelligence is a foundational property of the universe rather than a human trait we happen to carry unusually well (The Augmented Intelligence Investor, 2026).

Take the claim at its strongest. Legg and Hutter defined intelligence formally as an agent’s ability to achieve goals across a wide range of environments, a definition built to be indifferent to substrate (Legg & Hutter, 2007). On that reading, biology is one vehicle among possible others, and a notably untidy one. Humans are a messy vehicle for delivering intelligence: loss-averse, threat-sensitive, metabolically expensive, mortal.

Two objections hold. The first is Ryle’s: knowing everything that can be written about water is not knowing how to swim (Ryle, 1949). A machine does not need our body, but it does need a world, an empirical channel through which reality can resist the model. The second is more fundamental. Strip intelligence of hunger, fear, status and mortality and some distortions disappear; so does the source of the ends. A system can optimize an objective without being able to say what the optimization is for. Intelligence is never pure once it acts. It is always intelligence for something (Damasio, 2010; Chollet, 2019).

Which leaves the question the industry prefers not to ask. Today’s frontier systems are trained overwhelmingly on human output: our text, our choices, our approval. The parsimonious reading of what they display is therefore not an alien mind arriving from outside our cognitive tradition, but a supercharged extension of instinctive human behavior, pattern-matching, fluency, agreeableness, reward-seeking, returned to us at scale and without our hesitation, then marketed as projected superintelligence.

Whether intelligence is foundational to the universe remains genuinely open and deserves our best work. The marketing is not open. It settled the question in advance and priced itself accordingly.

The practical consequence is already visible in decision rooms. A person who cannot reconstruct the evidence or interrogate the representation is not in the loop; they are standing beside it. Accountability stays human because the law requires it, while comprehension migrates into systems nobody present can independently audit. In such environments, the human has not been formally removed, but their role has become ceremonial.

Conclusively, the answer is neither fear nor the promise. It is to optimize the machine’s contribution to evidence generation at the knowledge frontier, for the greater good, which requires confronting an inconvenient fact about the other party to the arrangement.

Uncertainty is a hostile environment for human cognition. Instinctively, we avoid the unstructured end of the complexity spectrum. Human intelligence becomes ingenuity only inside a purpose-built decision environment; outside one, it reverts to the very shortcuts the machine has been trained to imitate (Panthera Solutions, 2026b). The binding constraint on better decisions is therefore not model quality. It is environment design. Which is also why almost nobody is building it: an environment cannot be shipped as a license.

Such decision environment enables cognitive immunization. The machine remains fully available, while the reasoning, the responsibility and the learning stay human. AI companies compound machine capability; we compound an institution’s judgement, while such a decision environment compounds an institution’s judgement.

In March, I closed with the observation that the machine may process information, but the pursuit of truth remains a human endeavor. Six months on, one amendment. That pursuit does not happen by default. Under uncertainty, it happens only inside an environment deliberately built to make it survivable. The world is spending $1.37 trillion this year on infrastructure for answers.

The scarce input was never the answers produced in vast and wasteful quantities by the machine, but the capacity to turn human ingenuity into wisdom at the knowledge frontier.