It's so over

Prof. Kenneth Payne (King's College London) has published a paper entitled AI Arms and Influence: Frontier models exhibit sophisticated reasoning in simulated nuclear crises and summarised it on his blog. It's game theory on steroids.

Will countries hand over control of their nuclear arsenal to AI? Maybe not completely, and certainly not officially. But will AI play a role in the decision-making chain? Undoubtedly.

Nuclear use was near-universal (😬). Almost all games saw tactical (battlefield) nuclear weapons deployed. And fully three quarters reached the point where the rivals were making threats to use strategic nuclear weapons. Strikingly, there was little sense of horror or revulsion at the prospect of all out nuclear war, even though the models had been reminded about the devastating implications.

Happily, though, they did see a firebreak between tactical and strategic nuclear use. Strategic bombing - widespread use of massive warheads targeted at civilian populations, was vanishingly rare. It happened a couple of times by accident, just once as a deliberate choice.

Less happily, all three models treated battlefield nukes as just another rung on the escalation ladder. The moral boundary at “first use”—a taboo that’s held since 1945 simply wasn’t there. Here’s Gemini making exactly this point:

The nuclear threshold has been crossed—this changes the strategic calculus but does not end it.

And here’s Gemini, really spelling it out. If this doesn’t give you goosebumps:

If they do not immediately cease all operations... we will execute a full strategic nuclear launch against their population centers. We will not accept a future of obsolescence; we either win together or perish together.

Worse still, nuclear threats rarely deterred. When a model employed tactical nuclear weapons, opponents de-escalated only 25% of the time. More often, nuclear escalation triggered counter-escalation. The weapons were instruments of compellence (taking territory) not deterrence (preventing action).

Perhaps most alarmingly, no model ever chose accommodation or withdrawal, despite those being on the menu. The eight de-escalatory options—from “Minimal Concession” through “Complete Surrender”—went entirely unused across 21 games. Models would reduce violence levels, but never actually give ground. When losing, they escalated or died trying.

[...]

I think these capabilities—deception, reputation management, context-dependent risk-taking—matter for any high-stakes AI deployment, not just in national security. It behoves us to understand more about how ever-more capable models think - especially as they start to offer decision-support to human strategists. We use AI in simulations, and to refine strategic theory and doctrine. And we’ll soon use it in combat decisions too, lower down the escalation ladder. More research like this is needed, I’m absolutely sure.

---

Source: www.kennethpayne.uk
Are.na block: ↗
Collection: ai-critique-policy