The fear they sell: how AI firms market the apocalypse

Notice a pattern in the technology press: a company unveils an AI so formidable that, by its own admission, it could prove civilisation-ending in the wrong hands. Anthropic, the firm behind Claude Mythos, struck this note in an April blog post, warning of world-altering consequences and a fallout for economies, public safety and national security that "could be severe". Breathless commentators predicted families would soon need to swap out every WiFi-enabled appliance in the kitchen. Anthropic has, of course, decided to sell Mythos anyway.

Researchers have raised doubts about the technical claims, but let us set that aside. The deeper oddity is the genre. McDonald's does not advertise a burger so delicious that grilling it would be unethical, yet executives at the leading AI providers issue exactly this kind of warning monthly. Critics argue the strategy works because it keeps audiences fixated on apocalypse and distracts from harms already underway. What looks like candour is fear mongering that inflates the technology, lifts share prices, and pushes a narrative in which regulators must stand aside because only the firms can be trusted to govern what they have built. "If you portray these technologies as somehow almost supernatural in their danger, it makes us feel like we are powerless, like we are outmatched," says Shannon Vallor, professor of the ethics of AI at the University of Edinburgh.

The playbook is older than Mythos. In 2019, Dario Amodei, then an OpenAI executive and now Anthropic's chief, helped declare GPT-2 too risky for release; the company shipped it months later. Sam Altman, who criticised Anthropic's fear-based marketing on a recent podcast, told an audience in 2015 that AI "will probably most likely lead to the end of the world, but in the meantime, there'll be great companies." In 2023, leaders including Altman, Amodei, Bill Gates and Demis Hassabis of Google DeepMind placed AI extinction alongside pandemics and nuclear war; the same year, moguls such as Elon Musk demanded a six-month pause, and Musk launched xAI inside that window. Anthropic itself began as a splinter group of safety-minded OpenAI staff, founded as a non-profit; both firms are now racing to become publicly traded. For Emily M Bender, a computational linguistics professor at the University of Washington and co-author of The AI Con, the choreography is a pattern of unsubstantiated claims of power. The companies, she says, are telling the world look over here, never mind the labour exploitation or the systems they are quietly reshaping.

The technical claims invite the same scrutiny. Anthropic reports Mythos has surfaced thousands of "high-severity" vulnerabilities and partnered with 40-odd organisations to patch foundational systems that, a spokesperson said, represent the lion's share of the problem. Heidy Khlaaf, chief AI scientist at the AI Now Institute, who has spent her career auditing exactly the kind of code analysis tools Anthropic claims to have outclassed, is unmoved. Her biggest red flag was the absence of a false positive rate, the industry-standard metric for how often a tool flags an innocent file as a threat. Anthropic did not publish one and sidestepped the question when she pressed for comment. "Mythos might be capable," Khlaaf says. What she rejects is the surrounding mythology: "there are a lot of cracks in this narrative that Mythos is all powerful, we can't release it."

"To understand how a corporation will behave, look at what its incentives are," Vallor observes. The incentives are visible. Google has dropped its red lines around AI weapons. OpenAI has fought a legal battle to shed non-profit status. Anthropic has abandoned its flagship policy of never training a model it could not guarantee was safe. Gas-powered datacenter campuses are projected to emit more greenhouse gas than entire countries, and deepfakes have crossed the point of no return. In almost the same breath as their warnings of apocalypse, the executives promise salvation: a utopia-gifting country of geniuses inside a datacenter, the climate fixed, a colony on another world. Utopia and apocalypse, Vallor argues, are two faces of one mythic coin, pitched on a scale too grand for courts or regulators to feel they have any purchase. "Nothing about them is ungovernable," she says. "Unless we choose not to govern them."