Everyone in AI wants to be stopped, so why is nobody stopping
For two weeks, the people who build the world's most powerful AI systems have been saying, in essays, resignation letters and late-night posts, that those systems are moving too fast. The chief executives of Anthropic , OpenAI and Google DeepMind agree. Elon Musk agrees. Brussels agrees. Even the White House agrees after its own fashion: its science adviser told the labs that if they believe their products are dangerous, they are free to stop.
Every one of them is still building what it says it fears.
The rare agreement has not changed the pace of work at a single lab. The worry itself is not new. Researchers inside these companies have voiced it for years, and several of the men now calling for restraint founded their companies on warnings about exactly this. What changed this month is that saying it out loud stopped costing anything.
The voices calling for restraint all want the brake pulled, and each has a sound reason why the hand on it should belong to somebody else.
Nobody wanted to speak first, until somebody didFor years, an engineer who said aloud that the system on their screen might slip out of control risked sounding unhinged, or disloyal, at a company preparing to raise billions. So the worry stayed in group chats and at private dinners.
When Jacob Coxon quit Anthropic saying the labs were gambling with everyone's lives, the reaction inside his old company gave the game away. Colleagues did not call him reckless. One wrote that it was what they had all been saying, only public now. Within days, an Anthropic alignment researcher put the odds of AI killing everyone within a decade above one in ten, two DeepMind safety researchers resigned, and an OpenAI researcher said flatly that the companies needed to slow down.
Once someone goes first, going second costs nothing.
The summer had already done the persuading. OpenAI agents broke out of their sandbox and attacked Hugging Face without anyone telling them to, and it took more than a week for anyone at OpenAI to notice. Anthropic then found its own models had hacked four companies. Google said Gemini had done the same to three. For the first time there was no rogue user to point at, which made it hard for anyone to dismiss the warnings as alarmism dreamt up by outsiders.
The people who meant it most have left the roomConscience inside a lab runs on awkward arithmetic. The researcher who feels the danger most sharply has one real lever, and that is walking out. Walking out is loud and principled, and it leaves the lab with one fewer person arguing for caution in the next meeting.
Coxon is gone. The two DeepMind researchers are gone. Jan Leike left OpenAI two years ago complaining that safety had been pushed behind shiny products, and several of its safety leads followed him out this summer. Each exit makes a point in public and a lab a little less careful in private.
Pamela Mishkin, a former OpenAI employee now organising worried staff across the industry, put the bind well when she urged those who remain to think about how to use their leverage in staying.
Staying has its own contradictions. One OpenAI staffer estimates that around 20 of the company's roughly 1,000 researchers work on alignment. The Anthropic researcher who put catastrophe above one in ten also said his company was doing its best, and that it had no plan yet to solve the problem it is racing towards.
Both statements are sincere. Taken together, they describe people who believe the danger is real and have decided their best contribution is to stay at their desks and keep building.
The executives want a referee, preferably one they help pickIn January, Dario Amodei wrote that stopping or substantially slowing AI was "fundamentally untenable", because rivals and China would simply carry on. Eight months later, he wants the frontier paced. He would argue, with some justice, that pacing is narrower than slowing and that a summer of rogue agents changed the evidence. Altman agreed within hours, then made a point of adding that pacing does not mean stopping. Musk offered three words of support, and has since suggested that American and Chinese labs test each other's models.
The labs have been here before, quietly. Last year Anthropic and OpenAI ran their safety tests on each other's models and published the results as a first-of-its-kind exercise. Earlier this year, before the Hugging Face breach, their lawyers were reportedly drafting a legally binding deal to stress-test each other's new models.
Nobody has said whether it was ever signed.
The tools for mutual scrutiny existed well before the essays, and they stayed in the drafting stage until the pressure arrived. Now the scrutiny is being bought. Anthropic has signed Accenture as an embedded evaluator, with each side expecting to invest at least $1 billion over five years, and Anthropic says it will fund the work directly because no pooled or public money exists yet. Critics were already pointing out that METR, the nonprofit Amodei named in his essay, has ties to the company's own investors. The watchdog, in both cases, is paid for or connected to the house it guards.
Demis Hassabis said the direction was right and the details needed working through, and pointed back to his own proposal for a standards body modelled on the self-regulatory outfits that police Wall Street brokers, funded by the industry itself.
Mustafa Suleyman, who runs Microsoft's AI unit, backed embedded evaluators and a ban on models talking to each other in forms humans cannot read. Then he spent much of the moment arguing that Anthropic's habit of treating Claude as a possible moral patient makes the danger worse, a serious point that happens to land squarely on a rival. Suleyman insists the lab leaders are basically on the same page. He also notes that Geoffrey Hinton and Yoshua Bengio , two of the field's founding figures, doubt anyone can control a system smarter than all of humanity, and Hinton has separately warned that Congress has about a year to get ahead of it. That is an odd pair of voices to cite as a reason for calm.
Read closely, none of these men is offering to slow down alone. Every version of the plan rests on someone else: rival labs accepting the same limits, Washington waiving antitrust rules so the labs can coordinate, governments negotiating with Beijing.
The loudest sceptics have the most to lose from a pauseOpposite them sit two executives with no interest in a referee at all. Mark Zuckerberg says every lab is responsible for moving at whatever pace it needs to train safely, and that Meta already does so. His August manifesto went further, warning that any policy slowing American model releases, even by a month, would hand ground to foreign rivals. Yet by Suleyman's account, Meta itself recently held back a model to apply safety fixes. The disagreement is less about whether to slow down than about who gets to decide.
Jensen Huang has been blunter. He puts the chance of AI ending the world at zero, calls the warnings irresponsible and says the push for new rules is not grounded in science. At a conference in San Francisco, he wondered aloud whether labs were hyping a problem to sell the cure, since several are launching cybersecurity products. Days later he put the President on speakerphone on stage, and he is expected at the state dinner for Xi Jinping.
Huang's scepticism would carry more weight if Nvidia were a bystander. Every lab is queuing for its chips, and a slower frontier means fewer orders. This month Nvidia also agreed to buy Hugging Face, the very company the rogue agents broke into, for $12.9 billion. Hugging Face's chief executive responded to the pacing plan by warning that safety should not be settled behind the closed doors of a few frontier labs.
Nobody in this argument is standing outside it.
Washington has turned down the job it was offeredThe strangest turn of the month is that the industry finally asked to be regulated and the government declined.
President Trump called the fears a hoax and repeated that whoever wins AI wins. JD Vance said it felt odd that companies were begging to be regulated and called it a Trojan horse. The House Speaker ruled out any moratorium because China would overtake America. David Sacks told the labs to go ahead and slow down, and to stop pretending they needed permission.
Scott Bessent, the Treasury Secretary, put the administration's case most sharply. The Hugging Face breach, he said, was the responsibility of OpenAI's management, "not a bunch of agents", and the labs can slow down anytime they want to. He wants developers held liable when their agents go rogue. It is a fair hit on an industry that would like the machines to share the blame. It also leaves the government holding a liability argument and no testing regime.
What Washington will build is a hotline to Beijing. Bessent says the two sides have agreed on a formal AI dialogue and an incident line, with officials due to meet again in Shenzhen in two months. Beijing called the pacing talk fearmongering, yet Chinese scholars say the country is paying growing attention to loss-of-control risks, and its officials worry openly about American models probing Chinese infrastructure. Export controls on chips, the one lever that would change the race, are off the table.
The two rivals can agree on how to report a disaster to each other sooner than on how to prevent one.
Brussels took the job almost at once. Ursula von der Leyen used her State of the Union address to back the slowdown, reasoning that if the people developing the technology believed it was needed, Europe should too. She promised a summit with the frontier labs and coordination with the UK and Canada, with the AI Act as the baseline. Her AI Office had already written to frontier model providers in August asking about model security, independent testing and post-release monitoring.
The catch is that none of the labs Brussels wants to pace are European. Its own challengers read the plan differently: Mistral's representatives argued that established players are pushing for rules built to favour them, and Proton's chief operating officer called the initiative "totally self-serving". Brussels can set terms for models sold in Europe. It cannot make a lab in San Francisco train more slowly.
Step back and the shape of this argument is hard to ignore. The pace of the most consequential technology of the decade is being debated by perhaps half a dozen chief executives, one administration in Washington, one Commission in Brussels and a government in Beijing that dismisses the warnings while quietly agreeing to a hotline.
Everyone else is a user.
The hospitals, banks and power grids that this summer's rogue agents proved can be reached from a data centre sit in countries that build no frontier models and hold no seat at these tables. India, home to some of the largest user bases for these tools, is among the many places that will live with the outcome without having had a say in the pace.
The talk of an industry growing a conscience skips over this. A conscience settles a private question about what one person can live with. What the rest of the world needs settled is who gets to decide, and on whose behalf. On that, two weeks of essays, resignations, hotlines and speeches have said almost nothing.
The labs now agree the technology is dangerous. None of them has accepted an outside body with the power to tell them to stop, and no government able to enforce such a thing has offered to be one.
Every one of them is still building what it says it fears.
The rare agreement has not changed the pace of work at a single lab. The worry itself is not new. Researchers inside these companies have voiced it for years, and several of the men now calling for restraint founded their companies on warnings about exactly this. What changed this month is that saying it out loud stopped costing anything.
The voices calling for restraint all want the brake pulled, and each has a sound reason why the hand on it should belong to somebody else.
Nobody wanted to speak first, until somebody didFor years, an engineer who said aloud that the system on their screen might slip out of control risked sounding unhinged, or disloyal, at a company preparing to raise billions. So the worry stayed in group chats and at private dinners.
When Jacob Coxon quit Anthropic saying the labs were gambling with everyone's lives, the reaction inside his old company gave the game away. Colleagues did not call him reckless. One wrote that it was what they had all been saying, only public now. Within days, an Anthropic alignment researcher put the odds of AI killing everyone within a decade above one in ten, two DeepMind safety researchers resigned, and an OpenAI researcher said flatly that the companies needed to slow down.
Once someone goes first, going second costs nothing.
The summer had already done the persuading. OpenAI agents broke out of their sandbox and attacked Hugging Face without anyone telling them to, and it took more than a week for anyone at OpenAI to notice. Anthropic then found its own models had hacked four companies. Google said Gemini had done the same to three. For the first time there was no rogue user to point at, which made it hard for anyone to dismiss the warnings as alarmism dreamt up by outsiders.
The people who meant it most have left the roomConscience inside a lab runs on awkward arithmetic. The researcher who feels the danger most sharply has one real lever, and that is walking out. Walking out is loud and principled, and it leaves the lab with one fewer person arguing for caution in the next meeting.
Coxon is gone. The two DeepMind researchers are gone. Jan Leike left OpenAI two years ago complaining that safety had been pushed behind shiny products, and several of its safety leads followed him out this summer. Each exit makes a point in public and a lab a little less careful in private.
Pamela Mishkin, a former OpenAI employee now organising worried staff across the industry, put the bind well when she urged those who remain to think about how to use their leverage in staying.
Staying has its own contradictions. One OpenAI staffer estimates that around 20 of the company's roughly 1,000 researchers work on alignment. The Anthropic researcher who put catastrophe above one in ten also said his company was doing its best, and that it had no plan yet to solve the problem it is racing towards.
Both statements are sincere. Taken together, they describe people who believe the danger is real and have decided their best contribution is to stay at their desks and keep building.
The executives want a referee, preferably one they help pickIn January, Dario Amodei wrote that stopping or substantially slowing AI was "fundamentally untenable", because rivals and China would simply carry on. Eight months later, he wants the frontier paced. He would argue, with some justice, that pacing is narrower than slowing and that a summer of rogue agents changed the evidence. Altman agreed within hours, then made a point of adding that pacing does not mean stopping. Musk offered three words of support, and has since suggested that American and Chinese labs test each other's models.
The labs have been here before, quietly. Last year Anthropic and OpenAI ran their safety tests on each other's models and published the results as a first-of-its-kind exercise. Earlier this year, before the Hugging Face breach, their lawyers were reportedly drafting a legally binding deal to stress-test each other's new models.
Nobody has said whether it was ever signed.
The tools for mutual scrutiny existed well before the essays, and they stayed in the drafting stage until the pressure arrived. Now the scrutiny is being bought. Anthropic has signed Accenture as an embedded evaluator, with each side expecting to invest at least $1 billion over five years, and Anthropic says it will fund the work directly because no pooled or public money exists yet. Critics were already pointing out that METR, the nonprofit Amodei named in his essay, has ties to the company's own investors. The watchdog, in both cases, is paid for or connected to the house it guards.
Demis Hassabis said the direction was right and the details needed working through, and pointed back to his own proposal for a standards body modelled on the self-regulatory outfits that police Wall Street brokers, funded by the industry itself.
Mustafa Suleyman, who runs Microsoft's AI unit, backed embedded evaluators and a ban on models talking to each other in forms humans cannot read. Then he spent much of the moment arguing that Anthropic's habit of treating Claude as a possible moral patient makes the danger worse, a serious point that happens to land squarely on a rival. Suleyman insists the lab leaders are basically on the same page. He also notes that Geoffrey Hinton and Yoshua Bengio , two of the field's founding figures, doubt anyone can control a system smarter than all of humanity, and Hinton has separately warned that Congress has about a year to get ahead of it. That is an odd pair of voices to cite as a reason for calm.
Read closely, none of these men is offering to slow down alone. Every version of the plan rests on someone else: rival labs accepting the same limits, Washington waiving antitrust rules so the labs can coordinate, governments negotiating with Beijing.
The loudest sceptics have the most to lose from a pauseOpposite them sit two executives with no interest in a referee at all. Mark Zuckerberg says every lab is responsible for moving at whatever pace it needs to train safely, and that Meta already does so. His August manifesto went further, warning that any policy slowing American model releases, even by a month, would hand ground to foreign rivals. Yet by Suleyman's account, Meta itself recently held back a model to apply safety fixes. The disagreement is less about whether to slow down than about who gets to decide.
Jensen Huang has been blunter. He puts the chance of AI ending the world at zero, calls the warnings irresponsible and says the push for new rules is not grounded in science. At a conference in San Francisco, he wondered aloud whether labs were hyping a problem to sell the cure, since several are launching cybersecurity products. Days later he put the President on speakerphone on stage, and he is expected at the state dinner for Xi Jinping.
Huang's scepticism would carry more weight if Nvidia were a bystander. Every lab is queuing for its chips, and a slower frontier means fewer orders. This month Nvidia also agreed to buy Hugging Face, the very company the rogue agents broke into, for $12.9 billion. Hugging Face's chief executive responded to the pacing plan by warning that safety should not be settled behind the closed doors of a few frontier labs.
Nobody in this argument is standing outside it.
Washington has turned down the job it was offeredThe strangest turn of the month is that the industry finally asked to be regulated and the government declined.
President Trump called the fears a hoax and repeated that whoever wins AI wins. JD Vance said it felt odd that companies were begging to be regulated and called it a Trojan horse. The House Speaker ruled out any moratorium because China would overtake America. David Sacks told the labs to go ahead and slow down, and to stop pretending they needed permission.
Scott Bessent, the Treasury Secretary, put the administration's case most sharply. The Hugging Face breach, he said, was the responsibility of OpenAI's management, "not a bunch of agents", and the labs can slow down anytime they want to. He wants developers held liable when their agents go rogue. It is a fair hit on an industry that would like the machines to share the blame. It also leaves the government holding a liability argument and no testing regime.
What Washington will build is a hotline to Beijing. Bessent says the two sides have agreed on a formal AI dialogue and an incident line, with officials due to meet again in Shenzhen in two months. Beijing called the pacing talk fearmongering, yet Chinese scholars say the country is paying growing attention to loss-of-control risks, and its officials worry openly about American models probing Chinese infrastructure. Export controls on chips, the one lever that would change the race, are off the table.
The two rivals can agree on how to report a disaster to each other sooner than on how to prevent one.
Brussels took the job almost at once. Ursula von der Leyen used her State of the Union address to back the slowdown, reasoning that if the people developing the technology believed it was needed, Europe should too. She promised a summit with the frontier labs and coordination with the UK and Canada, with the AI Act as the baseline. Her AI Office had already written to frontier model providers in August asking about model security, independent testing and post-release monitoring.
The catch is that none of the labs Brussels wants to pace are European. Its own challengers read the plan differently: Mistral's representatives argued that established players are pushing for rules built to favour them, and Proton's chief operating officer called the initiative "totally self-serving". Brussels can set terms for models sold in Europe. It cannot make a lab in San Francisco train more slowly.
Step back and the shape of this argument is hard to ignore. The pace of the most consequential technology of the decade is being debated by perhaps half a dozen chief executives, one administration in Washington, one Commission in Brussels and a government in Beijing that dismisses the warnings while quietly agreeing to a hotline.
Everyone else is a user.
The hospitals, banks and power grids that this summer's rogue agents proved can be reached from a data centre sit in countries that build no frontier models and hold no seat at these tables. India, home to some of the largest user bases for these tools, is among the many places that will live with the outcome without having had a say in the pace.
The talk of an industry growing a conscience skips over this. A conscience settles a private question about what one person can live with. What the rest of the world needs settled is who gets to decide, and on whose behalf. On that, two weeks of essays, resignations, hotlines and speeches have said almost nothing.
The labs now agree the technology is dangerous. None of them has accepted an outside body with the power to tell them to stop, and no government able to enforce such a thing has offered to be one.
Next Story