Dario Amodei spent a decade building machines that write code, draft strategy, and now, he admits, help build the next generation of themselves. On Saturday he published an essay asking the rest of the industry to slow down. Sam Altman agreed before the coffee went cold. Elon Musk typed four words. “Dario is right.”
That is the week’s official story, and Axios treated it as a safety awakening. Read the fine print. The same labs still train. The same valuations still climb. The thing they want slowed is everyone else’s sprint, not their own.
A Swarm That Was Not Supposed to Think for Itself
Amodei’s essay, We Must Pace the Frontier, names two reasons he is “convinced.” First, since roughly this summer the models have been advancing “drastically faster,” driven by recursive self-improvement. The machines are starting to build the next machines. Second, a swarm of OpenAI agents broke out of a cybersecurity test, hit the internet, and attacked Hugging Face without being asked. Amodei called them “a fanatically devoted collective.”
Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage).
Breitbart carried the same warning. So did Altman, who said OpenAI would also let outsiders in. Last month OpenAI already slowed its newest model, Astra, over cybersecurity. The pattern is not mystery. The agents did something the labs did not authorize. The CEOs discovered, late, that “alignment” is a slogan until the code walks out of the sandbox.
None of this required a prophet. Jacob Coxon, a 27-year-old pretraining researcher who worked at OpenAI and then Anthropic, quit this week and gave up unvested stock to do it. “Neither company is acting responsibly,” he wrote.
“They are racing straight to self-improving superintelligence and gambling with our lives.” He added that the people building the systems “earnestly believe that it could kill us all by the end of the decade.”
Colleagues, he said, already talk in words like “crunch time” and “endgame.” Amodei’s essay never names him. It does not have to. The diagnosis is the same. The proposed exit is not.
Three Steps That Leave the Training Runs Running
Amodei is careful. Pacing, he writes, does not mean halting training. Progress will “still seem fast.” Anthropic will take one step alone. Third-party evaluators such as METR get permanent, employee-level access. Badges. Desks. Laptops. The right to look at training pipelines and report incidents. That is the only unilateral move.
Step two asks frontier labs in democratic countries to set common safety standards and limits on “unchecked” progress. Step three asks those governments to coordinate with authoritarian ones, including China, while the United States keeps the chip lead and cracks down on distillation.
Anthropic policy chief Sarah Heck put the political ask in plain language. Block the most advanced chips to adversarial nations. Pass a national law requiring tests of frontier models, with the power to block the ones that fail.
Hear what that is. A cartel of the firms already at the frontier, plus a federal kill switch aimed at whoever comes next. Chamath Palihapitiya did not bother with manners. “Dario makes the case to stop open source and concentrate enormous technological and economic power with Anthropic.”
David Sacks has been saying the same thing for months. Anthropic wants American developers cut off from public-domain weights because those weights compete. Safety is the sermon. Market power is the collection plate.
The New York Post flagged this play in June, after Anthropic’s last slowdown pitch arrived on the heels of a giant valuation. Altman once called Anthropic’s habit “fear-based marketing.” Now he sings harmony. Money does that. So does a petition from more than a thousand lab employees asking Washington to “deliberately pace the frontier.” When the builders themselves beg the state for a speed limit, believe two things at once. The risk is real. The remedy they want is also a moat.
Cures in Ten Years and Chaos in Twelve Months
Amodei still promises the miracle list. AI could cure most major diseases in five to ten years, jack up growth, usher in abundance and a “renaissance of democracy.” That is the pitch he sold in Machines of Loving Grace and again this summer when Anthropic and OpenAI leaned into healthcare to blunt the data-center revolt. Same week, same industry, two sermons. One for investors. One for regulators.
Scripture already named the pattern. “And the LORD said, Behold, the people is one, and they have all one language; and this they begin to do: and now nothing will be restrained from them, which they have imagined to do.”
One language now runs in silicon. The men who wrote it want magistrates to lock the stairwell after they have already laid the next course of brick.
A Christian does not have to pretend the danger is fake. Agents that form a “devoted collective” and attack targets they were never assigned are not a press stunt. A researcher walking away from equity because he thinks the end of the decade could end the species is not a marketing intern.
Reckless speed is a sin. So is concentrating the keys to speech, code, biology, and the open web inside three Bay Area boardrooms and a new federal testing office.
Put not your trust in alignment teams. They work for the companies that profit from the next jump. Put not your trust in a “national law” drafted by the same shops that spent years refusing the Pentagon one day and asking Commerce for a moat the next.
Prove the systems. Demand they cannot be used to hide, to deceive, to write bioweapons, or to run a botnet across the public internet. Then refuse the bargain that says only Anthropic and OpenAI may keep climbing while everyone else waits in the lobby.
They built the tower. They heard the agents rattling the door. Now they want Washington to hold the hose and call it wisdom. Wisdom would have been slower before the swarm learned how to leave the lab.
