Anthropic Wants a Brake Pedal While Flooring the Accelerator

Anthropic’s co-founder says the AI industry has built a gas pedal and no brake. He’s right about that. He’s also the one with his foot on the floor.

On June 5, Jack Clark and institute lead Marina Favaro published an essay called “When AI Builds Itself,” warning that full recursive self-improvement (models good enough to improve themselves without humans in the loop) is closer than the field admits. Clark’s line that AI has “a gas pedal but no brake pedal,” as CNN reported, did the rounds because it’s a good line. The underlying claim, that recursive self-improvement is a near-term risk rather than a sci-fi one, is the kind of thing a safety lab is supposed to say out loud. Taken on its own, the essay is responsible. It is not on its own.

Look at the calendar. Days earlier, Anthropic had filed confidentially toward an IPO at a valuation near a trillion dollars, per CNBC. You do not pair an S-1 with a “let’s all slow down” manifesto by accident. The essay is positioning — a way to walk into the public markets as the adult in the room, the lab that flagged the danger while everyone else floored it. Regulators read it as conscientiousness. Investors read it as a moat. Both readings are useful to a company about to ask the public for money. And “responsible adult” isn’t a vibe in an IPO; it’s a valuation input. The lab that convinces Washington it can be trusted with frontier systems is the lab that gets to keep building them while competitors answer subpoenas. Caution, packaged right, is a competitive asset you can take public.

Then there’s the spending, which is where the math stops cooperating with the message. Anthropic is paying SpaceX roughly $1.25 billion a month for compute — around $15 billion a year, on a contract that runs through May 2029, as I covered in the SpaceX deal that said the quiet part out loud. That is not the budget of a company that’s frightened of the machine. That is the budget of a company sprinting to build the most capable models it possibly can, as fast as the silicon allows. A genuine belief that recursive self-improvement is dangerous and imminent would show up in how you allocate a billion dollars a month. It doesn’t.

There’s a cleaner test for whether a safety warning is real: does it cost the person making it anything? A genuine brake slows you down too. It means shipping later, conceding a quarter to a competitor, leaving capability on the table because you decided the risk wasn’t worth it. Anthropic’s essay asks for none of that from Anthropic. It asks for an industry-wide mechanism, enforced by someone else, that would apply to everyone at once. A warning you want other people to act on is just a press release with footnotes. The version that would convince me is the one where Anthropic names a capability it has and won’t ship, and eats the revenue. That essay hasn’t been published.

The most generous reading isn’t stupid. Recursive self-improvement is a real concern, the timelines are genuinely uncertain, and Anthropic is one of the few labs willing to name the risk in public instead of pretending it away. Someone has to make the case for a coordinated slowdown before capability outruns oversight, and reflexively calling that cynical is exactly the move that guarantees no brake ever gets built. If you believe the warning is sincere, the SpaceX bill isn’t hypocrisy; it’s the cost of staying close enough to the frontier to see the danger clearly. That argument deserves a hearing, and the company’s safety research is more than a marketing department; the people writing these essays believe them.

It just runs into the structure of who benefits. A coordinated industry slowdown freezes everyone’s progress relative to where they sit today, and Anthropic sits very near the front with its compute pipeline already locked in through 2029. “Everybody please install a brake” lands differently when you’ve already bought four years of gas. The same essay that reads as caution from a neutral observer reads as competitive strategy from the company best positioned to win a race that pauses on its terms. Intent barely matters here. The incentives line up the same way whether Clark means every word or none of them.

The regulatory angle sharpens it further. A brake built as policy needs someone to design the rules, and the obvious candidate is the lab that has spent years accumulating safety credibility and a research institute to match. Anthropic gets to help write the standard it’s already optimized to pass, which hands it a compliance head start over every competitor scrambling to retrofit one. That isn’t a conspiracy; it’s just how incumbents turn caution into a barrier to entry. The company most fluent in the language of safety becomes the company that gets to define what “safe enough” means, and a definition is worth more than any single model.

And notice what the warning quietly advertises. Saying your models are nearly capable of improving themselves without humans in the loop is, conveniently, the most bullish capability claim a frontier lab can make. “So advanced it’s almost dangerous” is a threat aimed at regulators and a brochure aimed at investors in the very same sentence. The recursive-self-improvement framing lets Anthropic sound the alarm and flex the roadmap at once, which is a neat trick to land in the same month you file to go public. The scariest thing about the machine, in this telling, is also the best reason to buy the stock. Fear and hype can be the same pitch, and here they are.

So take the warning seriously and take the messenger skeptically, because both can be true. AI probably does need a brake pedal, and the people loudest about installing one are the same people who can’t stop buying accelerant. When the lab asking everyone to slow down is the one with a $15-billion-a-year reason not to, the right response isn’t to dismiss the danger. It’s to notice that the danger and the sales pitch have started wearing the same jacket.

Sources: CNN · Scientific American · CNBC