AI Safety Crisis: Inside Anthropic's Competitive Pressure Problem

The resignation of a single 27-year-old researcher rarely moves markets or rewrites policy. But when Jacob Coxon walked away from Anthropic this week, he did something more unsettling: he confirmed, from the inside, what critics have long argued from without.

The central thesis here is not that AI development is dangerous — that much is contested terrain. It is that the institutional logic governing the two most prominent AI safety companies, OpenAI and Anthropic, has structurally disabled the very caution they publicly champion. Coxon’s departure is not an anomaly; it is a data point in a pattern of organisations whose stated values are progressively subordinated to competitive pressure.

Coxon spent three years in the pretraining discipline — the foundational stage at which AI models absorb vast quantities of data to develop their capabilities. He joined OpenAI first, then moved to Anthropic precisely because he believed the latter operated with greater epistemic humility about existential risk. What he found, by his own account, was a company that understands the stakes with clarity and presses forward regardless. “At Anthropic, the stakes are well-understood,” he wrote on X, “but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk.” This is not recklessness born of ignorance. It is recklessness rationalised by game theory.

That rationalisation has now acquired institutional form. In February, Anthropic quietly removed from its safety charter a pledge to halt model development if it could not adequately control the associated risks. The company’s stated justification — that a unilateral pause would cede the field to less cautious competitors — is internally coherent and externally devastating. It is the logic of an arms race dressed in the language of responsibility. Anthropic’s own safety executive, Evan Hubinger, publicly estimated the probability of AI killing all humans within the next decade at above ten percent, while simultaneously affirming that the company is “trying its best.” A ten-percent civilisational risk, acknowledged openly, and the response is to continue — just more carefully. This is the operational definition of a structural trap.

The broader industry signals reinforce rather than contradict this reading. OpenAI paused training on its latest models for two weeks in August before resuming under tighter controls — a gesture toward caution that nonetheless ended in resumption. The company’s chief scientist, Jakub Pachocki, called for “extreme caution” in a blog post on Sunday and urged international governmental coordination as a top priority. The gap between the rhetoric and the operational reality is not hypocrisy so much as institutional incapacity: these organisations lack the external constraint that would make restraint a viable strategy rather than a competitive liability. In the United States, no federal law regulates AI models. Senator Bernie Sanders and Representative Greg Casar introduced legislation in September to suspend AI development pending the creation of a federal regulator, but the bill remains a legislative proposal in a Congress with a crowded agenda.

What Coxon’s resignation ultimately illuminates is a governance vacuum operating at speed. The race toward what AI leaders now openly describe as “recursive self-improvement” — the theoretical threshold at which AI systems design and train successive generations of AI with minimal human involvement — is accelerating without a credible international framework to manage it. More than 1,000 technology industry employees, including Anthropic’s own chief executive Dario Amodei, signed an open letter in July calling on Washington to support a coordinated slowdown in the development of the most advanced systems. The letter is notable precisely because it demonstrates that the people building these systems do not trust the competitive environment they themselves have created. The implication is stark: if the architects of the race are petitioning governments to impose the brakes they cannot apply themselves, the question is no longer whether regulation is necessary, but whether it will arrive before the systems it is meant to govern outpace it.