Anthropic researcher Jacob Coxon resigned from the AI company, saying he no longer wants to be a part of an increasingly dangerous race to build self-improving artificial intelligence.
Coxon, who spent the past three years conducting pretraining research at both Anthropic and OpenAI, announced his resignation on X. He accused both companies of moving too quickly toward ‘self-improving superintelligence,’ arguing that the potential consequences could extend far beyond the technology industry.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote, describing the concern as a genuine risk rather than a public-relations tactic.
Coxon argued that future AI systems could become capable of performing tasks far beyond human abilities, including sophisticated cyberattacks, rapid scientific and technological advances, and acquiring access to real-world resources.
His central concern is the development of systems capable of improving themselves. He questioned whether companies should continue pushing increasingly powerful models when researchers do not yet have a reliable understanding of how such systems would behave once they become significantly more capable.
Coxon also criticized the competitive dynamic between leading AI companies. He claimed that while the risks are understood more clearly at Anthropic, the company remains locked in a race because of fears that another organisation may develop advanced AI first.
Evan Hubinger, Anthropic’s Alignment Science Lead, responded to Coxon’s comments with an unusually direct assessment. Hubinger said Anthropic researchers genuinely believe AI could potentially kill all humans and gave his personal estimate of that risk at more than 10% within the next decade. He stressed that this was his own assessment rather than an official company-wide probability.
Hubinger also acknowledged that Anthropic does not yet have a complete solution for aligning superintelligent AI with human interests. He said current models pose relatively low risk, while recursive self-improvement is the larger concern.
Coxon urged researchers inside major AI laboratories to reconsider whether they should continue accelerating development simply because competitors are doing the same. He pointed to recent AI-related cybersecurity incidents, including an attack involving AI agents and Hugging Face infrastructure, as a warning that more coordination may be needed among frontier AI companies.
Coxon said preventing a global race could require costly measures, potentially including a temporary pause on improving model capabilities. His resignation adds another prominent internal warning to the growing debate over how quickly advanced AI should be developed and what safeguards should be in place before systems become significantly more autonomous.