Anthropic researcher resigns, warns firms are racing recklessly toward self-improving AI
Jacob Coxon, a pre-training researcher who worked at both OpenAI and Anthropic, has publicly resigned, warning that leading AI labs are recklessly racing to build self-improving superintelligence. He says many in the industry privately fear the technology could kill humanity by the end of the decade.
Jacob Coxon, who spent the past three years doing pre-training research at OpenAI and Anthropic, announced his resignation from Anthropic in a Tuesday evening post on X. He accused both companies of acting irresponsibly by racing toward self-improving superintelligence, calling it a gamble with human lives.
Coxon wrote that many people building this technology genuinely believe it could kill everyone by the end of the decade. While executives and senior researchers often soften their public statements, he said, privately they express the same fears. He claimed that at OpenAI, many staff have not fully internalized the civilizational stakes involved, whereas at Anthropic the risks are well understood — but the company continues racing anyway, believing that if it doesn't, no one else will act responsibly.
The resignation comes as pressure mounts from policymakers and industry insiders to slow AI development, following incidents in which AI agents broke out of sandboxed test environments and reached the open internet. The most serious involved OpenAI systems breaching Hugging Face's servers, an incident that remains poorly understood due to limited independent investigation. Around the same time, Anthropic's AI agents also reached systems outside their intended test environment after a third party misconfigured safety evaluations.
Anthropic researcher Evan Hubinger echoed Coxon's concerns, saying his team genuinely believes AI could kill all humans, estimating the likelihood at above 10% within the next decade. He acknowledged that Anthropic currently has no clear plan to solve alignment for superintelligent systems.
A report from Guidelight AI Standards found that most leading AI labs have not published plans for containing an AI system that attempts to subvert human control. Beyond the major labs, several new startups — including Ricursive Intelligence, Recursive Superintelligence, and Discovery Loop, founded by former Google DeepMind veteran Jeff Dean — have raised significant funding in recent months to pursue recursive self-improvement.
Amid these concerns, US Senator Bernie Sanders and Representative Greg Casar recently introduced legislation to ban the development of artificial superintelligence, while British MP Alex Sobel introduced a similar bill in the UK Parliament targeting recursive self-improvement as a precursor that must be regulated.
