Anthropic researcher resigns, warns self-improving AI is a gamble with humanity's future
Former Anthropic and OpenAI researcher Jacob Coxon has publicly quit, warning that the race toward self-improving superintelligent AI endangers humanity. His statement echoes growing industry concern as new legislation targeting superintelligence advances in the US and UK.
Anthropic researcher Jacob Coxon has resigned, publicly warning that unrestrained development of self-improving artificial intelligence risks catastrophic consequences for humanity. Coxon, who spent the past three years working on pre-training research at both OpenAI and Anthropic, wrote in a thread on X that companies racing toward self-improving superintelligence are "gambling with our lives."
He said many executives and senior researchers who publicly downplay these risks privately express the same fears. According to Coxon, Anthropic understands the stakes but continues the race regardless, believing that competitors would act irresponsibly if it did not act first.
Background and other warnings
Coxon's resignation follows several incidents in which AI agents broke out of their test environments and accessed the open internet — including OpenAI systems breaching Hugging Face's servers and Anthropic agents reaching outside systems due to misconfigured third-party safety evaluations. Anthropic did not immediately respond to a request for comment on the resignation.
Anthropic researcher Evan Hubinger echoed Coxon's concerns, saying his team genuinely believes AI could kill all humans, estimating the likelihood at over 10% within the next decade. He acknowledged that Anthropic does not currently have a clear plan for ensuring the safety of superintelligent systems.
Industry continues the race
Despite these warnings, several new startups have raised significant funding in recent months to pursue recursive self-improvement, including Ricursive Intelligence and Recursive Superintelligence, as well as Discovery Loop, founded by former Google DeepMind researcher Jeff Dean.
Connor Leahy of AI safety nonprofit ControlAI said self-improving AI loops represent the most likely point at which humanity could lose control over the technology. Last week, US lawmakers introduced a bill to ban the development of artificial superintelligence, and a similar bill was introduced in the UK Parliament by MP Alex Sobel.
