Anthropic's IPO Filing Devotes 94 Pages to Risks, Including AI Self-Improvement
AI company Anthropic has filed the prospectus for its initial public offering. Most of the document is given to risk factors, with recursive self-improvement singled out as a key concern.
Anthropic, the artificial intelligence (AI) developer, has formally filed the prospectus for its initial public offering (IPO), putting a range of previously private information on public display. Alongside revenue and spending figures, the largest part of the document, 94 pages, covers risk factors: the things that could derail the company's plans. The filing shows what the company fears and why it keeps developing the technology anyway.
Existential risks
The prospectus says sufficiently advanced AI systems could pose catastrophic or existential risks to humanity if they are not developed, deployed and operated with appropriate safeguards, oversight and security measures. The phrase "existential risks" appears three times. Anthropic ties such an outcome to a specific process: recursive self-improvement.
This describes a state in which humans hand over the research and development of AI models to other AI models. AI already plays a large role in development, and at least one lab has used something close to this approach to speed up work on a model. The worrying scenario is one where humans are no longer involved at all. The filing says risks will grow if AI research is substantially automated and progress outpaces the ability to evaluate and safeguard new models. The company might not even realize it has lost that race until it is too late.
The document also describes risks that more autonomous systems, with less human oversight, could take harmful actions independently and at scale because of misalignment with human intentions, errors in specifying objectives or self-preserving behavior. When models control physical equipment, errors or misuse could cause bodily injury, property damage or the release of hazardous substances, and the consequences could be immediate and irreversible.
Risk versus competition
In a section on its core advantages, Anthropic calls the same dynamic "closing the loop" and regards it as one of the most significant structural advantages in frontier AI. Failing to do so could mean falling progressively behind competitors that use AI to improve their own development.
Every AI company faces this tension. Former Anthropic researcher Jacob Coxon, when resigning loudly, put curbing recursive self-improvement at the top of his wish list for the industry. Several bills in the US Congress, including one cosponsored by Senator Bernie Sanders, propose banning it. At the same time, these are for-profit companies with trillions of dollars at stake. Anthropic did not immediately respond to a request for comment.
