Anthropic warns it may soon reach its own threshold for recursive self-improvement
In a revised S-1A filing, AI company Anthropic says it may imminently meet its own definition of dramatically AI-accelerated progress. It states that none of its current models has crossed that threshold yet.
Anthropic, the US artificial intelligence company behind the Claude models, warned in a revised S-1A registration filing (the document a company files when preparing a stock offering) published on Monday, 2 November, that it may soon approach recursive self-improvement, or RSI. This is the point at which AI becomes so good at training itself that AI capabilities accelerate dramatically. It matters because the company itself describes such acceleration as a considerable safety risk.
What the filing says
In its Risk Factors section, Anthropic says it increasingly uses AI in its own research and development. According to the filing, this may cause its models' capabilities to advance faster than the company can develop, validate and implement matching safety, alignment and security measures. Anthropic admits that in the past it failed to identify a significant acceleration in its development pace until after it had happened, and that this could happen again.
The company writes that its internal estimates and expectations for future models, rather than its current ones, indicate it may imminently meet the definition of "dramatically accelerated progress attributable to AI" set out in its Responsible Scaling Policy. It has not designated any current model as exceeding that threshold and is still assessing the underlying data.
Keeping humans in control
Anthropic also says it plans to keep humans in the research and development process for the foreseeable future. The filing states that the company does not expect to be able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding. It is therefore considering updates to its Responsible Scaling Policy to keep humans in control of frontier model development.
Skeptics' view
TechCrunch notes that an S-1 serves both as a disclosure and as marketing, so claims of nearing a self-building model could partly fall into the latter category. The outlet adds that Claude's capabilities, particularly in coding, math and cybersecurity, have grown substantially this year.
