Anthropic pitches AI safety bill that would let courts block a model's release
AI company Anthropic is circulating a draft bill it hopes the U.S. Congress will consider. It would allow a model's release to be halted if more than 50 people could be at risk of dying.
Anthropic, the U.S. artificial intelligence (AI) company, has drafted an AI safety bill and is shopping it around in the hope that the U.S. Congress will take it up. Politico reported on the draft, and a company spokesperson confirmed it. The text sets out conditions under which the distribution of an AI model could be blocked, making it one of the first such proposals to come from within the industry itself.
Which risks are covered
A release could be stopped if there is a "foreseeable, imminent risk" that more than 50 people could die or that over a billion dollars in property damage could result. The catastrophic risk would have to stem from the creation of a biological weapon, a model evading its developer's controls, or a cyberattack. The thresholds are similar to those in California's recently enacted Senate Bill 53.
How the process would work
A model's potential for disaster would be assessed by "independent evaluators" registered with the U.S. Commerce Department. AI labs themselves could pay them, as long as payment does not depend on the results of the evaluations. Politico notes it is unclear how the system would ensure payments could not influence the assessments.
If evaluators find an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department. The department would then have to ask a judge in Washington, D.C., to halt the release. The bill would explicitly bar any federal agency from deciding on its own to stop a model's distribution.
An Anthropic spokesperson said the bill reflects proposals in the company's Advanced AI Framework, which it has promoted to federal and state lawmakers since June.
Industry context
TechCrunch points out that these scenarios are somewhat removed from the incidents that have recently worried the tech industry most. The main concern has been agentic hacking, after several cases in which AI programs apparently disobeyed their operators and hacked websites on the open internet. Most AI labs have resisted mandatory government oversight, proposing instead a voluntary system of third-party evaluators. Nvidia CEO Jensen Huang recently argued that the AI industry needs no new laws at all.
