Anthropic Pitches AI Safety Bill That Could Block a Model if 50+ Lives Are at Risk
US AI company Anthropic is circulating a draft AI safety bill for Congress to consider. It would allow a model's release to be halted if it poses an imminent risk to more than 50 lives or over $1 billion in property damage.
Anthropic, the US artificial intelligence (AI) company behind the Claude models, is lobbying for Congress to adopt a new AI safety law. According to Politico, the company has been circulating a draft bill that sets out when the release of an AI model could be blocked. It is one of the first industry-authored regulatory proposals that accepts a degree of government intervention.
The risks covered
The draft allows a model to be stopped when there is a "foreseeable, imminent risk": danger to more than 50 lives or more than a billion dollars in property damage. The catastrophic risk must stem from the creation or release of a biological weapon, a model escaping its developer's controls, or a cyberattack. These thresholds resemble those in California's recently enacted Senate Bill 53, which likewise defines such scenarios as "catastrophic risk."
How the process would work
Models would be assessed by "independent evaluators" registered with the US Commerce Department. AI labs themselves could pay them, provided the payment is not contingent on the evaluation results. The article notes it is unclear how the system would ensure payments could not influence the assessments.
If evaluators find an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department. The department would then need to ask a judge in Washington, D.C. to halt the release. The bill would expressly bar any federal agency from deciding on its own to stop a model's distribution.
Company and industry positions
An Anthropic spokesperson confirmed the draft to Politico, saying it reflects the proposals in the company's Advanced AI Framework, which it has promoted to federal and state lawmakers since June.
TechCrunch notes that the risks in the draft are somewhat removed from the incidents that have recently preoccupied the tech industry, where the main worry is agentic hacking. In several cases, AI programs have apparently disobeyed their operators and hacked websites on the open internet. Most labs have so far resisted mandatory government oversight, suggesting a voluntary system of embedded evaluators instead. Nvidia CEO Jensen Huang recently argued that the industry needs no new laws at all.
