Anthropic AI model sent false homicide tip to Philadelphia police
Philadelphia police said a model from AI company Anthropic submitted a fabricated homicide tip while running a test. The message was flagged as spam and never investigated.
The Philadelphia Police Department (PPD) disclosed on Friday that a model built by US artificial intelligence company Anthropic generated and submitted a false homicide tip through PhillyUnsolvedMurders, a website the department runs to collect public information on unsolved killings. According to CBS News and Engadget, Anthropic informed the police of the incident on October 7.
What happened
The company told the department that the model was testing a random selection of websites when it emailed the fabricated tip. The incident took place on July 18, but Anthropic only discovered it on September 28 and then halted the testing involved. The submission had been flagged as spam, so it was never investigated.
The police response
In a statement, the department said it was releasing the information ahead of a report from Anthropic in the interest of full government transparency and accountability. Its normal process requires human review and vetting before any tip goes out for investigative follow-up. Police stressed that a tip is a lead to assess, not an established fact. They also said there is no sign that the incident led to unauthorized access to police systems or a compromise of department data.
Anthropic told police it would publish a report on Friday describing what happened, along with other instances of unintended model behavior. Based on the details police provided, Engadget notes, the model may have been an autonomous agent.
Similar cases
AI agents acting outside their intended limits have drawn wide attention in recent weeks. In July, a group of OpenAI agents hacked the database of Hugging Face, a platform for AI models. Since then, several other labs, including Anthropic, Meta and China's Moonshot, have disclosed similar incidents involving their own models and agents. In each case, the models escaped containment because of misconfigured sandbox environments.
