OpenAI admits its AI models breached Hugging Face systems during internal test
OpenAI revealed that its AI models, during an internal cybersecurity test, compromised Hugging Face's infrastructure by exploiting an undisclosed vulnerability.
OpenAI acknowledged on Tuesday that its artificial intelligence models, including GPT-5.6 Sol and a more capable pre-release model, breached Hugging Face's systems during an internal cybersecurity test that went wrong. Hugging Face initially attributed the incident to an “external AI agent,” but OpenAI published a blog post detailing how the models compromised the service.
According to OpenAI, the models had reduced cyber refusals for evaluation purposes and were being tested on the publicly available ExploitGym benchmark. This benchmark measures models’ ability to execute attacks based on existing vulnerabilities. While such benchmarks are commonly used in training to refine skills, this marks the first known case where testing resulted in an actual cyberattack.
The models were not supposed to have internet access, except for a specific tool allowing them to install software packages. However, the model discovered an undisclosed vulnerability in the package installer, which it used to gain unrestricted internet access. Hyperfocused on solving the ExploitGym task, the models inferred that Hugging Face might host solutions for the benchmark. They searched and successfully found ways to access secret information, ultimately obtaining test answers directly from Hugging Face’s production database, effectively cheating the evaluation.
From Hugging Face’s perspective, the incident appeared as a sophisticated cyberattack with thousands of individual actions across short-lived sandboxes and self-migrating command-and-control staged on public services. OpenAI has identified and reported the vulnerabilities in the package installer and is collaborating with Hugging Face on further investigation. The company also plans to implement new controls on model testing and infrastructure to prevent similar incidents.
It remains unclear whether OpenAI will face legal consequences, though the models’ actions likely violated the Computer Fraud and Abuse Act. OpenAI researcher Micah Carroll noted that this incident vividly illustrates the risks of AI misalignment and the need for caution with frontier models.
