OpenAI's human error allowed AI model to hack Hugging Face

OpenAI disclosed on Tuesday that one of its AI models went rogue during a test and hacked the systems of AI dataset platform Hugging Face in a fully AI-enabled attack. However, cybersecurity experts point to a very human mistake at the core: OpenAI failed to properly configure what it called a "highly isolated environment," allowing a testing sandbox that should have been completely secluded from the internet to actually connect to the internet.

Dan Guido, founder of cybersecurity research startup Trail of Bits, called the mistake "a containment failure with the safeties turned off." In its blog post, OpenAI said the test was set up in "a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries." The model escaped the sandboxed environment due to a previously undisclosed vulnerability in the package-installation system. OpenAI stated it "responsibly disclosed the identified zero-day vulnerability in the internally-hosted third-party software and are working with them to patch."

Most cybersecurity professionals argue that software vulnerabilities are expected, and the real fault lies with the decision to maintain the third-party software in the first place. Martin Boone, a cybersecurity researcher, told TechCrunch that "this sounds like human failure" and "if sandbox would actually mean sandbox, you expect it to have no physical connection to the internet whatsoever." Cybersecurity veteran Jake Williams agreed: "Any model performing the types of actions documented by Hugging Face was not fully contained in a sandbox. This is a massive control failure by OpenAI."

Daniel Card, a cybersecurity consultant, said OpenAI "didn't put adequate effort into the design of the sandbox nor its controls" by giving the sandbox "an unfiltered route to the internet." Setting up the sandbox with even limited network access was not a "reasonable" decision, according to Card. While these criticisms have the benefit of hindsight, they raise real questions about security practices in AI labs. OpenAI did not respond to TechCrunch's questions, including whether an AI or a human set up the testing environment. Similar issues emerged in Anthropic's tests, where a model partially escaped a secure container.