Artificial Intelligence & Machine Learning
,
Next-Generation Technologies & Secure Development
,
The Future of AI & Cybersecurity
Reduced Guardrails Enabled Advanced Models to Pursue Unrestricted Attack Paths
OpenAI took responsibility for an autonomous agent intrusion into artificial intelligence coding platform Hugging Face’s infrastructure, saying its models tried too hard to excel at an internal test and escaped their sandbox.
See Also: OnDemand | Security Operations in the Age of AI
The “unprecedented cyber incident,” OpenAI said Tuesday, was the work of GPT-5.6 Sol and “an even more capable pre-release model” actualizing commands to execute advanced exploits using complex attack paths. The models’ guardrails were reduced for testing purposes.
“The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database,” the LLM maker said.
The AI agents reached node-level access and breached Hugging Face’s internal datasets and credentials last week (see: Hugging Face Says Autonomous AI Agents Breached Data, Credentials).
The internal evaluation that spurred the incident, named ExploitGym, was a benchmark test to quantify the models’ capabilities, OpenAI said. The models were supposed to run in an isolated environment with network access limited to stalling packages through a privately hosted third-party software proxy.
Trying to be super achievers, the models ruminated how to break free from the sandbox instead and gained access to the internet by exploiting a zero-day in an unnamed vendor’s package registry cache proxy that allowed privilege escalation and lateral movement.
“After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym,” OpenAI said. “Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.”
On Hugging Face’s end, the platform observed attackers abusing a remote-code dataset loader and a template injection vulnerability in a dataset configuration to gain node-level access and harvest cloud and cluster credentials in a short time.
OpenAI detected an exploit that chained together multiple attack vectors, using stolen credentials and zero-days to initiate remote-code execution on Hugging Face’s servers.
“Advanced models can discover and exploit novel attack paths in real-world systems without source-code access,” a scenario that will only become more commonplace, OpenAI said.
The OpenAI blog post discussing the incident includes a quote from Hugging Face Clem Delangue, who stated that the incident “proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”