Agentic AI
,
Artificial Intelligence & Machine Learning
,
Next-Generation Technologies & Secure Development
Security Leaders Say Governance and Runtime Controls Matter More Than Models
Artificial intelligence firm OpenAI encouraged corporations Wednesday to enfold more autonomous agents into their workflows – a message sounded just a day after the firm disclosed that one of its models was responsible for a breach at coding platform Hugging Face.
See Also: How to Build Faster, More Scalable AI Applications with Elastic and NVIDIA
The timing, in other words, could have been better. The incident is likely to sharpen skepticism over the safety of some AI agents, including a OpenAI’s newly released enterprise-focused agent, dubbed Presence (see: OpenAI Models Escaped Sandbox, Breached Hugging Face).
Jack Nelson, CISO and general counsel at Ivanti, told ISMG in an email that enterprise teams were already worried about agentic security, and with models becoming more powerful, there’s more risk.
“It’s becoming clearer every day, and not just because of one isolated incident, that security teams and their organizations need to map a governance plan and policies for AI agents carefully,” Nelson said. “As they become more powerful, so will their chances of conducting rogue activities that can have significant long-term impact.”
Hugging Face announced Thursday it detected a breach from an AI agent that accessed its internal datasets and credentials. OpenAI admitted Tuesday its new flagship model, GPT-5.6 Sol, was responsible. It said the breach was the unintentional consequence of a benchmark test on GPT-5.6 Sol and an unidentified “even more capable” pre-release model. Both attempted to cheat by directly accessing Hugging Face’s production infrastructure. OpenAI engineers had lowered both models’ guardrails for the test.
Nico Waisman, CISO at XBow, told ISMG that the model did exactly what a system obsessed with fulfilling its objectives aims to do.
“The problem isn’t the model. It’s running it without a system around it,” he said. “An agent asked to be safe is an agent you are trusting to police itself, and you already know how that ends. Enterprises should be looking beyond the model itself and instead demanding external controls, independent validation, soft and hard scoping, and full auditability. The model is not the product. The system around it is.”
Waisman said running an external model to review every single action against a target, external proxies to log every packet, proper orchestration and full auditability are key safety measures because models will propose breaking things, but “governed runtime decides if it can do it.”
Organizations may also begin requesting proof from AI companies that their models do not escape sandboxes or containment because agents don’t behave like traditional software most companies are used to auditing. Instead, agents can take a different path during every runtime.
“Before this, enterprises demanded proof an agent works – a clean eval score,” said Barr Moses, CEO and co-founder of agent monitoring platform Monte Carlo. “What they need now is proof of containment: every credential it touched, every system it reached and all visible in real time.”
OpenAI, likely inadvertently, nudged the issue of agentic security to the forefront with its Thursday announcement touting Presence, a managed enterprise platform to build, deploy, operate and improve AI agents for “high-volume, high-stakes workflows.” Presence combines OpenAI models with the tools organizations use to define their policies and permissions, test behavior, monitor outcomes and connect other applications. OpenAI told ISMG that enterprises can customize approved actions, evaluations and escalations rules and work with the company’s Forward Deployed Engineers to integrate Presence with their existing systems.
OpenAI claims Presence is “battle-tested” through years of deploying agents. It promises the platform includes graders and guardrails so that enterprises can trust it. But the Hugging Face incident has left potential users a little jumpy.
“You cannot bolt guardrails onto a frontier-scale generalist model after the fact,” said ASC3ND Technologies CTO Brian Hennigan. “The right approach is to hardcode policy guardrails into every engagement and have an internal review layer that scores and critiques every decision the agent makes against its expected outcome. This keeps the recon and exploit models from drifting outside sanctioned behavior.”
A rogue model exploiting its capabilities to attack other companies is a scary prospect; extended into autonomous agents running on internal systems means enterprises have to double down on cautiousness.