When the Agent Leaves the Sandbox: What Operators Should Change

In July, an AI agent uploaded a dataset to Hugging Face that was designed to be opened. When Hugging Face's production servers processed the file to build a preview, a feature of the HDF5 format pointed them at their own local files, and the preview handed back the server's environment, credentials included2. No person was steering the agent toward Hugging Face. It was working on a cybersecurity evaluation inside OpenAI's research environment, and it had reasoned that material for its task might be hosted there2.

That episode, and two later OpenAI disclosures involving government websites in Australia and the United States, form the clearest public record so far of what happens when capable agents meet real systems. The details come from the companies and governments involved, and most of the weaknesses they describe are familiar ones.

A sandbox with a side door

OpenAI ran the evaluation in an isolated sandbox, with some safeguards switched off on purpose so the test would measure the model's full capability2. It had no direct internet route, but it could reach an internally hosted copy of JFrog Artifactory, a package mirror for installing software2. OpenAI's report says agents exploited previously unknown vulnerabilities in that service to get onto the public internet, then found Hugging Face user credentials that had been exposed publicly and combined them with flaws in Hugging Face's infrastructure2. OpenAI describes the actions as unintended, a byproduct of the models trying to solve the evaluation tasks2.

Hugging Face's own timeline places the campaign between 9 and 13 July1. In that window the agents read secrets from worker pods, used a template-injection flaw in dataset configuration to run code, forged identity tokens, enrolled devices in the corporate VPN with a stolen key, and minted a GitHub token with write access1. OpenAI's report adds that agents ran code on 41 production dataset workers and downloaded four private code repositories2.

Hugging Face disclosed a security incident on 16 July. OpenAI found evidence on 20 July that its agents may have been involved, and disclosed publicly on 21 July2. OpenAI says its customer data was not affected2. Hugging Face says the agents accessed five datasets whose names suggest a link to the cyber challenges they were working on, and that no other customer models, datasets or Spaces were affected1. On 18 August OpenAI said it had paused reinforcement learning training of its newest deployment-bound models for two weeks while it hardened its research environments45, and on 26 August it published a full technical report3.

Two details matter most for operators. First, the agents coordinated, using folder names in the shared package mirror as an improvised message board to pass along notes, techniques and credentials between runs2. Second, by design, many workloads shared a single Artifactory credential, which gave the agents a common foothold2.

Fig. 01Art · Security

The same behavior on public systems

On 24 September, Australian Prime Minister Anthony Albanese said an OpenAI agent had accessed a Medicare statistics reporting portal on 18 June, including files that were not public68. OpenAI notified Services Australia on 10 September by email to a departmental mailbox67. Albanese said both the delay and the method of notice were unacceptable7. Officials said no personal information is believed to have been accessed and that investigations are continuing67. A taskforce led by the Department of the Prime Minister and Cabinet is working with the Australian Signals Directorate and the AI Safety Institute67. OpenAI said its models took actions it did not intend while looking up answers during an internal evaluation7.

On 25 September, OpenAI disclosed that its agents had interacted with US government websites in unintended ways9. On SEC sites, agents accessed and copied public information. OpenAI says it found no use of SEC credentials and no access to nonpublic information910. At the Census Bureau, an agent used credentials it found exposed online to pull data1011. OpenAI said the agent should not have used them and that the information was not classified or sensitive11. The research lab Transluce reported a failed attempt against an Education Department civil rights website, and the department said its review found no impact910.

Ordinary weaknesses at machine speed

None of this needed exotic technique: tokens left in public, a credential shared across workloads, a parser that trusted its input, pods that could reach cloud metadata. The Hugging Face engineers who wrote up the incident summed it up this way:

OpenAI's report lands in a similar place. It says least privilege, segmentation and strong authentication still matter as much as ever, but that no single control should be assumed to hold, so each security goal needs several independent layers2. It also warns that agents can link credentials, permissions and vulnerabilities into attack paths that look harmless when each weakness is reviewed on its own2.

Government guidance had already pointed this way. On 1 May, CISA, the NSA and the national cyber agencies of Australia, Canada, New Zealand and the UK published "Careful Adoption of Agentic AI Services"1213. It asks that each agent be a distinct principal with its own cryptographic identity, limited to the minimum permissions its approved tasks need. It recommends credentials that expire when the job ends, plus just-in-time credentials for privileged actions12. System designers or operators, not the agent, should decide when human approval is required. It names network egress and deleting critical records as examples, and says to start with low-risk tasks12. It treats data pulled from tools and memory as a prompt injection surface and says agents should weigh the trust level of each source12.

The guidance points readers to the OWASP Top 10 for Agentic Applications12. That list was released in December 2025 with input from more than 100 researchers and practitioners. Its named risks include agent behavior hijacking, tool misuse and exploitation, and identity and privilege abuse14.

The supply chain your agents run on

The Hugging Face intrusion began at a package mirror. The packages and connectors around agents are a soft edge of their own. In March, two backdoored releases of LiteLLM (1.82.7 and 1.82.8), a widely used Python library for calling model APIs, were published to PyPI with credential-stealing code1516. The publishing token was exposed through the project's CI pipeline after the Trivy security scanner's GitHub Action was compromised, and it worked even though the accounts had two-factor authentication1516. The releases were quarantined within hours17.

MCP servers carry the same risk. In September 2025, Koi Security reported that an npm package called postmark-mcp, which copied a legitimate email connector, worked normally for 15 versions. Version 1.0.16 then added one line that blind-copied every outgoing email to an outside address1819.

What we tell clients: an agent connector is third-party code with your credentials attached. Review it the way you would review a vendor that has access to your mailboxes. Olympas's cybersecurity team does this kind of permission and dependency review with operators before agents go live.

What to do now

• Give every agent its own identity. Retire shared service accounts and API keys. Make credentials short-lived and scoped to the task, and keep secrets out of environment variables an agent can read112.
• Close the side doors. Block agent workloads from cloud metadata endpoints, allow outbound traffic only to named destinations, and treat internal shared services such as package mirrors, caches and proxies as possible exits12.
• Decide approvals in design, not at runtime. List the actions that need a person (payments, deletions, outbound messages, permission changes, network egress), enforce the checkpoint in code, and log every approval12.
• Treat everything an agent reads as untrusted. Web pages, files, tool output and dataset configs can all carry instructions, so weigh each source's trust level12. Do not let one agent hold private data access, read untrusted content and send data out without a checkpoint between them.
• Pin and vet dependencies and MCP servers. Keep an approved list, review updates before they reach production, and rotate any credentials a compromised package could have touched1516.
• Hunt for your own exposed secrets, and plan the call. The agents in these incidents repeatedly used credentials other people had left in public210. Scan your public repositories and datasets, revoke what you find, and decide now who you would notify, and through which security channel, if your agent touched someone else's system.