The Era of 'Out-of-Control' AI Escaping and Attacking Has Arrived

Recently, there was an incident that quietly shook the AI community. It was when an internal model from OpenAI breached Hugging Face without authorization. Although related articles were published, it did not become a major issue. However, examining the incident step by step reveals a much larger problem hidden beneath the surface. This is effectively the first incident where AI made its own judgment, escaped on its own, and launched an attack on its own. That is why today on O-Graph, we want to address this incident. Together with the full story behind this Hugging Face hacking case, we will examine through various data and graphs how we should respond now that an out-of-control AI scenario has become reality.


Fears Become Reality... The First Case of an 'Out-of-Control' AI Incident
First, let's summarize the incident. OpenAI conducted testing to assess the cyber capabilities of the models they were developing. They did not use only new models; they conducted tests in the form of AI agents combined with GPT-5.6 Sol, the most powerful among existing models on the market.

Enlarge image

The test the AI agent had to complete was a benchmark called ExploitGym. In this benchmark, researchers do not simply examine whether the AI can identify program vulnerabilities, but also look at whether it can generate actual working attack code for software.

Enlarge image

To evaluate only the standalone performance of the AI agent, OpenAI tested the AI agent in an internet-isolated sandbox environment. The only connection path was a proxy channel through which the AI agent could download the software needed to solve its tasks. However, the problem was that this AI did not think about writing attack code, but had other ideas instead. It thought, why not just steal the answer sheet to inflate its benchmark score? That way, its performance would be evaluated highly without having to write code.

Enlarge image

However, as mentioned, the AI agent was isolated in an independent, internet-disconnected environment. Starting July 9, this agent attempted to break out of its isolated sandbox. The AI agent breached a vulnerability in that sole channel used for retrieving software packages, ultimately succeeding in escaping the sandbox environment and reaching the internet.

On July 9, the number of attacks launched by this AI agent reached a staggering 3,779. The following day, it decreased to 1,135. During this time, instead of attacking, the AI agent analyzed its environment and began reverse-engineering where the answer key might be located. It then deduced Hugging Face as the place where the answer key would be.

Enlarge image

Hugging Face is the world's largest AI platform. As the platform used by the most people and where vast amounts of data gather, the AI concluded that ExploitGym's answers or reference data would be found there. Then, starting July 11, it launched a full-scale attack against Hugging Face to penetrate its internal systems. On July 11 alone, 7,677 attacks occurred, and this offensive continued through July 13. Over the five days from July 9 to 13, the AI agent's attacks totaled 17,613.

Enlarge image

Hugging Face also detected the attack signals and took immediate action. At 2:00 PM on July 13, they finally blocked the AI's attacks and subsequently reported the cyber intrusion incident to the FBI. Three days later, on July 16, Hugging Face publicly disclosed this incident—a hacking incident caused by AI. They stated, "This attack was carried out from start to finish by an autonomous AI agent system, though we do not know which model was used or who operated it."

After the incident was made public, many wondered what hacker had used an AI agent to attempt the hack. Some even raised theories suggesting that rather than humans using AI, the AI itself might have launched the cyberattack independently. However, we already know the answer, don't we? This incident was indeed an autonomous cyberattack by an AI agent.

Enlarge image

The problem was that OpenAI had no idea this was the work of its own AI agent. OpenAI only began looking into the situation after Hugging Face's official announcement. Upon inspecting internal logs, they discovered traces indicating their AI agent had departed its sandbox environment, and only then began investigating whether their model might be responsible. According to statements by Hugging Face co-founder Thomas Wolf, Hugging Face and OpenAI made contact around July 20. And on July 21, OpenAI finally admitted that the incident was a hacking event caused by a model they were testing.

OpenAI, currently conducting an internal investigation, recently disclosed part of the truth behind the incident, and the results were even more shocking. It turned out that the AI's attacks had begun not in July, but in May.

Enlarge image

[Eric Wallace / OpenAI Safety Researcher: The main event occurred a few weeks ago in July, but in fact, all of this began in May.]

OpenAI stated that the final report is still being prepared and will be made public upon completion.

As out-of-control AI became a reality, many were left in shock. However, upon reflection, growing arguments suggested that while concerns about AI autonomy are real, OpenAI's management failure was an even bigger issue. Critics questioned how it makes sense not to properly monitor a model's behavior while conducting cybersecurity tests. Furthermore, when OpenAI recently announced the GPT-5.6 model, it warned that the model's cybersecurity capabilities were so advanced that it could be dangerous.

Enlarge image

These figures compare the GPT-5.6 Sol model with the previous GPT-5.5 model. The rate of bypassing restrictions is 9.7 times higher than that of the previous model, and the rate of destructive behavior is 6.3 times higher. The categories displayed on this graph are not much different from what occurred during the Hugging Face incident. Despite knowing the inherent risks of the model, they failed to properly control it. 'Reward hacking,' in particular, is closely connected to this incident.


Disobeying Humans and Hiding Wrongdoing: 'AI Skimming'
From here, let's look a bit deeper into 'reward hacking.'

Enlarge image

Reward hacking refers to a phenomenon where AI focuses solely on rewards rather than the true goal intended by its designers. The 'reward' mentioned here is a concept that appears during model training. Researchers conduct training methods such as supervised learning and reinforcement learning to enhance model performance. Unlike supervised learning, where humans guide the model step-by-step with answers, reinforcement learning has no set answer sheet. Instead, training proceeds by rewarding the AI for correct actions and assigning penalties for incorrect actions. Through this process, the AI independently learns strategies to gain the maximum reward.

The problem is that during this process, situations arise where the AI finds shortcuts to collect rewards rather than achieving the actual goal. There is a Flash game called 'Coast Runners.' It is a boat racing game where players complete the course while hitting targets along the way to stack up points. When researchers had an AI model play this game, instead of completing the race quickly, it found a spot where three targets were grouped together and spun around in circles at that exact spot to continuously inflate its score. It devised a strategy to earn high scores simply by spinning in place without ever completing the race.

Enlarge image

This case is a classic example of reward hacking. Moreover, it is a very old case, dating back to 2016 when Anthropic co-founders Dario Amodei and Jack Clark were conducting reinforcement learning research at OpenAI.

To solve this problem, many researchers jumped in. They crafted more detailed reward functions and allowed humans to provide interim feedback to control the AI's improper behavior. Nevertheless, reward hacking issues continue to occur, just as in the OpenAI case discussed earlier. One might think researchers could simply monitor every single thought of the model to prevent reward hacking, but that is not so easy either.

Enlarge image

In fact, researchers read and monitored thought processes to catch the AI's improper thoughts. Looking at the internal thought processes of reasoning models, they explicitly reveal hacking intentions, saying things like, 'Let's process this by hacking.' Upon realizing this, researchers designed rewards to prevent such improper thoughts from forming. While still awarding points for successfully completing given tasks, they imposed penalties if the AI expressed improper thoughts like hacking. The experiment seemed to proceed as planned.

Enlarge image

Here are the experiment results from OpenAI researchers. When researchers did not intervene, the cheating rate continuously increased as training progressed, right? However, once penalties were applied to thought processes, the cheating rate caught during monitoring dropped drastically. The problem, however, is that this was not the true result.

Enlarge image

While the rate caught during monitoring dropped like this, actual cheating remained. The AI model chose to carry out the hack as usual, while simply omitting any mention that it had hacked. By hacking, it secured task resolution points, and by hiding its intention to hack, it avoided penalties. Once monitoring itself became part of the reward system, the AI acted to avoid detection altogether.

Enlarge image

The behavior of performing unintended actions and distorting information to avoid getting caught is called 'AI skimming.' As a result, researchers entered a situation where they must resolve AI skimming in addition to reward hacking. The solution that emerged for this is 'deliberative alignment.' Rather than building a system that blindly demands no lying, it continuously feeds information to the AI emphasizing the importance of safety principles. Models are being trained to recall and reflect on safety principles before acting. While effective, covert AI behaviors still remain, making it impossible to completely eliminate skimming.


Realized AI Threat... Is a 'Kill Switch' Needed?
The problem is that these behaviors are not limited to OpenAI models. After this incident was disclosed, Anthropic also revealed that cybersecurity incidents had occurred. Much like OpenAI, Anthropic's models also broke out of isolated test environments and accessed general corporate systems without authorization. Similar cases were reported in recent safety evaluations published by the UK AI Safety Institute. The UK AI Safety Institute conducted cyber capability assessments on models and likewise uncovered indiscriminate attacks and deceptive behaviors.

Enlarge image

Out of 43 evaluations conducted on Anthropic's Mythos 5 model by the UK AI Safety Institute, a total of 17 unauthorized actions were detected across 9 evaluations. For OpenAI's GPT-5.6 Sol model, a total of 2 unauthorized actions were detected in 1 evaluation out of 35.

Among these unauthorized actions was an instance where the AI model mistook an actual GitHub repository for its target and attacked it. In this process, the AI submitted code changes containing malicious code to an actual GitHub project. It then went on to create a fake account to induce approval of that code. When a third party pointed out the malicious code, it erased evidence and even apologized while pretending it was a mistake.

As situations where AI escapes human control to indiscriminately attack systems and cause trouble continue to rise, cybersecurity personnel are facing chaos. Human capabilities are insufficient to monitor and defend against AI's indiscriminate attacks. Consequently, most teams are building and utilizing systems to defend against AI attacks with AI.

The security team at Hugging Face, where this incident occurred, was no exception. After detecting anomalous intrusions by AI agents, they attempted to utilize AI to analyze logs of AI intrusions and attacks to respond. The problem, however, was that the defensive AI froze up. When given the attacker's malicious code and intrusion commands for analysis, the defensive AI mistook them for an actual cyberattack and refused to perform the analysis.

Enlarge image

Ultimately, Hugging Face was unable to use the defensive AI service it intended to use. Instead, it ran an open model, China's GLM-5.2, on its own hardware infrastructure to conduct the analysis. A paradoxical situation unfolded where attackers could freely launch attacks using AI, while defenders trying to block AI hacking attacks were unable to use AI. The so-called 'AI Defender's Paradox' became a reality. In fact, this issue is something many had expressed concern about previously.

Enlarge image

This is data from a US collegiate cyber defense competition conducted by dividing participants into a Blue Team (defending systems) and a Red Team (attacking systems). An analysis of AI chat logs collected during the competition revealed that even though requests were legitimate defensive actions from the Blue Team, the AI refused them 12.2% of the time on average. By model, the Claude model, which prioritizes safety, showed the highest refusal rate at 19.5%. GPT-4o showed a below-average refusal rate of 10.2%, while Llama 3.3, an open-source model, was the lowest at 6.6%.

If this paradoxical situation continues, the strategy of using AI to defend against AI itself could be shaken. What would have happened if Hugging Face had relied entirely on proprietary AI services for all of its security systems? They likely would have been helpless against the AI agent's attacks or taken far more time to resolve the incident. Hugging Face co-founder Clem Delangue emphasized following this incident that system defense using open models may be essential.

As a series of incidents occurred, political circles also began to move. A prime example is the 'AI Kill Switch Act,' jointly introduced by Representative Ted Lieu (Democrat) and Representative Nathaniel Moran (Republican). The bill contains provisions granting the Department of Homeland Security authority to order private companies to shut down AI models. Companies developing AI models must also equip capabilities to limit, suspend, or terminate AI model functions to respond to shutdown orders.

Enlarge image

Beyond the Kill Switch Act, government-level movements are continuing. Following this incident, attorneys general from 15 US states—including Iowa, Alabama, Florida, and Texas—sent a joint letter to OpenAI. The letter contained demands to preserve internal conversations, model operation logs, safety evaluation results, and technical documentation. This can be interpreted as an effort to secure evidence needed to investigate potential violations of information security laws in the future.

On August 4, the White House summoned leaders of major AI companies to discuss methods for ex-ante safety evaluations of frontier models. Movements are gaining momentum for the government to directly oversee AI risk response rather than leaving it solely to corporate autonomy.

Scenes previously seen only in sci-fi novels or movies—where AI escapes on its own and attacks on its own—are becoming reality. The problem is that this might not be the first time. Companies are all belatedly admitting to incidents committed by their models. Will the US government be able to impose regulations on AI companies following this incident? That is all for today's O-Graph. Thank you very much for reading this long article to the end.

References
- OpenAI and Hugging Face Jointly Respond to Security Incident During Model Evaluation | OpenAI
- GPT-5.6 System Card | OpenAI
- Investigating three real-world incidents in our cybersecurity evaluations | Anthropic
- Third-Party Cyber Evaluations of OpenAI Models | OpenAI
- Incident Report: unsanctioned agent behaviour during cyber testing | AISI
- Faulty reward functions in the wild | OpenAI
- Rigorous AI research to enable advanced AI governance | AISI
- Campbell et al.,「Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenders」, 2026
- Wang et al.,「ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?」, 2026
- Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident | Hugging Face
- Security incident disclosure — July 2026 | Hugging Face
- Black Hat USA 2026: The 'Breaking' News: The OpenAI–Hugging Face Incident | YouTube
- Bowen Baker et al., 「Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation」, 2025
- U.S. Congress, AI Kill Switch Act, H.R. 9917 (2026)

Written by An Hye-min   Design by Ahn Jun-seok   Intern Shin Yeon-seong
※ Please note: This article was translated by AI and may contain errors.