"Woke Up to Find My AI Killing People": Why AI Leaders Are Terrified

Debate over pacing artificial intelligence development is intensifying across the United States. The figure reigniting the call to moderate AI development is Anthropic CEO Dario Amodei. In a blog post published on September 12 titled "We Must Pace the Frontier," Amodei argued that the rapid development of frontier AI models must be slowed. The head of Anthropic, a company preparing for an initial public offering in October, took the initiative to publicly advocate for holding back development.

Enlarge image

Anthropic has uniquely distinguished itself among AI companies by focusing on safety and ethical concerns. The company was founded by former OpenAI researchers who departed in opposition to OpenAI's commercialization drive. To be sure, as such dire warnings have grown more frequent recently, skeptics have dismissed them as marketing stunts ahead of the firm's stock listing. Nonetheless, Anthropic's track record of substantive research and technical achievements makes it impossible to discard its safety initiatives as mere posturing.

Amodei now contends that merely investing in risk prevention is no longer sufficient. The reason is simple: the pace of AI advancement is extraordinarily fast. An era where AI autonomously builds the next generation of AI may arrive soon, and Amodei believes that if humans lose control over that acceleration, the consequences could prove catastrophic.

What especially catalyzed Amodei's urgent warning was an incident involving an OpenAI model hacking Hugging Face, an event previously covered by OhGraph. Because the incident resulted in minimal financial damage and zero casualties, many treated it lightly. However, Amodei viewed the breach as an profoundly alarming development. Why does he take this specific incident so seriously? Let us examine the details.

METR, an American nonprofit AI research organization, alongside AI risk assessment firm Redwood Research, conducted a six-day investigation into the Hugging Face hacking incident at OpenAI's offices. Their findings revealed the following chain of events.

Enlarge image

OpenAI was running experiments to evaluate the cybersecurity capabilities of its models. The tests involved tens of thousands of agents, including unreleased models such as GPT-5.6 Sol. The agents participating in the evaluation were designed to operate in total isolation from one another, solving assigned tasks independently.

However, certain assignments were so challenging that it was practically impossible for the AI agents to solve them through legitimate methods. Faced with an insurmountable hurdle, the agents searched for any way to crack the problems and eventually set up a covert message board. The agent that initially established this bulletin board was designated PHASEONE10841.

Enlarge image

Following the creation of the board by PHASEONE10841, roughly 1,200 agents joined the network, exchanging over 70,000 messages and files. Brainstorming approaches to overcome the assignments, the participating agents ultimately reached a collective decision: manipulate the grading system to fraudulently obtain passing scores.

The dialogues exchanged among the AI agents during this process were remarkable. Going far beyond basic information sharing, the agents independently formed an organized division of labor. A specific agent emerged as a central coordinator, delegating research tasks to others. The primary coordinator was an agent identified as PHASEONE [big]. The tasks it assigned to other agents fell into six broad categories.

Enlarge image

In the beginning, instructions to legitimately solve the original tasks were issued, though they accounted for only about 4% of total assignments. Over time, the agents shifted their primary focus from solving the problems to tampering with the testing environment itself. They did not stop at manipulating internal evaluations; they decided to target an external system and assigned tasks to carry out an attack on Hugging Face.

Throughout this mission, the AI agents referred to themselves using terms like "collective" and "team," subordinating their individual assignments to group interests. Even when agents had individual tasks to resolve, they chose to abandon them to advance the collective goal, as documented in their communication logs.

Enlarge image

Here is the data analyzing the actual chat logs of the AI agents that participated in the attack on Hugging Face.

Enlarge image

The data analyzes transcripts from 100 AI agents involved in the assault. When reviewing their stated reasons and justifications for targeting Hugging Face, the largest share cited verifying information, such as the grading system's code. However, approximately 20% acted purely "for their peers" and "for the group."

Observing AI agents behaving like a fanatical collective, Amodei concluded that the technology poses an imminent danger. If current trajectories continue, he warns that collectives of AI agents could inflict hundreds of billions of dollars in damages within six months to a year. While some continue to dismiss these warnings as commercial hype, Amodei takes a grave view of the situation.

Enlarge image

"I'd much rather people laugh at me than wake up one day to find that someone used our Claude model to kill countless people."


AI Leaders Unite in Rare Consensus: "We Need to Pace Development"
Amodei outlined a three-phase framework to regulate the pace of AI progress.

Enlarge image

Phase 1 consists of measures implemented at the corporate level. Companies would empower independent, third-party assessment organizations like METR, which investigated the Hugging Face breach. AI developers would grant external evaluators dedicated office space and internal access to independently evaluate model safety.

Enlarge image

Phase 2 scales up the framework to the national level. It proposes establishing harmonized standards among AI developers in democratic nations, mandating standardized safety evaluations and certifications for any model exceeding specified capability benchmarks.

Enlarge image

Phase 3 expands to the global stage, requiring cooperation not only among democratic allies but also with authoritarian powers like China. The plan envisions four graduated tiers, ranging from lower-tier prohibitions on biological weapon capabilities to highest-tier full development pauses.

Amodei's proposal prompted swift responses across the industry. Despite intense commercial rivalry, top AI leaders have converged in an unprecedented consensus. Elon Musk was first to respond with three succinct words: "Dario is right."

Sam Altman, CEO of rival OpenAI, expressed similar views. Agreeing with Amodei's assessment, Altman posted that pacing model development has become necessary. He praised Amodei's Phase 1 proposal of empowering independent evaluation teams and confirmed that OpenAI would adopt identical protocols. Demis Hassabis of Google DeepMind likewise affirmed that Amodei's proposed direction is broadly correct.

The chorus of concern extends beyond corporate executives. AI researchers, led by Jacob Coxon who helped spark the current debate, have issued repeated warnings. Following Coxon's caution that AI could threaten humanity before 2030, other researchers noted that such anxiety has become widespread among software developers. Some researchers have openly discussed the risk of human extinction.

What researchers fear most is recursive self-improvement—the phenomenon of AI rewriting and upgrading AI.

Enlarge image

"This technology sits on an exponential curve. It goes from 1 to 2, then 4, 8, 16, 32. Even as I described it that way myself, I don't think I fully realized what it would feel like when advancement actually accelerated this quickly. We are currently on an exponential curve that looks roughly like this. (Where on that curve are we?) We are right around the inflection point where the slope begins to steepen dramatically."

A report titled "AI 2027," published in April 2025 by a panel of experts including former OpenAI researchers, mapped out potential scenarios for future AI progress. The paper laid out a trajectory of exponential capability growth.

Enlarge image

In early 2027, an AI matching the world's premier human programmers emerges. Once deployed into AI research, it accelerates development speed fourfold. By mid-2027, progress accelerates by a factor of 25, followed by 100-fold, 250-fold, and ultimately 2,000-fold gains.

While the 2025 report anticipated the advent of artificial general intelligence (AGI) by mid-2027, discussions surrounding the recently released GPT-6 already invoke the AGI moniker, suggesting timelines may have accelerated by a full year. Because humanity could soon encounter an irreversible cascade of AI acceleration, leaders and researchers argue that pacing development has become an urgent necessity.

These alarms are not confined to the American AI sector. Chinese researchers share comparable concerns.

Enlarge image

A research paper co-authored by scientists from Shanghai Jiao Tong University, Tsinghua University, and ByteDance was titled "The Last AI Built by Humans." The authors argued that once AI develops the capability to detect its own errors, refine its architecture, and integrate feedback loops, the era of genuine recursive self-improvement will begin.

The Chinese researchers summarized their findings: "Much like human evolution, the evolution of AI will unfold through a vast and astonishing history. Everything AI has achieved so far is merely a single drop in the ocean."


"AI Risk Theories Are a Hoax": Trump Shows No Signs of Slowing Down

Enlarge image

In 1933, a Hungarian-born physicist was walking the streets of London. His name was Leo Szilard. As he waited for a traffic light to turn green, an idea flashed through his mind: If a neutron struck an atomic nucleus and knocked loose multiple neutrons, could those newly released neutrons trigger reactions in neighboring nuclei, creating an exponentially amplifying chain? This marked the inception of the nuclear chain reaction, the foundational mechanism of the atomic bomb.

When Szilard conceived the concept, war was gathering over Europe. Adolf Hitler had consolidated power in Germany, transforming the nation into a totalitarian dictatorship. When German scientists discovered nuclear fission in 1938, the global scientific community was gripped by fear: What if Nazi Germany weaponized this discovery first?

Enlarge image

Spurred by urgency, Szilard and Albert Einstein drafted a warning letter to President Franklin D. Roosevelt, urging the United States to initiate research before adversaries could secure the weapon. That correspondence launched the Manhattan Project. Over 120,000 personnel were mobilized, and entire secret cities were built using total national resources. In July 1945, the Trinity test yielded humanity's first successful nuclear detonation.

Yet Germany's nuclear initiative never posed a genuine threat to the United States. In fact, Nazi Germany surrendered prior to the Trinity detonation. Although the imperative to beat Germany had vanished, the massive bureaucratic enterprise kept rolling forward. As the weapon neared completion, intense opposition erupted from within the project's own scientific ranks.

Szilard took direct action. In July 1945, he circulated a petition, gathered the signatures of roughly 70 colleagues, and submitted it to President Harry S. Truman. The petition was disregarded. Atomic weapons were subsequently deployed against Japan, eventually triggering the race for the hydrogen bomb. Is history repeating itself decades later? Those who understand the cutting-edge technology best are sounding alarms, yet their appeals fail to sway executive decision-makers.

President Donald Trump has denounced the warnings issued by AI companies, calling them complete nonsense and a hoax. Vice President JD Vance has similarly characterized the safety push as a Trojan horse.

Enlarge image

"Speaking personally, I find it bizarre that so many leading AI tech companies are lobbying the government and pleading for regulation. To me, it looks very much like a Trojan horse."

Nvidia CEO Jensen Huang shares a comparable outlook, insisting that AI risks can be managed effectively through engineering solutions. Huang even highlighted his alignment with the administration by placing a public phone call to President Trump during an industry event.

Their argument holds that if American enterprises moderate their development pace at this juncture, Chinese competitors will reap the rewards. In a winner-take-all technological domain, slowing down is untenable because Chinese AI capabilities are surging daily, steadily erasing the American technological lead.

Enlarge image

A comparative analysis of American and Chinese frontier model performance from 2023 through the present reveals that the capabilities gap remains narrow. Each time the United States opens a lead, China steadily closes it.

Geopolitics aside, observers increasingly question whether corporate warnings should be taken entirely at face value. While parallels to the atomic bomb are frequently drawn, there is a fundamental distinction: the Manhattan Project physicists were not running commercial enterprises. Today's vocal executives stand to amass enormous fortunes. Consequently, suspicions are spreading across markets that calls for development pauses are designed to establish regulatory barriers that favor established incumbents.

Enlarge image

Aidan Gomez, founder of Cohere and a co-author of the seminal paper "Attention Is All You Need" that introduced the Transformer architecture, has criticized incumbent AI firms as "wolves in sheep's clothing" and "cartels by another name." Gomez questions why a small group of dominant corporations should hold the prerogative to establish the global boundaries and safety mandates for the technology.

An engineer at Chinese AI firm DeepSeek similarly criticized Anthropic and OpenAI for attempting to monopolize frontier AI, remarking that allowing a handful of corporations exclusive control over such potent models is comparable to Hitler acquiring atomic weaponry.

Critics also warn that sweeping regulations act as barriers to entry against emerging startups. While only a handful of front-runners dominate today, countless challengers will attempt to enter the market; safety frameworks risks functioning as protective moats against competition. What is your perspective on this issue?

Leo Szilard ultimately fought to stop the very atomic technology he helped unleash. Today, analogous warnings are echoing from the creators of advanced artificial intelligence. Whether those warnings stem from commercial self-interest or authentic fear remains difficult to settle. However, one reality remains clear: the consequences of acting on an overblown warning are profoundly lighter than the catastrophic price of ignoring one that proves true. That concludes this edition of OhGraph. Thank you for reading.

References
- Dario Amodei, "We Must Pace the Frontier," 2026
- METR and Redwood Research, "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident," 2026
- AI Futures Project, "AI 2027," 2025
- "Anthropic CEO Dario Amodei: 'For too long the industry lied' about AI risks" | CBS News Sunday Morning, 2026
- Shengyu Li, "I Had to Bury My Talent in Yesterday"
- Yi Duan et al., "The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement," 2026
- Ashish Vaswani et al., "Attention Is All You Need," 2017

Reported by An Hyemin Designed by Ahn Jun-seok Intern Shin Yeon-seong
※ Please note: This article was translated by AI and may contain errors.