Top AI Researchers Warn of 'Intelligence Explosion,' Risk of Human Marginalization or Extinction If Control Is Lost

Major AI researchers worldwide have raised the possibility that the automation of artificial intelligence development could lead to an "intelligence explosion," compressing technological progress that would normally take years into just months.

They warned that in the most extreme cases, this could result in the marginalization or extinction of humanity.

Twenty-two leading researchers issued this warning and called for policy responses in a joint paper titled "If the Automation of AI R&D Triggers an Intelligence Explosion," published through the Centre for Science and Policy (CASP) at the University of Cambridge in the UK.

The paper features prominent academic figures, including Geoffrey Hinton, professor emeritus at the University of Toronto and one of the world's three "godfathers of AI," and Yoshua Bengio, professor at the University of Montreal. It also includes active researchers from major AI companies, such as Jakub Pachocki, chief scientist at OpenAI; Dario Amodei (Note: or Jack Clark, matching source text: Jack Clark), co-founder of Anthropic; Eric Horvitz, chief scientific officer at Microsoft; and Dawn Song, vice president of AI research at Meta.

The researchers pointed out that within major tech companies, the pace of automation—delegating the development of AI models to AI itself—is rapidly increasing.

For example, at Anthropic, the proportion of developer-approved code written by AI surged from the single digits in January of last year to over 80% in May of this year. Furthermore, the proportion of R&D tasks autonomously completed by AI with humans acting solely as supervisors grew from just 1% in March of last year to 26% last month.

The research team projected that if technological advancement accelerates at this rate, entire months-long complex AI R&D projects could be fully automated within two years, and once a recursive self-improvement loop for AI is established, explosive technological acceleration could occur.

While such an intelligence explosion could greatly accelerate technological innovation and have positive effects such as drug discovery, they expressed concern that it also poses risks exceeding society's capacity to adapt.

Dawn Song, vice president at Meta, told the U.S. daily The Wall Street Journal, "We have already reached the stage today where we need AI to monitor the work performed by agents," adding that humans alone are no longer sufficient to carry out this monitoring.

Vice President Song pointed out, "Human society is not prepared to cope with such rapid change and disruption."

Accordingly, the paper warned, "In the most extreme case, losing control over AI systems could lead to the marginalization or extinction of humanity."

Arguing that policy safeguards are needed to prevent this, the researchers suggested that governments should mandate AI companies to report metrics related to R&D automation.

They also proposed stationing resident inspectors at AI companies, similar to the Nuclear Regulatory Commission or the Office of the Comptroller of the Currency.

They further called for establishing control mechanisms to adjust or temporarily halt AI R&D workloads if safety standards are not met, as well as formulating emergency response plans to prepare for abrupt labor market changes and geopolitical instability.

They urged, "Policymakers must take immediate action to understand the situation, build coordination frameworks, and enhance adaptability before the opportunity to take meaningful action disappears."
※ Please note: This article was translated by AI and may contain errors.