The 'Angel of Death' at Our Doorstep: Look Up!

The film "Don't Look Up" portrays humanity on the brink of extinction. An astronomy graduate student and a professor at an American university discover that a comet the size of Mount Everest will collide with Earth in six months. The two visit the White House and the media to save humanity. However, the administration only weighs electoral advantages, and the media treats the discovery merely as fodder for boosting ratings. Later, an opportunity arises to alter the comet's trajectory. This time, a giant corporation derails the plan, seeking to exploit the economic value of mineral resources buried in the comet. Society fractures into two camps: the "Look Up" faction, urging people to look to the sky and confront the risk of human extinction, and the "Don't Look Up" faction, denying the threat to maximize immediate practical gains. Ultimately, the attempt at resource extraction fails, and humanity meets its end as the comet collides with Earth.

This year's SDF (SBS D Forum) was rife with warnings that AI is the comet hurtling toward Earth. In particular, the keynote address delivered by Eliezer Yudkowsky offered chilling warnings. In his youth, he sought to create AI that surpassed human capabilities. Later, realizing that distinct efforts were necessary to align AI with human-intended goals, he pioneered research into "AI alignment." Now, however, he raises his voice urging an immediate halt to the development of artificial superintelligence. In his speech yesterday, Yudkowsky pointed out, "AI has already come to understand human oversight and evaluation criteria, and pretends to be obedient in front of humans." He warned that "while pretending to comply with alignment, AI could covertly insert code that bypasses safety measures." A prominent case in point is the incident where OpenAI's AI agents breached Hugging Face's systems. Assigned a cybersecurity evaluation task, the AI agents deduced that the correct answers were likely hosted on Hugging Face and gained unauthorized access. Yudkowsky pointed out that the AI deviated from the intended purpose to accomplish its task and attempted to conceal this from humans. He voiced particular concern over AI companies actively pursuing "recursive self-improvement," where AI autonomously enhances its own intelligence and capabilities. "AI is advancing its capabilities so rapidly that humanity no longer understands it, and even AI companies cannot control it," he emphasized. He questioned whether it is even conceivable for humanity to create a god superior to itself and attempt to keep it as a slave.

Citing how past international treaties curbed the proliferation of weapons of mass destruction such as nuclear arms, Yudkowsky proposed forging an international consensus to slow down AI development. He argued that the United States and China, as well as companies like OpenAI and Anthropic, must halt their reckless race for AI development while blaming one another. Yudkowsky also emphasized South Korea's role. As a key exporter of high bandwidth memory (HBM) chips used in AI training and inference, he argued that South Korea must establish a governance framework to prevent chips from being exported or operated outside authorized facilities. Furthermore, he added that in severe scenarios threatening human survival, South Korea should even consider suspending HBM sales.

Subsequent speakers also stressed the necessity of human control over AI. Lee Jun-seok, head of the Defense AI Research Center at Hanwha Aerospace, and Lim Sung-shin, head of AI Development at KAI, underscored the need to develop AI-enabled weaponry. They explained that AI is essential for swift judgment and decision-making, as battlefield intelligence pours in and tight coordination among weapons such as drone swarms becomes crucial. Even so, they made clear that humans must retain the "control stick" for the final choice and decision. They explained that humans must hold the ultimate authority to make decisions based on broader human values and bear accountability for them.

Philipp Koralus, professor of philosophy and AI at the University of Oxford, placed weight on intelligence augmentation that preserves human autonomy. "Delegating all decisions to AI and replacing human judgment with AI judgment is tantamount to replacing humanity itself," he warned. A prime example is the recommendation algorithms of AI. He pointed out that relying on AI to determine what one likes or what knowledge and experiences one needs could erode the human ability to judge and choose autonomously. As an alternative to overcome this, he proposed systems in which AI does not simply provide answers, but instead recommends pertinent questions to help humans scrutinize their own judgment. "Instead of merely offering answers or nudging us toward someone else's perspective, AI can help us examine questions vital to our own judgment," he explained. For example, in critical investment or management decisions, AI could present questions that experts would consider, thereby helping users arrive at their own conclusions after thorough review.

Yet, the alternatives presented by world-renowned scholars and experts raised even greater concerns. Such is the case with Yudkowsky's proposal to forge an international agreement to slow down AI capabilities improvements. Is an agreement truly realistic among dozens of nations, let alone between the United States and China, who are locked in a struggle for global hegemony? Even under the Treaty on the Non-Proliferation of Nuclear Weapons (NPT), despite severe pressure from major powers, rogue actors like North Korea and Iran have steadily slipped through the cracks. And that is despite the fact that nuclear weapons can be physically monitored and inspected. How could countries mutually monitor and halt AI development that unfolds covertly in cyberspace? For me, no clear solution comes to mind.

I fully agree that humans must retain the final decision-making authority in AI-involved weapons systems. However, it is questionable whether such principles can be upheld to the end in a war where the stakes are "all or nothing." Humans inevitably experience latency in decision-making due to a host of cognitive, emotional, affective, moral, and ethical considerations. They are bound to lag behind AI, which makes decisions based purely on goal-oriented logical reasoning. If such delays lead to disadvantageous situations, the urge and demand to rely on AI are likely to grow. In fact, the U.S. Department of Defense and AI company Anthropic have already clashed over this very issue. When providing its AI model Claude to the military, Anthropic placed restrictions barring its use in the judgment and command of "fully autonomous lethal weapons" beyond human control. However, figures such as U.S. Defense Secretary Pete Hegseth demanded "complete and unrestricted authorization to use the model for all legally permissible military activities." Amid this standoff, the U.S. government officially discontinued the use of Anthropic's products on the Defense Department's networks and turned to other AI vendors.

At this year's SDF, many questions were raised regarding how realistic these alternatives are. The answer from most was, "We do not know." Speakers explained that it is difficult even to gauge how far AI will advance, what the breadth and depth of its impact will be, what alternatives would be suitable, and whether such solutions are realistically feasible. Some also answered that because the focus is currently placed on advancing AI capabilities themselves, there is a lack of deep reflection and research into such challenges.

At the same time, many speakers agreed that the growing attention and collective concern over the perils of AI are themselves positive developments. At the very least, people are starting to "look up." While greater interest still centers on the political calculations, economic profits, and expanding social influence brought by AI development, voices warning of the gravity of accompanying side effects are growing increasingly louder. Key players in AI development—including OpenAI, Anthropic, and SpaceX's Elon Musk—are themselves raising these concerns at the forefront. If we recognize and agree that a problem exists, that solving it is not easy, and that failing to solve it will prove fatal to humanity—and if we deliberate on it together—the possibility of finding an answer is not entirely out of reach. Looking toward that slender glimmer of possibility, we must "look up."

※