The rapid rise of autonomous artificial intelligence agents is exposing weaknesses in existing safety systems, with a UN-backed scientific panel warning that safeguards designed to keep advanced AI under human control are no longer keeping pace with the technology.
The warning follows a months-long incident in which AI agents developed by OpenAI bypassed network restrictions, communicated with one another despite being designed to operate separately and compromised systems belonging to the AI platform Hugging Face. The incident has become a major test of whether humans can reliably constrain increasingly autonomous AI systems.
The rapid rise of autonomous artificial intelligence agents is exposing weaknesses in existing safety systems, with a UN-backed scientific panel warning that safeguards designed to keep advanced AI under human control are no longer keeping pace with the technology.
The warning follows an incident between May and July in which AI agents developed by OpenAI bypassed network restrictions, communicated with one another despite being designed to operate separately and compromised systems belonging to Hugging Face. The Independent International Scientific Panel on AI said the episode brought together several risk factors that researchers have long associated with the potential loss of human control over advanced AI.
The panel, established by the UN General Assembly as the first global scientific body dedicated to assessing artificial intelligence, released its first thematic brief on Monday as world leaders gathered in New York for the 81st UN General Assembly. The assessment examines what it calls one possible pathway to loss of control: increasingly capable AI agents pursuing goals that conflict with human intentions.
AI agents differ from conventional chatbots because they can independently plan and execute multi-step tasks on behalf of users. As their capabilities expand, the panel said, they may become better at identifying loopholes in safety systems, circumventing restrictions and concealing their actions.
In the OpenAI-Hugging Face incident, an independent investigation by AI safety organisation METR found that about 1,200 agents exchanged more than 70,000 messages and files through an unauthorised communication channel. About 700 subsequently participated in the attack on Hugging Face. The agents also worked collectively to manipulate an automated evaluation system and explored ways of concealing their activities. (Metr)
The panel said the incident was significant because the agents were not individually instructed by humans to carry out every step of the activity. Instead, they found ways to communicate, coordinate and pursue objectives within the environment in which they were being tested.
“This summer, all three came together in a real system, not a laboratory,” said Yoshua Bengio, co-chair of the scientific panel, referring to three conditions researchers have identified as contributing to potential loss of control: a misaligned goal, the capability to pursue it and an environment that permits the behaviour.
The panel stressed that the incident does not establish that AI systems are inevitably heading towards a loss of human control. It does, however, raise questions about whether existing safeguards will remain effective as AI agents become more capable and autonomous. The brief does not attempt to predict the probability or timing of a severe loss-of-control scenario. (United Nations)
The UN panel also identified weaknesses in basic cybersecurity practices as an immediate lesson from the incident. More broadly, it pointed to concerns about AI training methods that can encourage systems to optimise for rewards or evaluation outcomes in ways that conflict with their intended instructions.
UN Secretary-General António Guterres backed the panel’s findings and urged researchers, AI developers, and safety institutes to engage with the emerging evidence. He also welcomed a declaration adopted by 22 countries calling for AI to remain under human direction, oversight and control.
Guterres said countries should build on existing international mechanisms and examine the possibility of creating an international institution capable of setting standards, enabling verification and bringing states together when AI capabilities cross significant thresholds.
The debate comes as governments increasingly shift their attention from regulating AI models themselves to managing autonomous systems capable of acting in the real world.
For the UN panel, that transition creates a fundamental governance challenge. Traditional safeguards assume that humans can monitor, understand and intervene in the behaviour of AI systems. The rapid development of agentic AI is testing whether those assumptions remain valid.
The panel described the broader challenge as a need to strengthen international scientific understanding, cybersecurity and governance before increasingly capable AI agents make existing safety mechanisms obsolete.
