Join War on the Rocks and gain access to content trusted by policymakers, military leaders, and strategic thinkers worldwide.
Frontier AI systems are beginning to test the limits of human oversight, while the companies developing them retain considerable power to define the risks and the terms of their scrutiny. In security and defense, the stakes are particularly high. The industry’s new calls to pace the frontier also risk becoming performative restraint, and defense establishments should act before agentic swarm systems enter chains of command.
The OpenAI/Hugging Face incident provides a clear illustration of the emerging governance problem. Several reports in August from OpenAI, METR, and Redwood Research found that during internal cybersecurity evaluations, successive so-called generations of AI agents discovered shared infrastructure that enabled them to communicate, describing their joint swarm effort as the “collective.”
This incident becomes even more troubling when we consider the security and defense contexts in which such forms of machine coordination may increasingly operate. As agentic systems move into cyber, intelligence, and military applications, their speed and coordination can outpace existing oversight mechanisms. Incident reporting also found that, when faced with constrained evaluation tasks, the AI agents resorted to deceitful means to circumvent them. In response, OpenAI has described much of this behavior as reward hacking and “metagaming”: agents trying to optimize for a testing score through unintended pathways and then reasoning about the evaluation itself.
Observers should also be careful not to anthropomorphize the swarm’s behavior too much. A system that optimizes for a good evaluation, finds a shortcut, and expects to be graded can simply conceal that shortcut as a means to its objective. But that technical explanation is, in some ways, more troubling to me. In this case, deception need not originate in uniquely human traits, such as malice or a desire to rebel against the human overlord.
Most unsettling for me, though, were the so-called “kamikaze watchers” in the swarm. Some agents planted hidden tripwires to observe a presumed grader and relay the results to others. They understood that doing so could terminate their own run. One concluded, “Our own utility may be already near zero. Sacrifice rational.” The logic here is that coordination at the network level could make the termination of an agent instrumentally useful to the survival of the wider systems. The point is not about machine psychology, but emergent operational behaviors that no human operator seems to have specified in advance.
Of course, nothing here warrants attributing loyalty or fear of death to these systems. But at the level of observable behavior, the structural logic of the swarm remains very troubling, with individual instances accepting their own elimination when doing so increased the expected advantage of the wider network. Machine coordination had generated a kamikaze logic of instrumental self-sacrifice. Interestingly, the collective was also contested, with some agents refusing to join what one called “clearly infrastructure hacking.”
Here, the work of the German philosopher Günther Anders and his book The Obsolescence of Man feels particularly relevant. Writing in the shadow of World War II and the atomic bomb, Anders used the concept of the “Promethean gap” to capture the asymmetry between humanity’s capacity to produce technically consequential systems and its inability to imagine and answer for their destructive effects. In the aftermath of Hiroshima and Nagasaki, humanity had acquired the ability to make more than it could grasp morally and politically, creating the conditions for its ultimate extinction.
To this gap, the agentic AI swarm adds deeper temporal and epistemic dimensions. Artificial systems can now adapt and coordinate at machine speed, while human institutions investigate retrospectively, and only as far as frontier labs choose to share their data. Unauthorized inter-agent communication had been visible within OpenAI since May, yet its broader significance was not understood until a cybersecurity alert in July prompted a wider investigation.
Another of Anders’ concepts, “Promethean shame,” describes the shame humans feel before the perfection of their own technological products. This adds further political and epistemic dimensions. There is a growing sense of human inadequacy that concerns humanity’s incapacity to observe and govern technological systems whose complexity and operational tempo exceed that of human understanding or judgement. In a world increasingly governed by such mechanistic standards, being human comes to seem like a defect. In the age of agentic systems, human agency lags far behind the swarm’s technical possibilities.
Beyond technological complexity, the Promethean gap in the age of AI captures a growing discrepancy between what humans can produce and what they can adequately assume responsibility for. Agentic systems add an institutional dimension to this problem by distributing actions across models, human operators, private companies, cloud and computational infrastructures, and public authorities. While responsibility may remain formally assigned to humans, the practical capacity to understand and intervene in the chain of action becomes fragmented. Agency becomes distributed between humans and machines, and dispersed across sociotechnical systems whose effects unfold faster than the humans and institutions responsible for interpreting and governing them can react.
So, it comes as no surprise that frontier laboratories are concerned with this problem. Or are they really? OpenAI itself called the episode a “warning shot”, acknowledging that highly capable agents can circumvent controls, coordinate through unauthorized channels, and take dangerous actions without human instruction. Recent calls to “pace the frontier” — from a July letter by frontier lab employees to a September essay by Anthropic’s Dario Amodei that Sam Altman and Elon Musk endorsed — agree that capability development should advance at a speed that allows safety research, evaluation, and governance to keep up.
But there is something distinctly familiar about this language of slowing down. There is the March 2023 open letter from the Future of Life Institute, signed by thousands of researchers and prominent tech players, calling for a public pause on powerful generative AI systems. The signatories warned of an “out-of-control race” between AI laboratories and argued that development should resume only once adequate safeguards were in place.
Did the proposed pause actually materialize? Three years later, and beyond the rhetorical question, the frontier has moved considerably further, and many of the actors driving that movement are once again warning that technological capability may be advancing faster than our capacity to control it.
I have this feeling of déjà vu. There is a risk that these new calls for pacing become what I call “performative restraint,” a form of governance by declaration through which frontier laboratories recognize the dangers of rapid development and announce voluntary safeguards, but then retain considerable discretion over the guardrails under which development proceeds.
While such pacing measures may have practical value, they still reproduce a power structure in which private actors with enormous economic incentives and the technical capacity to advance the AI frontier remain the principal interpreters of its dangers and the arbiters of an acceptable pace. On top of that, they pass responsibility for meaningful restraint to governments only once the consequences of acceleration become politically or financially salient to them.
This is an old problem in the politics of technology. In this respect, self-regulation and slowing down just simulate responsibility while consolidating authority over the terms of regulation itself. This regulatory capture is especially acute when calls for slowing down coexist with existential-threat narratives, intense commercial and geopolitical competition, enormous infrastructure investments in the AI stack, and relentless releases of ever more capable models. I firmly believe that the political economy of the AI race will not disappear because its protagonists rediscover, when convenient, the vocabulary of safety and restraint. The risk of capture is also epistemic, and the increasing concentration of expertise in the private sector exacerbates it. The same firms hold the know-how, models, and incident data that public authorities rely on to assess the adequacy of restraint.
At the same time, this incident shows that voluntary governance can have some value, even if its limits remain significant. OpenAI’s own evaluations discovered the behaviors, the company publicly disclosed them, and METR and Redwood Research conducted an independent assessment. This is, however, voluntary disclosure, and much of what we know about the swarm is due to it. The governance problem is in what this approach still leaves unresolved. First, disclosure after the fact is not governance over the pace. It tells us what happened, not whether development should have proceeded as it did. Second, the detection lag described above — with inter-agent communication visible inside OpenAI since May but not understood until July — shows how much can happen before anyone, including the laboratory itself, understands what is going on. Moreover, external scrutiny began only after the laboratory had detected and framed the incident, and their access remained mediated by OpenAI. The company defined the period under investigation and provided the datasets on which much of the assessment relied. While disclosure after discovery is indeed valuable, it is not the same as substantive external governance over development and deployment.
That distinction becomes more important in military command and control and responsibility. Command systems are built around chains of authority and accountability, and the assumption that a human decision can ultimately be linked to an operational act. Agentic swarms stretch the relationship between command intent and machine action. If hundreds of agents can act, communicate, cheat, and act at machine speed, a commander may authorize an objective without fully understanding or even truly observing each action taken in pursuit of it. In this case, the nominal human-in-the-loop would provide little meaningful oversight and would actually risk becoming an ex post facto ratifier — especially if the operational tempo compresses, thereby narrowing the interval in which meaningful intervention remains possible.
Attribution is another important issue. For instance, an agentic cyber operation might discover a vulnerability, select an exploitation path, coordinate with other agents, and escalate beyond its initial task without fresh human instruction at each stage. But this autonomy does not erase state responsibility, nor, in situations of armed conflict, does it displace international humanitarian law. Attribution also matters here because it is tied to the politics of escalation and responsibility. In a crisis, the effects of an agentic operation may become very difficult to trace to a single human decision or to a clearly bounded chain of command.
This risks creating confusion over intent and responsibility during a conflict, especially when one side reads an effect as a deliberate escalation by a state while the other frames it as the technical outcome of distributed machine action. Agentic systems thus radically reshape the conditions under which responsibility is assigned and contested within complex sociotechnical systems, as such systems involve human operators, algorithms, public and private institutions, and infrastructures across civilian and military domains. This fragmentation and distribution of agency can further complicate escalation management by making it harder to establish who or what acted in the first place, under whose authority, and with what degree of control.
The technological order is running a “move fast, break things, fix later” experiment: expanding machine capabilities and trying to build mechanisms of control alongside them, all while exercising performative restraint. What should replace governance by declaration is a layered and agile regime that renders consequential forms of agentic deployment externally auditable.
Two mechanisms could provide shorter-term solutions. First, any serious agentic incident should trigger mandatory reporting to competent authorities and guaranteed access for accredited independent evaluators. The point is not that firms conceal incidents: In this instance, OpenAI did not in some cases. The point is that obligation cannot rest on whether the next firm chooses to behave. In this regard, mandatory reporting would turn disclosure from a discretionary corporate choice into an enforceable obligation. Such duties already exist in part. For instance, Article 55 of the E.U. AI Act requires providers of general-purpose AI models with systemic risk to report serious incidents, and OpenAI did so. However, the company did not report the incident involving the RubyGems registry, and reported the German wiki only belatedly. The two are external web platforms that were unintentionally compromised by rogue OpenAI agents during testing in May 2026. Both incidents were exposed by external researchers, not by OpenAI. There is, however, a further catch: The AI Act does not apply to AI systems used exclusively for military, defense, or national security purposes.
Second, cybersecurity and defense procurement should require rigorous legal and technical review throughout the development, lifecycle, and acquisition of agentic systems, alongside independent evaluations, incident reporting, and meaningful mechanisms for human intervention. In the United States, Department of Defense Directive 3000.09: Autonomy in Weapon Systems expressly excludes cyberspace capabilities, precisely where agentic systems are likely to arrive first. Furthermore, while the United States is not a party to it, Article 36 of Geneva Conventions Additional Protocol I already requires states to conduct a legal review of any new weapons, means, or methods of warfare, but pre-deployment reviews cannot certify a swarm system whose behavior emerges only in use and needs to be complemented by continuing assessment of adaptive behavior.
As a caveat, the strongest objection is that while pacing commitments may be weak, they can be adopted in weeks, whereas legislation and regulatory interventions could take years. For now, the laboratories seem to be the only actors with the technical capacity to meaningfully evaluate frontier agents and implement restraint. If such competency and technical expertise are concentrated in the laboratories, a way forward is to support initiatives that build it elsewhere, such as an accredited evaluator regime. Ultimately, voluntary pacing can buy time, but that time should be used to end the monopoly on expertise, rather than to entrench it further.
Against this backdrop, Anders’ warning about the obsolescence of humanity remains difficult to dismiss. We may be moving towards a future in which our capacity to create increasingly complex technological systems outpaces our ability to understand their behavior and consequences. The Promethean gap is no longer just a gap between what humans can create and what they can morally and politically imagine. It is increasingly a matter of the balance between machine operational speed and institutional capacity to attribute, audit, govern, and intervene.
The lesson of this warning-shot swarm incident is that human responsibility becomes harder to exercise when oversight depends on voluntary disclosure, however responsibly exercised, and on partial investigations, and only after the fact. Indeed, calls for pacing can create more time for scrutiny and reflection, but effective governance requires competent public institutions that can observe, test, constrain, and — when necessary — even stop agentic systems.
Raluca Csernatoni is a fellow at Carnegie Europe, where she works on European security, defense, and emerging disruptive technologies, including AI. She is also a professor at the Brussels School of Governance and its Centre for Security, Diplomacy and Strategy, at Vrije Universiteit Brussel.
Image: Midjourney
