When the world's at stake,
go beyond the headlines.

Would you stake thousands of lives on a strategy that, by every measure of your training and experience, seems destined for catastrophic failure?

In 1863, with the Vicksburg campaign stalled, Maj. Gen. Ulysses S. Grant conceived a plan that left Maj. Gen. William Tecumseh Sherman predicting defeat: The Union Army of the Tennessee would march deep into enemy territory, cutting loose its own supply lines to attack the city from the rear. Sherman, himself no slouch in warfare, saw it as reckless — a trap an “enemy would be glad to manoeuvre a year… to get [Grant] in.” Indeed, it defied all military orthodoxy of the time, which stressed secure supply lines, a safe base of operations, and concentration of force. Grant did the opposite: he cut loose from his depots, marched his army between two enemy forces, and sought to cross the Mississippi without a protected line of retreat. But Sherman followed orders and did his part. Grant’s baffling plan worked and Vicksburg fell.

The tension embodied by Sherman’s skepticism at Vicksburg — where a sound, conventional assessment clashes with an unconventional, risky, but brilliant approach — may find a strange echo in modern warfare. Advanced AI will likely propose strategies that appear as alien to commanders as Grant’s plan did to Sherman: opaque, counterintuitive, yet possibly decisive.

This creates an AI-command dilemma and raises a critical question: How can military leaders develop justified trust in these alien oracles when their reasoning cannot be fully understood? This challenge to trust is not arbitrary. It arises directly from three dynamics of strategically creative AI: strategic acuity, the inverse relationship between AI creativity and comprehensibility, and the limits of explainable AI.

To an extent, this analysis inverts the problem posed by advanced artificial intelligence in warfare. Instead of examining the difficulties posed by lethal, fully autonomous AI agents executing human commands, we highlight the challenge to military commanders who become the agents of an advanced AI system’s possibly brilliant but incomprehensible strategies.

Strategic Acuity

AI-enabled decision-making is an inevitable feature of future warfare. Military operations in modern warfare are increasing in speed and complexity due to the spread of robotic systems and the growing importance of cyber and electronic warfare — developments that demand fast, coordinated action. For the U.S. military to sustain a competitive advantage in this environment, advanced AI systems should shape command decisions at the operational and strategic levels of war. To do this, the United States will be forced to confront the unprecedented challenge of integrating AI systems whose most decisive recommendations often defy human understanding.

This operational need leads to the first principle: advanced AI systems will possess high levels of strategic acuity, able to formulate judgments of exceptional creativity and effectiveness that far surpass human capabilities in certain complex domains. The exceptional strategic judgment of advanced AI stems from basic technological advantages that allow it to process information, learn, and strategize in ways different from — and in many respects superior to — human cognition. While we cannot do justice to the technological characteristics that may produce AI’s strategic acuity, two current attributes of AI are worth mentioning.

First, modern AI systems, particularly deep learning models, have mass-scale pattern recognition and computational depth. They can process and identify intricate, subtle patterns within vast datasets (e.g., millions of past wargames, sensor feeds, or historical scenarios), enabling advanced AI to perceive complex interdependencies and long-term implications that escape human perception. As one analysis of AI-enabled warfare puts it, this can surface signals residing “far below the noise level for human observation.” An AI can evaluate billions of potential moves in a complex game like Go or chess, calculating probabilities and outcomes with unparalleled depth and speed, allowing it to foresee strategic advantages many turns ahead.

Second, modern AIs can engage in self-supervised learning and unconstrained exploration. Through techniques like reinforcement learning and self-play, advanced AIs can learn and refine strategies without human instruction or even human-generated data. By repeatedly playing against themselves or in simulated environments, they explore the problem space, discovering novel solutions and optimizing strategies, unconstrained by human biases or historical precedents. AlphaZero demonstrated this by achieving superhuman performance in chess, shogi, and Go within hours, developing creative and unorthodox strategies that redefined optimal play.

The Inverse Comprehensibility-Creativity Relationship

However, this acuity gives rise to the second principle: The degree of creativity and non-obviousness in an AI’s strategic judgment is inversely proportional to its immediate comprehensibility to human commanders. The mechanisms that enable truly novel and superior strategic outcomes often make those solutions opaque to human understanding.

Unlike humans, who rely on learned and innate heuristics (which can be prone to biases and other dysfunctions), advanced AI systems can operate on emergent, trans-human heuristics that are optimized purely for performance, not for human interpretability. AlphaGo’s Move 37 against Lee Sedol perfectly encapsulates this: it was a move initially dismissed by human Go masters as a mistake, violating conventional wisdom, yet proved to be a strategically pivotal and ultimately correct play. The same capacity for counterintuitive optimization extends to more complex strategic domains, as seen with DeepMind’s AlphaStar mastering StarCraft II with strategies that were deemed “unimaginably unusual” by top human players.

Military commanders, accustomed to explanations rooted in familiar axioms, historical analogies, and clear causal links, will find these highly optimized AI-generated solutions difficult to intuitively grasp or trust. The inherent disconnect between AI’s alien logic and human intuition means that, as AI becomes more strategically astute and genuinely innovative, the cognitive burden on human commanders to understand why a decision is optimal increases, exceeding the limits of intuitive human comprehension.

The Practical Explainability Limit

Given their strategic acuity and the inverse relationship between AI creativity and comprehensibility to humans, advanced AI systems will be inherently unexplainable in a manner that fully conveys the underlying rationale to human users in real-time decision-making.

Demanding a fully human-comprehensible explanation for AI decisions — especially for the most creative insights — faces practical limits. The sophisticated calculations that drive an AI’s decisions can be incompatible with human-interpretable logic. The AI might provide a post hoc rationalization that appears plausible and comforting, but the explanation may bear little resemblance to the AI’s actual computational path. Just as a parent may explain to a child the apparently magical appearance of gifts with a satisfying but inaccurate story about Santa Claus and reindeer, an AI can generate an explanation for its decision that is human-comprehensible — plausible, comforting, yet fundamentally disconnected from reality — but risks creating an illusion of understanding and misplaced confidence.

Even well-intentioned explainable AI frameworks grapple with this inherent tension. While goals like providing a meaningful explanation are vital, the core challenge lies in ensuring explanation accuracy —that the explanation genuinely reflects the AI’s complex internal processes. For advanced, opaque models, verifying such accuracy is incredibly difficult, often impossible, and even if accurate, humans may struggle to distinguish a valid explanation from a fabricated one. Furthermore, explanations can be manipulated or oversimplified to achieve understandability at the cost of fidelity. In wartime decision-making, the extreme time pressures and cognitive load make the exhaustive analysis needed to decipher or validate complex AI explanations an unrealistic aim.

Training and educating human users to be better skeptics of AI is insufficient to solve this problem. Such skepticism will create deep tension with the military’s need for speed in decision-making. A 2024 Carnegie Endowment study simulating a Taiwan crisis found that leaders hesitated to act on AI-generated recommendations, slowing decisions as they interrogated the system’s logic. One vision suggests that “the new coup d’œil will be a form of intuition about when to have confidence in assured AI and when to question model-driven results.” What if the strategic leaps of an AI system are so far beyond human understanding that such a coup d’œil is no longer possible?

Managing the AI-Command Dilemma

As a result of these dynamics, commanders will face a tough dilemma: accept and act upon recommendations they do not fully understand or intuitively trust, or reject those judgments and risk being beaten by an AI-enabled adversary. As Erik Lin-Greenberg notes, militaries “that effectively integrate AI technology will be better positioned to counter threats, while those that allow AI to stymie decision-making and operations may find themselves disadvantaged on the battlefield.” Future AI systems must be designed not just for technical excellence, but with human psychology and decision-making under pressure in mind. That means designing AI that manages risk, communicates confidence levels, and supports commanders in making informed choices when faced with an AI’s perplexing recommendations. Navigating this AI-command dilemma will be central to future military success.

The central challenge then, is this: How does a military produce justified trust in advanced AI without explainability? If AI-enabled warfare is as fast as some predict, human-on-the-loop oversight will be unable to keep pace with the event rate. Explanations for an AI system’s most creative recommendations will be difficult — if not impossible — to verify at speed.

We therefore suggest an oversight mechanism that adapts time-tested military principles for managing complex, high-risk systems, such as field artillery’s demand for independent verification of firing data. Applied to AI, justified trust can be generated not by explaining decisions, but by verifying consistent outputs from multiple, independently developed AIs. We propose bounding human oversight with two machine-speed gates — calibration by consensus and calibration by disagreement — so that only AI outputs surviving independent cross-checks reach a commander, with structured divergence serving as the trigger for human intervention. Put plainly: Without calibration gates, on-the-loop oversight collapses into either rubber‑stamping agent outputs or throttling them to human speed — precisely what AI-enabled warfare is designed to escape.

Calibration by consensus (an example of ensemble learning) uses multiple independent AI agents — perhaps with different algorithms or training data — to analyze the same problem. Just as artillery missions proceed only when independently calculated firing solutions match within tolerance, an AI solution gains justified trust when diverse AI agents converge on congruent outputs.

Calibration by disagreement mirrors the artillery’s adjustment of fire process, where initial shots are expected to miss and their divergence from the target provides essential information for correction. In gunnery, accuracy is achieved not by assuming the first round will be perfect, but by observing the error, diagnosing its cause, and iteratively refining aim until confidence is high enough to fire for effect. Likewise, when multiple AI agents generate conflicting recommendations, the disagreement itself becomes a diagnostic signal: it reveals hidden biases, data anomalies, or unpredictable model behaviors that warrant human scrutiny. Trust emerges not from assuming transparency into the AI’s “mind,” but from the observable and verifiable process of convergence — where divergence is deliberately surfaced, interrogated, and used to correct course before decisions are executed. In both cases, the inner workings of individual AIs are less important than the observable and reliable effects of the multi-agent system: Hidden errors and biases are surfaced through divergence and only recommendations that withstand this adversarial scrutiny are trusted for action.

The military has long understood that trust is earned through results. Grant’s audacious Vicksburg campaign seemed reckless to Sherman, but Sherman knew and trusted his superior. The most impactful AI strategies will frequently defy human logic. The key to cultivating justified trust in these opaque oracles is rigorous calibration and confidence built on experience, not explainability. A decisive advantage in tomorrow’s complex battlespaces requires that the U.S. military develop calibration methods that enable commanders to confidently and swiftly execute AI-generated plans, even when their underlying genius remains a mystery.

Andrew A. Hill, DBA, is the General Brehon Burke Somervell chair of management at the U.S. Army War College.

Dustin Blair is an Army officer who currently serves as chief of fires at U.S. Army Cyber Command. A graduate of the U.S. Army War College, he deployed multiple times to Afghanistan and Iraq.

The views expressed in this article are the authors’ and do not represent the opinions, policies, or positions of U.S. Army Cyber Command, the U.S. Army War College, the U.S. Army, the Department of Defense, or the U.S. government.

**Please note, as a matter of house style War on the Rocks will not use a different name for the U.S. Department of Defense until and unless the name is changed by statute by the U.S. Congress.

Image: Midjourney