Kessel Run Shows How to Bridge the Gap Between Development and Operations
Everyone seems to agree that the U.S. military needs to rapidly update its weapons by harnessing information technologies — software, computing, and data transmission — to make both systems and humans more effective. This consensus position is entrenched in the forthcoming National Defense Strategy, which directs the military to accelerate force development with a focus on fielding technology more quickly.
But despite the talk, change isn’t happening. Military structure continues as it did in the mid-20th century, optimized for industrial practices that are now sunsetting. It’s easy to complain about lethargy in the Department of Defense. New companies seeking to expand into military contracting highlight their competitors’ ossification, suggesting that the legacy defense industry’s focus on hardware is a key factor holding the United States back. Disgruntled former officials go on television to suggest that the United States is losing its edge, or losing to China, hurling criticism at a faceless bureaucracy and saying it must “do better.” While entreaties to go faster and adopt commercial technologies more rapidly are hard to argue with, general complaints aren’t actionable. Change is difficult and only becomes real through specific recommendations and actions.
The Department of Defense can successfully modernize by empowering software teams organized around a powerful new model that blends capability development with the software and network operations in order to drive rapid feedback, known as DevSecOps. This pivot can be made by following three principles of successful federated innovation organizations. First, start simple, focusing on clear operational challenges or problems; second, structure organizations to avoid artificial valleys of death; and third, ensure the team can execute on all aspects of DevSecOps, driving virtuous cycles of learning and rapid improvements.
These three principles all suggest the need for significant organizational changes. Military leaders have already learned to use DevSecOps as a buzzword, reciting that the blending of software development and software operation is critical to future software efforts. But they are satisfied with organizational structures that leave a gaping chasm between capability development and operational forces, thereby reinforcing the historical divide between “Dev” and ”Ops.” This structural bifurcation inhibits the exact change leaders are demanding. To fix it, the Defense Department should pivot away from historical practices that prioritize innovation tourism over structural changes. A future software-enabled force won’t come by wearing hoodies in an industrial-era organization — it comes from reorganizing to optimize for DevSecOps.
Hew to a Tightly Scoped Charter
The first principle is to maintain a sharp focus on a single military problem to be solved. It’s not a new concept. The Defense Advanced Research Projects Agency has a version of this as the first element of its famous Heilmeier catechism, and it is similar to the ideas behind design thinking and the “lean” method. Such discipline is easily jettisoned in practice, however. It is tempting to take a sweeping assessment of need — that artificial intelligence, for example, could be useful across hundreds of use cases and systems — and drive organizations, investment, and activity around it.
The Department of Defense’s choices to date have displayed a lack of focus and failure to understand what creates effective digital modernization. Efforts to share data are often lost behind 150-person coordination teams and confusing names such as joint all-domain command-and-control. Alternatively, new organizations focus on a technique and not an operational need, as was the case with the Joint Artificial Intelligence Center, now being sunset. Moving from overly-broad charters like “deliver information at the speed of relevance” or the “implementation of artificial intelligence” to tangible fielded technology depends on aligning the technology teams with the system’s delivery path and operational software support. Without that alignment, too much is effort spent on coordination or internal sales to cross the infamous valley of death. An alternative approach is more narrowly scoped charters and budgets aimed at clearly defined problems. I led one such team, the Air Force’s Kessel Run — an experiment in delivering modern software for aircraft operations that is now in its sixth year. Its lessons can be useful to other teams.
Often, an instinctive reflex for efficiency surfaces too early in the exploration process. The venture ecosystem has demonstrated that start-ups will pivot many times before finding traction and growth, exploring different product offerings around their central theory of success. Only then do they scale their proven model. This is incompatible with the Department of Defense’s current approach, which highly overvalues standardization and seeks one perfect tool for all jobs. It is neither healthy nor realistic to expect a Joint Artificial Intelligence Center to apply AI across the entire Defense Department or think that the small business innovation research program is always the right tool to onboard new industrial base players.
By contrast, over the last two years, Kessel Run intensely focused on a very narrow operational problem: providing the software for the 609th Air and Space Operations Center to build and execute the Air Tasking and Air Control Orders. We found it essential to have a narrow scope and a clear set of users.
This narrow charter stands in contrast to overly broad, committee-driven goals that often afflict modernization initiatives, such as achieving “all-domain superiority” or “connecting any sensor to any shooter in any domain at any time.” Defense and industry leaders often abuse commercial analogies — seeking a “military internet of things” to create “plug-and-play” capability. These visions are too high-level to address the mission-specific challenges and gloss over or ignore the unique aspects of defense problems like data classification. The alternative is to focus on specific military problems. In the case of Kessel Run, our initial focus was to deliver a single tool for planning tanker support — getting refueling aircraft up in the air at the right place and time when fighter aircraft would be running low on gas. Only after this was working did Kessel Run begin to expand its scope of work to other tools that help run an air operations center, gradually delivering an integrated suite of capabilities that would replace the legacy system.
Starting small might seem counterintuitive, especially when it is clear that emerging technology can and should have a broad and transformative impact. However, the complexity of engineering effective solutions for coupling humans and machines together requires an approach that can demonstrate value early, either failing fast or eventually building to success. This idea has been codified by system theorist John Gall: All complex systems that work evolve from simpler systems that worked.
Structure to Deliver Outcomes
Narrowing the scope of the problem statement also facilitates our second principle: Organizational structure for software development should own the entire path to production. In certain cases, organizational barriers or protected funding can provide valuable separation from vested interests and day-to-day operations — famously, the Defense Advanced Research Projects Agency funded early innovations like the predecessor to the internet which likely could not have been developed elsewhere. But often these firewalled organizations — like AFWERX or the Joint Artificial Intelligence Center — must convince another organization that owns fielding responsibility that the new technology is a good idea, worthy of upsetting their already crafted and funded plans.
Another downside of putting innovation into new, separate organizations is that it sends the message that innovation is someone else’s job and not part of what the program offices would be doing within their modernization efforts. Program offices need to be a part of the solution, not an obstacle to be circumvented.
The desire to stand up and fund a new organization is rational. The Department of Defense is plagued with budgeting practices that make it easier to fund a new team than to fund new approaches within existing programs. But the alternative — amplifying the desirable behavior of those teams within the existing organization — is ultimately far more scalable. Adding resources to forward-leaning program offices working to operationalize new technology creates strong organizational incentives for innovation as opposed to depending on a “transition hero” who must hurl new technology across the valley of death.
While there is room for diversity in organizational models, it is important to empower the program offices tasked with acquiring and delivering military capability with the ability to innovate. Kessel Run, almost uniquely, is an innovative team that is the program office. We only had to convince ourselves to transition our products to our existing program of record. Structured well — aligned to outcomes rather than adherence to an obsolete plan — a program office should continually seek new ideas and new technologies that can support those desired outcomes. The Department of Defense should reward delivery over plans.
The valley of death disappears when organizational incentives are aligned by putting the innovation teams within the group responsible to deliver capabilities. Kessel Run was an example of this structure. Despite its popular image, Kessel Run isn’t a cadre of hoodie-wearing hackers fixing other people’s software problems. Kessel Run is the program office for a formal acquisition program — the Air Operations Center — that chose a modernization strategy of continuous, iterative development and delivery.
Align Organizations with DevSecOps
This brings us to the third principle of organizational design. Any development effort that intends to use the tools of the digital era — including software and data — to deliver capability should be structured around seamlessly gathering feedback from operations and incorporating it into future development. Being able to change software in hours is only valuable if the development team can constantly learn from user feedback and push changes quickly. In the case of programs that deliver capability via networks — like Kessel Run — the program office itself should be a DevSecOps unit that manages the full path to production and the operational cloud environment for the software. The teams running the network, fixing the software, and ensuring the software is available should be aligned with the development team, not some disconnected team or oversight officials.
Things were different with hardware. The World War II-era B-29 Superfortress — the most complex weapon then built at its time — illustrates the distinctions between developmental and operational systems for hardware. In building the B-29, the goal was to have one perfect design produced in massive quantity. However, this meant that any change in the design would require an updated blueprint, which would then have to cascade back through a manufacturing line at enormous expense. When it was necessary to update planes that were already in the field, field mechanics would have to modify deployed aircraft. The fact that the B-29 worked at all in the face of many design changes is the stuff of history books.
But software is different. There is no blueprint that mediates between design and manufacture. The only specification is the source code, which can change hundreds of times a day. Using continuous delivery pipelines, updated software can be distributed to the field in minutes at near-zero cost. With well-designed software and delivery mechanisms, changing an operational system can be easy and inexpensive. This also allows for a culture of learning and continuous improvement through experimentation and knowledge gleaned from user data. Giving development teams insights that come from this data is necessary for delivering high reliability software products quickly.
Unfortunately, the organizations and habits of the Department of Defense are still better tuned to the industrial B-29 than today’s digital reality, as they continue to emphasize the value of good blueprints over operational feedback. There are nearly a dozen efforts at improving this by placing fewer restrictions on spending funds, adjusting acquisition rules, and delegating more responsibility for innovation. There is even an entire reform commission. But these efforts will take a long time to make their impact felt.
Every software system requires regular change, not only to improve its performance but also because of the need to patch hidden vulnerabilities. As a result, it is crucial that the same organizations, coders, and companies that update and patch software can also add new capabilities and features. Separating development from the operational running of software systems hurts the nation’s ability to respond in times of conflict.
Combining all three aspects of DevSecOps into a single Kessel Run team meant that we were able to replace the legacy planning and execution software for the Air Operations Center more effectively than was possible in prior attempts. Being a DevSecOps unit creates a constructive environment for rapid capability evolution and field repair. The last modernization program spent seven years and over a half-billion dollars and delivered no operationally usable software. A new organization more narrowly focused delivered software that eventually supported the Afghanistan evacuation in only five years. The Defense Department would benefit from more DevSecOps units, but there is no obvious home in today’s force structure.
The divide between acquisition and operations exists at the highest echelon of the military. In the Air Force, one Major Command runs development (Air Force Materiel Command), while separate Major Commands run operations (Air Combat Command and operational commands like the Pacific Air Forces). The distinction was useful for hardware systems like the B-29, but the services now need combined development and operational support units for software capabilities. Otherwise, the benefits of DevSecOps will never be realized because the first commander responsible for both the Dev and Ops teams is a four-star general in the Pentagon.
Conclusion
There is a vibrant future for a digitally enabled military that can constantly innovate with new combinations of tactics, decision support, and weapons. But getting there requires heeding the right lessons of the commercial software era.
Commercial software development has exploded because of the opportunity for companies and developers to hook together systems and code using evolving commodity infrastructure, then add something new on top of, or between, these pieces. For example, modern internet and the massive datacenters that power cloud computing are mostly built on versions of the Linux operating system, but no committee coordinates this. Instead, there are tens of thousands of different versions of the core Linux kernel alone, as innovators mix and match pieces to meet their particular needs — some stripping it down for high performance, and others adding features for security. In the era of the B-29, design potential was built around interchangeable bolts, rivets, and sheet metal gauges. In the digital era, those physical building blocks give way to limitless combinations of blocks of code and web services and system interfaces.
The Department of Defense can unleash a new wave of digital military innovation. This doesn’t start with an architecture definition, common standards, mandates for a common development platform, or buying new hoodies for the team. It begins with clearly defining important problems, and then empowering organizations with the right tools and structure to deliver better outcomes.
Brian Beachkofski is the former commander of Kessel Run (Air Force Life Cycle Management Center’s Detachment 12), which he led from April 2020 to April 2022. Before that, he was a managing director at Third Sector Capital Parters, Inc. He can be found on Twitter as @beachkofski.
Dan Patt is a senior fellow at the Hudson Institute, a former DARPA official, and the former CEO of Vecna Robotics.
