Fixing the Navy’s Software
Good software elevates human performance. Bad software kills people, even in peacetime.
Software development in the U.S. Navy today relies on contractors too far removed from the mission and slow, outdated processes to direct their efforts. This paradigm yields bad software that arrives late, fails to take into account user needs, and only has a veneer of security. Industry, the intelligence community, and the Air Force have all found much better ways of delivering value to servicemembers and civilian staff who use the applications. The Navy can do better by copying these entities’ successful development pipelines, expanding user influence on product development, and tackling software challenges with permanent teams instead of as one-off projects.
An Agile Navy
Consider two techniques for navigating a ship over great distances. One relies heavily on detailed planning, the other on broad forecasting and then dynamic execution.
In the first technique, every heading and speed change needed to get from point A to B is plotted ahead of time. You develop extremely sophisticated tools to predict weather over vast distances and time, since currents and winds will push your vessel off course over the journey. Then you must predict where other shipping traffic will be, and the turns needed to avoid it. Next, you forecast mechanical issues and position resources accordingly. After all this work, you leave port and take a nap until arrival at your destination.
With the second technique, you look at broad forecasts of future conditions, take reasonable precautions like bringing extra fuel, and plan to avoid forecasted storms. Then you set out, maintain a watch, and adjust your ship’s heading in response to the changing environment around you.
Which do you choose?
I hope the second, as no sane mariner would choose the first. It’s impossible to precisely predict currents and winds at every step of a long journey, or to forecast the precise future location of every other vessel. And it’s impossible to know when equipment will break with any meaningful accuracy. Rather than overly detailed planning, training competent mariners to make good decisions at the point of action is the best approach.
The U.S. Navy understands this need for flexibility when it comes to navigating warships. But the organization writ large hasn’t yet successfully applied this concept, in the form of agile development techniques, to build the software that its people use. The service builds extremely complex systems as if it can foretell the winds — writing detailed requirement documents, setting contractors to work with the overly complicated plan, and then expecting to arrive exactly where the end user needs. This is as unwise when navigating software development as it is when navigating a ship. Private industry transitioned many years ago to software development approaches like Agile and DevOps that put the user at the center and allow application requirements to change over the course of the project. The Navy is far behind.
The Navy isn’t missing the mark because it has bad people working on these problems. It’s missing the mark because it has bad incentives and processes. Let’s look at how it might fix them.
A Case Study
The Navy has started experimenting with agile software development in the P-8 anti-submarine warfare community. An organization called the iLoc Development Team, based within the P-8’s warfare development center, takes fleet operators and officers, trains them, and unleashes their creativity and experience to build effective applications for the fleet. The iLoc Development Team, which I lead, builds serverless web applications that don’t require an internet connection — which means they work even in communications-denied environments. This framework requires much less security overhead than other methods and enables applications to run on any computer with a web browser, including aircraft and ship operating systems. It gets more relevant capability in the hands of fleet users faster.
The products that this team fields have cut tactical reporting errors by 90 percent; doubled the effectiveness of anti-submarine searches in common scenarios; saved an average of 30 man-hours per week per squadron; and enabled junior personnel to perform as well as experienced tacticians at one of the hardest mission phases that P-8s perform. It accomplished this with essentially zero budget.
A few months ago, operational P-8 squadrons reported a mechanical defect in their equipment that was harming mission effectiveness. The Naval Air Systems Command developed troubleshooting procedures, but they were difficult to execute tactically. The iLoc Development Team was able to update its tactical planning software to account for the procedures in a couple of days, making it so any operator could easily execute them. This would not have been possible without developers who had recent, tactically relevant experience.
In the past, a warfare development center would have had to come up with elaborate guidance that would have been hard to train tacticians to apply. Some operators would have been able to apply the new guidance better than others. Today, the same warfare development centers can simply push a software update and the entire force will uniformly improve. The software they write acts as actionable extensions of the doctrine it writes. This new concept, which we call software-defined tactics, is a paradigm shift that requires continuous authorities to operate and warfighters cross-trained as programmers.
There are three main principles behind the success the iLoc Development Team has enjoyed: continuous delivery, informed evolution, and a prioritization of teams over applications. These principles lend themselves to repeatable processes that the U.S. Navy needs to make easier to execute. Let’s start with continuous delivery.
Continuous Delivery
In industry, the intelligence community, and the Air Force, the gold standard in software development is continuous delivery. This allows software to evolve as organizations change.
When a ship is built there are clear demarcations between development, construction, and maintenance. Once requirements are established, there’s very little that can fundamentally change about the ship.
With software, developers can continually improve a product as they learn what customers need and identify the appropriate performance metrics. Consider an application that seeks to automate navigation of a ship along a journey from planning to execution to docking. Building software that does all of that at once would be very hard and would likely fail.
Instead, it’s better to build some minimally useful tool, called a “minimum viable product” in industry. In this case, the tool might tackle some part of the process that takes sailors a lot of time to do manually, like plotting an initial track and automatically looking for hazards to navigation. The goal here is threefold: quickly field a tool that helps sailors, confirm that sailors want a tool to help them do this part of their job, and collect usage data to learn how to help sailors with the next version of the tool. With each successive iteration, you could make the tool incrementally more useful or pivot to a better concept. Over the same amount of time you could have failed to build it all at once, you can slowly add useful functionality that will actually work and is actually desired. This dramatically lowers the risk that you are building software that sailors don’t want and that does things they don’t need.
The Navy doesn’t build software this way because it is often as hard to get permission to deploy a slight modification to software as it is to deploy a completely new application.
The current standard in Department of Defense information technology security is called the risk management framework. Getting permission to deploy software within this framework traditionally means manually verifying that an application complies with more than 800 security controls. While the desire to secure military software is understandable, this manual process takes an incredible amount of time and precludes rapid software development. The Air Force and intelligence community have figured out how to issue continuous authorities to operate, though. This means that once an organization fields an initial version of a product, it can continually release new versions as long as it uses vetted processes that reduce cyber security risk, and verifies compliance with cyber security requirements using automated tools. This is colloquially called DevSecOps in industry and involves continuously conducting deep security analyses throughout the development cycle, not just at the end. This process works because a later version of an application likely inherits some security built into prior versions.
The good news is that the Navy doesn’t need to figure out how to do this. The tools were created with the Compile to Combat in 24 Hours and other initiatives. What’s missing is the political will to execute from Navy leadership.
Informed Evolution
If the Navy puts in place a process to develop software with agility, it needs a way to figure out what to build. Luckily, the solution is simple — start by understanding the user.
The most effective way you can ensure that software engineers understand the user is to make them the same person. The most effective way to do that is to cross-train a small subset of operators as software developers and product managers. The Air Force has started down this path by taking operators out of their units for 6-month orders and training them to build software. This has been extremely successful, yielding tools that save $12 million a month in fuel costs and fix a lot of the problems around ALIS, a multibillion-dollar software project acquired through traditional processes to manage F-35 maintenance.
The iLoc Development Team consists of just six warfare-qualified P-8 operators. It can produce extremely impactful software with so few people because of the comprehensive mission knowledge the team possesses. The only help the team received from outside entities came in the form of security-scanning support from Naval Air Systems Command, algorithm development from an enterprising group of midshipman computer science students at the Naval Academy, and training from the Defense Digital Service’s Rogue Squadron.
Rogue Squadron is a software factory located in Silicon Valley that builds software to counter the threat from unmanned aerial systems. Recently, it has provided guidance to other small software initiatives throughout the Pentagon. They started with an active-duty naval officer building software on his own and later scaled their work by hiring talented civilian developers who code collaboratively with military personnel who know the mission the software will support. This model of active-duty personnel building systems with agile processes alongside civilian developers has been working across the military for years. It just has not been properly resourced.
You don’t need savants to build software that makes an impact. You need to train the people who already know the business and enable them with better technical process. Doing this will yield far more success than forcing a program manager to explain what they think the operator needs today to a contractor who has never done the operator’s job.
Mission risk is reduced by keeping the people who know the mission at the center while using technical experts to overcome difficult obstacles and build infrastructure.
Buy Teams, Not Applications
There’s one final, critical paradigm change the Navy needs to make to reorient its software development efforts. The Navy needs to hire and train teams of developers who understand the mission they’re trying to improve. And good developers with autonomy can spend their time building tools that create the most impact, instead of those that a contracting office 1,000 miles away picked a year prior. Special Operations Command did this successfully almost 10 years ago.
Ideally, these teams would be co-located with operational (not test) units. And the teams would consist mainly of active-duty, warfare-qualified officers and sailors to bring recent, relevant experience.
When the Navy focuses on buying individual software applications, it misses the point. This forces the service to use acquisition rules that encourage separate development and sustainment phases of the contract even though that dichotomy is meaningless for continually delivered software.
I have personally seen sailors cross-trained in software engineering spend about 80 man-hours to produce tools that save units 1,000 hours per year. One such tool created visual analytics and optimization tools to help manage a squadron training syllabus. This took about 30 hours per week of work off the shoulders of four of the most talented aviators at a P-8 squadron. It leveraged a static database already funded by the Navy, saving millions of dollars in duplicative software development efforts.
I have also seen software developers without users in their team spend millions of dollars to produce tools that detract from mission effectiveness. One example is an allocation algorithm on the P-8 that ostensibly makes it easier to track submarines by determining which sonobuoys should be made available next. In practice, it doubles the operator’s workload since they have to undo the counterproductive things the algorithm does before doing what they actually want. That algorithm certainly sounded like a good idea in the requirements-definition stage, but observing just a few users struggle with it would have sent a clear signal it needed to be reworked or killed.
It can seem expensive to commanders to give up their people to train in a skill seemingly unrelated to their primary job. However, it is an investment that quickly pays for itself in man-hours, dollars, and mission-effectiveness saved. Agile software development processes without developers who fundamentally understand the problems they are trying to solve just results in moving quickly nowhere.
Putting It All Together
The Navy usually develops bad software because it tries to plan software development years in advance, cannot change software quickly enough, and relies on contractor support that is not sufficiently connected to the user. To make sure it develops more good and less bad software, the service has some work to do.
First, the Navy should replicate and operationalize the DevSecOps pipeline the Air Force has already built at places like Kessel Run, and that the Navy almost has built with Compile to Combat in 24 Hours. It should use this pipeline to enable widespread usage of continuous authorities to operate to enable rapid iteration and deployment of software. Second, it should train a small subset of fleet personnel to build applications that help them accomplish the missions they execute every day. This will help ensure the Navy builds the right tools in the right way. Finally, the Navy should bring these trained sailors and the DevSecOps pipeline together by co-locating them with operational units. Funding them as mission- and acquisitions-oriented organizations would allow them to work on projects that provide the most value.
The best places to execute this strategy are the warfare development centers the Navy already operates in proximity to operational units. The iLoc Development Team operates at the Maritime Patrol and Reconnaissance Weapons School (the P-8, EP-3, and MQ-4 warfare development center), which is co-located with six operational P-8 squadrons. This allows for a rapid feedback loop from ideation, to application creation, to operational usage, and finally to iteration.
Putting this all together won’t just help the Navy catch up to the Air Force and intelligence community — it will help it leapfrog them.
Sean Lavelle is a graduate of the U.S. Naval Academy and Johns Hopkins University. He is a P-8 Instructor TACCO and leads the iLoc Development Team in rapidly developing and fielding cutting-edge capability for the fleet. The views expressed in this article are those of the author and do not reflect the official policy or position of the Department of the Navy, Department of Defense, or the U.S. government.
Image: U.S. Navy (Photo by Rick Naystatt)
