The Battle for Mon Cala: Getting the Military to Deliver Its Own Tech Solutions

Somewhere in the back of the Defense Digital Service headquarters, among copies of cybersecurity briefs and Star Wars paraphernalia, lies a briefing document several inches thick and 185 slides long, detailing the Marine Corps’ plan to transition its software to the cloud. The document is an artifact of the protracted process that is military software acquisition.

At the Defense Digital Service, we often hear stories about quagmires created by the services when approaching modern software development. The services want to begin building software solutions in commercial cloud environments, but struggle with where to begin. The Defense Department’s first instinct is to pay a vendor to “transfer” current software to the cloud, rather than redirecting talent internally to build applications in a cloud-based environment.

Earlier this year, the Defense Digital Service and the Marine Corps had an opportunity to address the modernization of the software development process. In the spring of 2019, the Marine Corps had approached the Defense Digital Service for help in shifting their systems to a commercial cloud infrastructure. After all, the Defense Digital Service is the self-titled “SWAT team of nerds” tasked with advancing technology for national security.

Cue the 185-slide plan, handed over by the marines to the Defense Digital Service, composed of Pentagon civilians who have previously built careers in world-leading tech companies and start-ups. It was clear to Defense Digital Service engineers that the Marine Corps had accepted that any software update or acquisition would be infuriatingly slow, potentially insecure, and expensive.

The Defense Digital Service engineers and product managers convinced the Marine Corps that it not only has the talent to code, but by developing software internally (rather than purchasing cloud software from a commercial vendor), the Marine Corps could actually acquire a product of even greater quality and security.

Working together, the Defense Digital Service and the Marine Corps identified the right marines and mentored these engineers to produce the branch’s first cloud-based product, the System for Operational Logistics Orders, or SOLO. The joint project — which we named the “Battle for Mon Cala” after a fictional clash in the Star Wars universe — became an opportunity to reimagine software development in the department. We leveraged internal talent, we built the application in the cloud, and we tested a novel software authorization process to deliver a new cloud-based software application — all in just 90 days.

Scaling the Battle for Mon Cala

Initially, the Marine Corps struggled with recruiting people to work on the Battle for Mon Cala. Though the Marine Corps has a vast supply of talented software engineers, it needed to get the word out about the opportunity.

During the summer of 2019, the Defense Digital Service and the Marine Corps began a word-of-mouth campaign to recruit engineers. The most successful tactic proved to be a wide-reaching call for applications from the Marine Administrative Message, a centralized message bank for the entire Marine Corps workforce. This approach follows one of the Defense Digital Service’s recruiting guidelines: Go where the talent is. Soon, the team was flooded with dozens of responses from talented developers within the Marine Corps.

One of the applications was from a staff sergeant with a background in both cyberspace analysis and cryptologic signals intelligence who was fully prepared to rotate to his assigned duty station, where he would have served primarily as the urinalysis program coordinator. At the Defense Digital Service, we hear stories like this all the time — engineers doing clerical work or programmers checking facility badges. Hardware engineers, software developers, product managers, and other types of technical talent are routinely given non-technical assignments despite the fact that their skills are so desperately needed.

Our goal for recruiting software engineers from the Marine Corps ranks was to give them full immersion in modern software development using the commercial best practices that Defense Digital Service engineers learned from careers at some of the most prominent tech companies in the world.

After a rigorous vetting process and technical evaluation, we selected our Battle for Mon Cala team by mid-October 2019. The team included engineers experienced in all facets of software development, and all team members were active-duty marines or Marine Corps civilians.

Capt. Victor Castro, 1st Lt. Litthideth Phansiri, 1st Lt. Scott Pierce-Wrobel, Staff Sgt. Jeffrey Hackshaw, and civilian Garrett Haley arrived at the Pentagon in January 2020 to begin work as engineers on the Battle for Mon Cala team. They were issued Apple MacBooks and new email addresses, and given access to Slack, G-Suite, Github, and other collaborative tools that the Defense Digital Service uses to rapidly build software. At the end of that first day, they were ready to go to work.

Locating the Pain

The Defense Digital Service and the assembled talent wanted to develop a software prototype that would both solve an existing problem and serve as a pilot for cloud-based development.

The Global Combat Support System was an ideal test case. It serves as a single point of entry for all logistics requirements and is known to be the bane of many marines’ existence. Every transaction involving a government-approved item, from motors to uniforms to ammo, goes through the Global Combat Support System. The system is essential.

The system is supposed to allow for rapid and flexible operations. In reality, it suffers from poor user experience: timeout issues during the login process, latency issues as more users log in during the day, and data errors that must be corrected regularly by support services. Because the system was built on a platform released in 2004, it has not kept up with improved software, web, and infrastructure capabilities.

Processing the receipt of goods at a supply house became our focus area after observing firsthand the pain of processing deliveries at a military workforce supply house.

Three drums of motor lubricant came into the warehouse for the motor pool that day. It took a marine 20 minutes to log the receipt of the drums. When an officer walked the 100 yards from the motor pool to the warehouse for the drums, it took the two of them 15 minutes to file the hand-off. Once the officer was back at the motor pool, he logged into the same system to acknowledge the drums were delivered. Essentially, it took two marines the better part of an hour to manually do the work of a barcode.

It was settled: Receipt of goods was a daily task that impacted lots of users, fixing it allowed us to tackle the issues of logging into the system, it would not require changes to the legacy product, and it looked feasible to deliver in 90 days.

The Battle Begins

The first week consisted of a master class in modern commercial software development for the marines, with an emphasis on secure cloud infrastructure and continuous integration and deployment pipelines. During week two, the Battle for Mon Cala team, along with Defense Digital Service user experience experts, hit the road to Quantico, Virginia, to meet face-to-face with Global Combat Support System users.

After meeting with users, the team sketched out a project workflow, information architecture, and the journey map for both the user and data.

After multiple Defense Digital Service development, security, and operations experts reviewed the infrastructure and security for the proposed system architecture, the Battle for Mon Cala team spent February building out the software application. User experience discussions, design interviews, and individual storyboard exercises helped the team develop wireframes — or basic outlines — that were tested by users at the supply house in Quantico.

With the infrastructure built and the front-end designs fully tested, by late February it was time to combine the robust infrastructure with the web-based software application. Then, nearly 60 days into the Battle for Mon Cala, the team encountered a roadblock.

Travel restrictions imposed to contain COVID-19 took hold and the team returned to their home locations spread throughout the national capital region, California, and Okinawa, Japan. But the Battle for Mon Cala work continued without missing a beat.

The Case for Rapid Authorization

Somehow, the Defense Department’s dependably slow authorization process proved a greater drag on the team’s progress than the pandemic. But not for long.

Even with the receipt of goods functionality finished, the Battle for Mon Cala team was unable to integrate the software application with the Global Combat Support System until the “authority to operate” was released by an authorizing official for the Marine Corps.

An authority to operate is a permit to use a product issued in the federal government when implementing new technologies. Today, technology systems are put into production only after such an authority is issued. This process requires a technology team to respond to hundreds of security questions, or risk management framework controls. The process usually only begins when a system is complete or near-complete, resulting in a long lead time to implement new technologies.

The Battle for Mon Cala team planned to circumvent the slowdown by initiating the authority to operate at the same time as the software application functionality. The Marine Corps software compliance team decided to use SOLO to test out these parallel software development and compliance review processes, known as a rapid authority to operate. On April 29, an unprecedented 30 days after the team successfully demonstrated its application worked, the authorizing official issued the authority to operate.

Delivering SOLO

In early May, the Battle for Mon Cala delivered SOLO, a secure cloud-based application that improves the “receipt of goods” function in the Global Combat Support System. We dramatically reduced the steps it takes to receive goods into a warehouse and distribute them to the ordering unit. SOLO combined more than a dozen steps into a single process, trimming the task down from 20 minutes to a matter of seconds.

More than a function in itself, SOLO was a pilot to prove a process of developing cloud-based software for the military. The application itself addresses one problem — receipt of goods — but it also contains a software-based server, database, and network infrastructure that can support up to a thousand functions. The Marine Corps can add on functions at a pace set by its needs and resources, without having to wait. The authorization to operate for SOLO was built around its infrastructure, so as long as the infrastructure stays intact, additional functions or changes need no further authorization.

Lessons Learned

The Battle for Mon Cala team built SOLO within 90 days — an unheard-of feat for federal government software development. It represents a significant reduction of the multiyear timelines usually required to field a product from conception to completion to delivery.

The Defense Digital Service took on the Battle for Mon Cala to prove a few points. First, the Marine Corps does have the technical talent in its ranks needed to build first-class software (as do all branches of the U.S. military). Second, the Department of Defense can build software quickly and with greater flexibility for the future by looking inward, instead of to vendors. Third, modern software development requires adapting cumbersome processes like the Authority to Operate so engineers can work fast and iterate quickly to user and security needs.

This project should shape not only future Marine Corps efforts, but future Department of Defense projects for years to come. The team demonstrated a cost-effective, iterative procedure that can quickly field a precise capability needed to accomplish a task. Now is time to empower military talent to build the modern software development capabilities the Pentagon so desperately needs. There is much work to be done.

Brett Goldstein is the Director of the Defense Digital Service. After serving on the founding team of OpenTable, he joined the Chicago Police Department following the September 11th terrorist attacks. He became the first municipal chief data officer in the United States and later the chief information officer of the City of Chicago. Before joining Defense Digital Service, Brett served as special adviser to the U.S. Department of the Navy where he provided technical expertise to the Navy, the Joint Special Operations Command, and NATO’s Resolute Support Mission.

Maj. Gen. Matthew G. Glavy assumed command of U.S. Marine Corps Forces Cyberspace Command on July 2, 2018. He was commissioned in May 1986 through the United States Naval Academy, and upon receiving his wings in September of 1988, he was selected to fly the CH-46 Sea Knight. He holds master’s degrees in both Military Studies and Strategic Studies and a Bachelor of Science in Systems Engineering from the United States Naval Academy.

Image: U.S. Marine Corps photo by Sgt. Jesus Sepulveda Torres