How Innovation Dies: The Case of RATE
At the height of the pandemic, the Department of Defense developed an algorithm that used biometric data provided by commercially available rings and watches to predict COVID-19 infections, including asymptomatic cases, with 73 percent accuracy. Now the research study that developed this capability is ending, but the Department of Defense does not have a program of record office ready to adopt it. As a result, the development of a promising technology has been halted. Rather than move at the pace of COVID-19 outbreaks and new variants, the Department of Defense instead relied on drawn-out business practices that will render this technology irrelevant by the time it is finally fielded.
Over the past year and a half, I served as the program manager at the Defense Innovation Unit that created the Rapid Analysis of Threat Exposure (RATE) program. The story of how RATE was developed, and why it ended, offers several valuable lessons for the Department of Defense as it seeks to innovate better and faster in the future. Through RATE, I learned that if the department wants to accelerate innovation, it can take three crucial steps. First, it should hold staff officers accountable for inaction. Second, it should develop a more robust waiver process to deal with new and uncertain situations like a global pandemic. And third, the Department of Defense should create a parallel fast lane to transition technology through temporary program offices that can do an interim lift while the larger program of record offices catch up.
The Story of RATE: What Is RATE?
At its simplest, RATE is a set of commercially available watches and rings that check a suite of biometrics and feed data to an algorithm. This algorithm then generates a score showing the likelihood of having an illness, in this case COVID-19. RATE’s been called a “check engine light” — it doesn’t tell you specifically what is wrong, but rather that you deviated from your baseline biometrics in a manner consistent with an impending illness. An algorithm like RATE is preventive rather than reactive. The idea is to get far enough ahead of the illness that you can both aggressively reduce its spread to others and consider prophylactic measures to recover sooner or get less sick.
The development of RATE began three years earlier, in 2018, when a team of highly skilled professionals set out to predict when hospital patients were on the verge of getting sick. The original team involved the Defense Threat Reduction Agency, Philips Healthcare, Texas A&M, and the Defense Innovation Unit. As a starting point, the team sifted through hundreds of thousands of hospital patients’ data to focus on the tens of thousands of those patients who got sick while in the hospital. The original algorithm relied on bulky hospital-grade equipment, but was able to predict a wide variety of illnesses 48 hours prior to symptoms.
Months after this RATE study concluded, COVID-19 reared its ugly head and the same team reassembled. This time the team focused on using commercially available watches and rings with sensors that could measure some of the same biometric data as the hospital equipment. While imperfect, we decided to “bootstrap” the hospital algorithm onto two small wearables in hopes of detecting the COVID virus. Roll forward six months, and we had over 10,000 warfighters participating in a research study for RATE.
One interesting challenge we faced was that the Department of Defense had to determine whether the algorithm we were developing should be categorized as a medical device or not. This, ultimately, was the hurdle the department failed to clear.
This is a complicated, slightly wonky issue, but it was central to the department’s handling of RATE. The Department of Defense needs to label things in order to know what to compare them to and what regulations to apply. Is “it” a weapon, a platform, a system? If the department knows what “it” is, the department knows the risks associated with it and how to mitigate them. This mindset is fine for evolutionary innovation, but counterproductive in the case of revolutionary innovation — in other words, when “it” is not just a better widget, but a new function. In the revolutionary case of RATE, the Department of Defense struggled to determine what bin to place it in. Rather than have a RATE conversation with the Food and Drug Administration, which regularly makes medical device determinations, the department instead chose to have an internal discussion that was never designed with an outcome in mind. This resulted in months of inaction, no bin determination, and ultimately an inability to field RATE.
How to Make Innovation Happen
That is the story of RATE. To prevent other promising research from meeting the same fate, the Department of Defense should apply three lessons.
Hold the Middle Level Accountable for Inaction
The Department of Defense readily admits that it struggles to innovate more quickly than its adversaries. Both insiders and outsiders have tried to help. Their suggestions range from requirements reform to a third-offset strategy and a multi-year acquisitions overhaul. The department itself likes to focus the blame on acquisitions processes. But it should instead scrutinize a small group of people who are the log jam to defense innovation.
It is tempting to blame the highest echelons of the department for not supporting the lowest echelons’ efforts to innovate. But the real problem is with the middle echelon linking the two. In my journey with RATE, I saw amazing innovation at the “worker bee” level. We had extremely capable, curious, and motivated individuals in the federal civilian, academic, commercial industry, and military sectors who worked together as a team to make an algorithm that predicts COVID-19. The higher echelon was supportive as well. With RATE, the combative command general officer level was extremely eager to adopt our technology.
But between the highest and lowest echelons, there was a bottleneck in the middle. After we achieved a statistically validated algorithm in February 2021, we reached back to the combatant commands and operational cohorts to see if they were ready to adopt the technology to help fight the COVID-19 pandemic. They responded enthusiastically, asking us to integrate this technology as soon as possible and begin the process of standing up a program of record for RATE. This is when I saw first-hand where innovation dies: at the GS-15/O-6 staff officer level. The Department of Defense relies on staff officers to advise general officers on a variety of issues. When the Pentagon sends new technology to these staff officers to review, they examine legal ramifications, find existing policy guidance, and talk to subject matter experts. But this is mostly an informal and subjective process with no timeline and considerable discretion left to the staff officer. As a result, even in the midst of a pandemic, a small group of staff officers can drag their heels and, by doing so, kill innovation.
The Department of Defense should make officers accountable for inaction. The Federal Acquisitions Regulations encourage risk-taking. So rather than arbitrarily killing innovation at the middle echelon for perceived risk, they should instead codify that risk and promptly address it up to the O-10 level. Likewise, at the highest level, the O-10 community needs to hold their staff officers accountable for inaction in the form of reprimands and performance report downgrades. If the Department of Defense wants to get serious about innovation, it needs to get serious about the consequences of not innovating. This means the department needs to groom mid-level leaders to take calculated risk, while also holding accountable those who choose to stifle innovation through inaction.
Don’t Let Innovation Outpace Policy
The United States genuinely loves its regulations, adding an average of 1,800 federal rules a year. Yet with all these regulations, the Department of Defense still struggles with the question of what to do about situations that aren’t explicitly covered. In RATE’s case, the department was not capable of fielding this new technology because there was not a regulation that specifically allowed it. Rather than embrace innovative technology, the same middle echelon saw the lack of clear regulatory policy as a reason to condemn RATE to the innovation “valley of death.”
How can this be fixed? In the corporate world, when a company is deciding on a pathway for a medical device, they submit a 513(g) “Request for Information” to the Food and Drug Administration. Upon receipt of the 513(g), the agency has 60 workdays to respond with a classification for the proposed device. The Department of Defense can adopt a similar framework by incorporating a joint centralized process that quickly answers policy questions with the goal of allowing the earlier adoption of tech.
In the event the policy discussion becomes drawn out, there needs to be another parallel waiver process to allow innovation to progress. While the Department of Defense does have a waiver process for a fair number of policies, they don’t have a standing waiver authority for situations in which the problem stems from the lack of applicable policies. When RATE detected its first COVID-19 case in June 2020, we began discussions with various offices in the Defense Health Agency and within the Department of Defense at the Pentagon for a fast track to adopt this detection technology. In every case, these organizations were unsure of what “bin” to put RATE in, pointing us back to the lack of policy for RATE — a hard roadblock. In fact, when there’s no policy, even the Secretary of Defense or the Commander in Chief cannot issue a waiver.
If innovation outpaces policy, innovation dies. The Department of Defense needs to consider the ramifications of this. While RATE is not the end-all-be-all, it had and still has the ability to drastically reduce COVID-19 outbreaks. How many other innovative programs fell to the wayside because of lagging policy?
Reform the Program of Record
The final step in innovation occurs when the Department of Defense assigns a new technology to a “program of record” office. At this point there is enduring funding and a management office to support, sustain and scale the effort. Meanwhile, nearly every technology that is not assigned to a program of record office falls into the “valley of death”. In the case of RATE, we fought and lost the battle for a timely transition to a program of record office.
Why does this transition lead to innovation lag? Program of record offices are designed for technologies that have gone through the Joint Capabilities Integration and Development System process. This process takes years and is designed to give the department a turnkey solution — one that’s been thoroughly vetted and has a line of funding and manpower to support and sustain an enduring capability. It’s an approach that decreases risk at the expense of time and product flexibility. A 2017 Air Force study found that on average, programs were fielding capabilities a year and a half later than promised. The same study found that the average time to award most contracts was one year. Following the department’s prescribed pathway for a successful transition would take two and a half years from idea to fielding the solution. That is too slow.
The current Joint Capabilities Integration and Development System works for lengthy research and developmental efforts like fielding a new aircraft. But what happens if the department needs a solution within two years? Where does that program live? To speed up this process, the Department of Defense needs to identify an iterative approach to fielding new technology, perhaps with a different type of program of record office to receive new technology even before developmental efforts begin. This would involve a process more akin to Agile software development, where the product is constantly being updated based on user needs. Innovative technology shouldn’t have to wait years to transition to a program of record office. Instead, there should be a smooth transition from proof of concept to prototype to a temporary program office and, finally, to a full program of record office.
The Defense Innovation Unit could play an important role in facilitating a new approach to housing innovation — part of its “fast follower” commercial tech adoption strategy. It is one of a few joint entities that have expertise in “other transition authority” contracts. This means it can stand up a temporary program office to field an early version of a new technology at a limited scale and for a limited duration while the larger program of record office is created to catch this technology. “Temporary” means these individuals are dual purposed so that when their product fully transitions to a program of record office, they resume their previous duties. “Limited scale” caps the scope, so an entity like the Defense Innovation Unit doesn’t inadvertently become an enduring program of record office. “Limited duration” likewise caps the scope and ideally needs to be one to two years, given the time required to stand up new program of record offices.
This is just one approach. It is not only the Defense Innovation Unit that could field a temporary program office. Any combatant command or joint entity in the Department of Defense (the Defense Threat Reduction Agency, for example) that routinely manages “other transition authority” contracts and conducts rapid prototyping should also be able to stand up this capability.
Ripe for Change
Critics cite a number of reasons for the Pentagon’s innovation lag, including the lack of money, manpower, and talent pool. But none of these hindered RATE’s success. If the Department of Defense wants to out-innovate its adversaries, then the general officers need to start by holding staff officers accountable for inaction. That alone will start to break up the log jam and allow technology to quickly flow to the general officers for adoption. But the department should also incorporate standing waivers as a wedge to keep the innovation door open. This will ensure timely integration of new technology while written policy catches up. The last piece of the puzzle is the temporary program office. By allowing a program to rapidly prototype and iterate at a small scale, the Department of Defense can not only develop technology much earlier, but also gain more power to shape it as it transitions to a program of record office.
It is too late for RATE. But adopting these lessons will make sure that the next RATE won’t end up on the shelf. America needs to be able to operate effectively in wartime, peacetime, and during a COVID-19 outbreak. Will the Department of Defense continue pushing new technologies into the valley of death, or will it find ways to help them across?
Jeff “Mach” Schneider is a U.S. Air Force Reservist with over 2,000 hours flying the F-16. He has served at the Pentagon’s Requirements directorate and was a member of the Chief of Staff of the Air Force’s Squadron Revitalization initiative. The views presented here are his alone and do not represent those of the Defense Innovation Unit, the U.S. Air Force, or the Department of Defense.
