AI and Intelligence Analysis: Panacea or Peril?
In today’s chaotic world, professional intelligence analysts must contend with nearly endless data streams, which risk overwhelming them while also exacerbating the impact of cognitive biases. Is AI the answer, or do the flaws that currently afflict AI create yet more risks?
In fact, AI is neither a panacea nor a peril. Like other emerging technologies, AI is not an instant “out of the box” solution but rather a capability that continues to evolve. Today, AI can augment human capabilities and enhance the analysis process by tackling specific challenges. However, AI is not without issues. This means its value lies in serving as a complementary capability to the expertise and judgment of human intelligence analysts.
Before the wholesale adoption of AI in support of intelligence analysis, it is essential to understand the specific problems facing analysts: coping with large volumes of data; the acquisition of data from non-traditional sources; and, perhaps most vexing of all, the impacts of cognitive biases that impact the objectivity of intelligence assessments. AI can play a valuable role in alleviating these challenges, but only if humans are kept in the loop.
Data Overload, Integrity, and Bias
Separating the wheat from the chaff, or sorting through the volumes of information collected daily, is a primary problem vexing the U.S. intelligence community today. This is despite the investment in automated means of collection and the corresponding infrastructure to store, organize, and structure bulk data for future retrieval and examination by intelligence analysts. Though this impressive process likely consumes a significant portion of the U.S. intelligence community’s budget, information nonetheless “falls on the floor” or goes unanalyzed by human analysts.
A simplistic argument would contend that too much data is a condition preferable to not enough data. However, intelligence failures such as the 2003 misidentification of Iraqi weapons of mass destruction, the failure of Israeli intelligence to discern what became the imminent invasion of a combined Arab army led by Egypt and Syria in 1973 (leading to the Yom Kippur War), and possibly the October 7, 2023 surprise incursion and attacks by Hamas into Israel suggests that an overabundance could prove detrimental to the vitally important process of intelligence analysis, particularly if analysts are unable to sort and sift through the information available to identify those key indicators of an adversary’s intentions.
Processing a growing amount of information requires the intelligence analyst to comb through, identify, and synthesize disparate data points into a judgment — which, when done well, reduces uncertainty. However, cognitive biases coupled with the problem of too much data or poor data quality plague this process, leading to imprecise assessments that could contribute to policy and decision-making failures, increased risks to military operations, and other disadvantageous and cascading outcomes. Given the challenging nature of intelligence analysis, could AI help avoid these consequences and provide decision-makers with crucial, objective, and accurate information?
If the promise of AI holds true, then generative AI technologies such as ChatGPT, which are based upon large language models, can add efficiencies to the analysis process. For example, generative AI could summarize lengthy texts (e.g., foreign grey literature), translate foreign languages, conduct open-source sentiment analysis, and perform various other functions. Moreover, generative AI could assist in the development of intelligence assessments. This does not alleviate human intelligence analysts of their pivotal function. Still, generative AI could serve as an adjunct to the analysis process, aiding in identifying analytical flaws or inconsistencies.
While these are promising functions, and it is reasonable to assume that intelligence agencies have already incorporated such technologies into their everyday processes, generative AI is not without its faults. First, generative AI does little to alleviate the perennial problem of analytical bias. Generative AI technologies constructed on large language models rely upon preexisting data sets, which are inherently unstructured and potentially flawed. Linked to this point, today’s generative AI models are prone to mistakes and can provide false or inaccurate content. These “hallucinations” relate to the development of generative AI models; despite training using a large corpus of data, if the generative AI system encounters an unfamiliar word, phrase, or topic — or if the data is insufficient — it will make an inference based upon its understanding of language and will give an answer that the system deems logical, but which could be erroneous.
Second, the information needed to determine an adversary’s capabilities and intentions is no longer solely the purview of governments. Non-governmental organizations, private entities, social media companies, and others have emerged as important data brokers possessing the information required to understand the strategic environment and to construct accurate intelligence assessments. The use of generative AI in intelligence analysis needs to address the associated underlying issues of data access, quality, and bias.
Thinking about Thinking: The Villainous Nature of Mindsets
It is a fallacy to believe that humans fully control their thought processes. The human mind exhibits instinctive and unconscious tendencies to process new information. For instance, humans exhibit an inclination to seek patterns, such as cause-and-effect relationships, which aid in analyzing the unknown or when presented with new problem sets. Moreover, as a component of survival instincts, individuals subconsciously develop mental models or cognitive and perceptual biases that impact human judgment and decision-making. Mental models are components of the human subconscious that guide our daily actions through mental shortcuts that we execute without conscious control.
In terms of intelligence analysis, a mental model is a paradigm that guides an analyst’s thought process on how an adversary will act or how a situation will unfold. The advantages associated with mental models vary depending on context and circumstances. Nonetheless, mental models can promote critical thinking, aid in decision-making, and incorporate diverse perspectives into the analytical or decision-making process. Ideally, the combination of multiple mental models could contribute to more positive intelligence analysis outcomes.
Conversely, extensive experience or knowledge on a topic could prove detrimental, as a mental model may prompt an individual to reject new and contradictory information or to process it incorrectly. Perhaps the most concerning aspect of mental models is the tendency to resist change. Once an analyst has developed an assessment, especially if the assessment has undergone review and approval, an analyst’s mental model may preclude them from accepting new and substantive information that alters the character of their assessment.
A stark example of mental models aggravating analytical biases is the failure of Israel’s intelligence apparatus to correctly identify Arab intent in the lead up to 1973 Yom Kippur War. Following the success of the 1967 Six-Day War, Israel developed an organizational and institutional adherence to what Israeli intelligence referred to as the “conception.” The conception acted as an analytical framework for Israeli intelligence and policymakers, shaping their perspectives on Arab activities, which held that, given Israel’s overwhelming success in the 1967 war, Egypt and Syria would not attack absent the other, Arab forces would not attack absent surface-to-surface ballistic missiles that could threaten and place Israeli airfields and civilian populations at risk, and that the combined Arab armies would not attack unless they could contend with Israeli air superiority via the provision of Soviet air defense capabilities. Israeli intelligence analysts exhibited a predisposition to examine incoming intelligence solely in relation to the conception, and attempts to derive Arab intentions based upon strategic calculations rather than tactical realities fostered the production of imprecise assessments, leading to the surprise attack by the combined Arab forces.
AI to the Rescue?
Intelligence agencies are information-centered organizations, meaning that the organization’s foundation relies upon a consistent inflow of data. Analysts subsequently transform this “raw” data into coherent information for customer presentation or delivery. Rarely, if ever, do the totality of facts regarding an issue present themselves for a nearly definitive conclusion. Instead, an intelligence agency strives for accuracy in its assessments through reliance upon inconclusive facts ascertained through intelligence collection operations and assumptions derived from analysts’ knowledge, expertise and training. Despite being well-versed in their understanding of the historical aspects of the issue and, more importantly, their tradecraft, the adverse effects of mental models consistently loom over intelligence assessments. Given the potential unconscious and involuntary influence of mental models, could generative AI pierce through the inherent bias of human cognition to provide unbiased and objective assessments?
To address this question, it is first necessary to understand how generative AI can enable the intelligence analysis process. Perhaps the primary contributing function of generative AI to intelligence analysis is the ability to distill complexity into more manageable core components for an analyst. Specifically, AI can process large amounts of structured and unstructured data from multiple, disparate sources and determine linkages within the data that are not readily apparent to a human analyst. Moreover, presuming the capability of the generative AI’s large language model, it could fuse information from multiple intelligence disciplines (e.g., signals, human, geospatial intelligence, etc.), presenting a clearer depiction of the issue at a faster rate. The value of actionable intelligence, particularly in time-sensitive scenarios, is high, and thus, the vastly faster processing speeds of AI to identify patterns and correlations are quite valuable to the analyst. As an ancillary benefit, the greater timeliness associated with delivering actionable intelligence acts as a relationship-strengthening measure between an intelligence organization and its customers.
The ability of a human to interact with a generative AI system — even if the prompt contains confusing grammar, misspellings, and a lack of punctuation — and receive a useful response is a demonstration of natural language processing. Natural language processing facilitates the interaction between humans and AI, and enables systems to process and understand human text and speech. There exists an estimated 7,164 languages in the world today. Additionally, the amount of data produced daily is likely in the range of 300–400 million terabytes. From an intelligence perspective, analyzing even just a fraction of this data, especially when the source is in a foreign language, is a difficult and time-consuming task. Natural language processing alleviates the translation burden and can aid in extracting pertinent information from textual data such as articles, books, and other documentation. Perhaps even more important, as this technology matures, the need to recruit, train, and retain linguists in difficult and esoteric language sets diminishes.
It’s All About the Data
Data analysis and natural language processing represent just a sampling of generative AI’s applications to intelligence operations. Indeed, the promise of AI could yield manifold benefits in the field of intelligence analysis beyond these two functions. However, AI is not without issue. It is vitally important to highlight that the core functionality of generative AI derives from the data employed to train the model. If the dataset contains bias, the model will continue to promulgate and perhaps even amplify those biases. Thus, we return to the perennial problem of negative mental models impacting the analysis that could potentially feed generative AI systems. The primary consequence of leveraging pre-existing intelligence datasets is the unknown implication of biases contained in the finished analytical products. The injection of such datasets could continue the diffusion of skewed analysis, creating a cyclical process that exponentially adds to the compendium of imprecise and possibly dubious intelligence products.
The potential of generative AI systems to provide misleading outputs, or hallucinations, formed from incomplete or inaccurate data is a common problem and an inherent limitation of today’s AI technology. Generative AI systems assess the next word, phrase, image, or other outputs in a combination based on observable patterns in the training data. In the absence of data or the presence of extraneous data, generative AI will deduce the most likely sequence of content, which may contain falsehoods or simply bogus information. As such, human knowledge, experience, expertise, and intuition will continue to remain the vital components of intelligence tradecraft until this technology matures.
It follows that quality data is essential for using generative AI for intelligence analysis purposes. Perhaps just as important is acquiring the data. Data is certainly a commodity: a lucrative product for purchase, sale, or collection. Though intelligence organizations expend perhaps a disproportionate amount of their funding on sophisticated intelligence collection capabilities, which acquire highly classified material, with the proliferation of publicly available or open-source information, governments no longer possess a monopoly on data. Data in the private sector can prove just as valuable, if not more so, than data collected from highly technical means. Therefore, an intelligence organization should pursue the acquisition of such data. However, several challenges arise when a government attempts to acquire data from the private sector, which include trust issues, proprietary concerns, and compatibility problems.
Generally, the private sector collects individual data to improve a company’s products and services by personalizing them for its customers, to understand consumer behavior, and to improve customer retention. We have all likely experienced targeted marketing campaigns based on our browsing history, past purchases, and demographics. Additionally, multimedia streaming companies often offer customized experiences through playlists or recommendations on what to listen to or watch next. Ultimately, a consumer-based company analyzes data to develop and maintain customer relationships by appealing to their desires. While much of this data appears trivial, if the company freely shares data with the government, it may damage consumer trust and generate negative public perceptions of the company. Moreover, there are ethical concerns regarding sharing one’s data, particularly regarding the individual’s given (or lack of) consent to share their data with a government entity.
In addition to consumer-based companies, a vast industry exists that collects and sells data on individuals and businesses. Data brokers collect data from various public and private sources and, in turn, sell it for such purposes as marketing, risk analysis, and business intelligence. Additionally, think tanks, non-governmental organizations, educational institutions, and various others that conduct data collection and analysis activities produce a wealth of publicly available data. Sharing this data with a government entity could prove problematic, though. Private organizations may deem their data proprietary as it provides a competitive market advantage, and thus, sharing could impact their market positions. Moreover, private entities structure data specific to their uses, and as such, compatibility issues such as data schema variations, data integrity, and data security are likely to necessitate costly integration solutions.
When taken in aggregate, the compilation of this data has the potential to add to the compendium of intelligence data and provide rich insights to inform complex intelligence problems. It is difficult to argue against incorporating additional data, as it will enhance the intelligence analysis process. But the question of how an intelligence agency can acquire and utilize this information remains. The issues of trust, proprietary data, and compatibility will no doubt aggravate the acquisition of such information. Still, it would prove worthwhile as ingesting additional data sets into intelligence databases, particularly those enabled by generative AI, will enhance the analysis process. In doing so, intelligence organizations should exhibit caution to avoid inheriting the biases within the original data.
Conclusion
Intelligence issues are typically not the result of insufficient information collection but rather analysis. In the face of intelligence failure, deploying new and highly sophisticated collection capabilities is not necessarily the answer. AI can definitely help. The capability to rapidly identify relationships within large data sets will certainly increase the efficiency of intelligence analysis and lead to the construction of more precise assessments. Generative AI can also help, but not completely solve, the problems associated with cognitive biases and mental models. But this requires that intelligence organizations scrub the data used for training the model to ensure that it is representative of thoughtful and validated analytical methodologies that seek to avoid bias. Additionally, as data is the integral component of any AI system, intelligence organizations should seek legitimate and legal approaches to acquire private sector data while simultaneously recognizing the inherent issues of compatibility, structure, and trust associated with this data. This is a challenging undertaking, as the private sector will certainly exhibit a reluctance to provide data to a government entity for fear of impacting economic bottom lines or violating consumer trust. Addressing this condition requires robust engagement and strategies to ensure data acquisition while simultaneously balancing private organizations’ concerns.
To contend with the problem of hallucinations, higher-quality training data coupled with tools such as retrieval augmented generation (a feature that fact-checks sources) can help. In the near term, generative AI’s use for intelligence will still necessitate the analyst expending time validating and double-checking AI’s contribution, which calls into question the utility of this technology. However, the time investments now will certainly yield dividends in the long term as intelligence organizations test and experiment with various models. Adoption and experimentation of this technology will facilitate its maturation, and training data can improve, fostering greater integration of generative AI into intelligence tradecraft.
Perhaps the most pivotal facet undergirding the success of generative AI into the field of intelligence is acceptance by the analytical community. AI technologies are not replacements for humans; rather, they are enabling systems that still require a human “in the loop” to operate and to improve functionality. Intelligence operations contend with ambiguity and complexity to quickly identify shifts and changes in the environment and correspondingly to provide this information to those with the authority and power to make and execute decisions. Automation and speed do not absolve intelligence analysts of their primary duty to ask and consider the right questions at the right time to deliver timely and accurate intelligence. Intelligence professionals should not view AI as a panacea or a peril, but rather as a tool that will no doubt improve over time.
Noah B. Cooper is a career U.S. Army military intelligence officer with nearly 20 years of experience. He received an MA from John’s Hopkins University and an MA from King’s College London. The views expressed in this paper are those of the author and do not reflect the official policy or position of the U.S. Department of the Army, U.S. Department of Defense, or the U.S. government.
Image: Spc. Elaina Nieves
