Since its founding, the Defense Advanced Research Projects Agency (DARPA) has been a hub of innovation. While created as the research arm of the Department of Defense, DARPA has played an important role in some of the technologies that have become (or will become) fundamental to modern human societies.
In the 1960s and 1970s, DARPA (then known as ARPA), created ARPANET, the computer network that became the precursor to the internet. In 2003, DARP launched CALO, a project that ushered in the era of Siri and other voice-enabled assistants. In 2004, DARPA launched the Grand Challenge, a competition that set the stage for current developments and advances in self-driving cars. In 2013, DARPA launched the Brain Initiative, an ambitious project that brings together universities, tech companies and neuroscientists to discover how the brain works and develop technologies that enable the human brain to interact with the digital world.
Among DARPA’s many exciting projects is Explainable Artificial Intelligence (XAI), an initiative launched in 2016 aimed at solving one of the principal challenges of deep learning and neural networks, the subset of AI that is becoming increasing prominent in many different sectors.
In an interview with TechTalks, Dave Gunning, XAI’s Program Manager, explained the challenges faced by the industry, the goals that DARPA’s initiative aims to accomplish, and the progress that the agency has achieved so far.
AI’s black box challenge
“AI has exploded, especially machine learning and deep learning applications are everywhere, but the models are difficult for people to pick and understand,” Gunning says.
And he’s right. In the past decade, thanks to advances in deep learning and neural networks, artificial intelligence has moved from the confines of research labs and sci-fi movies and novels into practical fields such as content recommendation, machine translation and facial recognition. AI has also found its way into more critical domains such as healthcare, autonomous driving, criminal justice and the military.
But the problem with deep learning is that it is a black box, which means it is very difficult to investigate the reasoning behind the decisions it makes. The opacity of AI algorithms complicates their use, especially where mistakes can have severe impacts.
For instance, if a doctor wants to trust a treatment recommendation made by an AI algorithm, they have to know what is the reasoning behind it. The same goes for a judge who wants to pass sentence based on recidivism prediction made by a deep learning application. These are decisions that can have a deep impact on the life of the people affected by them, and the person assuming responsibility must have full visibility on the steps that go into those decisions.
Focus on intel analysis and autonomy
“DARPA always likes to work on technologies that are for the good of everybody, defense and civilians. But there’s certainly significant defense needs for this,” Gunning says.
As the testbed for its XAI initiative DARPA has chosen intelligence analysis, where analysts have to pore over huge volumes of data coming out of videos, cameras and other sources. This is an area where the development of machine learning and computer vision techniques have been very beneficial, helping automate many of the tasks analysts perform.
“Intel analysts use AI tools to keep up with the volume of data, but they don’t get an explanation of why the system is picking a particular image or a particular situation,” Gunning says.
Early in the program, Gunning met with intel analysts from the U.S. government to examine the reach and impact of the AI black box problem. “One of the users said the problem she had was the machine learning algorithms were giving her recommendations on what items to look at and investigate further, but she had to put her name on the recommendation that went forward, and if a recommendation was a mistake, she was the one that would be blamed, not the algorithm,” Gunning recalls.
Another area that DARPA’s XAI program is focusing on is autonomy. “We don’t quite see that problem out in the field yet because autonomous systems that are going to be trained through deep learning are not quite out on the field yet,” Gunning says. “But we’re anticipating that one day we’re going to have autonomous systems, in air vehicles, underwater vehicles, and automobiles that will be trained at least in part through these machine learning techniques. People would want to be able to understand what decisions they make.”
While autonomy in defense scenarios largely involves detecting objects in images and video, Gunning explains that XAI will also focus on developing broader explainability and transparency into decisions made by those autonomous systems.
“In the autonomy case, you’re training an autonomous system to have a decision policy. Most of those also involve vision or perception, but it’s really trying to explain the decision points and its decision logic as it is executing a mission,” Gunning says.
A focus on explaining AI decisions to the end user
DARPA is one of several organizations working on creating explainable AI projects. However, the XAI initiative will focus on end users, the people who will be using applications powered by deep learning and other artificial intelligence techniques.
“XAI is trying to create a portfolio of different techniques to tackle [the black box] problem, and explore how we might make these systems more understandable to end users. Early on we decided to focus on the lay user, the person who’s not a machine learning expert,” Gunning says.
This is an important point since deep learning algorithms are not only non-transparent toward users, but they’re also a mystery to their own creators. There are parallel efforts to create tools to make AI algorithms understandable to both users and developers of those systems.
“To some extent, you’re getting a lot of new deep learning development and visualization tools that are helping the developers better understand the systems, so I’m glad we didn’t focus on that,” Gunning says.
Last year, I had the chance to review some of those developer-focused efforts. One example is Seq2Seq-Vis, a tool developed by researchers at IBM and Harvard, which enables architects of AI-based machine translation systems to debug their models. Google has also launched a visual tool that helps developers probe the behavior of their AI models.
Different strategies to develop explainable AI methods
“When the program started, deep learning was starting to become prominent. But today, it’s sort of the only game in town. It’s what everybody’s interested in,” Gunning says. Therefore, XAI’s main focus will be on creating interpretation tools and techniques for deep learning algorithms and neural networks. “But we are also looking at techniques besides deep learning,” Gunning adds.
The teams working under XAI are working in three strategic domains:
Deep explanation: These are efforts aimed at altering deep learning models in ways that make them explainable. Some of the known techniques for doing this involve adding elements in the different layers of neural networks that can help better understand the complicated connections they develop during their training.
Building more interpretable models: Projects in this category focus on supplementing deep learning with other AI models that are inherently explainable. This can be probabilistic relational models or advanced decision trees, structures that were used before the advent of artificial neural networks. These complementary AI models help explain the system’s decisions, Gunning explains. I explored some of the methods that fall into this category in a feature I wrote for PCMag last year (though they were not part of the XAI initiative).
Model induction: These are model-agnostic methods at explaining AI decisions. “You just treat the model as a black box and you can experiment with it to see if you can infer something that explains its behavior,” Gunning says. Last year, I wrote an in-depth profile of RISE, a model-agnostic AI explanation method developed by researchers at Boston University under the XAI initiative. “[RISE] doesn’t necessarily have to dig inside the neural network and see what it’s paying attention to,” Gunning says. “They can just experiment with the input and develop these heatmaps that shows you what was the net paying attention to when it made those decisions.”
The explainable AI methods should be able to help generate “local” and “global” explanations. Local explanations pertain to interpreting individual decisions made by an AI system whereas global explanations provide the end user the general logic behind the behavior of the AI model.
“Part of the test here is to have the users employ any of these AI explanation systems. They should get some idea of both of local and global explanations. In some cases, the system will only give the user local explanations, but do that in a way that they build a mental model that gives them a global idea of how the system will perform. In other cases, it’s going to be an explicit global model,” Gunning says.
Parallel efforts to create explainable AI
XAI consists of 12 teams, 11 of which are working on parallel projects to create explainability methods and explainable AI models.
“The projects consist of at least two main components. First, they modify the machine learning process somehow so it produces a more explainable model, meaning if they’re using deep learning, they’re pulling out some features from the system that can be used for explanation, or maybe they’re learning one of these more interpretable models that has more semantics or more information in it that can be used for explanation,” Gunning says.
The second component is making those explanation components understandable to human users. “This includes the explanation interface, the right HCI (human-computer interaction), the right use of cognitive psychology, so you take the features of the content from the explainable model and generate the most understandable explanation you can for the end user,” Gunning says.
A team at UC Berkley is overseeing several projects being developed by XAI, including BU’s RISE. “They also have some interesting techniques on how to make the deep learning system itself more explainable,” Gunning says.
One of these techniques is to train a neural network in a way that it learns a modular architecture where each module is learning to recognize a particular concept that can then be used to generate an explanation. This is an interesting concept and a break from the current way neural networks work, in which there’s no top-down, human-imposed logic embedded into their structure.
The Berkley group is also pursuing what Gunning calls “the way out of this is more deep learning.”
“They will have one deep learning system trained to make the decisions. A second deep learning system will be trained to generate the explanation,” Gunning says.
Another effort is led by Charles River Analytics, which has a subcontract with researchers at the University of Massachusetts. “They’re using one of the model induction approaches, where they’re treating the machine learning system as a black box and then experiment with it. Their explanation system will run millions of simulation examples and try all sorts of inputs, see what the output is of the system and see if they can infer a model that can describe its behavior. And then they express that model as a probabilistic program, which is a more interpretable model, and use that to generate explanations,” Gunning says.
While most of the efforts focus on analyzing and explaining AI decisions made visual data, Gunning explains that some of the projects will be taking those efforts a step further. For instance, other teams that work with the UC Berkley group are combining RISE with techniques that use a second deep neural network to generate a verbal explanation of what the net is keying on.
“In some of their examples, they’ll show an image and ask a question about the image and the explanation will be both the RISE system highlighting what features of the image the system was looking at, but there will also be a verbal explanation of what conceptual features the net had found important,” Gunning says.
For instance, if an image classifier labels a photo as a particular breed of bird, it should be able to explain what physical features of the bird (beak length, feather structure…) contributed to the decision.
Another team at Texas A&M is working on misinformation. First, they try to train an AI model to identify fake news or misinformation by processing large amounts of text data, including social media content and news articles. The AI then employs explainability techniques to describe what features in a text document indicated that it was fake.
The 12th team in the XAI initiative is a group of cognitive psychologists at the Institute for Human-Machine Cognition in Florida. “They are not developing an explainable AI system. They are just there to dig through all the literature on the psychology of explanation and make the most practical knowledge from all that literature available to the developers. They also help us with figuring out how to measure the effectiveness of the explanations as we do the evaluation,” Gunning says.
Getting ready for XAI’s first evaluation phase
XAI’s first phase spans over 18 months of work since the program’s launch. At the end of this phase, each of the teams will do an evaluation of their work.
The evaluation will consist of two steps. First, each team will build an AI system, show it to the test user without the explanation, and use a variety of measures to evaluate how effective and reliable the model is.
Next, the team will show the same or a different group of users the AI system with the explanation, take the same set of measures, and evaluate how effective the explanation is.
The goal is to see if the explanation tools help users’ mental models and whether it enables them to better predict what the system would do in a new situation. The teams will also evaluate whether the AI explanation tools assist users in their task performance and whether users develop better trust of the AI system because of the explanation.
“The user should have a much more refined idea of when it can and can’t trust the system. So instead of blindly trusting or mistrusting the system, they have a much more calibrated idea of how to use the AI,” Gunning says.
In the first phase, the development teams design their own evaluation problems. In some cases, it’s a data analytics problem, such as a visual question-answering problem or recognizing patterns in videos and images.
In the autonomy case, it’s a variety of simulated autonomous systems. One of the groups has a simulation environment. Their problem is to explain decisions made by an autonomous system trained to find lost hikers in a national park. The UC Berkley group is using a driving simulator where they’re having the AI system generate explanation of why it is making particular turns or other decisions.
“In most of these cases, especially in this first year, the teams are finding Mechanical Turkers or college sophomores as the subjects of these experiments. Most of the problems are designed to be ones where you don’t need a lot of in-depth domain knowledge. Most of the people could be easily trained for the problem they’re trying to solve,” Gunning says.
In future phases, the projects will inch closer toward real DoD scenarios that must be tested by real military personnel. “We might actually get some test data of overhead satellite video that would be unclassified and would be closer to what a DoD intel analyst would be using,” Gunning says.
Also, as the program develops, the test problems will be consolidated and generalized into four or five common problems so the teams can get a comparison of the technologies and their performance.
XAI has already wrapped up the first phase of its work and the teams are processing their tests. “We’ll have our next program meeting in February at Berkley, where we’ll get presentations from all the groups. DARPA will put out a consolidated report with all those results,” Gunning says. “We’re getting some initial interesting results from all of these players. There are all sorts of rough edges that we’re dealing with in the bugs, the technology and the challenges, as well as how you measure the effectiveness of the explanation and what are really good test problems to tease that out. That’s not as obvious as you think it might be.”
After February, XAI will have another two and half years of the program to iterate. The teams will do another evaluation each year and gradually improve the technology, see what works and what doesn’t. “This is very much an exploration of a lot of techniques. We’ll see which ones are ready for prime time, then spin-off some transition or engineering projects to take the most promising ideas and insert them into some defense application,” Gunning says.
Monitoring the ever-changing artificial intelligence landscape
The people at XAI will have to navigate and adapt to a landscape that is developing at a very fast pace. Deep learning and neural networks have seen many innovations and changes in the past few years. This is one most fast-paced areas of the tech industry with hundreds of papers being published every year.
“A lot of the members of the team are leading researchers in the area, and we really count on them and want them to inject the latest techniques in the program,” Gunning says. “It’s a very fast-moving topic and we don’t want to develop technologies four years from now that are obsolete because the technology’s changed so much.”
Gunning also points out that a lot of the work going into the field is on how to create more robust AI systems that are less susceptible to adversarial examples, and create better architectures for deep learning. “Some of those techniques will make the system more explainable as well. So we’ll especially want to keep track of those,” Gunning says. “Similarly there’s a huge amount of work on trying to understand what’s really happening in these neural networks. Again, there will be a lot of developments there that will help us make these systems explainable.”
Another important finding in the industry is the limits of deep learning and neural networks, which have become more highlighted as the industry has matured in the past few years. This is an important topic different AI experts are debating. New York University professor and cognitive scientist Gary Marcus recently published an important essay on the topic.
“There’s a broader question in AI, that just learning alone has its limit. You need to integrate reasoning. You need more complete systems. There’s all these other cognitive capabilities that need to be added. DARPA certainly looks at that in a broader way with other programs,” Gunning says.
The controversies of applying AI to military applications
In the past year, there has been increasing focus on the ethical implications of using AI in the military and law enforcement. Employees of several large tech companies, including Google and Amazon, have protested their employers’ involvement in such projects. At Google, several engineers quit their jobs in protest. The company later declared that it would not be extending its contract in Project Maven, the DoD’s initiative in using AI in detecting objects in videos and images. Will this be affecting DARPA’s work on XAI?
“There’s a whole separate controversial issue about autonomous lethal systems. DoD’s policy is that there will always be human supervision of these systems,” Gunning clarifies. “There’s a whole separate policy question there that we don’t get engaged with at DARPA. But assume you’re going to want to use these AI systems for all sorts of non-lethal tasks. The more explainable they are, the more useful they’ll be to the military, the better the military will understand when to trust them and when not to trust them. Making these systems explainable is an important component to making them useful and ethical for DoD or anyone.”
Gunning also tells me DARPA has a separate project codenamed Assured Autonomy which is working on technologies and techniques that can make sure the behavior of AI systems remain within a specific boundary.
“There are a lot of research questions on AI safety, and DoD in general and DARPA are interested in that topic, as is the commercial industry. There’s going to continue to be a lot of work in those areas,” Gunning says.