Last August, several dozen military drones and tank-like robots took to the skies and roads 40 miles south of Seattle. Their mission: Find terrorists suspected of hiding among several buildings.
So many robots were involved in the operation that no human operator could keep a close eye on all of them. So they were given instructions to find—and eliminate—enemy combatants when necessary.
The mission was just an exercise, organized by the Defense Advanced Research Projects Agency, a blue-sky research division of the Pentagon; the robots were armed with nothing more lethal than radio transmitters designed to simulate interactions with both friendly and enemy robots.
The drill was one of several conducted last summer to test how artificial intelligence could help expand the use of automation in military systems, including in scenarios that are too complex and fast-moving for humans to make every critical decision. The demonstrations also reflect a subtle shift in the Pentagon’s thinking about autonomous weapons, as it becomes clearer that machines can outperform humans at parsing complex situations or operating at high speed.
US Army Futures Command General John Murray told an audience at the US Military Academy last month that swarms of robots will force military planners, policymakers, and society to think about whether a person should make every decision about using lethal force in new autonomous systems. “Is it within a human’s ability to pick out which ones have to be engaged,” and then make 100 individual decisions, Murray asked. “Is it even necessary to have a human in the loop?”
Other comments from military commanders suggest interest in giving autonomous weapons systems more agency. At a conference on AI in the Air Force last week, Michael Kanaan, director of operations for the Air Force Artificial Intelligence Accelerator at MIT and a leading voice on AI within the US military, said thinking is evolving. He says AI should perform more identifying and distinguishing potential targets while humans make high-level decisions. “I think that’s where we’re going,” Kanaan says.
At the same event, Lieutenant General Clinton Hinote, deputy chief of staff for strategy, integration, and requirements at the Pentagon, says that whether a person can be removed from the loop of a lethal autonomous system is “one of the most interesting debates that is coming, [and] has not been settled yet.”
This May, a report from the National Security Commission on Artificial Intelligence (NSCAI), an advisory group created by Congress, recommended, among other things, that the US resist calls for an international ban on the development of autonomous weapons.
Timothy Chung, the DARPA program manager in charge of the swarming project, says last summer’s exercises were designed to explore when a human drone operator should, and should not, make decisions for the autonomous systems. For example, when faced with attacks on several fronts, human control can sometimes get in the way of a mission because people are unable to react quickly enough. “Actually, the systems can do better from not having someone intervene,” Chung says.
The drones and the wheeled robots, each about the size of a large backpack, were given an overall objective, then tapped AI algorithms to devise a plan to achieve it. Some of them surrounded buildings while others carried out surveillance sweeps. A few were destroyed by simulated explosives; some identified beacons representing enemy combatants and chose to attack.
The US and other nations have used autonomy in weapons systems for decades. Some missiles can, for instance, autonomously identify and attack enemies within a given area. But rapid advances in AI algorithms will change how the military uses such systems. Off-the-shelf AI code capable of controlling robots and identifying landmarks and targets, often with high reliability, will make it possible to deploy more systems in a wider range of situations.
But as the drone demonstrations highlight, more widespread use of AI will sometimes make it more difficult to keep a human in the loop. This might prove problematic, because AI technology can harbor biases or behave unpredictably. A vision algorithm trained to recognize a particular uniform might mistakenly target someone wearing similar clothing. Chung says the swarm project presumes that AI algorithms will improve to a point where they can identify enemies with enough reliability to be trusted.
