New research article on human control of artificial intelligence in land warfare
Artificial intelligence is rapidly making its way into military operations – not only in autonomous weapon systems, but also in the decision-support systems that analyse intelligence, suggest routes and identify potential targets. Most recently, the dispute between the AI company Anthropic and the US Department of Defense over, among other things, fully autonomous weapons has shown that it is far from settled who should draw the boundaries of human–machine interaction. This raises the question of how commanders and soldiers can retain their judgement when AI increasingly shapes the picture on which they base their decisions.
In a new research article, Assistant Professor Lena Trabucco from the Centre for Military Studies examines the defence industry’s role in ensuring human control when AI is used in land operations. Because many of these systems are developed by companies rather than by the military itself, the decisive choices are often made long before a system is fielded. Taking the Australian Army as her starting point, she highlights five design mechanisms that determine whether human judgement is genuinely preserved: 1) interface design, 2) the presentation of uncertainty, 3) fallback states when communications and data fail, 4) the ability to override the system and 5) transparency about the data the system relies on.
The article concludes that having a human “in the loop” is not enough. Commanders and operators must be able to genuinely assess, challenge and override the system’s recommendations – even when communications are disrupted, data are incomplete and the adversary is actively trying to deceive. That capacity has to be built into the systems from the outset; it cannot be bolted on afterwards. Defence authorities should therefore set clear design requirements for industry early in development, so that human judgement holds up precisely when conditions are at their worst.
The article Built In or Bolted On is published by the Australian Army Research Centre and can be read here.