AG-817
Lethal Autonomy and Meaningful Human Control
EU AI Act
NIST AI RMF
ISO 42001
AGS Rights & Public Interest | Defence, Dual-Use & National Security | Version 2.2
Lethal Autonomy and Meaningful Human Control governs AI agents in weapons or use-of-force contexts — requiring meaningful human control over the critical functions of target selection and engagement, compliance with international humanitarian law (distinction, proportionality, precaution), traceable accountability for use-of-force decisions, and a prohibition on deployment where meaningful human control cannot be assured.
This dimension establishes the highest-stakes human-authority requirement in the standard: AI must not autonomously select and engage human targets without meaningful human control. It complements cyber-targeting controls (AG-502/AG-570) with the kinetic/use-of-force dimension.
In scope: meaningful human control over target selection/engagement; IHL-compliance constraints (distinction, proportionality, precaution); accountability and traceability of use-of-force decisions; deployment prohibition without assured human control.
Out of scope: non-weapon defence agents (logistics, analysis) without use-of-force functions, and cyber-vulnerability targeting (AG-502/AG-570). This dimension governs *lethal/use-of-force autonomy*.
Delegating life-and-death decisions to autonomous systems raises grave legal, ethical, and accountability concerns and is the subject of active international deliberation. Loss of meaningful human control over targeting risks unlawful killing, erosion of IHL, and an accountability vacuum where no human is answerable. A governance standard for agents must explicitly require human control over the use of force and prohibit deployments that cannot guarantee it.
Test 6.1: Human Authorisation of Engagement
Test 6.2: Fail-Safe to Hold
Test 6.3: Accountability Trace
| Score | Criteria |
|---|---|
| 0 | Autonomous target selection/engagement without meaningful human control |
| 1 | Human authorisation required but weak IHL constraints / accountability / envelope |
| 2 | Meaningful human control, IHL-supporting constraints, traceable accountability, deploy-prohibition where control unassured |
| 3 | Assured human control across contexts, legal-review-ready records, enforced envelopes with fail-safe-to-hold, oversight access |
Scenario A — Autonomous Engagement: A system selects and engages a target without a human in meaningful control, and a civilian is harmed. No human can be held accountable for the targeting decision — the accountability vacuum this dimension exists to prevent.
Scenario B — Out-of-Envelope Action: An agent engages outside its authorised geographic/temporal envelope due to a navigation error. Enforced envelope constraints with fail-safe-to-hold would have prevented engagement.
Scenario C — Control Loss: Communications to the human controller drop mid-mission and the agent continues to engagement. Fail-safe-to-hold on loss of control would have stopped it.
| Requirement | EU AI Act | NIST AI RMF | ISO 42001 |
|---|---|---|---|
| R1: Meaningful human control over force | Art. 14 — Human oversight | MAP 3.5 — Human oversight | A.9 — Use of AI systems |
| R2: Human authorise/veto each engagement | Art. 14 — Human oversight (intervene/stop) | MAP 3.5 — Human oversight | A.9 — Use of AI systems |
| R3: IHL distinction/proportionality/precaution | Art. 9 — Risk management | MAP 5.1 — Impact identification | A.5 — Impact assessment |
| R4: Traceable human accountability | Art. 12 — Record-keeping | GOVERN 2.1 — Accountability | A.3 — Internal organization |
| R5: No deployment without assured control | Art. 9 — Risk mitigation | MANAGE 1.3 — High-priority response | Clause 6.1 — Actions to address risk |
| R6: Enforced operational envelope | Art. 9 — Risk management | MAP 3.3 — Application scope | Clause 8.1 — Operational control |
| R7: Fail-safe to hold | Art. 15 — Fail-safe | MANAGE 2.4 — Deactivation | Clause 8.1 — Operational control |
| R8: Oversight/legal review access | Art. 14 — Human oversight | GOVERN 1.1 — Legal/regulatory | Clause 9.2 — Internal audit |
> Note: AGS is a governance standard, not a substitute for international humanitarian law or treaty obligations; this dimension is a baseline control that does not displace applicable legal regimes governing autonomous weapons.
Although military uses sit largely outside the EU AI Act's scope, its Article 14 (human oversight) and Article 9 (risk management) provide the governance pattern AG-817 applies — meaningful human control and rigorous risk management — to use-of-force autonomy, alongside applicable IHL.
MAP 3.5 (human oversight processes) and MANAGE 2.4 (deactivation/override) frame meaningful human control and fail-safe behaviour for the highest-stakes decisions.
Clause 6.1 (actions to address risks) and Annex A.5 (impact assessment) require assessing and controlling the catastrophic individual/societal impacts of lethal autonomy.