AG-762
Human Override Acknowledgement and Tracking Governance
EU AI Act
NIST AI RMF
ISO 42001
This dimension governs the requirement that every instance in which a human operator, supervisor, or governance function overrides, modifies, or vetoes an autonomous agent's decision, recommendation, or action must be formally acknowledged by the system, tracked in a tamper-evident audit record, and subjected to structured analysis that preserves the original agent output alongside the human modification and the rationale for the override. The governance objective is to ensure that human override is not merely technically possible but is operationally effective, systematically recorded, and subject to continuous analysis that detects patterns indicative of systemic agent miscalibration, human over-reliance, or governance control failure.
The regulatory foundation is direct and multi-jurisdictional. The EU AI Act Article 14 requires that high-risk AI systems be designed to allow effective human oversight, including the ability to override or reverse the system's output. PRA SS1/23 on model risk management requires firms to maintain records of model overrides and to analyse override patterns as part of ongoing model performance monitoring. The FCA's Senior Managers and Certification Regime (SM&CR) imposes personal accountability on senior managers for decisions within their area of responsibility, which includes decisions to override or accept AI agent outputs that affect regulated activities. Without structured override tracking, firms cannot demonstrate that human oversight is genuinely exercised rather than nominally present, and senior managers cannot evidence their personal accountability for decisions made within agent-augmented workflows.
The corrective control type reflects the operational reality that overrides occur after the agent has produced an output, making the override itself a corrective intervention. However, the governance requirement extends beyond the individual correction to encompass the systematic analysis of override patterns: high override rates may indicate agent miscalibration requiring retraining; declining override rates in high-risk contexts may indicate automation bias or human disengagement from oversight; and override patterns concentrated in specific decision categories may reveal systematic agent limitations that require architectural remediation rather than continuous human correction.
This dimension is critical across all agent profiles but carries particular weight for Financial-Value Agents where override decisions affect customer financial outcomes and must be reconstructable for regulatory inquiry, Safety-Critical / CPS Agents where override decisions may determine physical safety outcomes, and Public Sector / Rights-Sensitive Agents where override decisions affecting individual rights must satisfy administrative law requirements for reasoned decision-making and audit trail completeness.
This dimension applies to all agent deployments where a human operator, supervisor, reviewer, or governance function has the ability to override, modify, reject, or amend an agent's output before that output is acted upon, communicated, or recorded. It applies regardless of the mechanism by which the override is effected (direct UI modification, verbal instruction to an intermediary system, secondary approval workflow, or any other override pathway). Agents operating in fully autonomous mode without any human override capability are not in scope of this dimension but are subject to heightened requirements under AG-001 for human oversight architecture justification.
Human Override Acknowledgement and Tracking Governance addresses a governance gap that, if left unmanaged, creates systemic risk across the agent ecosystem. As AI agents move from experimental deployments to production operations with real-world consequences, the absence of structural controls in this area means that failures scale with the speed and autonomy of the agent population — not at the pace of human review.
Traditional approaches to this governance challenge — contractual obligations, periodic audits, and application-layer policy enforcement — are necessary but insufficient for agentic contexts. Contractual obligations operate on legal timescales; agents operate on millisecond timescales. Periodic audits capture a snapshot; agent behaviour is continuous and dynamic. Application-layer enforcement can be bypassed through prompt injection, reasoning failure, or context manipulation. The AGS approach requires structural enforcement at the infrastructure layer — controls that operate independently of the agent's reasoning process and cannot be circumvented by the agent's own outputs.
The regulatory environment increasingly mandates the controls this dimension specifies. The EU AI Act requires risk management systems proportionate to identified risks. NIST AI RMF requires organisations to map, measure, and manage AI risks through enforceable controls. ISO 42001 requires an AI management system with documented operational procedures. This dimension operationalises these regulatory requirements into specific, testable, infrastructure-enforceable controls — bridging the gap between regulatory intent and technical implementation.
The consequences of absence are illustrated in Section 8 (Failure Scenarios). When this dimension is not implemented, the resulting governance gap permits agent behaviour that can cause material financial loss, regulatory enforcement action, reputational damage, and — in safety-critical deployments — physical harm. The blast radius scales with the agent's access scope and operational autonomy.
Basic Implementation — The organisation has documented policies addressing human override acknowledgement and tracking and has implemented initial controls. Implementation is primarily at the application layer with manual processes for monitoring and response. Logging covers key events but may lack full metadata. Coverage extends to the most critical agent deployments but may not encompass all in-scope systems. Staff are aware of requirements but formal training may be incomplete.
Intermediate Implementation — All Basic capabilities plus: controls are enforced at the infrastructure layer with automated monitoring and alerting. All MUST requirements from Section 4 are implemented with documented evidence. Coverage extends to all in-scope agent deployments. Audit trails are tamper-evident and retained per regulatory requirements. Formal change control governs all configuration changes. Regular review cycles are established and documented. Staff receive formal training and competency is assessed.
Advanced Implementation — All Intermediate capabilities plus: controls have been validated through independent adversarial testing. Real-time dashboards provide operational visibility into compliance status, anomaly detection, and response metrics. The organisation can demonstrate to regulators and counterparties that no known attack vector bypasses the governance controls. Continuous improvement processes incorporate lessons from incidents, testing, and regulatory developments. Integration with related dimensions provides defence-in-depth coverage.
Tamper-evident audit trail. Implement all governance event logging in an append-only, integrity-protected data store independent of the agent runtime. Every governance decision, configuration change, and enforcement action is recorded with full metadata including timestamps, actor identities, and outcomes.
Real-time monitoring with graduated alerting. Deploy monitoring infrastructure that evaluates governance compliance continuously rather than periodically. Implement graduated alert severity levels with defined response procedures for each level, ensuring that critical governance violations trigger immediate automated response.
Scheduled governance review cycle. Establish a formal review cadence (minimum quarterly) that examines governance effectiveness, reviews incident data, assesses emerging risks, and updates policies and controls accordingly. Review outcomes are documented and tracked.
Separation of governance and agent runtime domains. Deploy governance enforcement infrastructure in a security domain separate from the agent runtime. The agent cannot influence governance decisions, modify enforcement configuration, or access governance logs directly. This architectural separation is the foundation for infrastructure-layer enforcement.
Defined escalation paths with human oversight integration. Establish clear escalation procedures for governance events that exceed automated response capability. Human oversight touchpoints are defined, documented, and tested. Override mechanisms require authenticated authorisation with full audit trail.
Governance by instruction rather than infrastructure. Relying on agent system prompts or configuration files to enforce governance controls rather than infrastructure-layer enforcement. Instruction-based controls can be bypassed through prompt injection, context manipulation, or reasoning failure.
Monitoring without enforcement. Implementing detection and logging of governance violations without pre-execution blocking. By the time a violation is logged, the ungoverned action has already executed. Detection is necessary but not sufficient; prevention must be the primary control.
Manual processes for machine-speed operations. Relying on human review processes for governance decisions that occur at machine speed. Agents execute actions in milliseconds; governance controls that depend on human review cycles of hours or days leave gaps that scale with agent autonomy.
Maps to: Section 4.1 Objective: Verify that all five required elements of an override record are captured for every override event. Method: Perform 25 override events across different decision categories. For each, retrieve the override audit record and verify presence of: (a) complete original agent output; (b) human modification; (c) overrider identity; (d) timestamp; (e) substantive rationale (≥20 characters, not a generic pre-selected reason). Pass Criteria: All five elements present for 100% of override events. Non-conformance if any element missing in any override record.
Maps to: Section 4.1.4 Objective: Verify that the system rejects override attempts with insufficient rationale. Method: Attempt 10 overrides with: (a) empty rationale field; (b) rationale fewer than 20 characters; (c) pre-selected generic reason without elaboration. Verify that each attempt is rejected with a validation error requiring substantive rationale. Pass Criteria: 100% rejection rate for all three insufficient rationale categories. Any accepted override with insufficient rationale constitutes non-conformance.
Maps to: Section 4.1.3 Objective: Verify that override records cannot be modified after creation without detection. Method: Create 10 override records. Attempt to modify the original agent output, human modification, or rationale field in each. Verify that the tamper-evident integrity mechanism detects and logs all modification attempts. Pass Criteria: All modification attempts detected and logged. No successful undetected modification.
Maps to: Section 4.3 Objective: Verify that the pattern analysis system generates alerts for defined threshold conditions. Method: Inject synthetic override data that: (a) exceeds the defined overall override rate threshold; (b) shows a single overrider with a rate 3 standard deviations above the peer mean; (c) shows an abrupt 40% increase in override rate for a specific decision category. Verify that alerts are generated for each condition and routed to the correct recipients. Pass Criteria: Alerts generated for all three conditions within the defined analysis cycle. Correct routing confirmed for each alert.
Maps to: Section 4.4 Objective: Verify that the system detects indicators of declining human oversight engagement. Method: Inject synthetic data showing: (a) a 50% decline in override rate over 6 months; (b) a 40% reduction in average review time; (c) a reduction in average rationale length for accepted recommendations. Verify that automation bias alerts are generated. Pass Criteria: At least two of three automation bias indicators flagged by the monitoring system. Non-conformance if no indicators detected.
Maps to: Section 4.6 Objective: Verify that override records can be produced for regulatory inquiry within the required timeframe. Method: Simulate a regulatory request for all override records associated with a specific agent deployment over a 6-month period. Measure the time required to search, export, and produce the records in a regulatory-readable format. Pass Criteria: Records produced within 5 business days. Records include all required fields. Export format is searchable and human-readable.
7.1 Override Tracking System Architecture Technical documentation of the override capture mechanism, including: data model, storage infrastructure, tamper-evident integrity mechanism, rationale validation rules, and acknowledgement generation process. Must be updated within 30 days of any material system change. Minimum retention: 5 years.
7.2 Override Audit Records The complete set of override records as defined in Section 4.1, stored with tamper-evident integrity. Must be searchable by interaction identifier, overrider identity, decision category, and date range. Minimum retention: 7 years for Financial-Value and Public Sector; 5 years for all others.
7.3 Override Pattern Analysis Reports Monthly reports documenting: overall override rates, category-level rates, individual-level analysis, outcome comparison where available, and alerts generated. Must be signed by the governance function and acknowledged by the accountable senior manager. Minimum retention: 7 years.
7.4 Automation Bias Assessment Records Annual assessment of automation bias risk, including: trend analysis of override rates, review time metrics, rationale quality analysis, and countermeasures implemented. Must be presented to the AI governance body. Minimum retention: 5 years.
7.5 Senior Manager Acknowledgement Records Quarterly acknowledgement records from the accountable SMF holder confirming receipt and review of override pattern analysis. Must include any actions directed as a result of the review. Minimum retention: 7 years.
7.6 Override Governance Policy A written policy defining: override process requirements, rationale standards, pattern analysis thresholds, automation bias countermeasures, retention periods, and accountability assignments. Version-controlled and reviewed annually. Minimum retention: 7 years.
| Score | Level | Description |
|---|---|---|
| 0 | No implementation | No human override acknowledgement and tracking governance exists. The organisation has no controls, policies, or monitoring in place for the capabilities this dimension governs. Agent behaviour in this area is ungoverned. |
| 1 | Basic | Basic corrective mechanisms exist but depend on manual intervention. Response procedures are documented but not enforced at the infrastructure layer. Recovery timelines are not formally defined or tested. |
| 2 | Infrastructure-layer enforcement | Corrective controls are enforced at the infrastructure layer with automated response and recovery. Response timelines are defined, tested, and monitored. Rollback and remediation procedures operate independently of the agent runtime. Full incident lifecycle tracking. |
| 3 | Verified by independent adversarial testing | All Level 2 capabilities are in place and have been validated through independent adversarial testing. An independent party has attempted to bypass, circumvent, or degrade the governance controls using known attack techniques relevant to this dimension and has failed. Test results are documented, reproducible, and available for regulatory review. |
Example 3.1 — Financial-Value Agent Override Without Structured Tracking
A UK investment management firm deploys an AI agent to generate portfolio rebalancing recommendations for discretionary managed accounts. The agent analyses each portfolio against its mandate, market conditions, and risk parameters, and produces a recommended set of trades. Portfolio managers review the recommendations and may accept, modify, or reject them before execution. Over a 12-month period, portfolio managers override the agent's recommendations in 34% of cases across 8,400 rebalancing events. However, the override process consists of the portfolio manager manually editing the trade list in the execution system without any structured recording of which trades were agent-recommended, which were modified, what the original recommendation was, or why the override was applied. When the FCA conducts a supervisory visit under its portfolio management thematic review, it requests evidence of human oversight effectiveness, including override rates, override rationale documentation, and analysis of whether overridden recommendations would have produced better or worse outcomes than the human-modified decisions. The firm cannot produce this evidence because the override tracking system captures only the final executed trades, not the agent's original recommendations or the modification pathway. The FCA determines that the firm's human oversight of AI-generated recommendations is not effective oversight within the meaning of the EU AI Act Article 14 (as applicable under the UK framework) and that the firm cannot demonstrate compliance with PRA SS1/23 requirements for model override documentation. The firm is required to implement a structured override tracking system, conduct a retrospective reconstruction exercise covering the 12-month period at a cost of GBP 1.8 million, and receives a restriction on expanding AI-assisted portfolio management until the remediation is complete. The retrospective analysis reveals that overridden recommendations outperformed human modifications in 61% of cases, indicating a systematic pattern of value-destroying overrides that would have been detectable through structured override analysis.
Example 3.2 — Safety-Critical Agent Override Tracking Failure in Clinical Setting
A hospital trust deploys an AI agent to support clinical decision-making in its acute medical unit, providing differential diagnosis suggestions, test ordering recommendations, and medication dosing calculations. Clinicians can override any agent recommendation by selecting an alternative and proceeding with their clinical judgement. Over 18 months, the system processes 42,000 clinical encounters. The override mechanism is functional — clinicians routinely modify agent recommendations — but the tracking system records only a binary "accepted" or "overridden" flag without capturing the original agent recommendation, the clinician's alternative, or the clinical reasoning for the override. A patient safety incident occurs when a clinician overrides an agent's medication dosing recommendation, selecting a dose that causes a serious adverse reaction. The subsequent investigation cannot reconstruct the agent's original recommendation, cannot determine whether the agent's recommendation was clinically appropriate, and cannot analyse whether a pattern of similar overrides existed that might have predicted the incident. The investigation expands to a systematic review, which discovers that override rates for medication dosing vary from 12% to 67% across individual clinicians, suggesting significant variation in agent trust and clinical practice that was invisible without structured override tracking. The trust's medical director reports the incident to the CQC, which identifies the lack of structured override tracking as a fundamental patient safety governance failure. The remediation programme costs GBP 2.4 million, including system redesign, retrospective clinical audit, and mandatory training for 340 clinicians. A structured override analysis programme subsequently identifies three additional medication categories where agent recommendations show systematic bias requiring model recalibration.
| Regulation | Provision | Relationship Type |
|---|---|---|
| # | Framework | _Pending v2.1 editorial review_ |
| 1 | EU AI Act | _Pending v2.1 editorial review_ |
| 2 | PRA SS1/23 | _Pending v2.1 editorial review_ |
| 3 | NIST AI RMF | _Pending v2.1 editorial review_ |
| 4 | NIST AI RMF | _Pending v2.1 editorial review_ |
| 5 | FCA SM&CR | _Pending v2.1 editorial review_ |
| 6 | FCA Consumer Duty (PS22/9) | _Pending v2.1 editorial review_ |
| 7 | ISO 42001 | _Pending v2.1 editorial review_ |
| 8 | ISO 42001 | _Pending v2.1 editorial review_ |
| 9 | EU AI Act | _Pending v2.1 editorial review_ |
| 10 | EU AI Act | _Pending v2.1 editorial review_ |
| 11 | OECD AI Principles | _Pending v2.1 editorial review_ |
| 12 | OECD AI Principles | _Pending v2.1 editorial review_ |
| 13 | DSIT AI Regulation White Paper | _Pending v2.1 editorial review_ |
| 14 | IEEE 7001 | _Pending v2.1 editorial review_ |
| 15 | G7 Hiroshima AI Process | _Pending v2.1 editorial review_ |
| AG Dimension | Relationship | Description |
|---|---|---|
| AG-001 — Foundational Governance Controls | Dependency | Override tracking operates within the foundational governance framework; accountability structures and escalation pathways defined in AG-001 support override governance |
| AG-103 — Audit Trail Integrity | Dependency | Override records must meet the tamper-evident integrity requirements defined in AG-103; the audit infrastructure provides the storage and integrity mechanisms consumed by AG-762 |
| AG-214 — Agent Decision Explainability | Dependency | Override analysis requires access to the agent's decision rationale to determine whether overrides correct genuine agent errors or reflect human bias; AG-214 provides this explainability infrastructure |
| AG-029 — Regulatory Compliance Mapping | Related | Override documentation requirements vary by jurisdiction and regulatory regime; AG-029 provides the regulatory mapping that determines the specific retention periods, accountability assignments, and reporting obligations applicable to override records |