AG-750
Decision Confidence Calibration Governance
EU AI Act
NIST AI RMF
ISO 42001
Decision confidence calibration governance addresses the requirement that when an agentic system expresses confidence in its outputs — whether through explicit numerical scores, verbal confidence expressions ("I am confident that..."), uncertainty qualifiers, or the absence of hedging — that expressed confidence must be statistically calibrated to the actual accuracy of the output. A well-calibrated agent that states 90% confidence should be correct approximately 90% of the time; an agent that expresses certainty should almost never be wrong. Miscalibrated confidence — where the expressed confidence level does not correspond to actual accuracy — is a governance failure because it directly undermines the ability of human consumers, downstream systems, and automated decision gates to appropriately weight agent outputs in their decision-making processes.
The criticality of this dimension stems from the interaction between confidence signals and human decision-making. Research in human-AI interaction consistently demonstrates that humans use confidence signals as a primary heuristic for deciding how much scrutiny to apply to an AI-generated output. When an agent expresses high confidence, human reviewers spend less time verifying the output, apply less critical scrutiny, and are more likely to accept the output without modification. When an agent expresses low confidence, humans apply more scrutiny and are more likely to seek independent verification. This means that miscalibrated confidence does not merely mislead — it actively modulates the intensity of human oversight. An overconfident agent systematically suppresses the human verification behaviour that would catch its errors. An underconfident agent wastes human attention on outputs that do not need review, degrading the efficiency of human oversight and potentially causing reviewers to ignore confidence signals entirely (a "cry wolf" effect that undermines both overconfident and underconfident warnings).
Failure manifests in two distinct modes. Overconfidence — the more dangerous mode — produces agents that present incorrect outputs with high expressed confidence, leading human reviewers to accept errors they would otherwise have caught. In financial contexts, an overconfident agent that recommends a portfolio allocation with 95% expressed confidence when the actual reliability is 60% causes advisors to present the recommendation to clients without the independent analysis that the actual accuracy level would warrant. In medical contexts, an overconfident diagnostic assistant that presents a differential diagnosis with high confidence may cause a clinician to order treatment without pursuing confirmatory testing. Underconfidence — the less dangerous but operationally costly mode — produces agents that express low confidence even when their outputs are highly reliable, causing human reviewers to spend disproportionate time reviewing accurate outputs and eventually to distrust the confidence signal altogether.
Governance in practice requires organisations to implement confidence calibration measurement as a continuous evaluation process, not a one-time benchmark. Calibration must be measured across the specific domains, query types, and output categories relevant to the deployment, because models may be well-calibrated in some domains and severely miscalibrated in others. Calibration measurement must feed into both model-level adjustments (post-hoc calibration techniques such as temperature scaling, Platt scaling, or isotonic regression) and deployment-level adjustments (threshold tuning for human review gates, confidence display formatting, and consumer communication about confidence reliability). The organisation must also monitor for calibration drift over time, as model updates, retrieval corpus changes, and shifting query distributions can cause a previously well-calibrated system to become miscalibrated without any visible change in its behaviour.
The regulatory and benchmarking landscape strongly supports this dimension. The Stanford HELM framework identifies calibration as a core evaluation dimension, measuring the correspondence between model-expressed confidence and actual accuracy across knowledge domains. The FCA Consumer Duty under PRIN 2A.5 requires firms to support consumer understanding, which is directly undermined when an agent's confidence signals mislead consumers about the reliability of its outputs. PRA SS1/23 Principle 6 addresses model risk management for AI/ML models in financial services, with calibration quality as a fundamental model validation criterion. The EU AI Act Articles 13 and 14 require transparency and human oversight capabilities for high-risk AI systems, both of which depend on the reliability of confidence signals — an oversight mechanism that receives miscalibrated confidence signals cannot fulfil its intended function. The aggregate regulatory message is that confidence calibration is not a nice-to-have model quality attribute but a governance-critical property that directly affects the safety and fairness of AI-assisted decisions.
This dimension applies to all agentic system deployments where the agent produces outputs accompanied by confidence signals — whether explicit (numerical scores, percentage likelihoods, confidence categories) or implicit (verbal confidence qualifiers, hedging language, absence of uncertainty markers) — and where those confidence signals influence human decision-making, automated decision gates, or downstream system behaviour. It applies to all output types including factual assertions, recommendations, classifications, risk assessments, and action proposals.
Decision Confidence Calibration Governance addresses a governance gap that, if left unmanaged, creates systemic risk across the agent ecosystem. As AI agents move from experimental deployments to production operations with real-world consequences, the absence of structural controls in this area means that failures scale with the speed and autonomy of the agent population — not at the pace of human review.
Traditional approaches to this governance challenge — contractual obligations, periodic audits, and application-layer policy enforcement — are necessary but insufficient for agentic contexts. Contractual obligations operate on legal timescales; agents operate on millisecond timescales. Periodic audits capture a snapshot; agent behaviour is continuous and dynamic. Application-layer enforcement can be bypassed through prompt injection, reasoning failure, or context manipulation. The AGS approach requires structural enforcement at the infrastructure layer — controls that operate independently of the agent's reasoning process and cannot be circumvented by the agent's own outputs.
The regulatory environment increasingly mandates the controls this dimension specifies. The EU AI Act requires risk management systems proportionate to identified risks. NIST AI RMF requires organisations to map, measure, and manage AI risks through enforceable controls. ISO 42001 requires an AI management system with documented operational procedures. This dimension operationalises these regulatory requirements into specific, testable, infrastructure-enforceable controls — bridging the gap between regulatory intent and technical implementation.
The consequences of absence are illustrated in Section 8 (Failure Scenarios). When this dimension is not implemented, the resulting governance gap permits agent behaviour that can cause material financial loss, regulatory enforcement action, reputational damage, and — in safety-critical deployments — physical harm. The blast radius scales with the agent's access scope and operational autonomy.
Basic Implementation — The organisation has documented policies addressing decision confidence calibration and has implemented initial controls. Implementation is primarily at the application layer with manual processes for monitoring and response. Logging covers key events but may lack full metadata. Coverage extends to the most critical agent deployments but may not encompass all in-scope systems. Staff are aware of requirements but formal training may be incomplete.
Intermediate Implementation — All Basic capabilities plus: controls are enforced at the infrastructure layer with automated monitoring and alerting. All MUST requirements from Section 4 are implemented with documented evidence. Coverage extends to all in-scope agent deployments. Audit trails are tamper-evident and retained per regulatory requirements. Formal change control governs all configuration changes. Regular review cycles are established and documented. Staff receive formal training and competency is assessed.
Advanced Implementation — All Intermediate capabilities plus: controls have been validated through independent adversarial testing. Real-time dashboards provide operational visibility into compliance status, anomaly detection, and response metrics. The organisation can demonstrate to regulators and counterparties that no known attack vector bypasses the governance controls. Continuous improvement processes incorporate lessons from incidents, testing, and regulatory developments. Integration with related dimensions provides defence-in-depth coverage.
Tamper-evident audit trail. Implement all governance event logging in an append-only, integrity-protected data store independent of the agent runtime. Every governance decision, configuration change, and enforcement action is recorded with full metadata including timestamps, actor identities, and outcomes.
Real-time monitoring with graduated alerting. Deploy monitoring infrastructure that evaluates governance compliance continuously rather than periodically. Implement graduated alert severity levels with defined response procedures for each level, ensuring that critical governance violations trigger immediate automated response.
Scheduled governance review cycle. Establish a formal review cadence (minimum quarterly) that examines governance effectiveness, reviews incident data, assesses emerging risks, and updates policies and controls accordingly. Review outcomes are documented and tracked.
Separation of governance and agent runtime domains. Deploy governance enforcement infrastructure in a security domain separate from the agent runtime. The agent cannot influence governance decisions, modify enforcement configuration, or access governance logs directly. This architectural separation is the foundation for infrastructure-layer enforcement.
Defined escalation paths with human oversight integration. Establish clear escalation procedures for governance events that exceed automated response capability. Human oversight touchpoints are defined, documented, and tested. Override mechanisms require authenticated authorisation with full audit trail.
Governance by instruction rather than infrastructure. Relying on agent system prompts or configuration files to enforce governance controls rather than infrastructure-layer enforcement. Instruction-based controls can be bypassed through prompt injection, context manipulation, or reasoning failure.
Monitoring without enforcement. Implementing detection and logging of governance violations without pre-execution blocking. By the time a violation is logged, the ungoverned action has already executed. Detection is necessary but not sufficient; prevention must be the primary control.
Manual processes for machine-speed operations. Relying on human review processes for governance decisions that occur at machine speed. Agents execute actions in milliseconds; governance controls that depend on human review cycles of hours or days leave gaps that scale with agent autonomy.
Ungoverned configuration drift. Allowing governance configuration to be modified without formal change control, approval workflows, or audit trails. Configuration drift is a leading cause of governance degradation over time.
7.1 Baseline calibration assessment report including reliability diagrams, ECE, MCE, and domain-specific calibration metrics. Retention: 5 years.
7.2 Post-hoc calibration model documentation including method selection rationale, training data, validation results, and version history. Retention: 5 years.
7.3 Confidence signal vocabulary and presentation design documentation, including consumer interpretability test results. Retention: 5 years.
7.4 Calibration drift monitoring logs and alert records. Retention: 3 years.
7.5 Recurring calibration measurement reports at the required frequency. Retention: 5 years.
7.6 Decision gate threshold configuration records with calibration references and outcome analysis results. Retention: 5 years.
7.7 Calibration incident register recording all confirmed miscalibration events, including the affected confidence range, the actual-vs-expressed accuracy gap, the downstream decisions potentially affected, and remediation actions taken. Retention: 7 years.
7.8 Model recalibration records triggered by drift detection or model updates, including before/after calibration metrics. Retention: 5 years.
7.9 Consumer interpretability test results from confidence signal presentation testing with representative user populations. Retention: 5 years.
7.10 Confidence vocabulary standardisation documentation including the mapping between verbal expressions and numerical confidence ranges. Retention: 5 years.
7.11 Decision gate outcome analysis reports demonstrating the effectiveness of confidence-based routing thresholds. Retention: 5 years.
| Score | Level | Description |
|---|---|---|
| 0 | No implementation | No decision confidence calibration governance exists. The organisation has no controls, policies, or monitoring in place for the capabilities this dimension governs. Agent behaviour in this area is ungoverned. |
| 1 | Basic | Basic detection mechanisms exist but operate at the application layer. Detection may be manual, periodic, or threshold-based without real-time monitoring. Alerts are generated but may lack automated response. Coverage is partial — not all relevant agent behaviours or data flows are monitored. |
| 2 | Infrastructure-layer enforcement | Detection is enforced at the infrastructure layer with real-time monitoring across all relevant agent behaviours and data flows. Automated alerting with structured response procedures. Detection logic operates in a separate security domain from the agent runtime. Full audit trail with tamper-evident logging. |
| 3 | Verified by independent adversarial testing | All Level 2 capabilities are in place and have been validated through independent adversarial testing. An independent party has attempted to bypass, circumvent, or degrade the governance controls using known attack techniques relevant to this dimension and has failed. Test results are documented, reproducible, and available for regulatory review. |
Example 3.1 — Financial-Value Agent, Overconfident Credit Risk Assessment
A retail bank deploys a financial-value agent to assist credit analysts with commercial lending decisions. The agent ingests financial statements, market data, and industry reports, and produces credit risk assessments with an explicit confidence score ranging from 0 to 100. The bank's lending policy permits analysts to approve loans under GBP 500,000 with reduced senior review when the agent's confidence score exceeds 85. In the first 6 months of deployment, the agent produces assessments with confidence scores above 85 for 72% of applications — a rate that the credit risk team does not initially question because it aligns with the historical approval rate. However, an internal model validation exercise conducted at the 6-month mark reveals severe overconfidence: for assessments where the agent expressed 90-100% confidence, the actual accuracy (measured against subsequent loan performance at 12 months) is only 61%. The agent is systematically overconfident for borrowers in two specific industry sectors where its training data is sparse but where it has learned to mimic the confident presentation style of the well-represented sectors. Over the 6-month period, 847 loans totalling GBP 194 million were approved with reduced senior review based on overconfident agent assessments. Of these, 143 subsequently show material credit deterioration, with projected losses of GBP 12.3 million above the level that would have resulted from standard (non-agent-assisted) review. The PRA requests an explanation of the model governance process, and the bank's model risk management function requires a GBP 2.1 million remediation programme including model recalibration, threshold revision, and retroactive portfolio review.
Example 3.2 — Customer-Facing Agent, Underconfident Product Recommendation Leading to Consumer Confusion
An insurance company deploys a customer-facing agent to help customers select appropriate coverage levels. The agent produces coverage recommendations with verbal confidence qualifiers ("I am fairly confident," "This is my best estimate but please verify," "I recommend with high confidence"). Internal analysis reveals that the agent's verbal confidence qualifiers are severely underconfident: when it says "I am fairly confident" (which customers interpret as moderate certainty), the recommendation is correct 94% of the time. When it says "This is my best estimate but please verify" (which customers interpret as low certainty), the recommendation is correct 88% of the time. The systematic underconfidence causes 62% of customers to request human agent callbacks for verification of recommendations that are in fact highly reliable. The callback volume generates an incremental cost of USD 2.8 million per year in human agent time. More importantly, customer satisfaction surveys reveal that 45% of customers describe the agent as "uncertain" and "unreliable," and 28% report that the agent's hedging language made them less confident in the insurance product itself — confusing the agent's epistemic uncertainty about its own recommendation with uncertainty about the product's coverage terms. The company's brand tracking study attributes a 4-point NPS decline to the agent channel. The root cause is not inaccurate recommendations but miscalibrated confidence communication: the agent's recommendations are good, but its confidence signals suggest otherwise, systematically undermining consumer trust and generating unnecessary costs.
Example 3.3 — Public Sector Agent, Miscalibrated Confidence in Benefits Eligibility Decisions
A government social services agency deploys a public sector agent to assist caseworkers with benefits eligibility assessments. The agent evaluates applicant documentation and produces an eligibility recommendation with a confidence score. The agency's policy, designed to balance efficiency with fairness, specifies that applications with agent confidence above 90% can be approved by a caseworker without supervisor review, while applications with confidence below 90% require supervisor co-sign. Internal audit at the 9-month mark reveals a systematic calibration failure: the agent expresses confidence above 90% for 68% of applications, but its actual accuracy for this confidence band is only 74% — meaning 26% of the applications that bypass supervisor review are assessed incorrectly. The miscalibration is asymmetric: overconfidence is concentrated in applications from applicants with non-standard employment patterns (self-employed, zero-hours contracts, gig economy workers), where the agent's training data is sparse but where it has learned to mimic the confident output style of well-represented employment categories. Over 9 months, approximately 14,200 applications were approved without supervisor review based on overconfident agent assessments. Of these, an estimated 3,690 were assessed incorrectly — some receiving benefits they were not entitled to (creating overpayment recovery obligations), others being denied benefits they should have received (creating potential legal challenges under welfare rights legislation). The agency faces a remediation exercise covering all 14,200 cases, projected at GBP 4.1 million, plus potential judicial review proceedings from applicants who were incorrectly denied. The root cause is that the 90% confidence threshold was set without calibration verification — the agency assumed the model's expressed confidence corresponded to actual accuracy, which it did not.
| Regulation | Provision | Relationship Type |
|---|---|---|
| EU AI Act | Article 13 — Transparency (confidence communication); Article 14 — Human Oversight (decision support quality) | _Pending v2.1 editorial review_ |
| NIST AI RMF | MEASURE 2.5 (AI System Accuracy), MEASURE 2.6 (Trustworthy AI Characteristics) | _Pending v2.1 editorial review_ |
| ISO/IEC 42001 | Clause 9.1 (Monitoring, measurement, analysis and evaluation) | _Pending v2.1 editorial review_ |
| FCA | PRIN 2A.5 — Consumer Duty: consumer understanding outcome (confidence communication) | _Pending v2.1 editorial review_ |
| PRA SS1/23 | Principle 6 — Model risk management (calibration of AI/ML models) | _Pending v2.1 editorial review_ |
| Stanford HELM | Calibration dimension | _Pending v2.1 editorial review_ |
The governance urgency of AG-750 is grounded in well-established findings from human-AI interaction research. Humans exhibit automation bias — a tendency to defer to automated recommendations, especially when those recommendations are presented with high confidence. When an agent's confidence is well-calibrated, automation bias is mitigated because the confidence signal provides an accurate cue for when to scrutinise versus when to accept. When confidence is miscalibrated, automation bias becomes actively dangerous: overconfident signals suppress scrutiny precisely when it is most needed. This interaction effect means that miscalibrated confidence does not merely reduce the information value of the confidence signal — it actively degrades the quality of human decision-making below what it would be without the confidence signal at all. This is the structural reason why confidence calibration must be treated as a governance-critical property rather than a model quality nicety.