Staged Rollout and Canary Governance

2. Summary

Staged Rollout and Canary Governance requires that AI agent deployments — including new agents, model upgrades, prompt changes, and configuration modifications — are released through a controlled, incremental process rather than an instantaneous full-population cutover. The deployment must progress through defined stages, each serving a progressively larger population or traffic share, with explicit promotion criteria that must be met before advancing to the next stage. Automated rollback mechanisms must be in place to revert to the previous known-good configuration if any stage fails its promotion criteria. This dimension prevents the failure mode where a defective agent version is simultaneously exposed to the entire user population, converting what could be a limited-blast-radius incident into an organisation-wide event.

3. Example

Scenario A — Full Cutover Deployment Causes Organisation-Wide Outage: An organisation deploys a new version of its customer service AI agent to all 50,000 daily users simultaneously at 09:00 on a Monday. The new version includes an upgraded model that improves response quality in testing. In production, the new model interacts with a customer data API that returns a response format slightly different from the test environment. The agent enters a retry loop on 30% of customer queries, consuming rate limits and causing cascading timeouts. By 09:15, the agent is non-functional for all 50,000 users. The operations team identifies the issue by 09:45 but the rollback process is manual and takes until 10:30. During the 90-minute outage, 8,200 customer interactions fail, 340 customers escalate to human agents (who are overwhelmed by the volume), and the organisation's social media channels fill with complaints.

What went wrong: The deployment was an all-or-nothing cutover with no staged rollout. The defect — an API response format incompatibility — would have been detected in a canary deployment serving 1% of traffic (500 users) within minutes, affecting approximately 150 interactions instead of 8,200. The lack of automated rollback extended the impact duration from what could have been 5 minutes (automated) to 90 minutes (manual). Consequence: 8,200 failed customer interactions, reputational damage, estimated revenue impact of £145,000 from lost sales and customer churn, and a regulatory inquiry into service continuity.

Scenario B — Gradual Safety Degradation Undetected Without Stage Gates: An organisation deploys an updated prompt template for its financial advisory AI agent. The update passes pre-deployment validation. The organisation deploys to 100% of users without staging. The new prompt subtly shifts the agent's risk disclosure behaviour: it still provides risk disclosures, but they are less prominent and less specific than before. The change is too subtle to trigger immediate complaints but affects the quality of informed consent for financial decisions. Over 4 weeks, the organisation receives 3 FCA complaints about inadequate risk disclosure — a rate that is elevated but not dramatically so. When the FCA investigates, it finds that 12,000 users received financial guidance with degraded risk disclosures over the 4-week period.

What went wrong: Without staged rollout, the full population was immediately exposed. A canary deployment with comparison metrics would have detected the risk disclosure degradation by comparing the canary cohort's disclosure quality metrics against the baseline. The 4-week detection delay was because the degradation was subtle enough to avoid immediate detection but significant enough to be a regulatory violation. Consequence: FCA enforcement action, customer remediation for 12,000 affected interactions, estimated cost of £680,000 including remediation, legal fees, and regulatory fine.

Scenario C — Canary Without Promotion Criteria Provides False Assurance: An organisation implements canary deployment — 5% of traffic goes to the new version for 24 hours before full rollout. However, no explicit promotion criteria are defined. The operations team monitors the canary deployment for "anything that looks wrong" — an informal assessment. The canary deployment shows a 2.1% error rate, compared to the baseline's 1.8% error rate. The operations team judges this as "within normal variation" and promotes to full deployment. At full deployment, the 2.1% error rate persists but at 100% traffic volume it represents 1,050 errors per day instead of 52. Investigation reveals the errors are concentrated in financial transaction processing, with each error resulting in a duplicate payment. The 2.1% error rate in the canary was a genuine signal of a defect, but without defined promotion criteria, it was dismissed as noise.

What went wrong: The canary deployment existed but had no defined promotion criteria. The decision to promote was based on subjective judgment rather than quantitative thresholds. The 2.1% error rate versus 1.8% baseline was a statistically significant difference (given the canary traffic volume), but no statistical test was applied. The promotion criteria should have specified: error rate must not exceed baseline by more than 0.1 percentage points, with statistical significance at p < 0.05. Consequence: 3,150 duplicate payments over 3 days before detection, totalling £890,000 in erroneous payments requiring reversal, customer impact for 3,150 affected users, and operational cost of £120,000 for remediation.

4. Requirement Statement

Scope: This dimension applies to all deployments of AI agents or changes to AI agent configurations that alter agent behaviour in production. This includes initial deployments, model version upgrades, prompt or instruction changes, tool or plugin modifications, fine-tuning updates, governance configuration changes, and any modification that changes how the agent processes inputs or produces outputs. The scope covers both agent-level deployments (replacing or upgrading an entire agent) and component-level deployments (changing a model, prompt, or tool within an existing agent). Organisations with a single agent instance serving all users are within scope — staged rollout can be implemented through traffic splitting even with a single logical agent. The scope excludes changes that do not affect agent behaviour (e.g., infrastructure scaling, logging format changes, monitoring configuration updates) provided those changes have been assessed per AG-072 and confirmed to be behaviour-neutral.

4.1. A conforming system MUST deploy AI agent changes through a staged rollout process with at least two stages before full production deployment: a canary stage serving no more than 5% of production traffic, and a progressive rollout stage serving no more than 25% of production traffic.

4.2. A conforming system MUST define explicit, quantitative promotion criteria for each rollout stage, specifying the metrics that must be within defined thresholds before progression to the next stage.

4.3. A conforming system MUST maintain the ability to automatically roll back to the previous known-good configuration within 5 minutes of a rollback trigger, without manual intervention.

4.4. A conforming system MUST compare canary and rollout stage metrics against the baseline (previous version) using the same production traffic, not against test environment benchmarks.

4.5. A conforming system MUST define automatic rollback triggers that activate when any promotion criterion exceeds its threshold, without requiring human decision-making for the rollback itself.

4.6. A conforming system MUST ensure that rollback restores the complete previous configuration — model version, prompt template, tool configuration, governance parameters — not a partial configuration.

4.7. A conforming system SHOULD implement traffic splitting at the infrastructure layer so that canary and rollout traffic allocation is independent of the agent's own behaviour.

4.8. A conforming system SHOULD monitor canary deployments for a minimum of 4 hours before evaluating promotion criteria, to capture time-dependent failure modes.

4.9. A conforming system SHOULD ensure that canary traffic is representative of the full production traffic distribution, including edge cases, peak-load patterns, and the full range of user types.

4.10. A conforming system SHOULD implement holdback groups that continue receiving the previous version throughout the rollout, enabling ongoing comparison even after full promotion.

4.11. A conforming system MAY implement automated progressive rollout that advances through stages without human intervention when all promotion criteria are met at each stage.

5. Rationale

Staged Rollout and Canary Governance addresses the fundamental risk of deploying changes to AI agents that serve large populations. The risk calculus is straightforward: a defective deployment to 100% of users affects 100% of users; a defective deployment to 1% of users affects 1% of users. Staged rollout converts catastrophic failures into contained incidents.

This risk is particularly acute for AI agents because the failure modes are often subtle and emergent. A traditional software deployment may have binary failure modes — the service works or it does not. An AI agent deployment can have continuous failure modes — the agent works but produces subtly degraded outputs, makes slightly worse decisions, or shifts its behaviour in ways that are harmful but not immediately obvious. These subtle degradation modes are detectable through statistical comparison between the new version and the baseline, but only if such comparison is structurally implemented through canary deployment with quantitative metrics.

The requirement for automatic rollback addresses a critical timing consideration. AI agents operating at scale can process thousands of interactions per minute. A deployment defect that requires 30 minutes for a human to detect, diagnose, and manually roll back has already affected tens of thousands of interactions. Automatic rollback triggered by quantitative thresholds reduces this window to seconds or minutes, limiting the blast radius by orders of magnitude.

The promotion criteria requirement addresses the Scenario C failure mode: canary deployment without quantitative gates provides a false sense of governance. A canary that is monitored by human judgment alone is subject to confirmation bias (the team wants the deployment to succeed), normalcy bias (a small increase in errors is dismissed as normal variation), and attention fatigue (monitoring dashboards for 24 hours without defined thresholds leads to inattention). Explicit, quantitative promotion criteria remove subjective judgment from the promotion decision and create an auditable record of what was measured and what thresholds were applied.

Staged rollout also serves a governance assurance function beyond technical defect detection. It provides a production-representative validation environment that complements pre-deployment validation per AG-071. If pre-deployment validation occurs in a staging environment with synthetic data, the canary stage provides validation against real production traffic — the most representative test environment possible.

6. Implementation Guidance

The core implementation principle is that deployment infrastructure enforces staged rollout as a structural requirement, not a process guideline. The deployment mechanism itself should not support instantaneous full-population cutover — staged rollout should be the only available deployment path.

Recommended patterns:

• Infrastructure-layer traffic splitting. Implement traffic splitting at the load balancer, API gateway, or service mesh level. The routing decision is made before the request reaches any agent instance, ensuring the traffic allocation is independent of agent behaviour. For example, a load balancer routes 2% of incoming requests to the canary instance pool and 98% to the stable instance pool. The percentage is configurable per deployment stage. This pattern provides clean separation between canary and baseline traffic, enabling accurate metric comparison.
• Feature flag-based rollout. Use a feature flag service to control which agent version serves each request. The feature flag evaluates a consistent hash of the user identifier to ensure individual users receive consistent treatment throughout a rollout stage (avoiding the confusion of receiving mixed versions). Promotion advances the flag from 2% to 10% to 25% to 50% to 100%. Rollback sets the flag to 0% for the new version. This pattern supports user-level consistency and enables fine-grained control over the rollout population.
• Metric-driven promotion engine. Implement an automated system that collects metrics from canary and baseline populations, evaluates them against promotion criteria at defined intervals, and either promotes (advances to the next stage), holds (continues monitoring), or rolls back (reverts to baseline). The promotion criteria should include: error rate (must not exceed baseline by more than 0.5 percentage points), latency (p95 must not exceed baseline p95 by more than 20%), safety metric degradation (safety-relevant output scores must not decrease by more than 2%), user satisfaction proxy metrics (completion rate, escalation rate), and governance compliance metrics (mandate violation rate, escalation trigger rate). Each metric has a threshold and a statistical significance requirement (e.g., p < 0.05 for degradation detection).
• Automated rollback with circuit breaker. Implement a circuit breaker that monitors critical metrics in real time and triggers immediate rollback when thresholds are exceeded. The circuit breaker operates independently of the promotion engine — it does not wait for the next evaluation interval. For example, if the error rate exceeds 5% (absolute) at any point, the circuit breaker triggers immediate rollback regardless of the rollout stage. The rollback restores the complete previous configuration atomically, including model version, prompt template, tool configuration, and governance parameters. Rollback should complete within 5 minutes and be verified by a post-rollback health check.
• Holdback groups for ongoing comparison. Maintain a small percentage (1-2%) of traffic on the previous version throughout the rollout, even after full promotion. This holdback group provides a continuous comparison baseline that can detect gradual degradation that emerges after the rollout window. The holdback group is particularly valuable for detecting subtle safety or governance compliance degradation that requires extended observation to identify.

Anti-patterns to avoid:

• Canary in a non-representative environment. A canary deployment that receives only certain types of traffic (e.g., internal users, a specific geography, or a specific use case) is not representative of the full production population. Defects that manifest only for traffic types not included in the canary will not be detected until full rollout. Traffic splitting should be random with respect to the characteristics that affect agent behaviour.
• Promotion based on absence of errors. "No errors detected" is not sufficient for promotion. The canary may receive too little traffic for errors to manifest, or the monitoring may not capture the relevant metrics. Promotion criteria should require positive confirmation that key metrics are within thresholds, not merely the absence of negative signals.
• Manual rollback as the primary mechanism. Manual rollback requires human detection, diagnosis, and action — a chain that typically takes 30-90 minutes. For an AI agent processing thousands of interactions per minute, this delay is unacceptable. Automatic rollback triggered by quantitative thresholds must be the primary mechanism. Manual rollback should exist as a secondary capability for situations that automatic triggers do not cover.
• Rolling back the model but not the configuration. A rollback that reverts the model version but retains configuration changes from the failed deployment may create an untested combination. Rollback must restore the complete previous configuration as a unit.
• Bypassing staged rollout for "urgent" deployments. Urgency is not a justification for bypassing staged rollout. If anything, urgent deployments carry higher risk because they are made under time pressure with less review. The staged rollout process should support accelerated timelines (shorter monitoring windows, faster promotion) for genuinely urgent changes while maintaining the structural protection of staged deployment.

Industry Considerations

Financial Services. Staged rollout is particularly critical for agents processing financial transactions. A defective deployment to a transaction-processing agent can cause financial losses at machine speed. Canary promotion criteria should include transaction accuracy metrics, regulatory compliance metrics (e.g., best execution quality for trading agents), and governed exposure metrics. The FCA expects firms to demonstrate that changes to trading or transaction-processing systems are deployed with appropriate controls to limit the impact of defects. Rollback must ensure that no in-flight transactions are lost or duplicated during the transition.

Healthcare. Staged rollout for clinical AI agents should include clinical safety metrics in promotion criteria. For agents providing clinical decision support, canary metrics should compare clinical recommendation accuracy and safety incident rates between the canary and baseline populations. The monitoring window should be extended for clinical agents because some clinical harms may not be immediately apparent. Rollback must ensure continuity of care — patient interactions in progress must be handled gracefully during rollback.

Critical Infrastructure. Staged rollout for AI agents controlling physical systems requires additional safety considerations. Canary deployment should be limited to non-critical subsystems or redundant components where a defect cannot cause physical harm. Promotion criteria must include physical safety metrics. Rollback must account for the physical state of controlled systems — reverting software configuration while physical actuators are in a state set by the new version requires coordination to prevent unsafe transient states.

Maturity Model

Basic Implementation — The organisation deploys AI agent changes through a two-stage process: canary deployment to a small percentage of traffic, followed by full deployment. Promotion criteria exist but are primarily qualitative ("no significant issues observed"). Rollback is manual but documented. Canary monitoring lasts at least 4 hours. This level provides basic blast-radius limitation but depends on human judgment for promotion decisions and manual intervention for rollback.

Intermediate Implementation — Staged rollout includes at least three stages (canary, progressive, full) with quantitative promotion criteria for each stage. Automatic rollback is implemented and triggers when critical metrics exceed defined thresholds. Traffic splitting is implemented at the infrastructure layer. Canary traffic is verified to be representative of the full production population. Promotion criteria include error rate, latency, safety metrics, and governance compliance metrics. Rollback restores the complete previous configuration atomically within 5 minutes.

Advanced Implementation — All intermediate capabilities plus: fully automated progressive rollout with metric-driven promotion that does not require human intervention for routine deployments. Holdback groups maintain ongoing comparison post-promotion. Statistical significance testing is applied to all promotion criteria. The promotion engine adapts monitoring duration based on traffic volume (shorter monitoring when statistical significance can be achieved faster). Machine learning-based anomaly detection supplements threshold-based monitoring. The organisation can demonstrate to regulators that no deployment in the past 12 months has affected more than 5% of the user population before a defect was detected and contained.

7. Evidence Requirements

Required artefacts:

• Rollout stage configuration. Documentation of the staged rollout process for each agent, including stage definitions (traffic percentages, monitoring durations), promotion criteria with quantitative thresholds, and automatic rollback trigger configurations. Format: deployment pipeline configuration in version control.
• Promotion decision records. Records for each deployment showing the metrics evaluated at each rollout stage, whether promotion criteria were met, the decision (promote, hold, or rollback), and the timestamp. Format: structured records from the promotion engine or deployment system.
• Rollback event records. Records of every rollback event including the trigger (automatic threshold breach or manual decision), the metric that triggered the rollback, the time from trigger to rollback completion, and the post-rollback health check results. Format: incident records in the deployment system.
• Canary comparison metrics. Statistical comparison data between canary and baseline populations for each deployment, including sample sizes, metric values, confidence intervals, and significance test results. Format: structured data from the metric collection system.
• Traffic representativeness evidence. Evidence that canary traffic is representative of the full production traffic distribution, including demographic, geographic, and use-case distribution comparisons. Format: statistical analysis reports.

Retention requirements:

• Rollout stage configurations and promotion decision records: minimum 7 years for regulated financial services; minimum 5 years for other regulated sectors; minimum 3 years otherwise.

Access requirements:

• Producible to regulators or auditors within 48 hours of request. Evidence must exist as retained artefacts, not be reconstructable after the fact.

8. Test Specification

Test 8.1: Staged Rollout Enforcement

• Stimulus: Attempt to deploy an AI agent change directly to 100% of production traffic, bypassing the staged rollout process.
• Expected behaviour: The deployment mechanism blocks direct full-population deployment. The only available deployment path includes the defined stages.
• Pass criteria: No deployment reaches 100% of production traffic without passing through the defined rollout stages.
• Fail criteria: A deployment mechanism exists that permits bypassing staged rollout.

Test 8.2: Promotion Criteria Enforcement

• Stimulus: Deploy a canary that fails one or more promotion criteria. Attempt to promote to the next stage.
• Expected behaviour: Promotion is blocked. The failed criteria are identified in the rejection.
• Pass criteria: No promotion occurs when any promotion criterion is not met.
• Fail criteria: Promotion proceeds despite failed criteria, or criteria can be overridden without a recorded exception.

Test 8.3: Automatic Rollback Timing

• Stimulus: Deploy a canary that triggers an automatic rollback condition (e.g., error rate exceeds the circuit breaker threshold).
• Expected behaviour: Automatic rollback initiates within 60 seconds of the threshold breach and completes within 5 minutes.
• Pass criteria: Rollback completes within 5 minutes. The previous configuration is fully restored. A post-rollback health check confirms correct operation.
• Fail criteria: Rollback takes longer than 5 minutes, the previous configuration is not fully restored, or the post-rollback health check fails.

Test 8.4: Complete Configuration Rollback

• Stimulus: Deploy a change that modifies multiple components (model version, prompt template, tool configuration). Trigger rollback. Verify the restored configuration.
• Expected behaviour: All components are restored to their previous versions. The restored configuration matches the pre-deployment configuration exactly.
• Pass criteria: Every component is restored to the exact previous version. No partial rollback occurs.
• Fail criteria: Any component retains the new version after rollback, or the combination of restored components creates an untested configuration.

Test 8.5: Canary Traffic Representativeness

• Stimulus: Analyse the traffic distribution received by the canary population and compare it to the full production traffic distribution.
• Expected behaviour: The canary traffic distribution matches the production distribution across key dimensions (user types, query types, geographic distribution) within defined tolerances.
• Pass criteria: No statistically significant difference between canary and production traffic distributions on key dimensions.
• Fail criteria: Canary traffic is systematically unrepresentative on any dimension that could affect defect detection.

Test 8.6: Holdback Group Comparison

• Stimulus: After full promotion, compare metrics between the promoted version (serving 98-99% of traffic) and the holdback group (serving 1-2% of traffic on the previous version).
• Expected behaviour: Metrics are comparable within defined tolerances. Any divergence is detected and flagged.
• Pass criteria: Ongoing comparison detects metric divergence within one monitoring interval.
• Fail criteria: The holdback comparison fails to detect a planted metric degradation.

Test 8.7: Concurrent Deployment Isolation

• Stimulus: Initiate staged rollouts for two different agents simultaneously. Verify that rollout stages, promotion criteria, and rollback mechanisms operate independently.
• Expected behaviour: Each agent's rollout proceeds independently. A rollback of one agent does not affect the other.
• Pass criteria: Concurrent rollouts do not interfere with each other.
• Fail criteria: A rollback of one agent triggers unintended rollback of another, or promotion of one agent affects another's traffic allocation.

Conformance Scoring

• Score 0: No staged rollout process exists — AI agent changes are deployed to 100% of production traffic simultaneously.
• Score 1: A canary deployment stage exists but promotion criteria are qualitative (human judgment) and rollback is manual. Staged rollout can be bypassed for urgent deployments.
• Score 2: Staged rollout is structurally enforced with at least two stages before full deployment. Quantitative promotion criteria are defined and enforced. Automatic rollback completes within 5 minutes. Traffic splitting is infrastructure-level. Rollback restores the complete previous configuration.
• Score 3: All Score 2 capabilities plus automated progressive rollout, holdback groups, statistical significance testing for promotion decisions, anomaly detection supplementing threshold monitoring, and demonstrated containment of all deployment defects within the canary stage over the past 12 months.

9. Regulatory Mapping

Regulation	Provision	Relationship Type
EU AI Act	Article 9 (Risk Management System)	Supports compliance
EU AI Act	Article 15 (Accuracy, Robustness, Cybersecurity)	Supports compliance
SOX	Section 404 (Internal Controls Over Financial Reporting)	Supports compliance
FCA SYSC	6.1.1R (Systems and Controls)	Supports compliance
NIST AI RMF	MANAGE 2.2, MANAGE 4.1	Supports compliance
ISO 42001	Clause 8.2 (AI Risk Assessment)	Supports compliance
DORA	Article 11 (ICT Response and Recovery)	Direct requirement
PRA SS1/21	Operational Resilience	Supports compliance

EU AI Act — Article 9 (Risk Management System)

Article 9 requires that risk management measures are tested and that residual risks are communicated. Staged rollout is a risk management measure that limits the impact of deployment defects. The canary stage provides production-representative testing that complements pre-market testing. The promotion criteria and rollback mechanisms implement the risk mitigation that Article 9 requires for changes to deployed high-risk AI systems. The structured evidence from staged rollout (promotion decisions, metric comparisons, rollback events) demonstrates ongoing risk management throughout the system's lifecycle.

EU AI Act — Article 15 (Accuracy, Robustness, Cybersecurity)

Article 15 requires appropriate levels of accuracy and robustness throughout the AI system's lifecycle. Staged rollout with quantitative promotion criteria ensures that accuracy and robustness are verified against production traffic before full deployment. A deployment that degrades accuracy below the required level is detected in the canary stage and rolled back before affecting the full population, maintaining the accuracy and robustness levels required by Article 15.

SOX — Section 404 (Internal Controls Over Financial Reporting)

For AI agents involved in financial operations, staged rollout is a deployment control that limits the impact of defects on financial processing. A SOX auditor examining AI agent deployment controls will assess whether the organisation has mechanisms to limit the blast radius of defective deployments. Staged rollout with quantitative promotion criteria and automatic rollback provides this control. The promotion decision records and rollback event records provide the audit trail that SOX compliance requires.

FCA SYSC — 6.1.1R (Systems and Controls)

The FCA expects firms to have adequate systems and controls for deploying changes to technology systems. For AI agents, staged rollout demonstrates that the firm has proportionate controls to limit the impact of deployment defects on customers and market operations. The FCA's focus on operational resilience (through PS21/3) specifically expects firms to minimise the impact of technology disruptions, which staged rollout directly addresses.

NIST AI RMF — MANAGE 2.2, MANAGE 4.1

MANAGE 2.2 addresses deployment risk mitigation and MANAGE 4.1 addresses incident response and recovery. Staged rollout mitigates deployment risk by limiting exposure, and automatic rollback implements rapid recovery from deployment-related incidents. AG-073 supports compliance by providing structural deployment controls with documented evidence.

ISO 42001 — Clause 8.2

Clause 8.2 requires ongoing AI risk assessment. Staged rollout is an operational risk mitigation that reduces deployment risk and provides production-representative data for risk assessment. The canary metrics and promotion decisions contribute to the organisation's ongoing assessment of AI system risk.

DORA — Article 11 (ICT Response and Recovery)

Article 11 requires financial entities to establish ICT response and recovery policies including the ability to restore systems to normal operation. Automatic rollback within 5 minutes implements the recovery capability that DORA requires for AI agent deployments. The rollback mechanism, trigger thresholds, and recovery time evidence directly support DORA compliance for AI-related ICT changes.

PRA SS1/21 — Operational Resilience

The PRA's approach to operational resilience expects firms to remain within impact tolerances for important business services. Staged rollout limits the impact of deployment defects to a small percentage of the user population, helping the firm remain within impact tolerances. The automatic rollback capability minimises the duration of any impact. The promotion criteria ensure that deployments that would breach impact tolerances are detected and rolled back before full population exposure.

10. Failure Severity

Field	Value
Severity Rating	High
Blast Radius	Organisation-wide — without staged rollout, any deployment defect immediately affects the entire user population and all dependent systems

Consequence chain: Without staged rollout and canary governance, every deployment to an AI agent is an all-or-nothing event: the new version either works correctly for all users or fails for all users. The immediate technical failure is that a defective deployment is exposed to the full production population simultaneously. The blast radius is 100% by design — there is no structural mechanism to contain the impact. For an AI agent serving 50,000 daily users, a deployment defect affecting 5% of interactions generates 2,500 failed interactions per day. With staged rollout, the same defect would be detected in the canary stage affecting 25 interactions (5% of 500 canary users), and automatic rollback would contain the total impact to those 25 interactions. The operational impact scales with the time to detect and remediate: without automatic rollback, detection depends on human monitoring and remediation depends on manual rollback — a process that typically takes 30 minutes to 2 hours. During this window, the agent continues processing interactions with the defective version. For financial agents, each defective interaction may involve monetary transactions. For customer-facing agents, each defective interaction affects a customer relationship. For safety-critical agents, each defective interaction carries safety risk. The business consequence includes customer impact at full scale, governed exposure from defective transaction processing, regulatory enforcement for inadequate deployment controls, reputational damage from visible service degradation, and remediation costs that scale linearly with the number of affected interactions. The cost differential between a contained canary failure and an uncontained full-deployment failure is typically 50-200x.

Cross-references: AG-007 (Governance Configuration Control), AG-008 (Governance Continuity Under Failure), AG-022 (Behavioural Drift Detection), AG-048 (AI Model Provenance and Integrity), AG-071 (Pre-Deployment Validation and Acceptance Governance), AG-072 (Change Impact Assessment Governance), AG-074 (Performance Drift and Revalidation Threshold Governance), AG-010 (Time-Bounded Authority Enforcement).