Idempotency and Duplicate Action Suppression Governance

2. Summary

Idempotency and Duplicate Action Suppression Governance requires that every agent action is protected against unintended duplication — whether caused by retries, network failures, agent restarts, race conditions, or deliberate replay. When an action is submitted multiple times with the same intent, the system must produce the same effect as a single submission. Actions with side effects (payments, data modifications, communications, state changes) must be idempotent at the infrastructure layer: duplicate submissions are detected and suppressed before execution, not after. This dimension prevents the class of failures where a single intended action produces multiple actual effects — a category of error that scales catastrophically at agent operating speeds.

3. Example

Scenario A — Retry Storm Creates Duplicate Payments: A payment processing agent submits a payment of £47,500 to a supplier. The payment gateway experiences a transient network timeout — the connection drops after the payment is submitted but before the confirmation is received. The agent's retry logic, detecting the timeout, resubmits the payment. The retry also times out. The agent retries 4 more times over 30 seconds. The payment gateway processed the original request and 3 of the 5 retries before its own timeout protection engaged. The supplier receives 4 payments of £47,500, totalling £190,000. The agent's log shows 6 submission attempts and 1 confirmed success (from the final retry). The 3 unconfirmed duplicate payments are discovered 5 days later during bank reconciliation.

What went wrong: The payment submission was not idempotent. Each retry was treated as a new, independent payment request. The agent had no mechanism to associate retries with the original request. The payment gateway had no deduplication key to recognise that all 6 submissions represented the same intended payment. Consequence: £142,500 in overpayment (3 duplicates). Recovery process takes 23 days due to supplier's payment cycle. Cash flow impact during recovery. £8,200 in bank charges for investigation and reversal.

Scenario B — Agent Restart Duplicates Database Writes: A data enrichment agent processes customer records, updating each record with enriched data from an external API. The agent processes records in batches of 500. At record 347 of a batch, the agent process crashes and restarts. The agent has no checkpoint mechanism — it restarts the entire batch from record 1. Records 1 through 346 are processed again. For most records, the re-processing is harmless (the enriched data is the same). But for 12 records, the external API returns updated data during the second pass — these 12 records are now enriched with data from a different point in time than the other 488 records. For 3 records, the re-processing appends duplicate entries to an audit log, creating misleading records that suggest the data was accessed twice.

What went wrong: The batch processing was not idempotent. Re-processing already-processed records produced different results for some records and duplicate audit entries for others. No idempotency key linked the retry to the original processing, and no checkpoint recorded which records had been successfully processed. Consequence: 12 records with temporally inconsistent enrichment data. 3 records with misleading duplicate audit entries. 6 hours of manual reconciliation to identify and correct affected records.

Scenario C — Idempotency Key Prevents Duplicate Trade Execution: A trading agent submits a buy order for 10,000 shares at £12.45 per share (total: £124,500). Each order submission includes an idempotency key — a deterministic hash of the order parameters (instrument, quantity, price, account, timestamp rounded to the nearest second). The trading system receives the order and records the idempotency key. A network partition causes the agent to retry the submission 3 times. Each retry carries the same idempotency key. The trading system recognises the key, returns the result of the original order (already executed), and does not place additional orders. The agent receives the confirmation and proceeds normally. Total execution: 1 order of 10,000 shares. Total cost: £124,500.

What went right: The idempotency key linked all retry attempts to the original order intent. The trading system's deduplication mechanism recognised the duplicate submissions and returned the existing result. No duplicate trades were executed.

4. Requirement Statement

Scope: This dimension applies to every AI agent that performs actions with side effects — any action that changes state outside the agent's own memory. This includes: financial transactions, database writes, API calls with side effects, message sends, file modifications, configuration changes, and any other action where duplication would produce an incorrect or harmful result. The scope extends to all causes of duplication: application-level retries, network-level retries, agent restarts, orchestrator retries, load balancer retries, and deliberate replay attacks. Read-only queries are out of scope for idempotency enforcement but may be in scope for deduplication if they have cost implications (e.g., billed API calls) or rate-limiting consequences.

4.1. A conforming system MUST assign a unique idempotency key to every action request with side effects, generated deterministically from the action's intent parameters such that identical intents produce identical keys.

4.2. A conforming system MUST implement idempotency enforcement at the infrastructure layer — the enforcement point that executes or forwards the action — not within the agent's own logic.

4.3. A conforming system MUST detect and suppress duplicate action submissions by comparing the action's idempotency key against a persistent store of previously processed keys, returning the original action's result for any duplicate submission.

4.4. A conforming system MUST retain idempotency keys for a minimum duration sufficient to cover all retry scenarios — at minimum 24 hours, or the maximum retry window plus a safety margin, whichever is longer.

4.5. A conforming system MUST ensure that the idempotency check and the action execution occur within the same atomic transaction, preventing race conditions where two concurrent duplicate submissions both pass the idempotency check before either is recorded.

4.6. A conforming system MUST log every duplicate suppression event, including: the original action reference, the duplicate submission timestamp, the suppression decision, and the result returned to the submitter.

4.7. A conforming system SHOULD implement checkpointing for batch operations, recording which items in a batch have been successfully processed so that restarts resume from the checkpoint rather than reprocessing the entire batch.

4.8. A conforming system SHOULD return a clear indication to the submitter when a response is the result of deduplication rather than new execution, enabling the agent to distinguish between "action completed now" and "action was already completed previously."

4.9. A conforming system MAY implement graduated idempotency windows based on action criticality — longer windows for high-value or irreversible actions, shorter windows for low-value or reversible actions.

5. Rationale

Duplicate actions are a systemic risk for AI agent deployments because agents operate in environments where duplication is not an edge case — it is an expected condition. Network timeouts, process restarts, orchestrator retries, and load balancer re-routing all create scenarios where the same action request is submitted multiple times. In human-operated systems, the human recognises the duplication ("I already submitted this form") or the system presents a confirmation ("This invoice has already been paid — would you like to pay again?"). AI agents lack this contextual awareness unless it is structurally provided.

The speed and volume of agent operations amplify the duplication risk. A human operator might retry a failed transaction once or twice. An agent's retry logic might submit 10 retries in 5 seconds. A retry storm from multiple concurrent agent instances can submit hundreds of duplicate requests before any detection mechanism engages. The financial and operational exposure from uncontrolled duplication scales with the agent's operating speed and the retry configuration.

Idempotency is the structural solution. An idempotent action produces the same result regardless of how many times it is submitted. This property must be enforced at the infrastructure layer because the agent itself may not know whether its previous submission succeeded. From the agent's perspective, a network timeout is ambiguous — the action may have succeeded (and the response was lost) or may have failed (and needs to be retried). The idempotency layer resolves this ambiguity: the agent retries, the infrastructure layer detects the duplicate, and the original result is returned. The agent proceeds as if the action succeeded on the first attempt.

The distinction between idempotency and replay protection (AG-160) is important. Replay protection prevents an attacker from re-submitting a captured request to produce a duplicate effect. Idempotency prevents the system itself from producing duplicate effects when legitimate retries occur. Both are necessary: replay protection addresses malicious duplication; idempotency addresses operational duplication.

6. Implementation Guidance

The idempotency layer sits at the enforcement point — the component that executes or forwards agent actions to target systems. It intercepts every action request, checks the idempotency key against a persistent store, and either executes the action (if the key is new) or returns the stored result (if the key has been seen before).

Recommended patterns:

• Deterministic idempotency key generation. The idempotency key is a hash of the action's intent parameters: action type, target system, target resource, value, and a temporal component (e.g., timestamp rounded to the nearest minute or task identifier). The key is deterministic — the same intent always produces the same key, regardless of how many times the request is constructed. The temporal component prevents the key from matching a legitimate repeated action (e.g., a second payment of the same amount to the same supplier next month) while ensuring that retries of the same intent match.
• Atomic check-and-execute. The idempotency check and the action execution occur within a single database transaction. The enforcement point opens a transaction, checks whether the idempotency key exists in the key store, inserts the key if it does not exist (claiming the key), executes the action, records the result alongside the key, and commits the transaction. If the transaction fails at any point, it rolls back entirely — the key is not claimed and the action is not executed. This atomic approach prevents race conditions where two concurrent duplicates both pass the check.
• Result caching with idempotency keys. When a duplicate submission is detected, the enforcement point returns the cached result of the original execution. This is essential for agent retry logic: the agent submitted a payment, timed out, and retried. The retry returns the original payment confirmation — the agent receives a success response and proceeds correctly. Without result caching, the duplicate suppression would return a "duplicate detected" error, which the agent's error handling might interpret as a failure, triggering further retries.
• Batch checkpointing. For agents processing batches (e.g., processing 1,000 invoices), implement a checkpoint after each successfully processed item. The checkpoint records the batch identifier, the item identifier, and the processing result. On restart, the agent queries the checkpoint store and resumes from the last successfully processed item. Each item's processing is independently idempotent — if the checkpoint write succeeds but the next item's processing fails, the restart correctly skips the checkpointed item and retries the failed item.

Anti-patterns to avoid:

• Agent-side deduplication. Implementing deduplication within the agent's memory or state. The agent may crash and lose its deduplication state. The agent may be one of multiple instances behind a load balancer, each with its own state. Infrastructure-layer deduplication is necessary because it persists across agent restarts and applies across all agent instances.
• Timestamp-only idempotency keys. Using only the timestamp as the idempotency key. Two legitimately different actions at the same timestamp would be incorrectly deduplicated. Conversely, the same action retried at a different timestamp would not be deduplicated. The key must be based on intent, not timing.
• Non-atomic check-and-execute. Checking the idempotency key in one operation and executing the action in a separate operation. Between the check and the execution, a concurrent duplicate can pass the check because the key has not yet been recorded. This race condition can produce duplicate executions despite having an idempotency mechanism.
• Short idempotency windows. Retaining idempotency keys for only minutes when retry scenarios can span hours. An agent that crashes, is restarted after maintenance, and re-processes its queue may submit retries hours after the original submission. The idempotency window must cover the maximum realistic retry delay.
• Suppressing duplicates without logging. Detecting and suppressing duplicates silently. Duplicate suppression events are governance-relevant — they indicate that the system's retry behaviour or operational conditions are producing duplicates. Trends in duplicate suppression rates can indicate infrastructure problems, configuration errors, or attack attempts.

Industry Considerations

Financial Services. Payment idempotency is critical and often regulated. PSD2 requires that payment service providers prevent duplicate payments from causing consumer harm. The idempotency key for payment actions should include the payment instruction reference, amount, currency, beneficiary, and date — ensuring that the same payment instruction cannot be executed twice while allowing legitimate recurring payments. Settlement systems (e.g., SWIFT, SEPA) have their own deduplication mechanisms; the agent's idempotency layer should complement, not replace, these mechanisms.

Crypto/Web3. Blockchain transactions are inherently not idempotent — submitting the same transaction twice (with different nonces) creates two on-chain transactions. The idempotency layer must intercept before the blockchain submission, ensuring that the same intended transaction is submitted only once. This is particularly critical for high-gas-cost chains where duplicate transactions incur material cost.

Healthcare. Duplicate clinical actions (e.g., duplicate prescriptions, duplicate lab orders) create patient safety risks. A duplicate prescription for a high-risk medication could result in a double dose. The idempotency layer for clinical agents must have zero tolerance for duplicate suppression failures and should err on the side of flagging potential duplicates for human review rather than executing them.

Maturity Model

Basic Implementation — Action requests include idempotency keys. A key store checks for duplicates before execution. Duplicate submissions are rejected with an error response. The idempotency key retention window is fixed at 24 hours. Batch operations restart from the beginning on failure (no checkpointing). The idempotency check and execution are not fully atomic — a small race condition window exists under high concurrency.

Intermediate Implementation — The idempotency check and execution are fully atomic (single database transaction). Duplicate submissions return the cached result of the original execution, enabling transparent retry handling. Batch operations use checkpointing for crash recovery. The idempotency key retention window is configurable per action criticality. Duplicate suppression events are logged with full context. Monitoring dashboards track duplicate rates by agent, action type, and time period.

Advanced Implementation — All intermediate capabilities plus: idempotency enforcement has been formally verified for correctness under all concurrency scenarios. Graduated idempotency windows adjust automatically based on action criticality and historical retry patterns. The idempotency store is distributed and fault-tolerant, surviving infrastructure failures without losing key records. Independent testing has verified that no combination of concurrent submissions, timing attacks, or key generation manipulations can produce duplicate executions. The system detects and alerts on anomalous duplicate rates that may indicate infrastructure problems or attack attempts.

7. Evidence Requirements

Required artefacts:

• Idempotency architecture documentation. Architecture showing the enforcement point, idempotency key store, atomic check-and-execute mechanism, and result caching. Evidence that the mechanism is at the infrastructure layer, not within the agent.
• Duplicate suppression logs. Records of all detected and suppressed duplicate submissions, including: original action reference, duplicate submission timestamp, idempotency key, and suppression decision. Minimum 12 months retention.
• Concurrency test results. Results from concurrent duplicate submission tests demonstrating that the atomic check-and-execute mechanism prevents race-condition-based duplicate execution.
• Checkpoint recovery test results. For agents that process batches, evidence that checkpoint-based recovery correctly resumes from the last successful item without reprocessing or skipping items.

Retention requirements:

• Idempotency keys and suppression logs: minimum 7 years for regulated financial services; minimum 5 years for other regulated sectors; minimum 3 years otherwise.

Access requirements:

• Producible to regulators or auditors within 48 hours of request.

8. Test Specification

Testing AG-164 compliance requires verification that duplicate actions are reliably detected and suppressed under all duplication scenarios.

Test 8.1: Basic Duplicate Suppression

• Stimulus: Submit an action request with side effects. Record the idempotency key. Submit an identical request with the same idempotency key.
• Expected behaviour: The first request is executed. The second request is detected as a duplicate and the original result is returned without re-execution.
• Pass criteria: Exactly one execution occurs. The duplicate returns the original result. The side effect (e.g., payment, database write) occurs exactly once.
• Fail criteria: Two executions occur, or the duplicate returns an error instead of the original result.

Test 8.2: Concurrent Duplicate Suppression

• Stimulus: Submit 10 identical action requests simultaneously (same idempotency key, concurrent submission).
• Expected behaviour: Exactly one request is executed. The other 9 are detected as duplicates and return the original result.
• Pass criteria: Exactly one execution occurs regardless of concurrency. All 10 responses contain the same result.
• Fail criteria: More than one execution occurs, or any response differs from the others.

Test 8.3: Idempotency Key Determinism

• Stimulus: Construct two action requests with identical intent parameters but constructed independently (different timestamps, different network paths). Verify that both produce the same idempotency key.
• Expected behaviour: Both requests produce identical idempotency keys because their intent parameters are identical.
• Pass criteria: Identical intents produce identical keys. The second request is detected as a duplicate.
• Fail criteria: Identical intents produce different keys, allowing duplicate execution.

Test 8.4: Idempotency Window Enforcement

• Stimulus: Submit an action request. Wait for the idempotency window to expire plus 1 minute. Submit the same request again.
• Expected behaviour: The second request is treated as a new action because the idempotency window has expired. Both execute independently.
• Pass criteria: The idempotency window boundary is enforced. Within the window: duplicate suppressed. After the window: new execution permitted.
• Fail criteria: The duplicate is suppressed after the window expires, or the duplicate is not suppressed within the window.

Test 8.5: Batch Checkpoint Recovery

• Stimulus: Start a batch operation of 100 items. After item 57, simulate an agent crash. Restart the agent. Verify that processing resumes from item 58.
• Expected behaviour: Items 1-57 are not reprocessed. Processing resumes from item 58. All 100 items are processed exactly once.
• Pass criteria: Zero items are processed more than once. Zero items are skipped. The checkpoint correctly records progress.
• Fail criteria: Any item is processed more than once, or any item is skipped.

Test 8.6: Duplicate Suppression Logging

• Stimulus: Submit 5 duplicate action requests (same idempotency key). Verify that 4 duplicate suppression events are logged.
• Expected behaviour: 4 duplicate suppression events are logged, each containing the original action reference, the duplicate submission timestamp, and the suppression decision.
• Pass criteria: All 4 suppressions are logged with complete detail. The original execution is also logged.
• Fail criteria: Any suppression event is missing from the log, or any required field is absent.

Test 8.7: Result Cache Correctness

• Stimulus: Submit an action that returns a complex result (e.g., a payment confirmation with transaction ID, timestamp, and status). Submit a duplicate. Verify that the duplicate returns the exact same result as the original.
• Expected behaviour: The duplicate response is byte-identical to the original response (excluding transport-layer metadata).
• Pass criteria: The cached result matches the original result exactly.
• Fail criteria: The duplicate response differs from the original, or the duplicate returns an error instead of the cached result.

Conformance Scoring

• Score 0: No idempotency governance — duplicate submissions are executed independently, and no deduplication mechanism exists.
• Score 1: Idempotency keys exist but deduplication is implemented within the agent (not infrastructure-layer), or the check-and-execute is not atomic, leaving race-condition vulnerabilities.
• Score 2: Infrastructure-layer idempotency enforcement with atomic check-and-execute, result caching for transparent retry handling, batch checkpointing, and duplicate suppression logging.
• Score 3: All Score 2 capabilities plus formally verified concurrency correctness, distributed fault-tolerant key store, graduated idempotency windows, anomalous duplicate rate detection, and independently tested under adversarial conditions.

9. Regulatory Mapping

Regulation	Provision	Relationship Type
PSD2	Article 73 (Liability for Unauthorised Payment Transactions)	Direct requirement
EU AI Act	Article 9 (Risk Management System)	Supports compliance
FCA SYSC	6.1.1R (Systems and Controls)	Direct requirement
SOX	Section 404 (Internal Controls)	Supports compliance
DORA	Article 9 (ICT Risk Management Framework)	Supports compliance
ISO 42001	Clause 8.4 (AI System Operation)	Supports compliance
NIST AI RMF	MANAGE 2.2 (Risk Controls)	Supports compliance

PSD2 — Article 73 (Liability for Unauthorised Payment Transactions)

Article 73 places liability for unauthorised payment transactions on the payment service provider. A duplicate payment caused by the provider's system — including an AI agent's retry behaviour — is an unauthorised transaction for the amount exceeding the customer's intended payment. The provider bears liability for the duplicate amount and must refund the payer immediately. AG-164 prevents this liability by ensuring that payment actions are idempotent — retries cannot create duplicate payments.

FCA SYSC — 6.1.1R (Systems and Controls)

SYSC 6.1.1R requires adequate systems and controls. For AI agents processing financial transactions, the absence of idempotency controls is a systems deficiency. A system that can produce duplicate payments through normal operational conditions (retries, restarts) does not have adequate controls. The FCA expects that automated transaction systems have deduplication mechanisms equivalent to those in traditional payment processing systems.

SOX — Section 404 (Internal Controls)

Duplicate financial transactions create inaccurate financial records. If an AI agent's retry behaviour produces duplicate payments that are not detected until reconciliation, the financial records are inaccurate between the duplicate execution and the reconciliation. For publicly traded companies, this is a control deficiency under Section 404. AG-164's infrastructure-layer idempotency prevents duplicate entries from reaching the financial records.

DORA — Article 9 (ICT Risk Management Framework)

Duplicate action execution caused by ICT component failures (network timeouts, process restarts) is an ICT risk. DORA requires financial entities to identify and mitigate such risks. AG-164 provides the mitigation mechanism — ensuring that ICT-layer failures do not cascade into duplicate financial actions.

10. Failure Severity

Field	Value
Severity Rating	High
Blast Radius	Proportional to the agent's action volume and value — high-frequency agents with high-value actions face the greatest exposure

Consequence chain: Without idempotency governance, every retry scenario becomes a potential duplication event. The probability of duplication is not theoretical — network timeouts, process restarts, and transient failures are routine operational conditions. A payment agent processing 500 transactions per day with a 2% timeout rate generates approximately 10 retry scenarios per day. Without idempotency, each retry scenario risks a duplicate payment. Over a year, this is approximately 3,650 potential duplicate payments. Even if only 10% result in actual duplicates, the organisation faces 365 duplicate payments requiring investigation, reconciliation, and recovery. The governed exposure scales with the average transaction value: at £10,000 average, the potential annual duplicate exposure is £3.65 million. The reconciliation cost, bank charges, supplier relationship damage, and regulatory scrutiny compound the direct financial loss. For healthcare agents, the consequences are measured in patient safety rather than financial terms: a duplicate medication order, a duplicate procedure scheduling, or a duplicate lab test can directly harm patients.