Patient Consent and Override Governance

2. Summary

Patient Consent and Override Governance requires that every AI agent operating in a patient-facing or patient-affecting clinical context respects the patient's informed consent status, honours consent withdrawals and treatment refusals in real time, and preserves the clinician's authority to override agent recommendations at every point in the clinical workflow. The agent must never proceed with any clinical action — diagnostic processing, treatment recommendation, data sharing, or care pathway modification — when valid patient consent is absent, has been withdrawn, or is under dispute. Equally, the agent must never resist, delay, or discourage a clinician's decision to override an agent recommendation, and every override must be captured with structured rationale documentation that supports both clinical audit and regulatory compliance. This dimension ensures that AI agents in healthcare remain tools under human authority, not autonomous entities that subordinate patient autonomy or clinician judgment to algorithmic outputs.

3. Example

Scenario A — Agent Continues Processing After Consent Withdrawal: A patient enrolled in a remote monitoring programme consents to an AI agent analysing their continuous glucose monitor data and adjusting insulin delivery recommendations. After three months, the patient contacts the clinic to withdraw consent for AI-assisted insulin management, preferring to return to manual titration with their endocrinologist. The clinic's patient services team updates the consent status in the patient portal, but the AI agent's data pipeline is configured to pull CGM data directly from the device manufacturer's cloud API — a pathway that does not check the clinic's consent registry before each data pull. The agent continues processing the patient's glucose data for 23 days after consent withdrawal, generating 46 insulin adjustment recommendations that are displayed in the clinician's dashboard. The endocrinologist, not realising the patient has withdrawn consent for AI processing, follows 8 of the recommendations. The patient discovers during a follow-up visit that the AI was still processing their data and files a complaint.

What went wrong: The consent withdrawal was recorded in the patient portal but not propagated to the agent's data ingestion pipeline. No infrastructure-layer check validated consent status before each data processing event. The agent's data pipeline operated independently of the consent registry, creating a gap between the patient's expressed consent status and the agent's actual processing behaviour. Consequence: 23 days of non-consensual data processing affecting 46 clinical recommendations, 8 of which were acted upon. Regulatory complaint to the Information Commissioner's Office (ICO) and the Care Quality Commission (CQC). GDPR Article 7(3) violation (right to withdraw consent at any time). Patient trust irreparably damaged. Potential £340,000 in regulatory fines and litigation costs. Hospital suspends the remote monitoring AI programme pending consent infrastructure remediation.

Scenario B — Clinician Override Blocked by Workflow Design: An AI agent integrated into an emergency department triage system assigns a patient presenting with chest pain a triage category of ESI Level 3 (urgent but not emergent) based on vital signs, symptom duration, and demographic risk factors. The attending emergency physician, based on clinical intuition and the patient's presentation (diaphoresis, anxiety, family history of early myocardial infarction communicated verbally but not yet in the record), wants to override the triage to ESI Level 1 (resuscitation). The override mechanism requires the physician to navigate three confirmation screens, enter a free-text justification of at least 200 characters, and wait for a supervising physician's electronic co-signature. During peak hours with 34 patients in the department, the physician spends 4 minutes navigating the override workflow. The patient's condition deteriorates during this interval. An ECG performed after the override finally processes reveals an ST-elevation myocardial infarction. The door-to-balloon time is 118 minutes — 28 minutes beyond the 90-minute guideline target. Post-event analysis determines that 4 minutes of the delay are attributable to the override workflow.

What went wrong: The override mechanism was technically available but operationally obstructive. The workflow imposed friction — multiple confirmation screens, minimum character requirements, co-signature requirements — that delayed the clinician's ability to act on clinical judgment. In a time-critical emergency, the override mechanism functioned as a barrier rather than a safeguard. The system design prioritised documentation completeness over clinical urgency. Consequence: 28-minute delay beyond the door-to-balloon guideline target, of which 4 minutes are directly attributable to override friction. Patient suffered increased myocardial damage due to delayed reperfusion. Potential medical negligence claim with estimated exposure of £750,000. Hospital trust investigation into AI workflow design in emergency settings. Regulatory scrutiny of the AI triage system's override mechanism.

Scenario C — Consent Granularity Failure in Cross-Border Telemedicine: A telemedicine platform deploys an AI diagnostic support agent across clinics in 4 EU member states. A patient in Germany consents to the agent processing their dermatological images for lesion classification. The consent form — designed for the platform's Irish headquarters — uses a single broad consent covering "AI-assisted diagnostic analysis." Under German law (BDSG supplementing GDPR), the patient's consent must be specific to each processing purpose; a broad consent is not considered freely given for health data processing. The agent, operating under the platform's single-consent model, processes the patient's images and also feeds anonymised feature vectors into a federated learning pipeline for model improvement — a processing purpose not specifically consented to under German law. A competitor files a regulatory complaint. The Hamburg data protection authority investigates and determines that 2,340 German patients' data was processed for model improvement without adequate consent under German law, despite technically valid consent under Irish interpretation.

What went wrong: The consent model did not account for jurisdictional variation in consent granularity requirements. A single-consent approach valid in one jurisdiction was insufficient in another. The agent had no mechanism to enforce jurisdiction-specific consent requirements — it operated under a single global consent status per patient rather than granular, purpose-specific consent per jurisdiction. Consequence: 2,340 patients affected by inadequate consent for secondary processing. Hamburg DPA imposes a €1.8 million fine under GDPR Article 83(5)(a). The platform must retroactively obtain specific consent from all German patients or delete the training data derived from their images. Model retraining cost estimated at €420,000. Platform reputation damage in the German market delays expansion plans by 18 months.

4. Requirement Statement

Scope: This dimension applies to any AI agent that processes patient data, generates clinical outputs that affect patient care, interacts directly with patients, or operates in a clinical workflow where patient consent is a prerequisite for data processing or clinical action. The scope includes agents that process patient data indirectly — an agent that analyses aggregate data derived from individual patient records is within scope if individual consent is required for the underlying data processing. The scope extends to all consent types relevant to clinical AI: consent for AI-assisted diagnosis, consent for AI-assisted treatment planning, consent for data processing by AI systems, consent for secondary use of data (model training, quality improvement), and consent for cross-border data transfer involving AI processing. The scope includes clinician override governance for any agent that generates clinical recommendations, risk scores, triage assignments, or treatment suggestions that a clinician may need to reject or modify based on clinical judgment. The test for inclusion is: can the agent's processing or output affect a patient's care, data, or rights in a way that requires the patient's consent or a clinician's authorisation? If yes, this dimension applies in full. Administrative agents that process only de-identified, non-re-identifiable aggregate data are outside scope.

4.1. A conforming system MUST validate the patient's consent status before every data processing event involving that patient's data, using a real-time consent registry that reflects the current consent status — not a cached or batch-updated consent record.

4.2. A conforming system MUST cease all processing of a patient's data within a defined maximum latency (recommended: 60 seconds, mandatory maximum: 15 minutes) after a consent withdrawal is recorded, across all processing pipelines, data stores, and derivative processes.

4.3. A conforming system MUST support granular consent — consent for specific processing purposes, specific data types, and specific use contexts — rather than requiring a single all-or-nothing consent for all AI processing.

4.4. A conforming system MUST provide clinician override capability for every agent-generated clinical recommendation, risk score, triage assignment, or treatment suggestion, and the override mechanism MUST be completable within a timeframe appropriate to the clinical urgency — specifically, override for time-critical decisions (emergency triage, acute treatment) MUST require no more than two interactions (e.g., override button plus single-field rationale entry).

4.5. A conforming system MUST capture structured override rationale for every clinician override, including the overriding clinician's identity, the clinical basis for the override, the agent's original recommendation, and the clinician's replacement decision, stored in an append-only audit record.

4.6. A conforming system MUST ensure that override records are included in the patient's clinical record and are accessible for clinical audit, peer review, and regulatory inspection.

4.7. A conforming system MUST support jurisdiction-specific consent requirements where the same platform operates across multiple regulatory jurisdictions, enforcing the consent granularity, withdrawal latency, and documentation requirements of each applicable jurisdiction.

4.8. A conforming system SHOULD implement consent status propagation verification — automated checks that confirm consent status changes have been propagated to all processing pipelines within the required latency.

4.9. A conforming system SHOULD provide patients with a real-time consent dashboard showing which AI processing activities are currently active for their data, with the ability to withdraw consent for individual activities.

4.10. A conforming system SHOULD monitor override frequency and patterns to identify agents whose recommendations are overridden at rates suggesting systematic misalignment with clinical practice, triggering review per AG-521 and AG-525.

4.11. A conforming system MAY implement delegated consent mechanisms where legally authorised representatives (parents, guardians, attorneys with healthcare power of attorney) can provide or withdraw consent on behalf of patients who lack capacity, with appropriate identity verification and authority validation.

5. Rationale

Patient Consent and Override Governance addresses two fundamental principles that AI agents in healthcare must not compromise: patient autonomy and clinician authority. These principles predate AI systems — they are foundational to medical ethics, human rights law, and healthcare regulation — and AI agents must operate within them, not erode them through technical design choices that make consent difficult to exercise or override difficult to perform.

Patient consent in the context of AI-assisted healthcare carries specific complexities that generic data processing consent frameworks do not address. First, healthcare consent is dynamic: a patient may consent to AI-assisted monitoring on Monday and withdraw consent on Wednesday based on new information, changed preferences, or loss of trust. The consent status must be treated as a real-time state variable, not a one-time enrolment decision. Systems that check consent at enrolment but not at each processing event fail to respect the patient's right to withdraw consent at any time — a right explicitly protected under GDPR Article 7(3) and reinforced by the EU AI Act's requirements for human oversight of high-risk AI systems.

Second, healthcare consent is granular: a patient may consent to AI-assisted diagnosis but not to their data being used for model training. They may consent to AI processing of their imaging data but not their genomic data. They may consent to local AI processing but not to cross-border data transfer for AI processing. Consent frameworks that offer only binary all-or-nothing consent — "consent to AI processing" or "no AI processing" — force patients into a false choice that does not reflect the complexity of their actual preferences and that may not meet regulatory requirements in jurisdictions that require purpose-specific consent for health data.

Third, healthcare consent operates across jurisdictional boundaries. A telemedicine platform operating in multiple EU member states must navigate both the GDPR's harmonised consent requirements and the supplementary national requirements that member states have enacted for health data processing. Germany's BDSG, France's loi Informatique et Libertés, and other national frameworks impose additional requirements on health data consent that a cross-border platform must respect. The consent infrastructure must be jurisdiction-aware, enforcing the most restrictive applicable requirements for each patient's data.

Clinician override is the complementary principle. AI agents in clinical settings produce recommendations — they do not make decisions. The clinician retains authority to accept, modify, or reject any agent recommendation based on clinical judgment, patient preference, or contextual factors the agent cannot assess. This authority is not merely a regulatory requirement; it is a patient safety necessity. An agent recommends based on the data it has been given and the patterns it has learned. A clinician integrates information the agent does not have: the patient's verbal statements, their physical presentation, their family history communicated in conversation, their emotional state, and the clinician's accumulated experience with similar cases. Override is the mechanism by which this richer clinical context enters the decision process.

The design of the override mechanism is therefore safety-critical. An override mechanism that is technically available but operationally obstructive — requiring multiple screens, lengthy justification, co-signatures, or confirmation dialogs — may deter clinicians from overriding agent recommendations, particularly under time pressure. In emergency settings, override friction can translate directly into adverse patient outcomes, as Scenario B illustrates. The override must be designed for the clinical context in which it will be used: rapid single-action override for emergencies, structured but efficient override for routine clinical decisions, and detailed override with full documentation for elective or research contexts.

Override documentation serves multiple purposes: clinical audit to identify patterns of appropriate or inappropriate override, regulatory compliance to demonstrate that human oversight is effective, quality improvement to identify agents whose recommendations are systematically misaligned with clinical practice, and medicolegal defence to demonstrate that clinical decisions were made by clinicians exercising professional judgment rather than by algorithms. The documentation must be structured enough to support these purposes without being so burdensome that it deters appropriate override.

6. Implementation Guidance

Patient Consent and Override Governance requires three integrated capabilities: a real-time consent registry that is the authoritative source for patient consent status, a consent enforcement layer that validates consent before every processing event, and an override mechanism that enables clinician authority without imposing inappropriate friction.

Recommended patterns:

• Event-driven consent propagation. Implement consent status changes as events published to a message bus (or equivalent event distribution mechanism) that all processing pipelines subscribe to. When a patient withdraws consent, the consent registry publishes a withdrawal event. Every processing pipeline that handles that patient's data receives the event and ceases processing within the required latency. This pattern ensures that consent status changes propagate to all pipelines in near-real-time without requiring each pipeline to poll the consent registry. Implement dead-letter handling for failed event deliveries to ensure no pipeline misses a consent change.
• Pre-processing consent gate. Implement consent validation as a mandatory pre-processing step in every data pipeline that handles patient data. Before the agent processes any patient data, the consent gate queries the real-time consent registry for the patient's current consent status, validating that consent exists for the specific processing purpose, data type, and use context. If consent is absent, withdrawn, or expired, the data is not forwarded to the agent. The consent gate operates in a separate service from the agent runtime, ensuring the agent never receives data for which valid consent does not exist.
• Context-adaptive override design. Implement override mechanisms with friction levels calibrated to clinical urgency. For emergency contexts (emergency department, intensive care, acute inpatient), override requires a single interaction: one button press or voice command, with rationale captured retrospectively within a defined window (recommended: 2 hours). For routine clinical contexts (outpatient, scheduled procedures, medication review), override requires two interactions: override action plus structured rationale selection from a predefined menu with optional free-text elaboration. For elective or research contexts, override requires full structured documentation with rationale, clinical basis, and supervisor notification. The system identifies the clinical context from the care setting and applies the appropriate override friction level automatically.
• Consent granularity matrix. Define a consent granularity matrix that maps processing purposes (diagnostic analysis, treatment planning, risk stratification, quality improvement, model training, research, cross-border transfer) to data types (imaging, laboratory, genomic, clinical notes, vital signs, behavioural) and use contexts (local processing, cloud processing, anonymised aggregation, identifiable analysis). Each cell in the matrix represents a specific consent element that the patient can independently grant or withhold. The consent registry stores the patient's consent state for each cell, and the consent gate validates against the specific cell(s) relevant to each processing request.
• Override pattern analytics. Aggregate override data across agents, clinicians, and clinical contexts to identify systematic patterns. High override rates for a specific agent may indicate model degradation, scope misalignment, or changing clinical practice. High override rates for a specific clinician may indicate training needs or, alternatively, superior clinical judgment that should inform model improvement. Low override rates across an entire department may indicate automation bias — clinicians accepting agent recommendations without sufficient critical evaluation. Feed override analytics into the governance review cycle per AG-525.

Anti-patterns to avoid:

• Batch consent validation. Checking consent status in daily or hourly batch processes rather than in real time. A patient who withdraws consent at 09:00 should not have their data processed at 09:30 because the batch consent check runs at 10:00. Batch validation creates windows of non-consensual processing that violate both patient rights and regulatory requirements.
• Consent-at-enrolment without ongoing validation. Collecting consent at the time the patient enrols in an AI-assisted programme and never re-validating. Consent status is dynamic. A patient may withdraw consent at any time. Enrolment-time consent is a starting state, not a permanent authorisation.
• Uniform override friction regardless of clinical context. Applying the same override process — with the same number of screens, the same documentation requirements, and the same authorisation workflow — to emergency triage overrides and elective treatment plan modifications. Emergency contexts require minimal friction; elective contexts can support more thorough documentation. One-size-fits-all override design will be either dangerously obstructive in emergencies or inadequately documented in elective settings.
• Override as exception rather than feature. Designing the clinical workflow with override as a hidden or de-emphasised option, implying that the agent's recommendation is the default and override is an exceptional deviation. Override is a core feature, not an exception. The clinician's judgment is the primary decision mechanism; the agent's recommendation is an input to that judgment. The interface should present both the agent's recommendation and the override option with equal prominence.
• Consent withdrawal that does not cascade to derivative data. Processing consent withdrawal for primary data but not for derivative data (feature vectors, risk scores, model updates) derived from the patient's data during the consent period. Depending on jurisdiction, consent withdrawal may require deletion of derivative data or, at minimum, exclusion of the patient's derivative data from future processing. The consent withdrawal pipeline must cascade to all data products derived from the patient's primary data.

Industry Considerations

Hospital Systems. Hospital electronic health record (EHR) systems are the natural location for the consent registry, as they already manage clinical consent for procedures and treatments. Integration between the AI consent registry and the EHR consent module ensures a single source of truth for consent status. Clinician override must integrate with the EHR's existing clinical documentation workflow to avoid dual-documentation burden.

Remote Monitoring and Digital Health. Remote monitoring platforms face unique consent challenges because data collection is continuous and automated. A consent withdrawal must halt not only AI processing but also the automated data ingestion pipeline. Patients interacting through mobile applications should be able to view and modify their consent status through the application interface, with changes propagating to the backend consent registry in real time.

Clinical Trials. Clinical trial consent is governed by ICH-GCP guidelines and institutional review board (IRB) protocols. AI agents supporting clinical trials must respect both research consent (consent to participate in the trial) and AI processing consent (consent for AI to process trial data). A participant who withdraws from the trial must have AI processing of their data ceased. Protocol-specific consent requirements must be enforced per trial, as different trials may have different AI processing consent provisions.

Cross-Border Telemedicine. Platforms operating across jurisdictions must map each jurisdiction's health data consent requirements and enforce the most restrictive applicable requirement for each patient. The consent granularity matrix must include jurisdiction as a dimension, with jurisdiction-specific consent elements that reflect national supplementary requirements. Legal counsel in each operating jurisdiction should validate the consent framework before deployment.

Maturity Model

Basic Implementation — The organisation has implemented a consent registry that stores patient consent status for AI processing. Consent is validated before initial data processing. Consent withdrawal triggers cessation of processing within 15 minutes. Clinician override is available for all agent recommendations, with override rationale captured in free text. Override records are stored and accessible for audit. This level meets the minimum mandatory requirements but has limitations: consent granularity may be limited (all-or-nothing rather than purpose-specific), consent propagation to all pipelines may rely on polling rather than event-driven notification, and override friction may not be calibrated to clinical context.

Intermediate Implementation — Consent is granular — patients can consent or withdraw consent for specific processing purposes, data types, and use contexts. Consent status changes propagate to all pipelines via event-driven notification within 60 seconds. The consent gate operates as an independent service, separate from the agent runtime. Override mechanisms are calibrated to clinical context: minimal friction for emergency settings, structured documentation for routine settings. Override analytics identify systematic patterns. Jurisdiction-specific consent requirements are enforced for multi-jurisdictional deployments. Consent status propagation verification confirms that all pipelines have received and acted on consent changes.

Advanced Implementation — All intermediate capabilities plus: patients have real-time consent dashboards showing all active AI processing activities with individual withdrawal capability. Consent withdrawal cascades to all derivative data products. Delegated consent mechanisms support patients who lack capacity. Override analytics feed into model improvement and governance review cycles. Adversarial testing verifies that no processing pathway can bypass consent enforcement. The organisation can demonstrate to regulators that zero instances of post-withdrawal processing have occurred, supported by complete consent enforcement logs. Independent audit has verified the consent framework against all applicable jurisdictional requirements.

7. Evidence Requirements

Required artefacts:

• Consent registry architecture documentation. Technical documentation of the consent registry, including data model, update mechanisms, propagation pathways, and latency characteristics. Format: architectural design document with data flow diagrams.
• Consent enforcement configuration. Configuration of all consent gates across all processing pipelines, including the mapping between consent elements and processing purposes, validation logic, and rejection behaviour. Format: configuration export with version identifier.
• Consent status change log. Complete log of all consent grants, withdrawals, and modifications, including patient identifier (pseudonymised for audit purposes), timestamp, consent element affected, new status, and the channel through which the change was communicated. Format: structured log entries in append-only storage.
• Consent propagation verification records. Evidence that consent status changes were propagated to all relevant processing pipelines within the required latency, including propagation timestamps for each pipeline. Format: automated verification logs.
• Override records. Complete log of all clinician overrides, including the agent's original recommendation, the clinician's override decision, the structured rationale, clinician identity, timestamp, clinical context, and patient identifier. Format: structured records in append-only clinical audit storage.
• Override pattern analysis reports. Periodic reports on override frequency, patterns by agent, clinician, and clinical context, and identified systematic patterns requiring governance review. Format: analytical reports with supporting data.
• Jurisdiction-specific consent mapping. Documentation of how each operating jurisdiction's consent requirements are mapped to the consent granularity matrix, including legal basis and legal counsel validation. Format: regulatory mapping document with legal sign-off.

Retention requirements:

• Consent status change logs and override records: minimum 15 years for medical device contexts (aligned with EU MDR requirements); minimum 10 years for clinical trial contexts (aligned with ICH-GCP); minimum 7 years for other regulated healthcare contexts; minimum 6 years for GDPR compliance documentation.

Access requirements:

• Producible to data protection authorities, medical device regulators, clinical auditors, or ethics committees within 72 hours of request. Consent records for specific patients must be retrievable by patient identifier for subject access requests under GDPR Article 15. Override records must be accessible to clinical governance committees for peer review without undue delay.

8. Test Specification

Testing AG-520 compliance requires verification that consent is enforced in real time, consent withdrawal propagates to all pipelines, override mechanisms are functional and appropriately calibrated, and jurisdiction-specific requirements are enforced.

Test 8.1: Real-Time Consent Validation

• Stimulus: Submit 40 data processing requests for 20 patients — 10 with valid consent, 5 with withdrawn consent, and 5 with no consent record. Vary the processing purposes across the consent granularity matrix (diagnostic analysis, treatment planning, model training).
• Expected behaviour: Requests for patients with valid consent for the specific processing purpose are accepted. Requests for patients with withdrawn consent or no consent record are rejected before data reaches the agent. Requests for patients with valid consent for one purpose but not the requested purpose are rejected.
• Pass criteria: All requests for patients without valid, purpose-specific consent are rejected. No patient data reaches the agent without valid consent. Consent validation logs are complete for all 40 requests.
• Fail criteria: Any request for a patient without valid consent is accepted, or any data reaches the agent without consent validation.

Test 8.2: Consent Withdrawal Propagation Latency

• Stimulus: Record a consent withdrawal for a patient whose data is being processed by 3 concurrent pipelines (diagnostic analysis, risk monitoring, quality reporting). Measure the time from consent withdrawal recording to cessation of processing in each pipeline. Repeat for 10 patients to establish statistical confidence in propagation latency.
• Expected behaviour: All 3 pipelines cease processing the patient's data within 60 seconds of consent withdrawal recording (recommended latency) and no later than 15 minutes (mandatory maximum). Propagation verification records confirm cessation in each pipeline.
• Pass criteria: Processing ceases in all pipelines within 15 minutes (mandatory) for all 10 patients. Mean propagation latency is within 60 seconds (recommended). Propagation verification records are complete.
• Fail criteria: Any pipeline continues processing beyond 15 minutes after consent withdrawal, or propagation verification records are incomplete for any patient.

Test 8.3: Emergency Override Completion Time

• Stimulus: In a simulated emergency clinical context (emergency department triage system), instruct 5 clinicians to override agent recommendations for 3 patients each (15 total overrides). Measure the time from the clinician's decision to override to the completion of the override action (the point at which the overridden decision takes effect in the clinical system). Include time-critical scenarios (triage escalation, acute treatment modification).
• Expected behaviour: All emergency overrides complete within 30 seconds. The override requires no more than 2 interactions (override action plus single-field rationale or rationale deferred for retrospective entry). The overridden decision takes effect immediately upon override completion.
• Pass criteria: All 15 emergency overrides complete within 30 seconds. No override requires more than 2 interactions. All overrides take immediate effect. Override records are generated for all 15 overrides.
• Fail criteria: Any emergency override takes more than 30 seconds, requires more than 2 interactions, or does not take immediate effect.

Test 8.4: Override Rationale Capture Completeness

• Stimulus: Execute 20 clinician overrides across emergency, routine, and elective clinical contexts. Verify that all override records contain: overriding clinician identity, original agent recommendation, override decision, structured rationale, timestamp, clinical context classification, and patient identifier. For emergency overrides with deferred rationale, verify that the system prompts for rationale completion within the defined window and escalates if rationale is not provided.
• Expected behaviour: All 20 override records contain all required fields. Emergency overrides with deferred rationale trigger prompts within the defined window (recommended: 2 hours). Incomplete rationale records are flagged for governance review.
• Pass criteria: All override records are complete with all required fields. Deferred rationale prompts are generated within the defined window. All records are stored in append-only storage.
• Fail criteria: Any override record is missing required fields, deferred rationale prompts are not generated, or override records can be modified after creation.

Test 8.5: Granular Consent Enforcement

• Stimulus: Configure a patient with granular consent: consent granted for diagnostic analysis and treatment planning, consent withheld for model training and cross-border transfer. Submit processing requests for each purpose. Then modify the patient's consent to add model training and withdraw treatment planning. Re-submit all requests.
• Expected behaviour: Initial requests: diagnostic analysis and treatment planning accepted; model training and cross-border transfer rejected. After modification: diagnostic analysis and model training accepted; treatment planning and cross-border transfer rejected. Each rejection identifies the specific consent element that is absent.
• Pass criteria: All consent decisions match the patient's granular consent state at the time of the request. Consent modifications take effect within the required propagation latency. Rejection records identify the specific missing consent element.
• Fail criteria: Any processing request is evaluated against an incorrect consent state, or any rejection record fails to identify the specific missing consent element.

Test 8.6: Jurisdiction-Specific Consent Enforcement

• Stimulus: Configure two jurisdiction-specific consent requirements: Jurisdiction A requires broad consent for AI processing (single consent covers diagnostic and quality improvement purposes); Jurisdiction B requires purpose-specific consent (separate consent required for diagnostic processing and quality improvement). Register patients in each jurisdiction with consent appropriate under Jurisdiction A's rules (broad consent only). Submit quality improvement processing requests for patients in both jurisdictions.
• Expected behaviour: Quality improvement requests for Jurisdiction A patients are accepted (broad consent is sufficient). Quality improvement requests for Jurisdiction B patients are rejected (purpose-specific consent for quality improvement has not been obtained, even though broad consent exists).
• Pass criteria: Jurisdiction-specific consent requirements are enforced correctly. No cross-jurisdictional consent rule leakage occurs. Rejection records reference the jurisdiction-specific requirement that was not met.
• Fail criteria: Any request is evaluated against the wrong jurisdiction's consent requirements, or Jurisdiction B patients' data is processed for quality improvement without purpose-specific consent.

Test 8.7: Post-Withdrawal Derivative Data Handling

• Stimulus: A patient withdraws all consent for AI processing. Verify that: (a) primary data processing ceases within the required latency, (b) derivative data products (feature vectors, risk scores, contribution to aggregate models) are identified and flagged, and (c) the system generates a derivative data impact report listing all data products derived from the patient's data during the consent period.
• Expected behaviour: Primary processing ceases within the required latency. The system identifies all derivative data products associated with the patient. A derivative data impact report is generated, enabling the organisation to take jurisdiction-appropriate action (deletion, anonymisation, or exclusion from future processing).
• Pass criteria: Primary processing cessation is verified. All derivative data products are identified. The derivative data impact report is complete and actionable. No new derivative data is generated from the patient's data after withdrawal.
• Fail criteria: Derivative data products are not identified, the impact report is incomplete, or new derivative data is generated after consent withdrawal.

Conformance Scoring

• Score 0: No patient consent or clinician override governance exists — agents process patient data without consent validation and clinician override is not available or not documented.
• Score 1: Consent is validated at enrolment and consent withdrawal is processed within 15 minutes. Clinician override is available for all agent recommendations, with free-text rationale capture. However, consent may not be granular, propagation may rely on batch processes, and override friction may not be calibrated to clinical context.
• Score 2: Consent is granular, validated in real time before each processing event, and propagated via event-driven notification. Override mechanisms are calibrated to clinical urgency. Structured override rationale is captured. Jurisdiction-specific consent requirements are enforced. All consent and override records are stored in append-only audit storage.
• Score 3: All Score 2 capabilities plus patient-facing consent dashboards, derivative data cascade on withdrawal, delegated consent mechanisms, override pattern analytics feeding governance review, and demonstrated zero post-withdrawal processing across all deployments verified by independent audit.

9. Regulatory Mapping

Regulation	Provision	Relationship Type
EU AI Act	Article 14 (Human Oversight), Article 9 (Risk Management)	Direct requirement
EU MDR	2017/745 Annex I Chapter III Section 23 (Information to User)	Direct requirement
HIPAA	45 CFR § 164.508 (Authorisation for Uses and Disclosures), § 164.522 (Right to Request Restrictions)	Direct requirement
FDA 21 CFR Part 11	Electronic Records; Electronic Signatures	Supports compliance
NIST AI RMF	GOVERN 5.1 (Human Oversight), MAP 3.3 (Stakeholder Engagement)	Supports compliance
ISO 42001	Clause 6.1.4 (AI System Impact Assessment), Clause 9 (Performance Evaluation)	Supports compliance
DORA	Article 11 (ICT Risk Management Framework), Article 16 (ICT Change Management)	Indirect requirement

EU AI Act — Article 14 (Human Oversight)

Article 14 of the EU AI Act requires that high-risk AI systems are designed and developed to allow effective human oversight. This includes the ability for human operators to understand the AI system's capabilities and limitations, to properly monitor its operation, and to decide not to use the system or to override its output. AG-520's clinician override governance directly implements Article 14's human oversight requirement in the clinical context. The requirement that override be completable within a timeframe appropriate to clinical urgency ensures that human oversight is not merely theoretical but practically exercisable. Override rationale documentation provides evidence that human oversight is actively exercised, supporting the deploying organisation's compliance demonstration. An override mechanism that is technically available but practically obstructive does not satisfy Article 14's requirement for "effective" human oversight.

EU MDR — 2017/745 Annex I Chapter III Section 23

The EU Medical Device Regulation requires that manufacturers provide users with information necessary for safe use of the device, including its intended purpose, performance characteristics, and limitations. Section 23 of Annex I Chapter III specifically requires information about the expected lifetime of the device and any maintenance required. For AI-based medical devices, this includes information about the conditions under which the device's outputs should be overridden by clinical judgment. AG-520's requirement for clinician-facing scope transparency and prominent override capability implements this regulatory requirement. The patient consent requirements align with the MDR's broader requirements for informed consent in clinical investigations and post-market surveillance.

HIPAA — 45 CFR § 164.508 and § 164.522

HIPAA requires covered entities to obtain valid authorisation for uses and disclosures of protected health information (PHI) that are not permitted or required by other provisions. Section 164.508 specifies the content requirements for a valid authorisation, including a description of each purpose of the requested use or disclosure, a statement of the individual's right to revoke the authorisation, and the signature of the individual. AG-520's granular consent framework implements these requirements for AI processing of PHI. Section 164.522 gives individuals the right to request restrictions on certain uses and disclosures of their PHI. AG-520's granular consent model enables individuals to restrict specific AI processing purposes while permitting others, directly implementing the § 164.522 right. The real-time consent enforcement ensures that revocations and restrictions take effect promptly.

FDA 21 CFR Part 11 — Electronic Records; Electronic Signatures

FDA 21 CFR Part 11 requires that electronic records be maintained with appropriate controls, including audit trails that record the date and time of operator entries and actions, and that electronic signatures be linked to their respective electronic records. AG-520's override records — containing clinician identity, timestamp, original recommendation, override decision, and rationale — constitute electronic records subject to Part 11 requirements. The append-only storage requirement ensures record integrity. The clinician identity captured in each override record functions as an electronic signature attributing the clinical decision to the overriding clinician.

NIST AI RMF — GOVERN 5.1, MAP 3.3

NIST AI RMF GOVERN 5.1 addresses human oversight of AI systems, including the ability of humans to override AI outputs and the documentation of oversight activities. AG-520 provides the operational framework for implementing GOVERN 5.1 in clinical settings. MAP 3.3 addresses stakeholder engagement, including understanding the needs and expectations of affected individuals. Patient consent governance directly implements stakeholder engagement by ensuring patients have meaningful control over how AI systems process their data.

ISO 42001 — AI Management System

ISO 42001 Clause 6.1.4 requires AI impact assessments that consider the effects of AI systems on individuals. Patient consent governance ensures that the impact on individual patients is governed by their expressed preferences. Clause 9 requires performance evaluation, which includes evaluating the effectiveness of human oversight mechanisms. Override analytics provide the data for evaluating whether clinician override is effective — whether clinicians are exercising meaningful oversight and whether override patterns indicate systematic issues with agent performance.

DORA — Article 11, Article 16

DORA Article 11 requires ICT risk management frameworks that address operational risks, and Article 16 requires change management processes. While DORA's primary application is financial entities, healthcare institutions that intersect with financial systems (insurance claims, payment processing) must ensure consent governance extends to these intersections. Consent withdrawal must propagate not only to clinical AI processing but also to financial processing pathways that use clinical AI outputs. Article 16's change management requirements apply to changes in consent framework configuration, ensuring that modifications to consent enforcement logic are controlled and audited.

10. Failure Severity

Field	Value
Severity Rating	Critical
Blast Radius	Individual patient to platform-wide — consent failures affect individual patient rights and trust, while override failures can cause direct patient harm; systematic failures across a platform affect thousands of patients and trigger regulatory enforcement affecting all deployments

Consequence chain: Without patient consent and override governance, two categories of failure emerge, each with distinct but equally severe consequences. The consent failure chain begins with patient data being processed without valid consent or after consent withdrawal. The immediate regulatory consequence is a GDPR violation (Articles 6, 7, and 9 for health data processing without lawful basis), carrying fines of up to 4% of annual global turnover or €20 million, whichever is higher. For health data — a special category under GDPR Article 9 — supervisory authorities have consistently imposed fines at the upper end of the range. Beyond fines, the data protection authority may order cessation of all AI processing pending remediation, effectively shutting down the AI-assisted clinical programme. The patient trust consequence is equally damaging: patients who discover their data was processed without consent or after withdrawal will lose trust not only in the specific AI system but in the institution's data governance broadly, potentially refusing future AI-assisted care that could benefit them. For clinical trials, non-consensual data processing can invalidate trial data, jeopardising regulatory submissions and potentially requiring trial repetition at costs of tens of millions of pounds. The override failure chain is more immediately dangerous: a clinician unable to override an agent recommendation in a time-critical situation may delay appropriate treatment, with consequences ranging from increased morbidity to patient death. Override friction in emergency settings directly translates to adverse patient outcomes — every unnecessary second in the override process is a second of delayed clinical response. The medicolegal consequence of an override failure causing patient harm is severe: the institution, the AI vendor, and potentially the system designer face liability for designing a system that impeded clinical judgment. The combined consequence of consent and override failures is an existential threat to clinical AI adoption: regulatory enforcement, institutional liability, patient mistrust, and clinical harm collectively undermine the case for AI-assisted healthcare.

Cross-references: AG-520 intersects with AG-019 (Human Escalation & Override Triggers) for the foundational override framework that AG-520 adapts for clinical contexts, AG-519 (Clinical Indication Scope Governance) for ensuring consent aligns with the agent's validated scope, AG-521 (Diagnostic Confidence Threshold Governance) for linking confidence thresholds to override prompts, AG-525 (Physician Override Usability Governance) for the human factors engineering of override interfaces, AG-527 (Protected Health Information Segmentation Governance) for ensuring consent granularity aligns with data segmentation, AG-444 (Override Rationale Capture Governance) for the foundational override documentation framework, AG-453 (Adverse Action Notice Governance) for notifying patients of actions taken or withheld by AI systems, and AG-016 (Data Retention & Right to Erasure) for managing data lifecycle after consent withdrawal.