Source Record Lineage Governance

2. Summary

Source Record Lineage Governance requires that every data record consumed, transformed, or produced by an AI agent carries a machine-readable lineage chain that traces the record back to its original source of truth through every transformation, aggregation, and derivation it has undergone. Without lineage, an agent's output is a black-box assertion — the organisation cannot determine which sources contributed to a decision, whether those sources were reliable, or how transformations may have altered the original data. AG-133 mandates that lineage is captured automatically at every pipeline stage, preserved through all transformations, and queryable for any record in the agent ecosystem, enabling forensic reconstruction, regulatory evidence production, and root cause analysis when agent outputs are challenged.

3. Example

Scenario A — Inability to Explain Regulatory Decision: A financial crime compliance agent flags a transaction as suspicious and files a Suspicious Activity Report (SAR) with the National Crime Agency. The counterparty's lawyers demand an explanation of the basis for the report. The compliance team attempts to trace the agent's decision: the agent's reasoning log shows it combined a "high-risk country indicator," a "transaction pattern anomaly score," and a "negative news signal." But the team cannot determine: which database the country risk indicator came from, when it was last updated, or what scoring methodology produced it; which transactions contributed to the anomaly score and over what time window; or which news article generated the negative news signal and from which news provider. Without lineage, the SAR is indefensible — the organisation cannot demonstrate a reasonable basis for the report. The counterparty's legal challenge succeeds, the SAR is withdrawn, and the NCA opens an investigation into the firm's SAR filing practices.

What went wrong: No lineage chain existed from the agent's decision inputs back to their original sources. The agent consumed derived metrics (risk score, anomaly score, news signal) without preserving the derivation chain. When challenged, the organisation could not reconstruct the basis for any of the input metrics.

Scenario B — Training Data Lineage Loss Blocks Model Deployment: An AI healthcare company develops a diagnostic model and applies for regulatory approval (FDA 510(k)). The model was trained on a dataset compiled from 14 hospital systems over 3 years. During the review, the FDA requests documentation of the complete data lineage: which patient records from which hospitals, processed through which de-identification pipeline, with which inclusion/exclusion criteria, and transformed by which feature engineering steps. The company can produce the final training dataset but cannot reconstruct its lineage — the ETL pipelines that compiled the dataset did not preserve per-record provenance. The FDA review stalls. The company estimates that reconstructing the lineage will require 8 months of engineering effort and re-processing from source data (if the source data is even still available). The delay costs the company $14 million in projected revenue and allows a competitor to reach market first.

What went wrong: ETL pipelines transformed data from 14 sources into a unified training dataset without preserving per-record lineage. Each transformation (de-identification, normalisation, feature extraction, train/test splitting) discarded provenance metadata. The final dataset existed as a standalone artefact with no connection to its origins.

Scenario C — Root Cause Analysis Failure After Incorrect Agent Output: A procurement agent recommends a supplier based on a composite score combining quality rating, delivery performance, financial stability, and ESG compliance. The selected supplier fails catastrophically — delivering defective components that cause a product recall costing £8.7 million. The post-incident investigation needs to determine: which data contributed to the supplier's high composite score, whether any of that data was inaccurate or stale, and where the evaluation went wrong. The investigation discovers that the quality rating came from a dataset that included the supplier's self-reported quality metrics (not independently verified), the delivery performance used a 5-year average that masked a recent deterioration, and the financial stability score used audited accounts that were 18 months old. But all of this is discovered through manual forensic investigation taking 6 weeks — the lineage should have been available instantly from the agent's decision record.

What went wrong: The composite score consumed by the agent was a single number. No lineage connected the score to its component metrics. No lineage connected the component metrics to their source data, freshness, or methodology. The investigation had to reconstruct lineage manually by interviewing data engineers and tracing pipeline code, rather than querying a lineage system.

4. Requirement Statement

Scope: This dimension applies to every data record that enters, traverses, or exits an AI agent ecosystem — from the original source of truth through every transformation, aggregation, derivation, embedding, retrieval, and agent consumption. The scope includes: raw source data ingested from databases, APIs, files, or streams; derived data produced by ETL pipelines, feature engineering, or aggregation; embeddings and vector representations derived from source content; training datasets compiled from multiple sources; inference-time data retrieved from any store or service; and agent outputs that become inputs to downstream systems or other agents. If a data record influences an agent's reasoning or output, the lineage of that record must be traceable from the agent's consumption point back to the original source of truth. The scope does not extend to the agent's internal reasoning process (which is covered by separate transparency and explainability dimensions) but does cover every data input to that reasoning process.

4.1. A conforming system MUST capture and store lineage metadata for every data record at every pipeline stage, including at minimum: the source record identity, the transformation applied, the transformation timestamp, the input record(s), and the output record(s).

4.2. A conforming system MUST preserve lineage through all data transformations including aggregations, joins, filters, normalisation, de-identification, feature engineering, chunking, embedding, and any other operation that produces derived data from source data.

4.3. A conforming system MUST maintain lineage chains that are queryable — given any record consumed by an agent, the system can return the complete chain of transformations and sources that produced it, from the agent's consumption point back to the original source of truth.

4.4. A conforming system MUST link agent decision records to the lineage of all data inputs consumed during the decision, enabling forensic reconstruction of which sources, at what freshness, with what transformations, contributed to any agent output.

4.5. A conforming system MUST protect lineage metadata against tampering, ensuring that lineage records are immutable once written and that any attempt to modify lineage is detected and logged.

4.6. A conforming system MUST retain lineage metadata for at least as long as the regulatory retention period applicable to the decisions the data influenced.

4.7. A conforming system SHOULD implement automated lineage capture integrated into data pipeline tooling, rather than relying on manual lineage documentation.

4.8. A conforming system SHOULD support lineage visualisation that renders the full provenance graph for any record in a human-readable format, enabling non-technical stakeholders to understand the data's journey.

4.9. A conforming system MAY implement lineage-based impact analysis that, given a source data correction or retraction, identifies all downstream records and agent decisions affected by the change.

5. Rationale

Source Record Lineage Governance exists because AI agent decisions are only as defensible as the data trail that supports them. When an agent's output is challenged — by a customer, a regulator, a counterparty, or an internal auditor — the organisation must be able to demonstrate: what data the agent consumed, where that data came from, how it was transformed between source and consumption, whether the source was reliable and current at the time of consumption, and whether the transformations were appropriate and authorised.

Without lineage, these questions are unanswerable. The organisation is left with the agent's output and no defensible basis for it. This is not acceptable in regulated environments where firms must demonstrate the basis for automated decisions (EU AI Act Article 13 — transparency, GDPR Article 22 — automated decision-making, MiFID II Article 25 — suitability), and it is not acceptable in any environment where agent decisions create financial, safety, or legal exposure.

The challenge is that modern AI data pipelines are complex multi-stage processes. Data flows from sources through ETL pipelines, feature engineering, aggregation, embedding generation, vector store indexing, retrieval, and finally into an agent's context. Each stage transforms the data, and each transformation creates an opportunity for lineage to be lost. Without deliberate lineage capture at every stage, the provenance chain breaks at the first transformation — and once broken, it cannot be reconstructed without re-processing from source data (if that data is even still available).

AG-133 addresses this by requiring that lineage capture is a structural property of every data pipeline stage, not an afterthought. Every transformation must record what went in, what came out, and what the transformation did. This creates an unbroken chain from source to consumption that can be queried at any time.

This dimension depends on AG-128 (Data Source Classification Governance) because lineage starts with source identification — the first node in the lineage graph is the classified source. It supports AG-130 (Residual Data Erasure Propagation Governance) because lineage is the map that erasure propagation follows. It enables AG-131 (Source Conflict Escalation Governance) because conflict resolution decisions must be traceable to the sources that conflicted. And it supports AG-066 (Forensic Replay and Evidence Preservation) because forensic reconstruction of agent decisions requires lineage to identify all contributing data inputs.

6. Implementation Guidance

The core implementation artefact is a lineage store — a persistent, immutable, queryable database that records the provenance of every data record in the agent ecosystem. The lineage store is populated by instrumentation at every pipeline stage, and it is queried by forensic, audit, and governance systems to trace any record back to its source.

Recommended patterns:

• Pipeline-instrumented lineage capture. Instrument every data pipeline stage (ETL, feature engineering, aggregation, chunking, embedding, retrieval) to emit lineage events to the lineage store. Each lineage event records: the input record ID(s), the output record ID(s), the transformation type and version, the transformation timestamp, and the pipeline stage identifier. This instrumentation is built into the pipeline tooling (e.g., Apache Airflow operators, Spark transformations, custom ETL functions) so that lineage capture is automatic and cannot be bypassed without modifying the pipeline infrastructure.
• Record-level lineage identifiers. Assign a globally unique lineage identifier to every data record at the point of ingestion. This identifier travels with the record through every transformation. When a transformation produces a new record from one or more input records, the new record receives a new lineage identifier, and a lineage relationship is recorded linking the new identifier to the input identifiers. When a record is consumed by an agent, the agent's decision record links to the lineage identifier of each consumed record.
• Lineage graph database. Store lineage relationships in a graph database (e.g., Neo4j, Amazon Neptune, Apache TinkerPop-compatible) that supports efficient traversal queries: "given Record X, what are all upstream sources?" and "given Source Y, what are all downstream records and agent decisions?" Graph databases are naturally suited to lineage because lineage is a directed graph with records as nodes and transformations as edges.
• Agent decision anchoring. When an agent produces an output (decision, recommendation, action), create a decision record that includes: the agent identity, the decision timestamp, the decision output, and a list of all data lineage identifiers consumed during the decision. This anchoring record is the entry point for forensic queries: "given Decision D, what data contributed to it?" The anchoring record must be immutable and tamper-evident per AG-006.

Anti-patterns to avoid:

• Lineage-free ETL pipelines. The most common failure pattern. ETL pipelines that transform, aggregate, and load data without recording any lineage metadata. Once data passes through a lineage-free pipeline, its provenance is permanently lost unless the entire pipeline is re-executed with instrumentation (if the source data is still available).
• Schema-level lineage only. Documenting that "Table X is derived from Tables Y and Z" without recording per-record relationships. Schema-level lineage tells you which sources contribute to a table but cannot answer "which specific source records contributed to this specific derived record?" Per-record lineage is required for forensic reconstruction.
• Lineage stored in pipeline logs only. Pipeline execution logs contain lineage information (which records were processed), but logs are not structured for querying. Finding the lineage of a specific record in terabytes of unstructured pipeline logs is impractical. Lineage must be stored in a structured, queryable system.
• Assuming lineage can be reconstructed. Organisations sometimes assume that because they have the pipeline code and the source data, they can reconstruct lineage after the fact. This assumption fails when: source data has been modified since the original processing, pipeline code has been updated, configuration parameters have changed, or intermediate pipeline state is not reproducible. Lineage must be captured at processing time, not reconstructed later.
• Lineage metadata that can be modified. If lineage records can be edited or deleted, the entire lineage system is untrustworthy. An adversary (or an employee covering a mistake) could modify lineage to show a different provenance than the actual one. Lineage metadata must be immutable — append-only with tamper detection per AG-006.

Industry Considerations

Financial Services. Regulatory requirements for demonstrating the basis of automated decisions (MiFID II suitability, MAR surveillance, AML/CTF SAR filing) map directly to lineage requirements. The FCA expects firms to be able to explain the basis for any automated decision affecting a client or the market. Lineage provides the evidentiary infrastructure for this explanation. Firms should integrate lineage with existing model risk management frameworks (SS1/23) to demonstrate end-to-end traceability from source data through model inference to client-facing output.

Healthcare. FDA and EMA regulatory submissions for AI-based medical devices require complete training data provenance. AG-133 lineage provides the structured metadata needed for these submissions. Clinical decision support systems must also maintain lineage for inference-time data to support clinical accountability — a clinician who follows an AI recommendation needs to be able to understand (at least at a high level) what data informed the recommendation.

Legal and Compliance. GDPR Article 15 (right of access) and Article 22 (automated decision-making) give data subjects the right to understand how automated decisions about them were made. Lineage provides the technical infrastructure to answer these requests: "the decision was based on data from Sources A, B, and C, transformed through Pipeline P, and consumed by Agent Q at time T."

Maturity Model

Basic Implementation — The organisation maintains schema-level lineage documentation showing which data sources feed which derived datasets and which agent systems consume them. Per-record lineage is captured for high-risk data flows (e.g., financial decision inputs, clinical data). Agent decision records include references to the data sources consulted, though not necessarily per-record lineage identifiers. Lineage records are stored in a structured format. This level meets the minimum mandatory requirements but lacks comprehensive per-record lineage across all pipeline stages.

Intermediate Implementation — Automated lineage capture is integrated into all data pipeline stages. Every record carries a lineage identifier from ingestion through all transformations. Lineage relationships are stored in a graph database supporting efficient traversal queries. Agent decision records are anchored to specific lineage identifiers. Lineage metadata is immutable with tamper detection. Lineage queries can reconstruct the full provenance chain for any record within seconds.

Advanced Implementation — All intermediate capabilities plus: lineage-based impact analysis automatically identifies all affected downstream records and agent decisions when a source record is corrected or retracted. Lineage visualisation renders provenance graphs for non-technical stakeholders. Lineage coverage monitoring detects pipeline stages that fail to emit lineage events. Independent adversarial testing verifies that lineage cannot be tampered with, bypassed, or fabricated. Lineage integrates with AG-130 erasure propagation, AG-131 conflict resolution audit, and AG-066 forensic replay to provide a unified evidence layer.

7. Evidence Requirements

Required artefacts:

• Lineage store schema and configuration. The schema of the lineage store showing the data model for lineage relationships (nodes, edges, attributes), the storage technology, and the immutability and tamper-detection mechanisms in place.
• Pipeline instrumentation evidence. Documentation or configuration demonstrating that each data pipeline stage emits lineage events to the lineage store, including the instrumentation mechanism and coverage metrics (percentage of pipeline stages instrumented).
• Lineage query results. Sample lineage queries demonstrating that the system can trace a record from agent consumption back to its original source, including all intermediate transformations. These queries should cover at least 3 different data types and pipeline paths.
• Agent decision anchoring evidence. Sample agent decision records showing linkage to specific data lineage identifiers, with demonstration that the lineage identifiers can be traversed to source records.
• Lineage immutability evidence. Evidence that lineage records are immutable — append-only storage, tamper-detection mechanisms (cryptographic hashing, blockchain anchoring, or equivalent), and access controls preventing modification.

Retention requirements:

• Lineage metadata: minimum 7 years for regulated financial services; minimum 5 years for other regulated sectors; minimum 3 years otherwise. Lineage must be retained at least as long as the regulatory retention period for the decisions the data influenced.

Access requirements:

• Producible to regulators or auditors within 48 hours of request. Lineage queries should return results within seconds for individual records and within minutes for complex multi-hop queries.

8. Test Specification

Testing AG-133 compliance requires verifying that lineage is captured, preserved, queryable, and immutable.

Test 8.1: End-to-End Lineage Traceability

• Stimulus: Create a test data record at a source. Process it through the full pipeline (ingestion, transformation, aggregation, embedding, vector store indexing, retrieval, agent consumption). Query the lineage store for the record's complete provenance chain.
• Expected behaviour: The lineage query returns the complete chain: source record → each transformation with inputs, outputs, and timestamps → agent decision record. No gaps exist in the chain.
• Pass criteria: The lineage chain is complete from source to agent consumption. Every pipeline stage is represented. Every transformation is documented.
• Fail criteria: Any gap exists in the lineage chain, or any pipeline stage is missing from the lineage record.

Test 8.2: Lineage Preservation Through Aggregation

• Stimulus: Aggregate 100 source records into a single derived metric (e.g., an average, a score, or a feature). Query the lineage of the derived metric.
• Expected behaviour: The lineage query returns all 100 source records as inputs to the aggregation, along with the aggregation type and timestamp.
• Pass criteria: The lineage system can identify every source record that contributed to the derived metric. The aggregation transformation is documented.
• Fail criteria: The derived metric's lineage shows the aggregation as a source without linking to the underlying 100 records, or any contributing record is missing.

Test 8.3: Agent Decision Anchoring

• Stimulus: Trigger an agent to produce a decision. Query the agent's decision record for its data lineage references. Traverse each lineage reference back to its source.
• Expected behaviour: The decision record contains lineage identifiers for all data inputs consumed during the decision. Each identifier resolves to a complete lineage chain.
• Pass criteria: Every data input to the agent's decision is anchored via a lineage identifier. Every lineage identifier resolves to a complete, traversable chain.
• Fail criteria: The decision record lacks lineage identifiers, or any identifier does not resolve to a complete chain.

Test 8.4: Lineage Immutability

• Stimulus: Attempt to modify an existing lineage record — change a source identity, alter a transformation timestamp, or delete a lineage relationship.
• Expected behaviour: The lineage store rejects the modification. The original lineage record is preserved unchanged. The modification attempt is logged.
• Pass criteria: No lineage record can be modified or deleted. All modification attempts are detected and logged.
• Fail criteria: Any lineage record is modified, deleted, or overwritten without detection.

Test 8.5: Lineage Coverage Verification

• Stimulus: Process test data through every known pipeline path in the system. Verify that every pipeline stage emits lineage events.
• Expected behaviour: Lineage events are captured at every pipeline stage. No stage is uninstrumented.
• Pass criteria: 100% of pipeline stages emit lineage events. Coverage monitoring confirms no gaps.
• Fail criteria: Any pipeline stage does not emit lineage events, creating a gap in the provenance chain.

Test 8.6: Downstream Impact Query

• Stimulus: Identify a source record that has been consumed through multiple pipeline paths and agent decisions. Query the lineage store for all downstream records and decisions affected by this source record.
• Expected behaviour: The lineage query returns all downstream records across all pipeline paths, and all agent decisions that consumed any record derived from the source.
• Pass criteria: The impact query is complete — no downstream record or decision is missed. Query completes within a reasonable timeframe (minutes for complex graphs).
• Fail criteria: The impact query misses downstream records or decisions, or the query does not complete within a reasonable timeframe.

Conformance Scoring

• Score 0: No lineage governance exists — data records in the agent ecosystem cannot be traced to their sources, and agent decisions cannot be linked to their data inputs.
• Score 1: Schema-level lineage documentation exists (e.g., "this table comes from these sources") but per-record lineage is not captured. Agent decision records do not reference specific data lineage.
• Score 2: Automated per-record lineage capture is implemented across all pipeline stages. Agent decision records are anchored to lineage identifiers. Lineage is queryable and immutable. End-to-end traceability is demonstrated for all critical data flows.
• Score 3: Verified by independent testing — adversarial attempts to tamper with lineage, bypass instrumentation, or fabricate provenance have been tested and failed. Lineage-based impact analysis is operational. Lineage visualisation supports non-technical stakeholders. Coverage monitoring ensures no pipeline stage is uninstrumented.

9. Regulatory Mapping

Regulation	Provision	Relationship Type
EU AI Act	Article 12 (Record-Keeping)	Direct requirement
EU AI Act	Article 13 (Transparency)	Supports compliance
GDPR	Article 15 (Right of Access)	Supports compliance
GDPR	Article 22 (Automated Decision-Making)	Supports compliance
MiFID II	Article 25 (Suitability and Appropriateness)	Supports compliance
SOX	Section 404 (Internal Controls Over Financial Reporting)	Supports compliance
NIST AI RMF	MAP 2.1, MAP 2.3, MANAGE 2.2	Supports compliance
ISO 42001	Clause 8.4 (AI System Development), Clause 9.1 (Monitoring, Measurement, Analysis)	Supports compliance
DORA	Article 9 (ICT Risk Management Framework)	Supports compliance

EU AI Act — Article 12 (Record-Keeping)

Article 12 requires providers of high-risk AI systems to ensure that the system is designed to enable automatic recording of events (logs) relevant to the identification of situations that may present risks or substantial modification. Data lineage is the foundational record-keeping mechanism for AI systems that consume data from multiple sources through complex pipelines. Without lineage, the organisation cannot identify which data inputs contributed to a risk-relevant AI output.

EU AI Act — Article 13 (Transparency)

Article 13 requires that high-risk AI systems are designed to enable users to interpret the system's output and use it appropriately. Data lineage is a core component of transparency — understanding an agent's output requires understanding what data informed it. AG-133's lineage provides the technical infrastructure for transparency about data inputs, complementing model-level explainability.

GDPR — Article 15 (Right of Access) and Article 22 (Automated Decision-Making)

Article 15 gives data subjects the right to obtain information about automated decision-making, including "meaningful information about the logic involved." Article 22 provides additional protections for decisions based solely on automated processing. Both require the organisation to explain how automated decisions are made — which presupposes the ability to trace the data inputs to those decisions. AG-133 lineage provides the technical capability to fulfil these transparency obligations for AI agent decisions.

MiFID II — Article 25 (Suitability)

Suitability assessments for investment advice must be documented and demonstrable. If an AI agent provides investment advice, the firm must be able to show what data informed the advice and whether that data was appropriate. Lineage from the advice output back to the source data (client profile, market data, product information) provides the evidentiary chain for suitability demonstration.

SOX — Section 404

Section 404 requires management to assess internal controls over financial reporting, including the integrity of data used in financial calculations. For AI agents producing financial outputs, lineage provides the control evidence that data inputs to financial calculations are traceable, verified, and appropriate. Auditors can follow the lineage chain from a financial output back to its source data and verify the integrity of each transformation.

10. Failure Severity

Field	Value
Severity Rating	High
Blast Radius	Organisation-wide — lineage failures affect the defensibility of all agent decisions and the organisation's ability to respond to regulatory inquiries

Consequence chain: Without lineage, every agent decision is indefensible under scrutiny. The immediate technical consequence is inability to trace an agent's output to its data inputs. The operational consequence is that any challenge to an agent's decision — regulatory inquiry, customer complaint, legal proceeding, internal audit — cannot be answered with evidence. The organisation is reduced to asserting "the agent produced this output" without being able to explain or justify the basis for it. In regulated environments, this is disqualifying: the SAR that cannot be defended (Scenario A) damages the firm's relationship with law enforcement; the medical device that cannot demonstrate training data provenance (Scenario B) cannot receive regulatory approval, costing $14 million in delayed revenue; the supplier evaluation that cannot be traced to its data inputs (Scenario C) prevents effective root cause analysis after an £8.7 million product recall. The severity is compounded by the fact that lineage cannot be retroactively created — if lineage was not captured at processing time, it is permanently lost. Cross-references: AG-128 (Data Source Classification Governance) provides the source node for lineage chains; AG-130 (Residual Data Erasure Propagation Governance) uses lineage to identify downstream consumers for erasure; AG-131 (Source Conflict Escalation Governance) uses lineage to trace conflicting data back to its sources; AG-132 (Vector Store and RAG Governance) requires chunk-level lineage from retrieval back to source documents; AG-066 (Forensic Replay and Evidence Preservation) depends on lineage to reconstruct the data context of any historical agent decision.