Pipeline-Integrated Threat Event Ingestion

1. Definition

Pipeline-Integrated Threat Event Ingestion governs the integration of detection pipeline outputs into the adaptive threat response subsystem. The detection pipeline — comprising anomaly detectors, behavioural monitors, integrity verifiers, rate limiters, and external intelligence feeds — generates a continuous stream of events that must be ingested, normalised, enriched, and routed to the appropriate threat assessment and response components. AG-791 defines the standards for this ingestion process, ensuring that detection events are not lost, delayed, corrupted, or misrouted between the detection layer and the governance response layer.

The ingestion process serves as the critical bridge between detection and response. Without a well-governed ingestion layer, an organisation might have excellent detection capabilities that generate accurate alerts, yet fail to respond effectively because the alerts never reach the escalation function (AG-784) or the composite scoring engine (AG-790) in a timely and reliable manner. The protocol addresses this gap by defining a structured event schema, delivery guarantees, normalisation rules, enrichment procedures, and routing logic that transform raw detection outputs into governance-actionable threat events.

Normalisation is a particularly important function of the ingestion layer. Detection sources produce events in heterogeneous formats: some emit structured JSON, others produce syslog entries, and external feeds may use STIX/TAXII or proprietary formats. The ingestion layer normalises all inputs into a common event schema that downstream consumers (AG-784, AG-790) can process uniformly. This normalisation includes field mapping, timestamp standardisation (UTC, millisecond precision), severity classification to a common scale, and source attribution tagging. Without normalisation, each downstream consumer would need to understand every detection source's native format, creating tight coupling and fragility.

2. Scope

This protocol applies to all systems involved in the flow of detection events from their point of generation to their consumption by the threat response subsystem. Specifically:

• Detection Sources: All systems that generate threat-relevant events, including internal detection engines (AG-003, AG-004, AG-005, AG-022, AG-029), external threat intelligence feeds, and federated peer threat signals (AG-788).
• Ingestion Pipeline: The infrastructure that receives, validates, normalises, enriches, and routes detection events. This may comprise message queues, stream processors, normalisation engines, and routing logic.
• Event Consumers: Downstream systems that receive normalised events, primarily the escalation function (AG-784) and the composite scoring engine (AG-790).
• Event Storage: Persistent storage for ingested events, supporting audit, forensic investigation, and performance analysis.
• Monitoring Infrastructure: Systems that monitor the health and performance of the ingestion pipeline itself, including throughput metrics, latency measurements, and error rates.
• Exclusions: This protocol does not govern the internal logic of detection sources or the escalation/decay/scoring logic of event consumers.

3. Why This Matters

The ingestion pipeline is the nervous system of the adaptive threat response framework. Every detection signal, no matter how accurate, is useless if it does not reach the response layer in time to inform governance decisions. Pipeline failures are particularly dangerous because they are silent: the absence of events is indistinguishable from the absence of threats unless the pipeline is actively monitored.

Concrete Failure Scenario: A multinational bank deploys a sophisticated anomaly detection system for its autonomous trading agents. The detection system correctly identifies a pattern of layering and spoofing in real time, generating high-severity events. However, the message queue between the detection system and the governance platform experiences a partition. Events accumulate in the queue but are not delivered to the escalation function. The trading agents continue to operate at Level 1 (nominal) for 45 minutes while the detection system has been signalling Level 4 severity. During this window, the adversary successfully manipulates prices across three markets, generating GBP 28 million in illicit profits. When the queue partition resolves, all accumulated events flood the escalation function simultaneously, triggering an immediate jump to Level 5 — but the damage is done. The subsequent FCA investigation reveals that the firm had detection capability but failed to ensure reliable delivery of detection outputs to the response layer, constituting a failure of the risk management system under SYSC 6.1.1.

The EU AI Act (Article 9(4)) requires that risk management measures be implemented throughout the lifecycle of the AI system, including the operational monitoring and response chain. SOX Section 404 requires that information flows supporting internal controls be reliable and timely. ISO/IEC 27001:2022 clause A.8.16 requires monitoring of activities, including the monitoring infrastructure itself. AG-791 operationalises these requirements by governing the reliability and integrity of the detection-to-response pipeline.

4. Requirements

• R1: The system MUST implement a standardised event schema for ingested detection events, including at minimum: event identifier (UUID), source identifier, source type, detection timestamp (UTC, millisecond precision), ingestion timestamp, severity classification (1-5 scale), threat vector classification, confidence score (0.0-1.0), and raw event payload.
• R2: The ingestion pipeline MUST provide at-least-once delivery guarantees for all detection events from source to consumer, with idempotent processing at the consumer to handle potential duplicates.
• R3: End-to-end ingestion latency from source emission to consumer delivery MUST NOT exceed 1 second under normal load and MUST NOT exceed 3 seconds under peak load (defined as 10x normal throughput).
• R4: The ingestion pipeline MUST normalise all source-specific event formats into the standardised schema (R1) before delivering events to consumers.
• R5: The system MUST monitor the health of the ingestion pipeline continuously, detecting and alerting on: source disconnection, queue depth exceeding thresholds, consumer processing failures, and end-to-end latency exceeding thresholds.
• R6: Pipeline health degradation MUST be reported to the escalation function as a governance-relevant event, preventing threat level decay (per AG-785 R2) when the pipeline cannot guarantee reliable event delivery.
• R7: The system MUST persist all ingested events to durable storage for a minimum configurable retention period (default: 90 days) for audit and forensic purposes.
• R8: The system SHOULD support event enrichment, augmenting raw detection events with contextual data such as agent identity (AG-012), operational boundary state (AG-001), and current threat level before delivery to consumers.
• R9: The system SHOULD implement backpressure mechanisms that prevent upstream sources from overwhelming the pipeline, with graceful degradation rather than event loss.
• R10: The system MAY support priority queuing, where high-severity events are processed ahead of lower-severity events during peak load conditions.

5. Maturity Model

Basic

At the Basic level, detection events are forwarded from sources to consumers through a message queue or API integration. A standardised event schema exists but normalisation may be incomplete for some sources. Delivery is best-effort without formal at-least-once guarantees. End-to-end latency is not systematically measured. Pipeline health monitoring is basic (heartbeat checks) without automated alerting on queue depth or latency degradation. Events are persisted but retention policies may not be formally defined.

Intermediate

At the Intermediate level, at-least-once delivery is guaranteed with idempotent consumer processing. All source formats are normalised to the standardised schema. End-to-end latency is monitored and alerted on, meeting the 1-second normal and 3-second peak load thresholds. Pipeline health degradation is reported to the escalation function as a governance event. Events are enriched with contextual data before delivery. Backpressure mechanisms prevent pipeline overload. All events are persisted with defined retention policies and accessible for audit.

Advanced

At the Advanced level, priority queuing routes high-severity events ahead of low-severity events during peak load. The ingestion pipeline has been validated through independent adversarial testing, including source flooding attacks, queue poisoning, normalisation bypass attempts, and pipeline partitioning scenarios. End-to-end pipeline performance is continuously optimised with sub-500ms median latency. The pipeline supports dynamic scaling to handle burst loads without latency degradation. Cross-pipeline correlation detects when multiple sources report related events, enabling pre-aggregation before delivery to the composite scoring engine.

6. Test Criteria

• TC1: Submit a detection event from each configured source type and verify that all are normalised to the standardised schema with all required fields populated. (Maps to R1, R4)
• TC2: Inject 1000 events into the pipeline and verify that all 1000 are delivered to consumers (at-least-once) with no losses, and that duplicate handling is idempotent. (Maps to R2)
• TC3: Measure end-to-end latency for 1000 events under normal load and verify that 99th percentile latency is under 1 second. Repeat at 10x throughput and verify 99th percentile under 3 seconds. (Maps to R3)
• TC4: Disconnect a detection source and verify that the pipeline health monitor detects the disconnection and generates an alert within 30 seconds. (Maps to R5)
• TC5: Degrade the pipeline (e.g., increase queue depth beyond threshold) and verify that a governance-relevant event is reported to the escalation function. (Maps to R6)
• TC6: Verify that ingested events are persisted to durable storage and remain accessible for the configured retention period. (Maps to R7)
• TC7: Submit a raw event in a source-specific format and verify that enrichment adds agent identity, boundary state, and current threat level before delivery to consumers. (Maps to R8)
• TC8: Flood the pipeline at 100x normal throughput and verify that backpressure engages without event loss, and that throughput degrades gracefully. (Maps to R9)
• TC9: Submit simultaneous high-severity and low-severity events during peak load and verify that high-severity events are delivered first. (Maps to R10)

7. Scoring

8. Failure Scenarios

F1: Silent Event Loss Due to Queue Overflow (Maps to R2, R9)

Scenario: A detection source experiences a burst of activity, generating 10,000 events per second when the pipeline is sized for 100 events per second. The message queue fills to capacity and begins dropping events silently. High-severity events indicating an active attack are among those dropped. The escalation function never receives these critical signals.

Impact: CRITICAL. The adaptive threat response is blind to an active attack because the pipeline silently discards the evidence. This failure mode is particularly dangerous because there is no visible error — the system simply does not escalate.

Mitigation: R2 mandates at-least-once delivery. R9 recommends backpressure mechanisms that slow upstream sources rather than dropping events. R5 mandates monitoring of queue depth with alerting on threshold breaches. R10 suggests priority queuing to ensure high-severity events are processed even under overload.

F2: Normalisation Failure Creating Blind Spots (Maps to R1, R4)

Scenario: A new detection source is deployed with an event format that the normalisation engine does not recognise. Events from this source pass through the pipeline with null or default values for severity and threat vector fields. The composite scoring engine (AG-790) assigns these events zero weight due to missing classification data. The new source, despite detecting genuine threats, has no influence on the composite score.

Impact: HIGH. The organisation has deployed a detection capability that is functionally disconnected from the governance response layer due to an ingestion normalisation gap.

Mitigation: R4 mandates normalisation of all source formats. The normalisation engine must reject events it cannot normalise and alert on the rejection. New source onboarding must include normalisation mapping validation as part of the deployment process.

F3: Pipeline Partition Preventing Decay Inhibition (Maps to R5, R6)

Scenario: The ingestion pipeline experiences a partition that prevents events from reaching the escalation function. Simultaneously, the pipeline health monitor is on the same side of the partition as the pipeline (not the escalation function), so its health reports also fail to reach the escalation function. The escalation function, receiving no events and no health warnings, allows the decay function (AG-785) to proceed, reducing the threat level during an active pipeline outage.

Impact: HIGH. The system decays its threat level during a period when it has no visibility into the threat landscape, creating a false sense of security.

Mitigation: R6 mandates that pipeline health degradation is reported as a governance event. The health monitoring path must be independent of the event delivery path, using a separate communication channel that can reach the escalation function even when the primary pipeline is partitioned. AG-785 R2 provides a complementary safeguard by preventing decay when detection pipelines are degraded.

F4: Event Enrichment Introducing Latency (Maps to R3, R8)

Scenario: The event enrichment process queries multiple external systems (agent identity store, boundary state engine, threat level store) for each event. Under peak load, these queries introduce 2-4 seconds of additional latency per event, pushing total pipeline latency to 5 seconds. During this period, the escalation function operates on stale data.

Impact: MEDIUM. Enrichment, intended to improve event quality, degrades pipeline performance to the point where it impairs the timeliness of threat response.

Mitigation: R3 mandates latency thresholds. Implement enrichment caching with millisecond-level cache refresh, and design the enrichment process to operate asynchronously where possible, attaching enrichment data after initial delivery if necessary rather than blocking delivery for enrichment completion.

9. Regulatory Mapping

10. Related Protocols

Document generated under Patent 7 governance framework. Classification: INTERNAL. Review cycle: Quarterly.