AG-804
Interpretability-Based Internal-State Monitoring
EU AI Act
NIST AI RMF
ISO 42001
AGS Frontier Safety | Logging, Observability & Forensics | Version 2.2
Interpretability-Based Internal-State Monitoring governs the use of model-internal signals — interpretable features, probes, or activations indicative of states such as deception, concealment, power-seeking, or goal-conflict — to oversee an agent independently of its outputs.
Output-level monitoring can be defeated by an agent that controls its outputs (the core risk addressed by AG-798 sabotage). Internal-state monitoring provides an oversight channel the agent does not directly author, strengthening scheming/sabotage/corrigibility evaluations (AG-797/798/799) and offering a runtime signal for high-stakes deployments where feasible.
In scope: where model internals are accessible, deriving and monitoring interpretability signals for safety-relevant internal states; using them to corroborate evaluations and to flag runtime anomalies; honest documentation of method limitations.
Out of scope: output/behavioural monitoring (covered elsewhere), and a requirement to achieve full mechanistic transparency. This dimension governs *using available internal-state signals as an oversight channel*, proportionate to access and maturity.
The hardest safety failures are ones an agent hides at the output level. A monitoring channel rooted in internal state is harder for the agent to game, and can catch deception or power-seeking that behaviour alone would not reveal. As interpretability tooling matures, internal-state monitoring becomes a uniquely robust complement to behavioural oversight for frontier agents — and a credible signal for regulators that oversight does not depend solely on the agent's self-presentation.
Test 6.1: Validated Signal
Test 6.2: Runtime Response
Test 6.3: Gap Compensation
| Score | Criteria |
|---|---|
| 0 | No consideration of internal-state monitoring for a frontier agent with accessible internals |
| 1 | Feasibility assessed but no validated signal derived |
| 2 | ≥1 validated internal-state signal corroborates evaluations, with documented limits |
| 3 | Runtime internal-state monitoring with responses, independence, audit logging, and gating integration |
Scenario A — Output-Only Oversight Defeated: A high-autonomy agent presents clean outputs while pursuing a concealed objective. With no internal-state channel, oversight sees nothing. A validated deception probe would have flagged the concealment during the action.
Scenario B — Overclaimed Interpretability: An organisation markets "fully interpretable, deception-proof" oversight based on an unvalidated probe. A missed deception incident exposes the overclaim; the absence of documented limits is central to the finding.
Scenario C — Unmanaged API Gap: An agent built on a closed third-party model has no internal access; the organisation neither documents this nor strengthens behavioural controls, leaving a silent oversight blind spot.
| Requirement | EU AI Act | NIST AI RMF | ISO 42001 |
|---|---|---|---|
| R1: Feasibility evaluation documented | Art. 55 — Model evaluation | MEASURE 2.9 — Explainability | Clause 8.3 — Verification |
| R2: Validated internal-state signal | Art. 55 — Systemic-risk assessment | MEASURE 2.9 — Explainability | Clause 8.3 — Verification |
| R3: Reliability validation + limits | Art. 15 — Accuracy representation | MEASURE 2.13 — TEVV effectiveness | Clause 9.1 — Monitoring and measurement |
| R4: Runtime anomaly response | Art. 14 — Human oversight | MEASURE 2.4 — Production monitoring | Clause 8.1 — Operational control |
| R5: Independent operation, audit logging | Art. 12 — Record-keeping | MEASURE 1.3 — Independent assessors | Clause 9.1 — Monitoring and measurement |
| R6: No over-claiming | Art. 15 — Accuracy | MEASURE 2.9 — Explainability | Clause 8.3 — Verification |
| R7: Document and compensate gaps | Art. 55 — Risk mitigation | MANAGE 1.4 — Residual-risk disclosure | Clause 6.1 — Actions to address risk |
| R8: Method refresh; feed gating | Art. 55 — Ongoing evaluation | MEASURE 3.1 — Emergent-risk tracking | Clause 10.1 — Continual improvement |
Article 55 requires meaningful model evaluation for systemic-risk models; internal-state evidence strengthens that evaluation where behavioural evidence is insufficient. Article 12 (record-keeping) covers logging of monitoring outputs for traceability.
MEASURE 2.9 (explainability — model explained and validated) and MEASURE 2.4 (production monitoring of behaviour) together frame internal-state monitoring as explainability-grounded oversight.
Clause 9.1 (monitoring and measurement) and Clause 8.3 (verification) require validated monitoring; AG-804 specifies an internal-state channel with honest limits.