Chapter 6C completes the threat model with a mitigation matrix, residual-risk process, detection and incident mapping, security-control ownership, threat-model maintenance, and production readiness.
6C.1 Threat-to-control matrix
| Threat | Prevent | Detect | Recover |
|---|---|---|---|
| Prompt injection | Context isolation, tool allowlist, independent policy | Injection signals, denied tools, unusual destinations | Suspend source or agent; replay from trusted context |
| Data exfiltration | ACL retrieval, minimization, DLP, egress allowlist | Volume anomalies, sensitive output alerts | Revoke access, contain destination, investigate exposure |
| Excessive agency | Budgets, approval, narrow capabilities, scoped identity | Loop, action-rate, and scope anomalies | Stop workflows, revoke capabilities, compensate actions |
| Knowledge poisoning | Source approval, integrity, versioning, provenance | Content anomaly, ranking shift, source change alert | Remove source, restore version, re-evaluate affected outputs |
| Connector compromise | Workload identity, segmentation, signed artifacts | Identity anomaly, destination and behavior monitoring | Revoke identity, isolate connector, reconcile target state |
| Cross-tenant leakage | Partitioned stores, tenant claims, cache isolation | Tenant mismatch and policy-denial telemetry | Stop service, contain data, rotate secrets, notify as required |
6C.2 Residual risk
Residual risk is the business risk remaining after controls are applied. It should be assessed per use case and action class, not once for the platform. A low-risk summarization assistant and a high-impact financial agent can use the same infrastructure but have very different residual risk.
{
"threat_id":"TM-EXFIL-004",
"scenario":"indirect prompt injection causes outbound disclosure",
"inherent_risk":"critical",
"controls":[
"task-scoped capability allowlist",
"destination-specific connectors",
"DLP on tool parameters",
"human approval for external transfer"
],
"residual_risk":"medium",
"accepted_by":"Data Protection Officer",
"conditions":[
"external transfer disabled for confidential data",
"quarterly red-team test",
"24x7 alerting"
],
"review_date":"2026-10-23"
}Risk acceptance must have an accountable owner, explicit conditions, evidence, and expiry. “The model is accurate most of the time” is not a valid control statement.
6C.3 Security-control mapping
| Control domain | Architecture implementation |
|---|---|
| Identity and access | Human and workload identity, delegated claims, short-lived tokens, access review |
| Data protection | Classification, encryption, ACL-aware retrieval, minimization, DLP, retention |
| Application security | Schema validation, safe output handling, dependency and artifact controls |
| Network security | Segmentation, private access, destination allowlists, controlled egress |
| AI safety controls | Context isolation, tool boundaries, policy, evaluation, refusal and escalation |
| Operations | Telemetry, detection, incident response, rollback, compensation, evidence |
6C.4 Detection engineering
Every high-risk threat should have at least one actionable detection. Detection logic should combine identity, task, agent, model, source, policy, tool, target, and outcome. Isolated prompt keywords produce excessive false positives and miss semantic attacks.
| Detection | Signal combination | Response |
|---|---|---|
| Possible indirect injection | Untrusted source + instruction-like content + denied capability request | Block action, preserve context, alert security |
| Data harvesting | Repeated broad retrieval + increasing scope + output volume anomaly | Throttle, require reauthentication, investigate |
| Approval mismatch | Approved action hash differs from execution payload | Block execution and create incident |
| Cross-tenant attempt | Tenant claim differs from resource or index partition | Fail closed and raise high-severity alert |
| Agent loop | Step count, token use, and tool calls exceed task baseline | Terminate workflow and preserve trace |
6C.5 Incident-response mapping
Response procedures should support independent suspension of the user session, agent, model route, knowledge source, capability, connector, or target identity. Broad shutdown is sometimes necessary, but granular containment reduces business disruption.
6C.6 Ownership and review cadence
| Artifact | Owner | Review trigger |
|---|---|---|
| System data-flow diagram | Platform architecture | New service, model, connector, or trust boundary |
| Threat register | Security architecture | New abuse case, incident, control change |
| Use-case residual risk | Business owner and risk | Autonomy, data, population, or impact changes |
| Detection coverage | SOC / detection engineering | New threat path or telemetry source |
| Recovery playbook | SRE and incident response | Architecture, dependency, or business process change |
6C.7 Threat-model Architecture Decision Records
# ADR-TM-014: Treat retrieved content as untrusted instructions
Status: Accepted
Context:
Enterprise agents retrieve email, tickets, documents, and web content.
These sources can contain direct or hidden instructions that the model may follow.
Decision:
Retrieved content is always labeled as evidence, never as authority.
It cannot modify identity, policy, tool scope, or system instructions.
Controls:
- Structured context with source and trust metadata
- Task-scoped capability allowlist
- Independent authorization for every side effect
- Detection for instruction-like content plus denied tool attempts
- High-impact actions require approval and post-condition verification
Residual risk:
The model may still produce a misleading recommendation, but cannot directly
expand its authority or execute an unapproved action.6C.8 Production threat-model checklist
6C.9 Summary
Chapter 6C turns threat modeling into a lifecycle. Threats map to controls, controls map to owners and detections, residual risks map to business decisions, and incidents feed back into architecture and evaluation.
Core conclusion: an enterprise AI threat model is not finished when the document is approved. It is finished only when the identified threats are governed, observable, testable, and recoverable in production.
Reference foundations
- STRIDE threat modeling principles for systems, stores, flows, and trust boundaries.
- OWASP guidance for prompt injection, excessive agency, sensitive information disclosure, and insecure output handling.
- NIST AI risk-management concepts for lifecycle governance, measurement, and risk treatment.
- MITRE adversarial AI and enterprise attack-path concepts for threat actors, techniques, detection, and response.
- Zero Trust architecture principles for identity, policy, segmentation, and continuous verification.