Threat modeling defines how an enterprise AI system can be abused, which assets are exposed, where trust changes, and which controls must interrupt the attack path. Chapter 6A establishes the scope, method, actors, assets, and data flows that the rest of the threat model builds on.
6A.1 Scope and security objectives
The threat model covers the full enterprise AI lifecycle: user request, gateway, identity, context retrieval, model execution, orchestration, tool selection, capability authorization, connectors, target systems, telemetry, human approval, and recovery. It includes both intentional attacks and high-impact misuse caused by configuration errors, model mistakes, stale data, or uncontrolled autonomy.
Prevent unauthorized disclosure across prompts, context, memory, logs, models, tools, and tenants.
Prevent manipulation of context, plans, approvals, capabilities, records, and evidence.
Ensure that models cannot create identity, expand privilege, or perform actions outside approved policy.
6A.2 Threat modeling methodology
The model combines several perspectives. STRIDE identifies classic software threats. AI-specific abuse cases focus on prompt injection, model manipulation, data leakage, excessive agency, and unsafe outputs. Attack-path analysis traces how a malicious input becomes an enterprise effect. Risk scoring evaluates likelihood, business impact, detectability, and control strength.
| Method | Purpose | AI application |
|---|---|---|
| STRIDE | Systematic software threat classification | Identity, services, stores, APIs, agents, connectors |
| Abuse cases | Describe attacker goals and misuse paths | Injection, exfiltration, privilege expansion, unsafe automation |
| Data-flow analysis | Locate trust boundaries and sensitive transfers | Prompts, retrieved context, tool parameters, outputs, telemetry |
| Attack trees | Show alternative paths to one objective | Unauthorized payment, production change, data disclosure |
| Residual-risk review | Assess risk after controls | Human oversight, detection, recovery, business acceptance |
6A.3 Assets
| Asset class | Examples | Primary consequence |
|---|---|---|
| Identity and credentials | User claims, agent identities, service tokens, approvals | Impersonation and unauthorized action |
| Enterprise data | Financial, personal, legal, customer, employee, operational records | Disclosure, corruption, regulatory impact |
| Model and prompt assets | System prompts, routing, safety rules, model deployments | Behavior manipulation and control bypass |
| Knowledge and memory | Indexes, embeddings, documents, summaries, conversation memory | Poisoning, leakage, stale or false decisions |
| Capabilities and connectors | Payments, account changes, production operations, communications | Direct enterprise impact |
| Evidence and logs | Traces, policy decisions, approvals, audit records | Loss of attribution and incident reconstruction |
6A.4 Threat actors
| Actor | Access | Typical objective |
|---|---|---|
| External attacker | Public interface, shared content, external service | Injection, exfiltration, resource abuse, fraud |
| Malicious insider | Legitimate user access and domain knowledge | Privilege abuse, covert extraction, record manipulation |
| Compromised account | Valid session or token | Use agent authority at scale |
| Compromised content source | Document, email, ticket, web page, knowledge base | Indirect prompt injection or poisoning |
| Compromised connector or dependency | Trusted integration path | Credential theft, data manipulation, lateral movement |
| Misconfigured agent | Broad permissions or unsafe routing | Unintentional high-impact behavior |
6A.5 Data-flow diagram
Each arrow carries a security contract: authenticated caller, tenant, data classification, resource scope, integrity protection, allowed destination, trace identifier, and retention policy. The threat model should treat missing or ambiguous metadata as a weakness.
6A.6 Trust boundaries
- External user or application to enterprise gateway.
- Enterprise gateway to model provider or model serving zone.
- Context service to enterprise knowledge and operational data.
- Agent runtime to capability router.
- Capability router to connector.
- Connector to system of record or production environment.
- Tenant, region, or business domain to another isolation domain.
- Production workloads to logging, analytics, and evaluation environments.
Threat modeling rule: every trust boundary must have an explicit security decision.
Encryption alone does not define trust. Identity, authorization, classification, ownership, and expected behavior must also be known.
6A.7 STRIDE baseline
| STRIDE class | Enterprise AI example |
|---|---|
| Spoofing | Agent or connector impersonates a user or workload |
| Tampering | Knowledge, policy, prompt, approval, or tool parameters are modified |
| Repudiation | Action cannot be attributed to initiator, agent, model, or approver |
| Information disclosure | Protected context or memory appears in output, logs, or external calls |
| Denial of service | Prompt loops, expensive retrieval, token exhaustion, queue saturation |
| Elevation of privilege | Agent invokes broader capabilities or crosses resource scope |
6A.8 Foundation checklist
6A.9 Summary
Chapter 6A defines the threat-model foundation. It establishes scope, security objectives, assets, threat actors, data flows, trust boundaries, and the STRIDE baseline required for deeper abuse-case analysis.
Core conclusion: a useful threat model begins with architecture and business impact, not a generic list of AI risks.