A reference architecture is not a mandatory product stack. It is a set of responsibilities, boundaries, and information flows that remain necessary regardless of whether the implementation uses managed cloud services, open-source infrastructure, on-premises systems, or a hybrid environment.
3.1 Architecture principles
Enterprise AI architecture should begin with the operating model and risk boundary, not with a model endpoint. The architecture must account for users, applications, automated events, data ownership, agent identity, model behavior, enterprise tools, network controls, evidence, and recovery. The design should remain useful when a model or orchestration framework is replaced.
A model may propose an action, but policy, identity, domain validation, and approval determine whether it can execute.
Agents receive approved business capabilities rather than unrestricted access to applications, databases, shells, or APIs.
Workflow progress, timers, approvals, retries, and compensation are persisted outside the model conversation.
Indexes, summaries, and memory support reasoning but never silently replace authoritative enterprise records.
The system records who initiated it, which agent acted, what sources were used, which policy approved it, and what changed.
Direct model calls, workflows, single agents, and multi-agent systems are selected according to need, not fashion.
Microsoft’s current agent architecture guidance identifies a client, application and storage infrastructure, orchestrator, model, catalog, and tool calling as core components. Its multi-agent reference architecture expands the picture with registries, memory, communication, observability, evaluation, security, and governance. AWS organizes enterprise agentic AI into layered services with security, discoverability, and observability spanning the architecture. Google’s recent multi-tenant agentic architecture similarly emphasizes centralized platform control while allowing decentralized teams to develop specialized agents. The reference architecture below synthesizes these recurring responsibilities into a vendor-neutral model.
3.2 Logical reference architecture
The architecture consists of two kinds of components. Runtime components process requests and execute tasks. Cross-cutting control components govern identity, policy, security, observability, evaluation, lifecycle, and cost. The logical view below shows both.
Primary trust boundary: the model and agent orchestration layer may interpret context and propose work, but they do not directly define enterprise authority.
Authorization, credential issuance, schema enforcement, high-impact approval, and final execution are performed by deterministic services that can be tested independently and audited after the fact.
The architecture does not require multiple agents. A single-agent workflow can use the same gateway, identity, knowledge, capability, observability, and evaluation planes. Multi-agent coordination is justified when specialization, independent security boundaries, parallel work, or distinct model and tool configurations provide measurable value.
3.3 Architecture layers and responsibilities
Web, mobile, chat, embedded application, API, batch process, event trigger, or partner interface. This layer captures user intent and presents evidence, approvals, and outcomes.
Authentication, WAF, DDoS protection, request size, rate limits, tenant routing, content controls, model quotas, egress policy, and correlation IDs.
Task classification, planning, agent selection, workflow state, context assembly, tool selection, model calls, and human-task coordination.
Foundation models, embeddings, rerankers, classifiers, safety models, structured extraction, model routing, caching, and fallback logic.
Source ingestion, indexing, retrieval, provenance, conversation state, workflow memory, domain memory, permission filtering, and data freshness.
Approved business functions, connectors, events, queues, workflow adapters, transaction validation, idempotency, compensation, and post-condition verification.
Identity, delegated authority, policy, secrets, network segmentation, configuration, registries, lifecycle, approval rules, and governance.
Observability, evaluation, incident response, audit evidence, release controls, red-team testing, business metrics, cost management, recovery, and continuity.
These layers are logical, not necessarily separate products. A managed platform may provide several responsibilities. A custom enterprise platform may implement them through independent services. The architecture review should verify that each responsibility exists and that overlapping products do not create gaps or contradictory policy.
3.4 End-to-end request and action flow
A production request moves through explicit stages. The stages can be implemented synchronously for low-risk read operations or as a durable workflow for long-running and write-capable tasks.
- Ingress and authentication. The gateway authenticates the user or workload, assigns a correlation ID, identifies the tenant, and applies edge controls.
- Task classification. The system identifies domain, risk tier, required data, and whether the task is informational, advisory, draft-producing, approval-driven, or autonomous.
- Context policy. The orchestrator determines which sources and memory scopes may be used for the authenticated principal and task.
- Retrieval and grounding. Permission-aware connectors retrieve authoritative or approved content with provenance, freshness, and classification metadata.
- Planning. A model or deterministic workflow generates a candidate plan using only registered agents and capabilities.
- Capability authorization. The policy engine evaluates initiator, agent, resource, environment, requested action, evidence, and risk.
- Validation. Schema, domain rules, limits, separation of duties, and preconditions are checked outside the model.
- Human approval. When required, an accountable person reviews evidence and approves, modifies, or rejects the action.
- Execution. A scoped connector invokes the target system using short-lived credentials and an idempotency key.
- Verification. The platform reads back target state or uses an independent signal to confirm the intended post-condition.
- Response and evidence. The user receives an explanation that distinguishes source facts, model interpretation, actions, approvals, and unresolved uncertainty.
- Learning loop. Traces, evaluation outcomes, incidents, corrections, and business results feed controlled improvement processes.
{
"trace_id": "tr_01J7...",
"tenant": "enterprise-eu",
"initiator": "user_19482",
"agent_id": "procurement-assistant",
"agent_version": "4.2.1",
"task_class": "purchase_request_draft",
"risk_tier": "medium",
"sources": [
{"id": "contract:8842:v3", "authoritative": true},
{"id": "supplier:219", "authoritative": true}
],
"model": {"provider": "routed", "deployment": "reasoning-standard"},
"capabilities_requested": ["erp.purchase-request.create-draft"],
"policy_decision": "allow_with_approval",
"approver": "user_7721",
"operation_id": "op_44192",
"post_condition": "draft_exists",
"outcome": "verified"
}3.5 The AI gateway and experience boundary
The AI gateway is the controlled entry point for model and agent traffic. It may be implemented through an API gateway, application gateway, service mesh, model gateway, or a combination. Its purpose is not simply to proxy requests. It applies common policy and creates a stable boundary between user-facing applications, agent runtimes, model providers, and external tools.
| Gateway responsibility | Why it matters | Typical controls |
|---|---|---|
| Authentication and tenant routing | Prevents cross-tenant and anonymous access | OIDC, workload identity, tenant claims, route isolation |
| Traffic protection | Protects availability and budget | WAF, DDoS, request limits, token quotas, concurrency caps |
| Model policy | Controls which models process which data and tasks | Provider allowlist, region, classification, risk-tier routing |
| Content controls | Reduces unsafe input and output propagation | Malware scanning, data-loss prevention, moderation, size limits |
| Observability | Creates consistent request evidence | Correlation IDs, token counts, latency, errors, model metadata |
| Egress governance | Prevents uncontrolled external calls | Firewall, proxy, DNS policy, destination allowlist, inspection |
Azure’s baseline enterprise chat architecture uses private endpoints, controlled outbound routing, workload-managed storage, and firewall-enforced egress. Microsoft’s dynamic-agent architecture similarly routes public APIs and MCP servers through controlled outbound infrastructure. These are implementation examples of a general principle: agent traffic should not gain unrestricted internet or enterprise network access merely because the runtime can call tools.
3.6 Orchestration plane
The orchestration plane coordinates tasks, but it should not become a monolithic source of business truth. Its core responsibilities are intent recognition, workflow selection, context management, model and agent invocation, tool sequencing, error handling, human-task coordination, and state transitions.
Choose the lowest sufficient orchestration complexity
| Pattern | Use when | Main risk |
|---|---|---|
| Direct model call | Single bounded transformation such as summarization or classification | Unnecessary tool access or hidden state |
| Deterministic workflow with model steps | Process sequence and control points are known | Model output not validated between steps |
| Single agent with tools | One domain, limited tool set, adaptive planning is useful | Tool overload and excessive permissions |
| Multi-agent orchestration | Distinct domains, security boundaries, parallel specialization, or independent policies | Coordination overhead, latency, state confusion, cascading errors |
Microsoft’s orchestration guidance explicitly recommends the lowest level of complexity that reliably satisfies the scenario. Multi-agent systems add specialization and modularity, but also increase cost, latency, coordination state, and failure modes. The architecture should require a reason for each agent boundary: domain ownership, security scope, model specialization, independent lifecycle, or parallel execution.
Agent registry and factory
A registry defines approved agents, owners, versions, capabilities, required data classifications, model policies, and lifecycle state. A factory creates or loads agent implementations without coupling orchestration logic to one representation. Microsoft’s dynamic-agent pattern highlights an agent factory that can instantiate agents from code or declarative templates and centralize validation.
agent:
id: customer-retention-specialist
version: 3.4.0
owner: Customer Operations
lifecycle: production
risk_tier: medium
allowed_tenants: [enterprise-eu]
knowledge_scopes:
- crm.customer-profile
- contracts.active
- support.case-history
capabilities:
- crm.note.create-draft
- retention.offer.recommend
model_policy: reasoning-medium
maximum_context_classification: confidential
human_approval:
required_for:
- retention.offer.commit
evaluation_suite: retention-agent-v83.7 Model and intelligence services
The model layer should be treated as a portfolio of capabilities rather than one universal endpoint. Different tasks may require different latency, cost, context size, reasoning quality, data residency, modality, or safety characteristics. A model router can select an approved deployment according to task class, risk, tenant, region, and current availability.
Use smaller models for extraction and classification; reserve expensive reasoning models for complex decisions.
Model eligibility depends on classification, geography, contractual terms, and retention requirements.
Fallbacks must preserve safety and semantics; a cheaper or available model may not be approved for the same task.
The architecture should record model provider, deployment, version or snapshot where available, system instructions, tool schema versions, temperature or reasoning settings, and safety configuration. Reproducibility is not perfect for probabilistic systems, but operational evidence should be sufficient to compare behavior across releases and investigate incidents.
Caching can reduce latency and cost, but cache keys must include relevant policy, tenant, model, prompt, source versions, and user scope. Reusing a response across users or data versions can create security and freshness defects. Semantic caching is particularly sensitive because approximate similarity can return a response generated for a materially different question.
3.8 Knowledge, retrieval, and memory plane
The knowledge plane transforms enterprise content and operational data into context that can be safely consumed by models. It includes ingestion, parsing, enrichment, indexing, retrieval, access filtering, provenance, freshness, and deletion. Google’s current RAG architectures treat retrieval as an end-to-end infrastructure flow rather than a single vector-store query.
Separate memory types
| Memory type | Purpose | Retention and risk |
|---|---|---|
| Conversation memory | Maintain continuity within a user interaction | Contains user data and inferred intent; apply access and expiry |
| Workflow state | Persist task steps, approvals, timers, and outcomes | Business record with stronger durability and audit requirements |
| Agent working memory | Hold temporary plans, intermediate results, and tool outputs | Minimize retention; may contain sensitive derived content |
| Domain memory | Reuse approved facts, preferences, or learned patterns | Requires owner, correction process, provenance, and deletion |
| Evaluation memory | Store test cases, reviewer feedback, incidents, and corrections | May contain production examples; de-identify where possible |
3.9 Capability and enterprise integration plane
The capability plane is the most important boundary between adaptive AI behavior and enterprise side effects. It presents approved, versioned business functions to agents while hiding raw system complexity. A capability may be implemented through an API, event, workflow, queue, database adapter, robotic automation, or human task.
Capability contract requirements
Tool protocols can standardize how capabilities are described and invoked, but they do not replace enterprise policy. An MCP server, plugin, function, or API can expose a capability. The enterprise still decides which agent and user may use it, what data may be passed, which network route is allowed, and whether approval is required.
Do not expose raw administrative primitives as general-purpose agent tools.
Arbitrary SQL, shell execution, unrestricted HTTP, broad file access, and generic “call any API” tools dramatically expand the attack surface. Replace them with narrow business capabilities that validate purpose, resource, parameters, and post-condition.
3.10 Identity, delegation, policy, and secrets
The control plane distinguishes the initiating user or process, the agent identity, the orchestrator workload, and the target workload identity. A high-quality audit trail preserves this chain. The architecture should avoid one permanent credential that represents every user and agent.
Microsoft Entra’s Agent ID guidance treats agent identity, owners, sponsors, lifecycle, access governance, and protection as first-class concerns. The broader architectural lesson is that agents should be discoverable and governable identities, not anonymous code using inherited secrets.
Policy enforcement pattern
input:
initiator:
role: service_manager
tenant: enterprise-eu
agent:
id: incident-remediation-agent
owner: IT Operations
capability: service.change.execute
resource:
service: payments-api
environment: production
evidence:
incident_severity: SEV-1
approved_runbook: true
approval:
provided: true
approver_role: production_change_approver
decision:
result: allow
credential_scope:
- service:payments-api
- action:enable-fallback-mode
expires_in_seconds: 300
audit_policy: high-impact-change-v4Secrets should be issued or retrieved at execution time by the connector or workload identity, not placed in prompts, agent memory, tool descriptions, or model-visible configuration. The model should receive only the information needed to select and parameterize a capability.
3.11 Observability, audit, and security telemetry
Traditional application telemetry is necessary but insufficient. Enterprise AI observability must connect user intent, retrieved sources, model calls, agent decisions, tool requests, policy outcomes, approvals, target-system results, quality evaluations, and business outcomes.
| Telemetry domain | Examples | Primary consumer |
|---|---|---|
| Application | Latency, availability, errors, queue depth, workflow state | SRE / Platform |
| Model | Provider, deployment, tokens, context size, refusal, fallback | AI Engineering / FinOps |
| Retrieval | Queries, sources, ranking, freshness, permission filtering | Data / Knowledge owners |
| Agent | Plan steps, selected capabilities, loops, handoffs, termination | AI Engineering |
| Security | Prompt injection signals, denied tools, data-loss events, egress | SOC / Security |
| Governance | Policy version, approval, owner, model and agent lifecycle | Risk / Compliance |
| Business | Cycle time, resolution, avoided work, correction, customer effect | Capability owner |
Sensitive prompts and tool responses should not automatically be copied into every log. The architecture needs structured logging, classification-aware redaction, encryption, controlled access, and retention rules. Investigators require enough evidence to reconstruct the event without turning the observability platform into a secondary sensitive-data repository.
3.12 Evaluation architecture
Evaluation is a production subsystem, not a pre-launch document. It combines offline test suites, adversarial tests, simulation, human review, online quality signals, business outcomes, and incident feedback. Each agent and capability version should be associated with an approved evaluation suite.
Evaluate the system, not only the answer
A natural-language response can be correct while the execution path is unsafe. Evaluation should therefore test source selection, permission filtering, tool choice, argument validity, policy enforcement, approval behavior, retry behavior, post-condition verification, and refusal. A useful production metric is not merely “answer quality,” but the percentage of tasks completed correctly without unauthorized access, avoidable escalation, repeated action, or manual correction.
3.13 Deployment topologies
The logical architecture can be deployed in several topologies. The correct choice depends on data residency, network boundaries, regulatory obligations, latency, available platform services, and operational capability.
| Topology | Strengths | Primary trade-offs |
|---|---|---|
| Managed cloud | Fast platform adoption, managed scaling, integrated identity and monitoring | Provider dependency, service availability by region, contractual review |
| Private cloud / isolated cloud | Greater network and data control with cloud operating model | Higher platform complexity and limited managed-service choice |
| On-premises | Local data boundary, legacy proximity, specialized sovereignty | Model operations, capacity, patching, resilience, and cost ownership |
| Hybrid | Keep sensitive systems local while using managed models or services | Network latency, identity federation, egress governance, split operations |
| Multi-cloud | Provider choice, regional coverage, workload specialization | Policy consistency, observability fragmentation, portability cost |
Production reference designs commonly use private endpoints, controlled egress, workload identities, managed secrets, zone redundancy, dedicated state stores, and centralized monitoring. These are not uniquely “AI” requirements; AI workloads inherit the reliability and security requirements of enterprise applications and add model, context, and tool-specific controls.
3.14 Resilience, continuity, and recovery
Enterprise AI systems fail across multiple dependencies: model endpoints, search services, data sources, workflow stores, queues, connectors, policy engines, identity providers, and external APIs. The architecture should define service-level objectives and degradation behavior for each task class.
When a source is unavailable, disclose the missing source, avoid false completeness, and use approved cached data only within freshness policy.
Do not execute when identity, policy, approval, schema validation, or target preconditions cannot be confirmed.
Use alternative models only when they are approved for the same data class and task risk.
Record intended operation, idempotency key, and state before invoking a write-capable connector.
Recovery testing should include model outage, retrieval outage, identity failure, delayed event, duplicated message, target-system timeout after commit, corrupted index, policy-engine unavailability, region failure, and operator error. The system should preserve enough state to resume or reconcile without asking the model to infer what probably happened.
3.15 Multi-tenancy and decentralized agent development
As adoption scales, business units will want specialized agents with different knowledge, tools, policies, and owners. A centralized platform should not force all teams into one monolithic agent. It should provide common controls while allowing domain teams to own their agents and capabilities.
Google’s current multi-tenant agentic architecture highlights this tension: decentralized teams need autonomy, but fragmented agent silos create governance gaps, operational overhead, and data-exposure risk. The platform should centralize identity, policy, registries, observability, evaluation standards, and deployment controls while isolating tenant data, memory, indexes, secrets, quotas, and capability access.
| Shared platform concern | Tenant-specific concern |
|---|---|
| Gateway, common identity integration, policy framework | Users, groups, delegated roles, approved models |
| Agent and capability registries | Domain agents, versions, owners, lifecycle |
| Observability schema and evaluation framework | Tenant traces, test sets, thresholds, business metrics |
| Deployment pipeline and security baseline | Domain configuration, connectors, knowledge sources |
| Cost allocation and quota framework | Budgets, token quotas, concurrency, priority |
3.16 Architecture decisions that must be explicit
Architecture Decision Records should capture choices that materially affect safety, portability, cost, and operations. The following decisions should not remain implicit in framework defaults.
# ADR-014: Production tool execution boundary
Status: Accepted
Context:
Agents require access to CRM and billing functions. Raw APIs contain broad
administrative operations and inconsistent authorization semantics.
Decision:
Expose versioned business capabilities through the enterprise capability gateway.
Agents cannot call raw CRM, billing, SQL, shell, or unrestricted HTTP interfaces.
Controls:
- Per-action policy decision
- User and agent attribution
- Short-lived workload credential
- Schema and domain validation
- Approval for high-impact actions
- Idempotency and post-condition verification
Consequences:
Additional adapter development is required, but tool scope, auditability,
testing, and provider portability improve.3.17 Implementation checklist
3.18 Chapter summary
A production enterprise AI platform is not a model wrapped in a chat interface. It is a governed distributed system with explicit channels, gateways, orchestration, model services, knowledge pipelines, business capabilities, identity, policy, observability, evaluation, deployment, and recovery.
The architecture should allow models and agents to remain adaptive where adaptation creates value while keeping enterprise authority deterministic. Models interpret. Orchestrators coordinate. Knowledge services ground. Policy services authorize. Connectors execute. Workflow services preserve state. Observability and evaluation provide evidence that the system behaves as intended.
Core conclusion: the reference architecture succeeds when changing the model does not require redesigning enterprise identity, policy, capability contracts, audit, or recovery.
Those durable controls are the foundation of Enterprise Agent Engineering and the difference between an impressive prototype and an enterprise system that can be trusted in production.
Sources and further reading
- Microsoft, Multi-Agent Reference Architecture. Conceptual and practical guidance on building blocks, registries, memory, communication, observability, evaluation, security, governance, and reference architecture.
- Microsoft Learn, Agent architecture components. Core components including client, infrastructure, orchestrator, language model, catalog, and tool calling.
- Microsoft Azure Architecture Center, Dynamic AI agents at scale pattern. Guidance on agent factories, agent selection, private networking, managed identity, observability, and controlled external access.
- Microsoft Azure Architecture Center, AI agent orchestration patterns. Guidance to use the lowest sufficient complexity and choose among orchestration patterns.
- Microsoft Azure Architecture Center, Baseline Microsoft Foundry chat reference architecture. Production design considerations for private networking, managed state, identity, controlled egress, reliability, and observability.
- AWS Prescriptive Guidance, Agentic AI architecture in the enterprise. Layered enterprise architecture with security, discoverability, and observability as cross-layer concerns.
- Google Cloud Architecture Center, Multi-tenant agentic AI system. Reference architecture for centralized governance and decentralized specialized agent development.
- Google Cloud Architecture Center, Generative AI with RAG reference architectures. Infrastructure guidance for end-to-end retrieval pipelines and enterprise grounding.