Data, observability, governance, and operational recovery

4C.1 Layer 10 — Data and knowledge operations

The data layer supports operational reads, analytical queries, knowledge retrieval, workflow state, memory, and evaluation. These stores have different authority and retention requirements. A vector index, conversation memory, warehouse, and ERP record should not be governed as though they were equivalent.

Data categories

Data minimization

The platform should not copy every enterprise dataset into a new AI repository. Copy only what the use case requires, preserve access attributes, define retention, and remove derived content when the source is deleted or access changes. Broad replication increases breach impact and creates a shadow system of record.

Freshness and consistency

Every context item should include retrieval time and source version. Freshness policy depends on the task: hours may be acceptable for trend analysis, while seconds may be required for entitlement or transaction decisions. When sources disagree, the system should apply explicit precedence or create a reconciliation task rather than allowing the model to choose the most plausible answer.

{
  "record_id": "account:AC-1092",
  "source_system": "crm-core",
  "authority": "system_of_record",
  "schema_version": "5.3",
  "retrieved_at": "2026-07-23T17:18:00Z",
  "classification": "confidential",
  "tenant": "enterprise-eu",
  "access_attributes": ["account-team","region:eu"],
  "freshness_policy": "max_age:300s",
  "lineage": ["crm.account","canonical.customer"]
}

4C.2 Layer 11 — Observability, audit, and cost

Enterprise AI observability connects the user request to the final business outcome. It must span gateway, identity, retrieval, model calls, agent decisions, policy results, capability invocation, connector behavior, approval, and verification.

Telemetry domains

Structured tracing

trace_id: tr_01J8...
spans:
  - name: gateway.request
    attributes:
      tenant: enterprise-eu
      task_class: incident_remediation
  - name: retrieval.query
    attributes:
      sources: [cmdb, monitoring, runbooks]
      permission_filter: applied
  - name: model.reason
    attributes:
      deployment: reasoning-enterprise
      tokens_in: 18342
  - name: policy.evaluate
    attributes:
      decision: allow_with_approval
      policy_version: prod-change-v4
  - name: connector.execute
    attributes:
      operation_id: op_44192
      status: verified

Prompt and tool content should not be copied indiscriminately into logs. Use structured fields, redaction, encryption, restricted access, and separate evidence storage when full content is required for audit. Observability must not become an uncontrolled secondary data lake.

Cost observability

Track cost by tenant, use case, model, workflow, agent, and business outcome. Token cost alone is incomplete. Include retrieval, compute, connector, storage, evaluation, human review, and failure-recovery cost. A lower-cost model that increases corrections or escalation may have worse total economics.

4C.3 Layer 12 — Governance and lifecycle

Governance defines who may create, approve, deploy, modify, monitor, and retire agents, models, prompts, policies, capabilities, and knowledge sources. It should be integrated with delivery pipelines rather than maintained only as documentation.

Governed assets

A central registry improves discoverability and accountability, but governance should not create a bottleneck for every low-risk change. Use risk tiers. A read-only internal assistant and an autonomous financial agent should not require identical approval paths.

Release gates

• Owner and sponsor are active.
• Threat model and data classification are current.
• Agent and capability versions are immutable and traceable.
• Evaluation thresholds are met for normal and adversarial cases.
• Policy, connector, and rollback changes are included in the release review.
• Canary, monitoring, and rollback criteria are defined.

4C.4 Cross-layer security

Security is not one layer because threats cross the entire stack. An indirect prompt injection begins in retrieved content, influences a model, selects a capability, and attempts to exfiltrate data through a connector. Controls must interrupt the chain at several points.

Threat-control matrix

4C.5 Continuous evaluation and assurance

Evaluation should cover system behavior, not only answer quality. It should test source selection, access filtering, tool choice, parameter validation, policy behavior, approval, retry, verification, refusal, and business outcome.

Evaluation dimensions

4C.6 Failure recovery and continuity

Recovery design must cover model outage, data-source outage, policy failure, connector timeout, duplicate event, corrupted index, expired approval, region failure, and security incident. The system should have an explicit degraded mode for each task class.

Rollback hierarchy

1. Configuration rollback: restore previous agent, prompt, policy, or model route.
2. Workflow rollback: stop new instances and resume from a safe checkpoint.
3. Business compensation: reverse or reconcile enterprise side effects.
4. Access rollback: revoke agent, connector, or model permissions.
5. Knowledge rollback: remove corrupted or unauthorized indexed content.

4C.7 Operating model and responsibilities

A production platform requires clear ownership across business, AI engineering, platform, security, data, and risk. The agent owner is responsible for the use case and behavior; the capability owner is responsible for the business action; the platform team operates shared infrastructure; security owns threat response; data owners govern authority and access.

4C.8 Architecture Decision Records

Important integration decisions should be recorded because framework defaults change and organizational assumptions are easily forgotten. ADRs should state context, decision, alternatives, controls, consequences, owner, and review date.

# ADR-021: Separate workflow state from agent memory

Status: Accepted

Context:
Long-running procurement tasks require approvals, deadlines, retries,
and compensation. Agent conversation memory is not durable or auditable enough.

Decision:
Store workflow state in the enterprise workflow service. Agent memory may
contain temporary reasoning context but cannot determine committed process state.

Controls:
- Every side effect references workflow_id and operation_id
- Approval is stored as a workflow event
- Agent restart cannot repeat completed operations
- Recovery uses workflow history, not generated summaries

Consequences:
Additional integration effort is required, but replay, audit, recovery,
and model portability improve.

4C.9 Production readiness checklist

4C.10 Summary

Chapter 4D completes the enterprise AI integration stack. The data layer preserves authority and lifecycle. Observability connects request to outcome. Governance manages assets and accountability. Security applies controls across every boundary. Evaluation measures behavior continuously. Recovery ensures that failures remain containable and reversible.