Brian Jagger
When I first started drawing AI architectures on whiteboards, I made the same mistake almost every architect makes early in their career. I treated the model as the centre of the universe, and everything else as plumbing. After a dozen production systems and a few painful incidents, I learned that the model is the easy part. The data flowing into it, the inference layer serving it, the application wrapping it, and the governance constraining it are where real architectures live or die.
The AI Reference Architecture: Layers, Patterns & Examples
In this article, I want to walk you through the AI reference architecture I now use as my mental model for almost every engagement. It is a five-layer model that maps neatly onto how enterprise AI systems behave in production, and it pairs naturally with six common architectural patterns that I see again and again across industries.
I will keep this practical. Expect concrete components, the tools I reach for, the trade-offs I weigh, and a few example architectures I have personally shipped. If you are designing your first AI platform, or refactoring one that has grown organically, this should give you a structured way to think about the choices ahead.
Why a reference architecture matters
A reference architecture is not a blueprint you copy. It is a shared vocabulary that lets a cross-functional team reason about an AI system at the right level of abstraction. Without one, conversations drift between vendor pitches, model benchmarks, and infrastructure debates that never converge.
I have watched organisations spend months arguing about whether to use Pinecone or pgvector when the real question was whether retrieval belonged in their architecture at all. A reference architecture forces those upstream decisions first. It also gives leadership a way to understand cost, risk, and dependencies without drowning in implementation detail.
Three benefits I consistently see when teams adopt a clear reference architecture:
- Faster decisions. New components get slotted into known layers instead of triggering open-ended debate.
- Cleaner accountability. Each layer has an owner, which matters when an incident spans data, model, and application boundaries.
- Better cost modelling. Costs are tracked per layer, making it obvious where optimisation effort will pay off.
A good reference architecture is opinionated enough to guide decisions, but flexible enough to accommodate the six patterns I will cover later. The five-layer model I use does exactly that.
The five-layer AI reference architecture
The reference architecture I use has five horizontal layers, with governance and observability cutting vertically across all of them. Each layer has a clear responsibility, well-defined interfaces, and its own tooling ecosystem.
| Layer | Responsibility | Owners |
|---|---|---|
| Data | Ingest, transform, store features and embeddings | Data engineering |
| Model | Train, fine-tune, register, version | ML engineering |
| Inference and serving | Host models, route requests, scale capacity | Platform engineering |
| Application | Surface AI capabilities through APIs and UI | Product engineering |
| Governance and observability | Enforce policy, monitor behaviour, audit usage | Risk and SRE |
The boundaries between layers matter more than the contents. Strong contracts between layers let teams move independently and let you swap implementations without rewriting the world.
I draw this architecture on a whiteboard for almost every kick-off meeting I run. It gives stakeholders a place to put their concerns. The legal team gravitates to governance. The CFO asks questions about inference economics. Product leadership focuses on the application layer. Engineering teams find their home in data and model layers. Each conversation becomes more productive because nobody is talking past anyone else.
Now let us walk through each layer in detail.
The data layer is the foundation, and it is where I find most enterprise AI systems are weakest. Generative AI has not eliminated the need for clean, well-governed data. If anything, it has raised the stakes because hallucinated outputs from poor inputs reach customers directly.
Core components of the data layer:
- Source connectors for transactional systems, document repositories, CRM, knowledge bases, and event streams.
- Transformation pipelines that clean, deduplicate, and normalise records, often built on dbt, Spark, or modern Python frameworks.
- Feature store for tabular ML features, with point-in-time correctness for training and low-latency reads for serving.
- Vector store for embeddings used in retrieval, semantic search, and similarity workloads.
- Object storage for raw documents, images, audio, and intermediate artefacts.
- Metadata catalogue for lineage, schema, ownership, and sensitivity classification.
Tools I reach for here include Databricks or Snowflake for the analytical backbone, Feast or Tecton for feature stores, Pinecone, Weaviate, or pgvector for vectors, and OpenMetadata or Unity Catalog for the catalogue.
The single most underrated investment in this layer is metadata. When an auditor asks where the answer to a customer query came from, you want to trace from the response back through retrieval, embedding, document, source system, and access policy. Without strong metadata, that trace is impossible.
Layer 2: Model
The model layer is where models are produced, registered, and managed. Even in heavily LLM-driven architectures, this layer is not optional. You still have classification models for routing, embedding models, reranking models, safety classifiers, and increasingly, fine-tuned variants of foundation models.
Key components:
- Training infrastructure for fine-tuning, distillation, and adapter training.
- Experiment tracking so you can compare runs and reproduce results.
- Model registry with versioning, lineage, and approval workflows.
- Evaluation harness that runs offline and red-team tests before promotion.
- Model catalogue that catalogues both internal models and approved third-party models.
| Component | Common tools |
|---|---|
| Training | PyTorch, JAX, HuggingFace, Determined |
| Experiment tracking | MLflow, Weights and Biases, Comet |
| Registry | MLflow, SageMaker Model Registry, Vertex AI Model Registry |
| Evaluation | Promptfoo, Ragas, custom eval pipelines |
I treat third-party models, like OpenAI or Anthropic endpoints, as first-class entries in the model registry. They have versions, evaluation results, cost characteristics, and owners. Treating them as opaque external dependencies leads to surprises when providers deprecate models or change behaviour.
Layer 3: Inference and serving
This is the layer that gets the most attention in operational reviews because it sits in the request path. Latency, cost, and reliability live or die here.
The inference layer typically contains:
- Model gateway that abstracts model providers behind a unified API, handles authentication, and enforces routing rules.
- Inference runtimes for self-hosted models, such as vLLM, TGI, or Triton.
- Caching tiers for prompt-response caching, embedding caching, and semantic caching.
- Rate limiting and quota management to prevent runaway costs and protect shared capacity.
- Routing logic that selects between models based on cost, latency, or capability.
I almost always introduce a model gateway, even when there is only one model. It pays for itself the first time you need to fail over to a backup provider, switch models without changing application code, or apply a token budget. Tools like LiteLLM, Portkey, and homegrown gateways built on FastAPI all work fine.
Layer 4: Application
The application layer is where AI capabilities meet users. It includes the orchestration logic, prompts, tool integrations, and user interfaces that turn raw model outputs into useful product features.
I think of this layer in three sub-layers:
- Orchestration. Frameworks like LangChain, LlamaIndex, or homegrown orchestrators that compose models, retrieval, tools, and prompts.
- Tool integration. Connections to internal APIs that let agents take action, query databases, or call business systems.
- Experience layer. Chat UIs, embedded assistants, batch processing endpoints, and API surfaces that consumers interact with.
A pattern I encourage: keep prompts, tool definitions, and orchestration logic in version-controlled code, not in databases or configuration files that bypass code review. Prompt changes are software changes. They deserve the same diligence.
Layer 5: Governance and observability
The fifth layer is technically two concerns that I bundle together because they share the same vertical cross-cutting nature. Governance enforces what the system is allowed to do. Observability tells you what it actually did.
Governance components:
- Policy engine that enforces access controls, content policies, and usage limits.
- Audit log that records every prompt, response, retrieval, and tool call with sufficient detail to reconstruct events.
- PII detection and redaction at ingest, retrieval, and response time.
- Approval workflows for new use cases, prompts, and models.
Observability components:
- Tracing that follows a request across orchestration, retrieval, and model calls.
- Quality metrics including hallucination rates, refusal rates, and user feedback signals.
- Cost telemetry broken down by tenant, feature, and model.
- Drift detection for embeddings, inputs, and outputs.
If you cannot answer the question “what did our AI tell that customer yesterday, and why?” within minutes, your observability is not production-ready.
Pattern 1: Single LLM application
The simplest pattern is a single LLM behind a thin application. No retrieval, no agents, just a model with carefully designed prompts.
This pattern works well for:
- Internal productivity tools where users supply context themselves.
- Stateless transformations like summarisation, translation, and classification.
- Prototypes and proofs of concept where you want to validate value before adding complexity.
Trade-offs to weigh:
- Pros. Lowest complexity, fastest to ship, easiest to debug.
- Cons. Limited by the model’s training data. No grounding in proprietary knowledge. Vulnerable to hallucination on domain-specific questions.
I see teams skip this pattern because it feels too simple, then build something more complex than they needed. Start here unless you have a clear reason not to.
Pattern 2: Retrieval-augmented generation
RAG is the workhorse pattern of enterprise AI. The architecture combines a retrieval system with a generative model so the model can answer questions grounded in your data.
Reference flow:
- User submits a query.
- The query is embedded and used to retrieve relevant chunks from a vector store, often combined with keyword search.
- Retrieved chunks are reranked, filtered for permissions, and assembled into a prompt.
- The LLM generates a response grounded in the retrieved content.
- Citations link back to source documents.
Trade-offs:
- Pros. Grounds responses in current, proprietary data. Easier than fine-tuning. Updates without retraining.
- Cons. Retrieval quality dominates output quality. Latency adds up. Chunking is fiddly. Multi-hop reasoning is hard.
This is the pattern I deploy most often, and the one I cover in depth in our enterprise RAG article.
Pattern 3: Agentic architectures
Agentic patterns introduce an LLM that can plan, choose tools, and iterate toward a goal. The architecture adds a planning loop, a tool catalogue, and state management on top of the basic LLM call.
Components specific to agentic systems:
- Planner that decomposes a goal into steps.
- Tool catalogue with structured schemas for each callable action.
- Memory for short-term scratchpads and long-term context.
- Guardrails that limit which tools can be called and under what conditions.
- Trajectory logging to capture the full reasoning trace.
Trade-offs:
- Pros. Handles complex, multi-step tasks. Adapts to varied inputs. Composes existing services in novel ways.
- Cons. Reasoning errors compound. Evaluation is hard. Cost and latency are unpredictable. Failure modes are creative.
I treat agentic architectures as a graduation from RAG, not a replacement. Start with constrained tool use and expand only as evaluation confidence grows.
Pattern 4: Multi-modal architectures
Multi-modal patterns handle text, image, audio, video, or structured data in the same architecture. The reference architecture barely changes, but the data and model layers expand significantly.
Specific additions:
- Modality-specific preprocessing for image resizing, audio chunking, and video frame extraction.
- Multi-modal embeddings for cross-modal retrieval.
- Modality-specific models alongside multi-modal foundation models.
- Output composition that may produce text, images, or speech.
Common use cases include document understanding, visual quality inspection, multi-modal search, and conversational agents that handle voice. The biggest architectural challenge is usually data volume: video and audio are orders of magnitude larger than text, and storage and bandwidth become first-class concerns.
Pattern 5: Fine-tuned domain models
When prompt engineering and RAG hit their limits, fine-tuning enters the conversation. The architecture changes in two ways: the model layer grows training infrastructure, and the governance layer takes on additional model risk responsibilities.
When fine-tuning earns its place:
- High-volume tasks where per-token cost matters.
- Strong domain vocabulary that base models do not handle well.
- Latency-sensitive workloads where smaller fine-tuned models replace larger general ones.
- Privacy constraints that prevent sending data to external APIs.
Architectural impact:
- Training and evaluation pipelines become permanent fixtures.
- A model registry with rigorous approval workflows is essential.
- Self-hosted inference, often on GPUs, joins the inference layer.
- Drift monitoring becomes more important because the model and the data both evolve.
Pattern 6: Hybrid on-prem plus cloud
Regulated industries and large enterprises often need a hybrid pattern where sensitive data stays on-premises while less sensitive workloads use cloud APIs. The architecture splits along data sensitivity lines.
A typical hybrid layout:
- On-prem zone hosts sensitive data, embedding models, vector stores, and self-hosted LLMs.
- Cloud zone hosts general-purpose LLMs, scaling capacity, and non-sensitive workloads.
- Routing layer classifies requests and routes them to the appropriate zone.
- Federated audit consolidates logs across zones for compliance.
| Concern | On-prem | Cloud |
|---|---|---|
| Sensitive data | Yes | Redacted only |
| Cost per token | Higher | Lower |
| Model quality | Self-hosted models | Frontier models |
| Latency | Lower for nearby users | Variable |
| Scaling | Capital-bound | Elastic |
I have shipped hybrid architectures for banking and healthcare. The hardest part is not the technology, it is the routing policy. Misclassification can leak data to the wrong zone, so the classifier deserves the same scrutiny as the rest of the system.
Choosing the right pattern: a decision guide
When I help teams pick a pattern, I work through these questions in order:
- Does the task require proprietary data the model does not know? If no, single LLM may be enough.
- Is the knowledge mostly static or rapidly changing? Static favours fine-tuning, dynamic favours RAG.
- Does the task require taking actions across multiple systems? That is agentic.
- Are non-text modalities essential? Plan for multi-modal.
- Are there data residency or sovereignty constraints? Hybrid is likely required.
- What is the volume and latency target? High volume and tight latency push toward fine-tuned smaller models.
Most production systems end up combining patterns. A customer service assistant might be RAG over knowledge articles, plus an agentic layer for taking actions like resetting passwords, plus a fine-tuned classifier for intent routing, all hosted in a hybrid topology.
The reference architecture stays the same. The patterns slot into the layers. That is the whole point.
