Brian Jagger
Most of my work over the past three years has involved designing retrieval-augmented generation systems for enterprise customers. I have shipped RAG for banks, hospitals, manufacturers, government departments, and several large software companies. Each project taught me something new, but the patterns and pitfalls have become remarkably consistent. RAG is the workhorse pattern of enterprise AI in 2026, and getting it right repays the investment many times over.
Designing Enterprise RAG Systems: An AI Architect’s Guide
What surprises me is how often teams underestimate enterprise RAG. The basic idea, retrieve relevant context and pass it to a language model, sounds simple enough that organisations attempt it without architectural discipline. The result is a system that demos beautifully and then fails to scale, fails compliance, or simply fails to give useful answers when users ask questions that vary from the rehearsed demo.
This article is the architectural playbook I wish I had had when I started. It walks through the full RAG architecture, the enterprise constraints that shape it, the design choices you will have to make around chunking, retrieval, and evaluation, and the anti-patterns I have seen repeatedly. By the end, you should have a clear mental model for designing RAG systems that survive contact with reality.
What enterprise RAG actually is
RAG combines a retrieval system with a generative model so that responses are grounded in your own data rather than the model’s training corpus. In its simplest form, a user query is embedded, the most similar document chunks are retrieved, and they are passed to a model as context for generating a response.
Enterprise RAG is the same idea executed under harder constraints. It must handle:
- Heterogeneous data sources: documents, knowledge bases, ticket systems, code, structured tables.
- Permissions that vary by user, role, region, and sensitivity classification.
- Auditability for compliance and regulatory review.
- Performance and reliability requirements measured in tens of milliseconds and four or five nines.
- Cost budgets that turn token efficiency into a first-class concern.
The architectural complexity grows fast. A small consumer chatbot can be RAG with three components. An enterprise customer service assistant routinely has fifteen or twenty.
The reference RAG architecture
I draw enterprise RAG as a pipeline with eight stages, plus a parallel evaluation and observability track.
| Stage | Responsibility |
|---|---|
| Ingestion | Connect to sources and pull content |
| Processing | Parse, normalise, redact, classify |
| Chunking | Split content into retrieval units |
| Embedding | Convert chunks into vectors |
| Indexing | Store vectors and metadata for retrieval |
| Retrieval | Find relevant chunks for a query |
| Reranking | Refine results with a more expensive model |
| Generation | Produce a response grounded in retrieved content |
Around the pipeline run evaluation, monitoring, audit, and governance. Each stage can fail in its own way and needs its own quality bar. The architectural skill is not just connecting the stages, but making each one observable and replaceable.
A RAG system is only as good as its weakest stage. I have seen excellent generation undermined by poor chunking, and clever retrieval undone by stale ingestion.
Ingestion and document processing
Ingestion is the unglamorous foundation of every RAG system. Get it wrong and everything downstream suffers. Get it right and the rest of the system becomes much easier.
Key concerns:
- Source coverage. Which systems are in scope? Confluence, SharePoint, Salesforce, ticketing tools, code repositories, file shares, and content management systems each have their own access patterns.
- Incremental updates. Pulling everything every night is wasteful and may breach SLAs. Change data capture, webhooks, and modified-since cursors are better.
- Document parsing. PDFs, Office documents, HTML, Markdown, and image-heavy content all parse differently. Plan for document type detection and specialised parsers.
- Normalisation. Trim boilerplate, deduplicate near-duplicates, fix encoding issues, and standardise metadata.
- Classification. Tag each document with sensitivity, owner, retention, and access rules.
I treat ingestion as a first-class data engineering workload. Tools like Unstructured, LlamaParse, and custom pipelines built on Apache Tika or PyMuPDF do most of the heavy lifting. The discipline is consistent: every document has a source, an owner, a sensitivity label, and a refresh policy.
Chunking strategies that scale
Chunking is where many RAG systems quietly fail. Naive splitting by character count produces chunks that cut across sentences, lose context, or fail to capture the semantic unit users care about.
Strategies I use, roughly in order of preference:
- Semantic chunking. Split at natural boundaries like paragraphs, sections, or sentence groups. Often combined with overlap to preserve continuity.
- Document-structured chunking. Use the document’s own structure: headings for articles, function boundaries for code, slide boundaries for decks.
- Recursive chunking. Start with large units and recursively split until they fit a target size.
- Hierarchical chunking. Store multiple chunk sizes and retrieve the appropriate level for the query.
- Late chunking. Embed at the document level and resolve to chunk level at retrieval. Useful when context matters across boundaries.
| Strategy | Pros | Cons |
|---|---|---|
| Semantic | Preserves meaning | Variable chunk sizes |
| Structured | Aligns with document layout | Depends on clean structure |
| Recursive | Flexible | Risk of arbitrary cuts |
| Hierarchical | Versatile | More storage and complexity |
| Late chunking | Context-rich | Higher compute at retrieval |
The trap is over-engineering. Start with semantic or structured chunking, measure retrieval quality, and only add complexity when the evaluation justifies it.
Embedding models and choice criteria
The embedding model determines what counts as similar. Choose it carefully.
Factors I weigh:
- Quality on your domain. Run a benchmark on your actual data. Generic benchmarks are weak proxies.
- Dimensionality. Higher dimensions can capture more nuance but increase storage and query cost.
- Multilingual support. Important for organisations operating across languages.
- Self-hosting versus API. Trade off control, latency, and cost.
- Licence. Commercial use, fine-tuning, and modification rights vary widely.
- Versioning. Embeddings from different versions are not comparable. Plan for migrations.
In 2026 the strong contenders include OpenAI’s embedding family, Cohere’s embed family, Voyage AI, BGE and E5 in the open-source space, and increasingly specialised models for code and multi-modal content. I usually evaluate at least three on the target domain before committing.
Embedding model migrations are expensive. A small reduction in quality at choice time can save months of rework later.
Vector storage and indexing
Vectors live in a vector store. The choice depends on scale, ops capacity, and feature requirements. I cover the landscape in detail in our vector databases article, but a few enterprise considerations belong here.
Critical requirements:
- Permission filtering at retrieval. The store must apply filters efficiently. Post-retrieval filtering is too slow and often incorrect.
- Multi-tenant isolation. Per-tenant collections or namespaces, with quotas and audit.
- Hybrid search. Combine vector similarity with keyword search and structured filters in one query.
- Snapshot and restore. Treat the vector store as production data with proper backup.
- Versioning. Track embedding model versions to avoid mixing incompatible vectors.
Enterprise-ready options include Pinecone, Weaviate, Qdrant, Milvus, Vespa, Elasticsearch, MongoDB Atlas Vector Search, and pgvector. Each has trade-offs, and the right answer is usually shaped by the team’s operational maturity rather than benchmark numbers.
Retrieval, hybrid search, and reranking
Retrieval is where the system finds candidate chunks for a query. The simplest implementation, a single vector similarity search, leaves significant quality on the table.
Patterns that materially improve quality:
- Hybrid search. Combine dense vector similarity with sparse keyword retrieval, typically BM25. Use reciprocal rank fusion or learned combinations.
- Query rewriting. Expand the user’s query into multiple search queries, especially for ambiguous or short inputs.
- HyDE. Hypothetical document embeddings generate a draft answer and embed it for retrieval. Useful when queries are short and documents are long.
- Reranking. Use a cross-encoder or LLM-based reranker on the top fifty results to produce a refined top five.
- Permission-aware retrieval. Apply permissions as part of the retrieval filter so users never see content they should not.
Reranking is the single highest-leverage addition I make to most RAG systems. The latency cost is real but usually acceptable, and the quality improvement is often dramatic.
Generation, citations, and structured outputs
The generation stage produces the response from the retrieved context. The architectural concerns are quality, faithfulness, and safety.
What I include in nearly every system:
- System prompts that establish role, scope, and rules.
- Citation requirements so responses reference the chunks they used. This supports trust and audit.
- Structured output schemas when the consumer is a downstream system rather than a human.
- Guardrails for content safety, PII leakage, and competitor content.
- Refusal patterns that produce safe fallback responses when retrieval fails or content is out of scope.
Citations are not a nice-to-have. They are how users build trust with the system and how auditors verify it. Make them a first-class output.
Evaluation: Ragas, custom suites, and human review
Evaluation is what separates RAG demos from RAG products. Without it, you cannot tell whether a change improved or degraded the system.
Layers of evaluation I run:
- Retrieval metrics. Recall, precision, mean reciprocal rank against a labelled dataset.
- Generation metrics. Faithfulness, answer relevance, citation accuracy. Tools like Ragas, TruLens, and DeepEval are useful starting points.
- End-to-end task metrics. Did the user achieve their goal? Did the answer reduce ticket volume?
- Red-team evaluation. Adversarial prompts, jailbreak attempts, and edge cases.
- Human review. Periodic deep dives by domain experts. No automated metric replaces this.
I treat evaluation as a CI-style pipeline. Every change runs against the suite. Regressions block deployment. The discipline is what separates teams that improve their RAG system from teams that argue about it.
Enterprise constraints that shape design
Enterprise RAG is shaped as much by constraints as by quality goals. The constraints that show up most often:
- PII and sensitive data. Detection at ingestion, redaction during retrieval, and policy enforcement at generation.
- Multi-tenant isolation. Strong boundaries between customers or business units.
- Audit trails. Every query, retrieval, and generation logged with sufficient detail for compliance review.
- Latency targets. End-to-end SLAs measured at the ninety-fifth or ninety-ninth percentile.
- Data residency. Documents may be required to stay in specific geographies.
- Regulatory exposure. Sector rules, the EU AI Act, and contractual obligations all apply.
These constraints often dictate architecture more than performance considerations. An on-premises RAG system for a government department looks very different from a cloud-hosted RAG for a consumer app, even if the core pipeline is the same.
Multi-tenancy, PII, and access control
Multi-tenancy is one of the hardest aspects of enterprise RAG. The right model depends on the trust boundary.
Common patterns:
- Logical isolation. Shared infrastructure with tenant identifiers on every record. Simplest but requires careful enforcement.
- Physical isolation. Separate indexes per tenant. Stronger isolation but higher ops overhead.
- Hybrid isolation. Logical isolation for low-sensitivity tenants and physical isolation for high-sensitivity ones.
PII handling deserves its own attention. I default to:
- Detection at ingestion using a combination of regex, named entity recognition, and dedicated classifiers.
- Configurable handling: redaction, masking, or routing to a separate sensitive store.
- Policy enforcement at retrieval based on user role and request context.
- Audit logging of any PII access.
Access control is enforced at retrieval, never at generation alone. Filtering after the model has seen the content is too late.
Cost modelling for RAG at scale
Cost surprises kill more RAG projects than quality issues. A simple model helps.
| Cost component | Driver | Order of magnitude |
|---|---|---|
| Embedding generation | Document count and refresh rate | One-time plus delta |
| Vector storage | Number of vectors and replicas | Steady monthly |
| Retrieval queries | Query volume | Per request |
| Reranking | Query volume times top-k | Per request, higher |
| Generation | Tokens per response times volume | Largest variable |
| Observability and logging | Storage and processing | Often underestimated |
The dominant cost in most enterprise RAG systems is generation tokens. Reduce prompt size aggressively. Cache common responses. Use cheaper models for routine queries with escalation to stronger models when needed.
I build a per-tenant cost dashboard as part of the initial architecture. The visibility shapes design decisions and prevents budget surprises.
Common anti-patterns and how to avoid them
The mistakes I see most often in enterprise RAG.
- No evaluation harness. Teams ship and pray. Without measurement, the system cannot improve.
- Single chunking strategy across all content. Articles, code, transcripts, and tables benefit from different chunking.
- Ignoring metadata. Pure vector search loses crucial filters. Always combine with metadata.
- No reranking. Top-k from vector similarity alone leaves quality on the table.
- Prompt engineering as configuration. Prompts live in a database, bypassing code review. They are software and should be versioned.
- Permissions enforced after retrieval. Too late and often incorrect. Filter at the index.
- Cost discovered in production. Build the cost model in design, not after the first invoice.
- Hallucinated citations. Models invent source IDs. Verify citations match real retrieved chunks.
- Treating the model as the system. The model is one component. The system is everything around it.
Every one of these anti-patterns is fixable. The cost of fixing them late is several times the cost of avoiding them from the start.
A pragmatic roadmap from prototype to production
The path I usually recommend for enterprise RAG:
Weeks 1 to 2: scoped prototype. One use case, one data source, basic chunking, off-the-shelf embeddings, simple retrieval, generation with a major API model.
Weeks 3 to 6: evaluation foundation. Build a labelled dataset of representative queries. Stand up Ragas or a custom evaluation harness. Establish a baseline.
Weeks 7 to 12: production-ready core. Add reranking, hybrid search, structured citations, and observability. Migrate to a production vector store. Implement permission filtering.
Months 4 to 6: enterprise hardening. Multi-tenancy, audit logging, PII handling, cost telemetry, red-team evaluation, and runbooks.
Months 7 onwards: scale and refine. Expand to additional use cases, optimise cost, fine-tune the generation model, and iterate on retrieval quality.
The mistake is to compress this roadmap into weeks. Enterprise RAG is built on accumulated rigour, not single sprints.
