Brian Jagger
The first enterprise AI system I designed end-to-end was a customer-service assistant for a mid-sized retailer. It was 2023, the team was excited, the budget was generous, and we shipped a beautiful demo in six weeks. We then spent eight months unwinding the architecture, because the demo had been designed for a different reality than production. The cost model did not survive contact with peak traffic. The retrieval layer did not survive the marketing team’s content updates. The evaluation harness did not survive the first regulatory enquiry. I have not designed an enterprise AI system the same way since.
How AI Solution Architects Design Enterprise AI Systems
This article is the process I now use, refined across roughly fifteen production systems in finance, retail and healthcare. It is not a methodology in the consulting sense. It is a working sequence that produces the artefacts I need to defend my choices to a CFO, a CISO, a product manager and a regulator, sometimes in the same week. I have tried to be specific about deliverables, decision frameworks and the order of operations, because abstract process advice is not useful.
I will walk through the full lifecycle, then a worked example of a customer-service AI system. I write in British English, I keep the language plain, and I show my homework where it matters.
The Eight-Phase Design Process
My working sequence has eight phases. They are not strictly linear; phases three through six iterate, sometimes painfully. But the sequence holds well enough that I use it as a mental scaffold for every new system.
- Intake: capture the problem, the stakeholders and the constraints.
- Requirements: convert intake into functional and non-functional requirements.
- Options Analysis: enumerate two to four credible architectural options.
- Architecture Decision Records: record the chosen direction with trade-offs.
- Prototype: build a thin slice end-to-end to validate the riskiest assumptions.
- Scale: harden the prototype into a production system.
- Govern: wrap the system in evaluation, observability and compliance.
- Operate: run the system, learn from production, and evolve the architecture.
The total elapsed time from intake to first production traffic is typically three to six months for a meaningful system. Trying to shorten this to weeks is almost always a false economy, because the shortcuts taken in phases one and two compound expensively in phases six and seven.
Phase 1: Intake and Problem Framing
Intake is the most undervalued phase. The mistake I made on my first system was treating intake as a kick-off meeting rather than a discovery exercise.
What I now do during intake, over two to three weeks:
- Meet the primary business sponsor and capture the problem in their language.
- Meet at least two end users and observe a session of their current workflow.
- Meet finance to understand the budget envelope and the unit economics that will be acceptable.
- Meet legal and risk to understand any regulatory exposure (PII, financial advice, medical advice, employment decisions).
- Meet security to understand the data classification and the threat model.
- Meet platform engineering to understand what infrastructure exists already.
The output of intake is a one-page problem statement and a stakeholder map. The problem statement answers four questions: what is the business outcome, who is the primary user, what is the budget envelope, and what is the timeline.
If you cannot fit the problem statement on one page, you do not yet understand the problem.
I have come to think of intake as the single highest-leverage phase. Every hour spent here saves five hours later.
Phase 2: Requirements and Non-Functional Requirements
Once intake is done, I move to a structured requirements document. This is where many AI projects go wrong, because the team focuses on the functional requirements (“the assistant should answer questions about returns”) and skips the non-functional requirements (“the assistant should answer in under two seconds, ninety-five per cent of the time, at a cost per interaction below two cents, with a refusal rate above eighty per cent on out-of-scope queries”).
The non-functional requirements I capture for every AI system:
| NFR | Typical Target |
|---|---|
| Latency (p50) | Under 1.5 seconds for interactive |
| Latency (p95) | Under 3.5 seconds for interactive |
| Throughput | Peak QPS, expected and burst |
| Availability | 99.5% or 99.9% depending on criticality |
| Cost per interaction | Stated explicitly in cents |
| Quality threshold | Defined evaluation metric, with floor |
| Safety threshold | Refusal rate, hallucination rate, toxicity rate |
| Privacy posture | PII handling, residency, retention |
| Auditability | Decision logging, prompt versioning |
| Recovery | RPO and RTO for stateful components |
Without explicit NFRs, the system will be designed to the wrong constraints. I have lost count of the times a team optimised for accuracy and then discovered the system was twice the acceptable cost.
Phase 3: Options Analysis
With requirements in hand, I enumerate two to four architectural options. The discipline is to make each option genuinely distinct, not three variations on the same theme.
For a typical generative AI system, the canonical option set might be:
- Option A: Pure prompt engineering against a frontier model, no retrieval, no fine-tuning.
- Option B: Retrieval-augmented generation with a hybrid retriever, against a frontier model.
- Option C: Fine-tuned smaller model with retrieval, hosted on managed inference.
- Option D: Multi-agent system with a planner, multiple specialised tools, and a frontier reasoning model.
For each option, I produce a one-page summary covering: capability fit, expected latency, expected cost per interaction, security and compliance posture, vendor risk, and engineering effort. I then score the options against the requirements.
The most common mistake is to enumerate options that all converge on the same architecture. If the score is close to identical, the options are not really different. Go back and find a more genuine alternative.
Phase 4: Architecture Decision Records
Once an option is selected, I write the ADR. The ADR is the canonical record of why the system looks the way it does. Every senior engineer on the project reads it. Every reviewer in calibration reads it. Future architects who inherit the system rely on it.
My ADR template:
- Status: proposed, accepted, deprecated, superseded.
- Context: the situation that drove the decision, including the constraints.
- Decision: the choice made, stated unambiguously.
- Alternatives considered: the other options, with reasons for rejection.
- Consequences: the positive and negative outcomes accepted.
- Review date: when this decision should be revisited.
I write ten to fifteen ADRs over the lifecycle of a meaningful system. The big ones cover the retrieval strategy, the model choice, the orchestration framework, the cost ceiling and the evaluation approach. Smaller ones cover narrower choices that nonetheless deserve a record.
I have a personal rule: if I cannot explain the decision in writing in under two pages, I do not understand it well enough yet.
Phase 5: Prototype and Validation
With ADRs in hand, I build a thin slice prototype. The goal of the prototype is not to demo a happy path. It is to validate the riskiest assumptions before the team commits to the full build.
The riskiest assumptions vary by system, but typically include:
- The retrieval quality on real, representative content.
- The cost per interaction at production traffic shape.
- The latency at p95 under load.
- The behaviour of the model on adversarial inputs.
- The integration with downstream systems.
I budget two to four weeks for the prototype, and I am explicit that it is a learning artefact, not a starting point for production code. I instrument the prototype heavily, particularly around cost and latency. I run an evaluation harness against a representative dataset and capture the baseline metrics.
The output of the prototype phase is a memo that updates the ADRs based on what we learned. Some assumptions survive contact with reality, some do not. The architects who skip this phase end up rewriting the system in production.
Phase 6: Scaling to Production
Scaling is the longest phase, often three to four months. The work breaks down into a relatively predictable set of streams:
- Infrastructure: production-grade hosting, autoscaling, observability, secret management.
- Data: ingestion pipelines for retrieval content, freshness guarantees, deduplication.
- Inference: model gateway, fallback chains, caching, batching where applicable.
- Evaluation: continuous evaluation harness, A/B testing infrastructure, red-teaming.
- Observability: tracing across components, logging, metrics, dashboards.
- Cost controls: token ceilings, rate limits, alerting on cost anomalies.
- Security: identity, authorisation, prompt-injection mitigations, data loss prevention.
- Resilience: degradation paths, circuit breakers, graceful failure modes.
The architect’s role in this phase shifts from authorship to review. The engineering team executes; the architect reviews the implementation against the ADRs and intervenes when reality diverges from design.
The hardest decisions in scaling are not technical. They are about what to cut to ship on time.
I keep a running list of design debt incurred during scaling, with a commitment to revisit each item within ninety days of launch.
Phase 7: Governance and Compliance Wrap
Before any meaningful system ships, I wrap it in the governance scaffolding. This includes:
- Model card: a one-page summary of the model, its training data, its evaluation results and its known limitations.
- System card: a one-page summary of the system, its components, its data flows and its risk classification.
- Evaluation report: the latest pre-deployment evaluation results, including safety and quality metrics.
- Audit trail: confirmation that prompt versions, model versions and decision logs are captured.
- Incident playbook: the response plan for AI-specific failures.
- DPIA or equivalent: data protection impact assessment where required by jurisdiction.
For regulated industries, I expect a formal sign-off from risk, legal and security before the system goes live. For lighter touch contexts, the equivalent is an internal review against the same artefacts.
Governance is not a tax. It is the difference between a system that ships and a system that ships and survives.
Phase 8: Operate and Evolve
The architect’s job does not end at launch. The first ninety days post-launch are critical, and I stay close to the system during this period.
What I do in the operate phase:
- Review the daily evaluation telemetry and watch for drift.
- Co-own the incident commander rota with engineering for the first ninety days.
- Hold a thirty-day, sixty-day and ninety-day post-launch review with stakeholders.
- Update the cost model with actual production data and identify optimisation opportunities.
- Schedule the first formal architecture review at the six-month mark.
The systems that evolve well are the ones with explicit operate-phase ownership. The systems that decay are the ones that get handed over and forgotten.
Reference Architecture Patterns I Reuse
Across the last fifteen systems, a few patterns have proved durable enough to become defaults.
The Hybrid RAG Pattern: BM25 plus dense retrieval, with a reranker, against a managed vector database. Works for most knowledge-grounded use cases.
The Two-Tier Model Pattern: a small, cheap model handles routing and easy cases; a frontier model handles hard cases. Cuts cost by thirty to sixty per cent.
The Cached Prompt Pattern: a prompt cache layer in front of the model, keyed on user intent and content hash. Cuts cost by twenty to forty per cent on chat workloads.
The Streaming Orchestration Pattern: first token in under one second, then stream the rest. Hides latency in interactive use cases.
The Evaluation Sidecar Pattern: every production interaction is sampled into an evaluation pipeline, with the results aggregated into dashboards. Catches drift early.
The Safety Filter Pattern: input and output filters around the model, configurable per use case, with the choices recorded in an ADR.
I do not start a new design from a blank page. I start from these patterns and modify them.
Decision Framework: Build vs Buy
The framework I apply to every component decision:
| Factor | Lean Buy | Lean Build |
|---|---|---|
| Capability rate of change | Fast | Slow |
| Operational load | High | Low |
| Strategic differentiation | Low | High |
| Vendor lock-in tolerance | High | Low |
| Data residency constraints | Loose | Strict |
| Internal team capacity | Limited | Strong |
I score the component on each factor, weight the scores, and make a call. The discipline of writing the scoring down catches the cases where my instinct disagrees with the evidence.
In practice, I buy retrieval infrastructure, evaluation tooling and observability. I build the orchestration layer and the integration with proprietary systems. I am cautious about buying the agent orchestration layer, because lock-in risk is high and the capability is evolving fast.
Decision Framework: Fine-Tune vs RAG vs Prompt
The most common question I get from product teams is “should we fine-tune?” The answer is almost always no, and the framework I use is:
- Use prompt engineering first, if the model can be steered with instructions and few-shot examples to the required quality.
- Add RAG if the model needs facts it does not know, or facts that change over time.
- Add fine-tuning only if there is a sustained quality gap that prompt engineering and RAG cannot close, and the volume of work justifies the investment.
Fine-tuning has costs that are easy to underestimate: data preparation, training, evaluation, ongoing retraining as the base model evolves, and the risk of lock-in to a specific model. For most enterprise use cases, RAG plus careful prompt engineering closes ninety per cent of the gap.
There are exceptions: classification at very high volume, format-strict generation, and brand-voice transformations. For these, fine-tuning a smaller model often beats prompting a larger one on both quality and cost.
Worked Example: Customer-Service AI System
To make the process concrete, here is the worked example of a customer-service AI system for a mid-sized retailer. I have simplified some details, but the shape is real.
Intake: The business sponsor wanted to reduce contact volume to human agents by twenty per cent. The primary users were customers and internal agents. The budget envelope was three cents per interaction. The timeline was one quarter to first production traffic.
Requirements: Functional requirements included answering questions about orders, returns, products and store locations. NFRs included p95 latency under three seconds, ninety-nine per cent availability, an explicit refusal rate of at least seventy per cent on out-of-scope queries, and full traceability for any answer that influenced a refund decision.
Options Analysis: Four options considered. Option B (RAG with hybrid retrieval against a frontier model) and Option C (fine-tuned smaller model plus RAG) scored highest. Option C was rejected because of timeline and data availability.
ADRs: Twelve ADRs authored, covering retrieval strategy, model choice, orchestration framework, escalation logic, evaluation approach, cost ceiling and PII handling.
Prototype: Four-week prototype against three thousand representative tickets. Baseline accuracy of seventy-four per cent, cost per interaction of 2.1 cents, p95 latency of 2.4 seconds. Two ADRs revised based on findings.
Scale: Twelve weeks to production. Notable choices: prompt cache layer cut cost by twenty-eight per cent; a router model handled forty per cent of easy queries at one-tenth the cost; degradation path to canned responses if the model gateway exceeded its budget.
Govern: Model and system cards published. DPIA completed. Evaluation harness running daily on five hundred sampled interactions. Sign-off from risk and legal obtained.
Operate: Launch traffic at ten per cent for two weeks, then twenty-five per cent for two weeks, then full traffic. Post-launch reviews at thirty, sixty and ninety days. By ninety days, contact volume to human agents had reduced by seventeen per cent and cost per interaction had stabilised at 1.8 cents.
Deliverables produced: One-page problem statement; full requirements document; options analysis; twelve ADRs; HLD; LLD; cost model; evaluation report; model card; system card; incident playbook; ninety-day review memo.
The full design process took seventeen weeks from intake to full production traffic. The artefacts now form the reference for two adjacent systems being designed by other architects.
Deliverables Checklist
A consolidated reference for the artefacts I produce for every meaningful system.
| Phase | Deliverable |
|---|---|
| Intake | One-page problem statement, stakeholder map |
| Requirements | Functional and non-functional requirements document |
| Options | Options analysis, scored against requirements |
| ADRs | One ADR per major decision, typically 10 to 15 |
| Design | High-level design, low-level design |
| Prototype | Prototype memo with revised ADRs |
| Cost | Cost model, including unit economics and forward forecast |
| Evaluation | Evaluation plan, harness, baseline report |
| Governance | Model card, system card, DPIA, incident playbook |
| Operate | Thirty, sixty and ninety day post-launch reviews |
Not every system needs every artefact. Smaller systems can collapse several of these into a single document. But the discipline of asking “do we have the equivalent of each?” is what keeps the architecture honest.
