Fine-Tuning vs RAG vs Prompt Engineering: Architect Framework

When I started leading AI architecture reviews three years ago, almost every conversation began with the same question. A product team would arrive convinced they needed to fine-tune a model, then a platform lead would push for retrieval-augmented generation, and somewhere in the middle a sceptical engineer would mutter that they could probably solve the whole thing with a better prompt. I have watched the same argument play out across banks, insurers, healthcare providers and software vendors, and the truth is that none of these techniques is universally correct.

In this article I want to give you the decision framework I now use when an organisation asks whether they should fine-tune, retrieve or prompt their way to a working system. I will compare the three approaches across capability, knowledge and behaviour, look at cost and complexity, explore hybrid patterns, and finish with an evaluation framework and a decision matrix that you can adapt for your own programme.

My aim is not to declare a winner. My aim is to help you recognise which technique each problem actually needs, because in modern enterprise systems you will almost always end up combining at least two of them.

Large language models arrive with three constraints that architects must learn to manage. They have a fixed knowledge cut-off, they have a generic style and reasoning behaviour, and they have a context window that is finite even when generous. Each of the three techniques addresses one of those constraints more directly than the others.

Prompt engineering adjusts what the model sees at inference time without changing weights or adding retrieval. Retrieval-augmented generation injects fresh, organisation-specific knowledge into the prompt by querying a vector store or hybrid search system. Fine-tuning changes the model’s weights so that capability or behaviour is encoded permanently.

I think of these as three dials. Prompt engineering is the cheapest and fastest dial. Retrieval is the dial for knowledge breadth. Fine-tuning is the dial for behavioural consistency and specialised capability. Architects who treat them as competitors miss the point. They are complementary tools for different problems.

The biggest mistake I see is teams choosing a technique before defining the problem. Start with the deficiency you are trying to fix, then pick the dial.

Before comparing them, it helps to be precise about what each technique modifies. I use a three-axis model: capability, knowledge and behaviour.

If your problem is that the model does not know about your private claims policy, no amount of clever prompting will fix it. If your problem is that the model writes in a generic style when you need a specific regulatory tone, RAG will not help much. If your problem is that the model occasionally hallucinates, fine-tuning alone rarely solves it.

The three axes give you a vocabulary for diagnosing the gap. I encourage every architect to write down which axis a project is targeting before any engineering work begins.

I always recommend that teams begin with prompt engineering, even when they are confident a deeper intervention is needed. The reason is pragmatic. Prompt engineering forces you to articulate the task, the inputs, the outputs and the failure modes in concrete language. That artefact then becomes the specification for everything else.

Effective prompt engineering in 2026 is no longer just a clever instruction. It includes structured system prompts, few-shot examples, output schemas, tool definitions, guardrail layers and explicit refusal patterns. A well-engineered prompt is often versioned, tested and observed in the same way as code.

I have seen teams reach the production bar with prompt engineering alone for tasks like summarisation, classification, simple extraction and content rewriting. The technique is cheapest, fastest to iterate and easiest to roll back. It also gives you a baseline that lets you measure whether RAG or fine-tuning actually improves the system.

• Use system prompts to encode role, constraints and refusal behaviour
• Use few-shot examples to demonstrate format and edge cases
• Use output schemas (JSON Schema or Pydantic) to constrain structure
• Use evaluation suites to catch regressions when prompts change

When prompt engineering hits a ceiling, the failure usually points clearly to either a knowledge gap or a behavioural gap, which tells you which dial to turn next.

Retrieval-augmented generation solves the knowledge problem. When the model needs to answer using information it was not trained on, you retrieve relevant passages at inference time and place them in the prompt. The model then grounds its answer in that context.

I recommend RAG when you have one or more of these conditions: the knowledge changes frequently, the corpus is large enough that you cannot stuff it into a prompt, the knowledge is private or proprietary, or the answers must cite sources for auditability. Examples include internal policy assistants, customer support copilots, legal research tools and clinical guideline lookup systems.

The architectural pieces of a serious RAG system include ingestion pipelines, chunking strategies, embedding models, vector or hybrid stores, retrieval orchestration, reranking and prompt assembly. Each piece can be tuned independently, which is both a strength and a complication.

RAG is the only technique on this list that lets you update the system’s knowledge without retraining anything. That property alone makes it the default for most enterprise use cases.

The limits of RAG are real. It does not teach the model new skills, it does not change tone or behaviour deeply, and it can be expensive at scale because every request pays the retrieval and longer-context tax. When teams complain that RAG is not enough, they are usually describing a behavioural or capability problem, not a knowledge problem.

Fine-tuning changes the model’s weights. In modern practice this almost always means parameter-efficient methods such as LoRA or QLoRA, or supervised fine-tuning on a hosted endpoint. Full fine-tuning of frontier models is rare outside of major AI labs.

I reach for fine-tuning when the problem is consistency or capability rather than knowledge. Examples include enforcing a strict output format across millions of calls, teaching a specialised classification taxonomy, replicating an in-house writing style, or compressing a complex prompt into a smaller, faster model.

Fine-tuning is also the right answer when you want to reduce inference costs by moving from a large general model to a smaller specialist model that performs as well on your narrow task. I have seen organisations cut their inference bill by an order of magnitude this way.

The trade-offs are significant. Fine-tuning requires curated training data, an evaluation harness, infrastructure for training runs, and a versioning discipline for models. It locks in knowledge at the time of training, which makes it a poor fit for fast-changing information. It is also harder to audit, because the change lives in weights rather than in a visible prompt or document.

I tell architects: fine-tune behaviour, retrieve knowledge, prompt for everything else.

When I sit down with a finance partner to compare these techniques, we look at three cost categories: build cost, run cost and change cost. The picture is rarely intuitive at first glance.

The pattern I have seen consistently is that prompt engineering is cheap to build and cheap to run, RAG is moderately expensive on both axes, and fine-tuning is expensive to build but can be the cheapest to run if you commit to it for high-volume narrow tasks.

The cost story changes dramatically with scale. At one million calls per day, a fine-tuned smaller model often beats a prompt-engineered frontier model on total cost, even after amortising the training investment. At ten thousand calls per day, that math rarely works.

Latency matters more than most architects initially admit. Users notice the difference between a 500 ms response and a 3 second response, and downstream systems often time out long before the model finishes generating.

Prompt engineering with a frontier model gives you the model’s native latency, which is usually adequate for chat-style applications but can be problematic for high-frequency backend calls. RAG adds retrieval latency, reranking latency and longer-context processing latency, easily doubling or tripling end-to-end response time.

Fine-tuning on a smaller specialist model is often the only path to low-latency, high-throughput production systems. A 7B parameter fine-tune can serve traffic at a fraction of the latency of a frontier model while matching or exceeding its quality on the specific task.

• Prompt-engineered frontier model: simplest, highest latency, highest cost per call
• RAG over frontier model: best knowledge coverage, highest total latency
• Fine-tuned specialist model: lowest latency at scale, narrowest scope
• Fine-tuned model plus targeted RAG: the sweet spot for many enterprise systems

I always ask teams to specify a latency budget before choosing a technique. The answer to that single question often eliminates half the architectural options.

Each technique has a different appetite for data and a different governance footprint. Architects who ignore this end up with surprises during compliance review.

Prompt engineering needs almost no training data. It needs an evaluation dataset, ideally with hundreds of examples, and a versioning system for prompts. Its governance story is relatively simple because the change is visible and reversible.

RAG needs a curated corpus. That corpus must be ingested, chunked, embedded and indexed, and it must be kept in sync with the source of truth. Governance issues include data classification, access control at retrieval time, personally identifiable information handling, and the question of whether retrieved content can leak through model outputs.

Fine-tuning needs labelled training data, often thousands of examples. It also needs a held-out evaluation set, an approval workflow for new model versions, and a model registry to track lineage. Governance issues include data provenance, consent for training use, bias evaluation and the question of how to retire a fine-tuned model when its training data is no longer permissible.

The most expensive part of a fine-tuning programme is rarely the GPU bill. It is the data labelling and the governance overhead.

In practice, the most powerful enterprise systems I have built combine RAG and fine-tuning. The pattern is straightforward: fine-tune for behaviour and structure, retrieve for knowledge.

A concrete example is a regulatory compliance assistant I helped architect for a global bank. We fine-tuned a mid-sized model to follow a strict citation format, refuse out-of-scope questions and respond in the bank’s house style. We then used RAG to inject the relevant regulatory text at inference time. The fine-tuning made the behaviour predictable, and the retrieval kept the knowledge current.

Another common hybrid is fine-tuning a small embedding or reranking model on domain-specific relevance judgements, then using it inside a RAG pipeline that calls a general-purpose generation model. This improves retrieval quality without touching the generation model at all.

I also see hybrid patterns where prompt engineering does the routing. A lightweight prompt classifies the request, then sends it to either a fine-tuned specialist model, a RAG pipeline or a general-purpose model. This is essentially the LLM-routing pattern, and it is a clean way to combine all three techniques.

The lesson I draw is that the techniques are layered, not exclusive.

When a team asks me to help choose, I walk them through a five-step evaluation that almost always produces a clear answer.

1. Define the deficiency. Is the model missing knowledge, capability or behaviour? Be specific.
2. Establish the baseline. Build the simplest possible prompt-engineered version and measure it on a real evaluation set.
3. Identify the constraints. What is the latency budget? Cost per call? Compliance requirement? Update frequency?
4. Hypothesise the intervention. Based on the deficiency, propose the smallest change that could close the gap.
5. Test against the baseline. Measure quality, latency, cost and operational complexity. Decide whether the lift is worth the cost.

This framework forces teams to use evidence rather than fashion. I have lost count of how many fine-tuning projects I have killed at step two when the baseline turned out to be acceptable. I have also seen teams discover that their real problem was retrieval quality, not generation quality, which changed the entire engineering plan.

The evaluation set is the single most valuable artefact in this process. Invest in it before anything else. A well-constructed evaluation set of two to three hundred examples is worth more than a thousand hours of speculation.

Here is the matrix I use when I sketch options on a whiteboard. The recommended technique is the starting point, not the final answer.

For an internal code review assistant we fine-tuned a 13B model on the organisation’s review history. For the policy Q&A we kept the frontier model and built a strong RAG layer. For the routing system we fine-tuned a 3B model that ran cheaply at high volume.

The patterns repeat. Behaviour and consistency push you toward fine-tuning. Fresh, large or private knowledge pushes you toward RAG. Everything else can usually start with prompt engineering.

I have seen the same mistakes often enough to give them names.

• The premature fine-tune. Teams fine-tune before they have an evaluation set, then cannot tell whether the fine-tune helped.
• The infinite prompt. Teams keep adding instructions to a system prompt until it is thousands of tokens long and impossible to reason about.
• The RAG monoculture. Teams treat every problem as a retrieval problem, even when the real issue is behaviour or structure.
• The vanity fine-tune. Teams fine-tune to feel sophisticated, not because it solves a measured gap.
• The hidden coupling. Teams fine-tune on data that includes facts that change frequently, then discover that the model is permanently wrong about those facts.
• The unobservable system. Teams ship without prompt versioning, retrieval logging or generation observability, then cannot debug regressions.

Each anti-pattern has a simple antidote. Build an evaluation set first. Keep prompts under control. Diagnose before treating. Measure before celebrating. Separate stable knowledge from volatile knowledge.

If you cannot say what you would measure to know your change worked, you are not ready to make the change.

In serious enterprise systems, the answer is almost always to combine techniques. Here is a layering pattern that has worked well for me across multiple programmes.

The base layer is a fine-tuned model that encodes house style, output format and refusal behaviour. The middle layer is a RAG pipeline that brings in current knowledge with citations. The top layer is prompt engineering that handles task framing, few-shot examples and request-specific instructions.

This layering gives you the strengths of each technique. Behaviour is consistent because it lives in weights. Knowledge is current because it lives in retrieval. Task instructions are flexible because they live in the prompt.

The operational benefit is that each layer can be updated independently. A policy team can update the retrieval corpus without involving ML engineers. The prompt team can iterate on instructions without retraining. The ML team can ship a new fine-tune behind a feature flag without disrupting either.

I have also used routing patterns where a lightweight classifier sends requests to different combinations of layers. A simple FAQ might bypass retrieval entirely. A complex compliance question might use all three layers plus a verification step.

The point is that you do not need to choose one technique. You need to design a system where each technique does what it is best at.

The decision is not a one-time event. Each technique has its own lifecycle that you must plan for.

Prompts need versioning, testing and review. I recommend storing prompts as code, with peer review, automated tests and rollback. Treat a prompt change with the same seriousness as a code change because it has the same blast radius.

Retrieval indexes need refresh schedules, monitoring and quality checks. A RAG system that is six months out of date is worse than no system at all because users trust it. Build retrieval observability that tracks recall, precision and freshness.

Fine-tuned models need a registry, an approval workflow and a retirement plan. Every model version should have a model card describing training data, evaluation results, intended use and limitations. Plan for model retirement before you deploy.

Across all three, you need drift detection, evaluation regression, cost monitoring and incident response. The operational maturity required for a fine-tuned RAG system is closer to a payments platform than a typical web application.

If your organisation cannot commit to that operational maturity, prefer the simpler technique. A well-operated prompt-engineered system will outperform a poorly operated fine-tuned one every time.