LLM Routing & Orchestration: Patterns & Best Practices

When I started architecting LLM-powered systems three years ago, the routing layer was almost an afterthought. We picked one model, sent every request to it, and moved on. Today, I treat routing and orchestration as one of the highest-leverage decisions in any AI architecture. The model you call, the order you call them in, and what you do when something fails will determine whether your system feels delightful or disastrous in production.

I have seen organisations cut their LLM bill by 60 percent simply by introducing intelligent routing. I have also watched teams burn through six-figure budgets in weeks because every query, no matter how trivial, hit the most expensive frontier model. The difference between those outcomes is rarely the model. It is the orchestration architecture sitting in front of it.

In this guide, I want to walk through the patterns I rely on when designing LLM routing and orchestration layers for production systems. I will cover why routing matters, the strategies I evaluate, the orchestration patterns that hold up under load, the tooling landscape, and the observability practices that keep the whole thing honest. Whether you are scaling a single-model prototype or wrangling a fleet of providers, these are the lessons I wish I had internalised earlier.

The model landscape has fragmented in a way that makes naive single-model architectures untenable. In 2026, I am routinely working with five or six frontier providers, a handful of open-weights models hosted on internal infrastructure, and specialised fine-tunes for narrow domains. Each has different price points, latency profiles, context windows, and capability ceilings.

Routing matters because the assumption underpinning single-model deployments has collapsed. There is no longer a clear winner across every task. A 2026 frontier model might be the best at long-context reasoning but four times more expensive than a smaller model that handles 80 percent of your queries just as well. Picking one to rule them all leaves money, latency, and quality on the table.

The job of a routing layer is to make the question “which model should answer this?” answerable in milliseconds, with logic that I can explain to a sceptical CFO.

I also think about routing as a hedge against vendor concentration risk. When a major provider has an outage, and they all do, your routing layer is the difference between a degraded experience and a downed product.

Every routing decision I make trades off three things: cost per request, latency to first token or completion, and capability fit for the task. I keep a simple matrix at the start of any routing design exercise.

When I design routing for a customer-facing chat product, latency dominates. For a nightly batch summarisation job, cost dominates. For a legal analysis pipeline, capability dominates. The mistake I see most often is teams optimising for a single lever and being surprised when the other two erode.

A useful exercise: write down the cost ceiling, the latency budget, and the minimum acceptable quality score for each route in your system. If you cannot articulate those numbers, your routing layer is operating on vibes.

Rules-based routing is where I start every project, and where I often end up staying longer than expected. The premise is simple: a deterministic function inspects request metadata and selects a model.

Typical rules I deploy:

• Route by user tier. Free users hit the cheaper model, paid users get the premium tier.
• Route by feature. The summarisation endpoint goes to one model, the code generation endpoint to another.
• Route by input length. Short queries go to a fast small model, anything above 8K tokens goes to a long-context provider.
• Route by region. EU traffic stays on EU-hosted endpoints to satisfy data residency.
• Route by time of day. Batch workloads shift to cheaper providers during off-peak windows.

The advantage of rules-based routing is debuggability. When something goes wrong, I can trace the decision in seconds. The disadvantage is brittleness as the rule set grows. Once you have more than twenty rules, you have a maintenance problem and probably need a smarter layer on top. I treat rules as the foundation, not the ceiling, of any routing strategy.

When rules run out of expressive power, I reach for classifier-based routing. The idea is to train or prompt a lightweight model to look at the incoming request and decide which downstream model should handle it.

I have built classifiers in three flavours:

• Prompt-based classifiers. A small, cheap LLM gets a system prompt describing the available routes and outputs a route label. Fast to iterate, easy to update, but adds latency.
• Fine-tuned classifiers. A specialised model like DistilBERT trained on labelled routing data. Much faster than an LLM call, but requires a labelled dataset.
• Hybrid classifiers. Rules first, then a classifier for the ambiguous cases. This is my default for mature systems.

Classifier-based routing shines when the routing logic depends on the semantic content of the request rather than its metadata. “Help me write a Python function” and “Explain quantum tunnelling to my eight-year-old” need different models, and no metadata rule will tell them apart.

The key to making classifier routing work is treating the classifier as a first-class component with its own evaluation suite, latency SLO, and rollback strategy. I have seen teams deploy a classifier, never measure its accuracy, and end up routing 30 percent of traffic to the wrong model.

Embedding similarity routing is a technique I have grown fond of for systems with stable, recurring query patterns. The premise: embed the incoming query, compare it to a library of reference queries with known optimal routes, and route accordingly.

Here is the pattern I use:

• Collect a corpus of historical queries with their best-performing model.
• Embed each query and store in a vector index.
• For each new query, embed it, find the k nearest neighbours, and route to the majority model.

This works well when your query distribution is concentrated rather than long-tailed. For a customer support bot handling the same 200 question patterns thousands of times a day, embedding similarity routing is fast, cheap, and self-improving.

I treat the embedding library as a living asset. Every week, I refresh it with the latest queries and their actual performance scores. The router gets smarter without any code changes.

The trap to avoid: embedding routers do poorly on novel queries. I always pair them with a fallback rule that catches low-similarity scores and routes those to a more capable default model.

Sequential orchestration is the pattern where multiple model calls happen one after another, with each step’s output feeding the next. The classic example is retrieval-augmented generation: retrieve, then generate.

I design sequential pipelines around three principles. First, each step should have a clear contract: known inputs, known outputs, known failure modes. Second, intermediate state should be persisted so that I can replay from any step. Third, the slowest step in the chain dictates the overall latency budget, so I optimise that step first.

A sequential chain I deployed recently for a legal research product:

1. Query reformulation (small fast model)
2. Document retrieval (vector search)
3. Relevance reranking (specialised reranker)
4. Synthesis (frontier model)
5. Citation verification (small fast model)

Each step uses a different model selected for the job. The reformulation step turns a vague question into a precise search query. The reranker filters noise. The frontier model only sees the most relevant context. The verifier catches hallucinated citations before the response leaves the system.

The architectural discipline that matters here is treating each step as independently observable, retryable, and replaceable.

Parallel orchestration calls multiple models simultaneously and reconciles their outputs. I use it less often than sequential, but in the right places it is transformative.

Patterns I have shipped:

• Voting ensembles. Three models answer the same question, the majority wins. Useful for high-stakes classification.
• Specialist fan-out. A coordinator decomposes a task into subtasks, dispatches each to a specialist model in parallel, then aggregates.
• Race-to-respond. Fire the request at two providers, return whichever responds first. Useful when latency matters more than cost.
• Diversity sampling. Multiple models generate alternatives, a downstream selector picks the best.

Parallel patterns have two costs: real money, because you pay for every call, and operational complexity, because you now have multiple failure modes to handle. I reserve them for use cases where the quality lift justifies the spend.

The race-to-respond pattern in particular has saved me during partial provider outages. When one provider is degraded and slow, the second one wins the race and the user never notices.

Conditional chains decide what to do next based on the output of the previous step. Fallback chains decide what to do when the previous step failed.

My default fallback chain looks like this:

primary model
 -> on error or timeout, retry with backoff
   -> on persistent error, route to secondary provider
     -> on secondary failure, return cached response if available
       -> on no cache, return graceful degraded response

Every layer in this chain has been earned in production. I have lost confidence in any architecture that does not have an explicit answer for what happens when the primary call fails.

Conditional logic, where the path through the system depends on intermediate output, is more nuanced. I use it for things like routing low-confidence answers to a human review queue, or escalating from a fast model to a slow one when the fast model declines to answer. The key discipline is making the conditional logic visible in your observability stack so that you can audit why a particular request took the path it did.

Model cascading is a specific orchestration pattern that has paid for itself many times over. The idea: try the cheapest capable model first, and only escalate to a more expensive model if the cheap one fails to produce a satisfactory answer.

A cascade I run in production:

The art is defining what counts as “the cheap model failed”. I use a mix of signals: explicit confidence scores when available, output-length heuristics, structured-output validation failures, and downstream task success.

When tuned correctly, a cascade can route 70 to 80 percent of traffic to tier 1, with only the genuinely hard queries paying the premium price. I have seen cascades cut cost per query by 5x while improving overall quality, because the frontier model is no longer wasted on trivial requests it could solve in its sleep.

I rarely build routing infrastructure from scratch anymore. The open-source and commercial ecosystem has matured.

• LiteLLM gives me a unified API across providers. I write code against the OpenAI interface and LiteLLM handles translation to Anthropic, Google, Mistral, or my self-hosted endpoints. The router module supports load balancing, fallbacks, and cost-aware selection.
• Portkey offers a managed gateway with built-in routing rules, semantic caching, and observability. I reach for it when I want a turnkey solution and am happy to pay for it.
• Helicone focuses on observability and cost analytics, with routing as a supporting feature. I pair it with LiteLLM when I need deep usage insight.

Other options I have evaluated include OpenRouter for breadth of model access, Martian for capability-based routing, and homegrown routers built on top of LangGraph or DSPy.

The toolchain choice matters less than the discipline of having a routing layer at all. A simple custom router with good logging beats a fancy commercial tool you do not understand.

I treat the routing layer as critical infrastructure. It deserves the same engineering rigour as any other production service.

A pattern I have shipped multiple times. The system receives a customer support message and needs to classify intent, retrieve relevant context, draft a response, and route to a human if necessary.

The orchestration looks like:

1. A small classifier model categorises the message (refund, technical, account, other).
2. Rules-based routing sends each category to a specialist pipeline.
3. The technical pipeline runs RAG against the product documentation.
4. The refund pipeline runs a structured workflow with policy checks.
5. A confidence scorer evaluates the draft response.
6. High-confidence responses go directly to the customer with audit logging.
7. Low-confidence responses are routed to a human agent with the draft as a starting point.

The routing layer here is doing four jobs: capability matching (refund vs technical), cost management (small model for triage, larger model for synthesis), latency control (tight SLO on user-facing path), and quality gating (confidence-based escalation).

The biggest lesson from shipping these systems: instrument every routing decision. When a customer complains, I need to reconstruct exactly which path their query took, why, and what each model returned.

A second pattern I deployed for a financial services client. The system ingests a 200-page document and produces a structured summary, risk assessment, and set of action items.

The orchestration is a hybrid of sequential and parallel:

1. Chunking and indexing (deterministic code, no LLM).
2. Per-chunk extraction running in parallel across a cheap fast model.
3. Cross-chunk synthesis using a long-context frontier model.
4. Risk classification using a fine-tuned specialist model.
5. Action item extraction using a structured output model.
6. Final review pass for consistency and completeness.

The interesting routing decision was step 2. Running the cheap model in parallel across hundreds of chunks beat running a single frontier model on the entire document by every metric: cost, latency, and quality. The frontier model in step 3 only had to synthesise pre-extracted facts, not parse raw text.

This is the orchestration mindset I want to encourage. Decompose the problem, route each subtask to the model best suited to it, and use the expensive models for the work only they can do.

A routing layer you cannot observe is a routing layer you cannot operate. The metrics I track on every router I ship:

• Request volume per route, per model, per provider.
• Latency distributions per route (P50, P95, P99, plus TTFT for streaming).
• Cost per request and aggregate spend per route.
• Error rate and failure modes per provider.
• Fallback invocation rate.
• Quality metrics per route, when measurable.

I also log every routing decision with the inputs that drove it. When a customer reports a bad experience, I want to answer “what model was used, why was it chosen, and what did it return” without needing to reproduce the request.

Dashboards are necessary but not sufficient. I set alerts on routing anomalies: a sudden drop in tier 1 traffic in a cascade suggests the cheap model has regressed. A spike in fallback invocations suggests a primary provider is degraded. The router is the place where the health of your entire AI system becomes visible.

A few mistakes I encounter often enough to call out explicitly.

• Single-provider lock-in. Building deeply against one provider’s quirks makes future routing impossible. I keep provider-specific logic at the edges and design my internal interfaces to be provider-agnostic.
• Routing without measurement. Teams add a router, ship it, and never measure whether the routing decisions are good. A bad router is worse than no router.
• Cascade thresholds set once, never tuned. Models drift, your traffic mix drifts, and your cascade thresholds need to drift with them. I review thresholds quarterly.
• No load testing of fallback paths. The fallback only matters if it actually works under load. I run periodic chaos tests that force traffic onto the fallback.
• Routing logic in the application layer. Embedding routing decisions in business code makes the system harder to evolve. I keep routing as a dedicated service or library.

The common thread is treating routing as a one-time setup task rather than an evolving capability. The right mental model is closer to traffic engineering than to configuration.