Devansh Kamlani
I started my career in MLOps when the discipline was mostly about retraining gradient-boosted trees on a schedule and shipping pickle files into Flask containers. The patterns we built then were good, and most of them still apply, but generative AI has bent the field in ways that take some getting used to. There is no fit-predict loop. Prompts are code. Evals are tests. Cost is a first-class metric. Most of the systems I helped design five years ago would simply collapse under the operational shape of a modern LLM application.
MLOps for LLMs: Deployment, Monitoring & Versioning
In this article I want to share the playbook I now use for mlops for llms in production. I will cover how LLMOps differs from classical MLOps, the deployment patterns I see across managed and self-hosted environments, the versioning strategies that prevent silent regressions, and the monitoring stack that keeps the lights on. I will also recommend the tools I actually reach for, including LangSmith, Helicone, Arize and Weights and Biases.
This piece is written for solution architects, platform engineers and senior ML practitioners who are being asked to put a generative system into production and keep it healthy for years. If you are coming from a classic MLOps background and wondering what changes, this should be your map.
What LLMOps actually is
LLMOps is the operational discipline of building, deploying, monitoring and improving systems that use large language models. It is a sibling of MLOps, not a replacement, and it is best understood as the application of operational thinking to a stack where the model is usually a fixed dependency, prompts are the variable, and evaluation is continuous.
When I introduce LLMOps to an organisation that already has MLOps, I emphasise three differences. First, there is rarely a retraining loop; the foundation model is an API or a downloaded artefact and you do not own the training process. Second, the unit of change is usually a prompt, a tool definition or a retrieval index, not a model weight. Third, the cost surface is dominated by inference tokens, which means cost is a metric you monitor in real time rather than a quarterly invoice.
LLMOps is what happens when the model becomes a stable platform component and the application logic moves into prompts and orchestration.
The implication is that LLMOps tooling tends to live closer to the application than classical MLOps tooling did. Observability, evaluation and registries all need to understand prompts, traces and tool calls, not just feature distributions and confusion matrices.
How LLMOps differs from classical MLOps
I find it useful to lay the two side by side. The differences are easy to skim but worth internalising.
| Concern | Classical MLOps | LLMOps |
|---|---|---|
| Unit of change | Model weights | Prompt, retrieval index, tool spec |
| Quality measure | Accuracy, AUC, F1 | LLM-judge score, eval pass rate, user thumbs |
| Retraining loop | Continuous or periodic | Rare; fine-tuning is occasional |
| Drift | Feature and label drift | Behavioural and semantic drift |
| Cost driver | Training compute | Inference tokens |
| Test artefact | Held-out dataset | Eval suite with assertions |
| Failure mode | Mispredictions | Hallucinations, jailbreaks, regressions |
The biggest mental shift is treating the prompt as the artefact under management. In classical MLOps the prompt did not exist; in LLMOps the prompt is your code, and you need to version, test, deploy and roll back prompts with the same rigour you applied to models.
The three deployment patterns I keep seeing
Across recent engagements I see three deployment patterns dominate. Most organisations end up with at least two of them in production.
- Managed API consumption. The application calls an external provider such as OpenAI, Anthropic, Google or AWS Bedrock. The provider runs the model.
- Self-hosted inference. The team deploys an open-weight model on its own infrastructure using vLLM, TGI, Triton or similar.
- Fine-tuned cloud-hosted models. The team fine-tunes a foundation model in the cloud and either consumes it through the provider’s API or pulls weights back for self-hosting.
I will walk through each, because the operational shape is meaningfully different.
The choice between these is rarely about model quality alone. It is about latency, sovereignty, cost, the team’s operational capacity and how much customisation the application needs. A solution architect should be comfortable mixing all three in the same enterprise.
Managed APIs in production
Managed APIs are the easiest entry point and the hardest to scale economically. The operational model is straightforward: you authenticate, you send a prompt, you receive a completion, you pay per token.
The patterns that matter in production are:
- A routing layer (LiteLLM, AWS Bedrock or a bespoke router) that abstracts model providers and lets you swap or fall back without changing application code.
- Aggressive caching for deterministic prompts, often with a content-hash key that includes the prompt template version.
- A token-accounting service that records cost per request and rolls up by tenant, feature and user.
- Rate-limit shaping so that one runaway tenant cannot consume the global quota.
In a recent engagement with a media client, we cut their managed-API spend by thirty-six per cent simply by introducing a deterministic cache and a routing layer that downgraded simple queries to a smaller model. No change in application behaviour, large change in invoice.
Self-hosted inference with vLLM and TGI
Self-hosting is the right choice for high-volume, latency-sensitive or sovereignty-constrained workloads. My default runtime is vLLM for high-throughput LLMs, with Hugging Face Text Generation Inference (TGI) as an alternative when the team wants a more opinionated production setup.
A self-hosted production deployment usually includes:
A horizontally scalable inference cluster, often on Kubernetes with a GPU node pool.
A model registry that signs and stores weights with their provenance and licence.
A blue-green or canary deployment strategy for new model versions.
Autoscaling that understands GPU memory and KV cache rather than just CPU.
The tricky part is capacity planning. Unlike CPU workloads, GPU inference does not scale linearly with concurrent requests because the KV cache is bounded. I usually plan for seventy per cent peak utilisation and use a queueing layer to smooth bursts.
deployment:
runtime: vllm
model: meta-llama/Llama-3.1-70B-Instruct
tensor_parallel_size: 4
max_num_seqs: 64
gpu_memory_utilization: 0.9
rollout:
strategy: canary
initial_traffic: 0.05
promotion_gate: eval_pass_rate >= 0.95
Cloud-hosted fine-tuning pipelines
Fine-tuning sits in an interesting place. It is rarely a permanent feature of the operational stack, but it is a recurring activity that needs the same hygiene as any other production process.
The pipeline I recommend looks like:
- Curate a training dataset with provenance, licence and PII review.
- Run fine-tuning on managed cloud capacity (Bedrock, Vertex, OpenAI fine-tuning, Modal, Together).
- Evaluate the fine-tuned model against a regression eval suite.
- Compare to the base model and any previous fine-tuned versions.
- Promote or reject based on automated gates and human approval.
- Register the artefact with metadata, evals, and a rollback plan.
The output of this pipeline is a model artefact ready for either managed deployment or self-hosting. Either way, it lands in a registry that the deployment system can read.
The mistake I see most often is fine-tuning without a baseline eval. If you cannot prove the fine-tuned model is better than the base model on your tasks, do not deploy it.
Versioning prompts, models and configurations
The single most under-invested area in LLMOps is versioning. I have seen organisations spend millions on infrastructure while shipping prompts via copy-paste from a shared document.
The minimum I would accept in production:
- Prompts live in a version-controlled repository or a dedicated prompt registry.
- Every prompt change is reviewed and tagged with a semantic version.
- The model identifier, runtime version and tool spec are bound to the prompt version.
- The application records the prompt version with every request, so traces can be replayed.
A useful schema for a prompt artefact:
id: claim-summary-v3.2.1
description: Summarise an insurance claim into 3 bullet points
model: anthropic.claude-3-7-sonnet
temperature: 0.2
system: |
You are a claims analyst...
template: |
Summarise the following claim:
{{claim_text}}
parameters:
max_tokens: 512
evals:
- eval_suite: claim_summary_v3
pass_rate_min: 0.92
Treating this as code lets you diff prompts, roll back to a previous version and run regression evals when something changes.
The prompt-as-code discipline
I want to dwell on prompt-as-code because it is the cultural shift that separates the teams who scale LLM applications from the teams who do not. Every change to a prompt is a deployable change. It deserves the same review, testing and rollout discipline as a code change.
The teams that get this right tend to share a few habits:
- A single source of truth for prompts (registry or repo), not scattered in application code.
- Pull requests for prompt changes with diffs, reviewers and CI evals.
- A staging environment where new prompts are evaluated against a representative dataset before promotion.
- Feature flags that allow gradual rollout of new prompts to subsets of users.
The teams that do not get this right end up with mysterious quality regressions, hours of debugging through stale traces and angry product managers wondering why the assistant started saying something odd last Tuesday.
Monitoring latency, quality and cost
Monitoring an LLM system is a three-dimensional problem: latency, quality and cost. Classical MLOps monitoring captured the first and a thin version of the second. LLMOps monitoring needs to capture all three at request granularity.
The minimum dashboard I would ship to a production team:
- p50, p95 and p99 latency per route, per model, per prompt version.
- Cost per request and per route, rolled up by tenant and by feature.
- Quality signals: automated eval pass rate, LLM-judge score, user thumbs, escalation rate.
- Error and refusal rates, including content-policy refusals and tool-call errors.
I also push for alerting on combinations rather than single metrics. A latency spike is not interesting on its own; a latency spike with a refusal-rate spike usually means the model is timing out on a particular content pattern, which is a real incident.
Drift detection in generative systems
Drift in LLM systems is different from drift in classical ML. The model weights do not change (unless the provider silently updates them), but everything around the model can change: prompts, retrieval indices, user behaviour, downstream tools. The signal of drift is usually a change in the distribution of completions, not inputs.
I monitor three drift signals:
- Behavioural drift. A change in the eval pass rate or LLM-judge score over time, holding the prompt version constant.
- Topic drift. A change in the distribution of user queries, often a leading indicator of quality regressions.
- Cost drift. A change in the average tokens per request, which can indicate prompt bloat or a change in retrieval recall.
When a provider silently updates a model (this happens more often than vendors admit), the behavioural-drift signal is your only early warning. I run a small smoke-test eval every hour against a canary prompt set, and I alert on any statistically significant change.
Tooling: LangSmith, Helicone, Arize, W&B
The tooling landscape has consolidated enough that I can recommend a short list with confidence.
- LangSmith. Strong tracing and evaluation tightly integrated with LangChain and increasingly framework-agnostic. My default when teams want one tool for traces, evals and prompt registry.
- Helicone. Lightweight observability with an excellent cost-tracking story. Good when the team wants a thin proxy in front of multiple providers.
- Arize Phoenix. Production observability with deep support for embeddings analysis and drift detection. Strong open-source story.
- Weights and Biases. The best choice for teams already invested in W&B for ML experiments who want to extend into LLMOps.
I almost always pair an observability tool with an independent evaluation framework. Tying evaluation and observability together inside one vendor is convenient, but it creates a single point of failure if you ever need to migrate.
Building an evaluation-driven CI pipeline
Eval-driven CI is the practice that separates serious LLM teams from prototype teams. The idea is simple: every change to a prompt, model or retrieval index runs through an evaluation suite before it can be promoted.
A typical CI pipeline:
- A pull request changes a prompt template.
- CI loads the new prompt and runs it against a fixed evaluation dataset.
- Results are compared to the baseline (current production prompt).
- The pipeline fails if the new prompt regresses on any guarded metric.
- If the pipeline passes, the prompt is promoted to staging with a feature flag.
- Staging traffic is sampled and scored against a smaller eval set.
- After a soak period, the prompt is promoted to production.
The eval suite itself is a product. It needs curation, version control and review. The teams that treat their eval suite as an afterthought end up trusting it less and less over time.
Incident response for LLM systems
Incidents in LLM systems look different from incidents in classical systems. Latency spikes still happen, but the more common incidents are quality regressions, content-policy issues and cost runaways.
The playbook I drill into teams:
- A canonical traces dashboard with the last 24 hours of requests, filterable by prompt version, model and route.
- A “freeze” switch that pins the production system to the last known good prompt and model version.
- A cost circuit breaker that throttles or rejects requests when daily spend crosses a threshold.
- A content-policy dashboard with refusal and flagged-content rates per route.
- A clear rollback path for prompt versions, with no more than two clicks.
The first time a senior leader asks “why did the assistant say that?” you will be grateful for traces with full prompt, completion and tool-call detail.
Organisational shape and team patterns
LLMOps requires a team shape that classical MLOps did not. The teams I have seen succeed have three roles:
- Platform engineers who own infrastructure, runtimes, registries and observability.
- Application engineers who own prompts, orchestration, tools and product behaviour.
- Evaluation engineers who own eval suites, judge models and quality measurement.
The third role is the newest and the most often missed. Without dedicated evaluation expertise, the team ends up trusting vibes instead of metrics, which is fine in a prototype and fatal in production.
In smaller organisations these roles compress into a single person, but the responsibilities still need to be named. I usually advise clients to add evaluation as a discipline before they need a dedicated hire, so the muscle is built when the team scales.
