Devansh Kamlani
I have lost count of how many times a client has shown me an LLM application in production with no traces, no eval scores and no cost telemetry. The dashboard, usually, is a single graph of API latency from the upstream provider. When I ask “how do you know the assistant is doing a good job?” the answer is often “users seem happy”. That is not observability. That is hope.
AI Observability: Tools, Metrics & Dashboards 2026
In this article I want to lay out the AI observability stack I actually deploy in production, the metrics that matter, the tools that have earned their place, and the dashboard designs that help operations teams act fast when things go wrong. I will cover LangSmith, Arize Phoenix, Helicone, Weights and Biases and a handful of supporting tools. I will also share the alert thresholds I use, the eval integrations that catch quality regressions before users do, and the common mistakes that turn observability projects into expensive screensavers.
If you are a solution architect or a platform engineer responsible for production generative systems in 2026, this should give you a working blueprint. I will write from the perspective of someone who has had to walk into the war room at three in the morning and explain why the cost graph just went vertical.
What AI observability actually means
Observability for AI systems is the ability to understand the behaviour of the system from the outside, using the data the system itself emits. The classical observability triad of metrics, logs and traces still applies, but the contents are different. A trace is no longer just a span graph; it is a record of prompts, completions, tool calls, retrieval results and evaluation scores.
I find it helps to separate three concerns: system observability, which covers the infrastructure the model runs on; model observability, which covers the behaviour of the model itself; and product observability, which covers the user-facing outcomes. A mature stack covers all three with the same data.
If your observability stops at “the API returned 200 OK”, you are not observing the AI; you are observing the network.
The shift in mindset is to treat the model’s output as the primary signal. A system can be technically healthy at every layer of the stack and still produce nonsense. Observability for AI needs to surface the nonsense.
The five pillars: latency, cost, quality, errors, satisfaction
I anchor every observability conversation around five pillars. The five together describe the health of a generative system.
| Pillar | What it tells you | Typical metrics |
|---|---|---|
| Latency | Is the system fast enough? | TTFT, tokens/sec, p95 end-to-end |
| Cost | Is the system economically sustainable? | tokens/request, cost/request, cost/tenant |
| Quality | Is the output good? | eval pass rate, judge score, hallucination rate |
| Errors | Is the system failing? | refusal rate, tool errors, provider errors |
| Satisfaction | Are users happy? | thumbs, retention, escalation rate |
The trap is to monitor one pillar and ignore the others. A system can have great latency, terrible quality and silent cost overruns all at once. The dashboard should make all five visible at a glance.
I treat the five pillars as the source of truth for every alert. If an alert does not map to one of them, it is probably noise.
Traces, spans and the structure of an LLM trace
A modern LLM trace is the foundation of the observability stack. The trace captures the full life of a single user request, from the inbound API call through retrieval, tool use, model calls and the final response.
A canonical trace structure:
- Root span: incoming user request, including session ID, user ID and feature flag context.
- Retrieval span: vector store query, retrieved documents with scores, embedding model version.
- Tool spans: each tool call with inputs, outputs and latency.
- Model spans: each LLM call with prompt template version, model identifier, token counts and cost.
- Evaluation spans: any inline evals run during the request.
When I review a production incident, the first thing I look at is the trace structure. A well-formed trace tells the story of the request; a malformed trace leaves you guessing.
Tracing tools that have invested in LLM-aware schemas (LangSmith, Phoenix, Langfuse) are far more useful here than general-purpose APM tools that treat the prompt as an opaque string.
Latency metrics worth tracking
Latency for LLM systems is more nuanced than for classical APIs. I track at least three numbers per route.
- Time to first token (TTFT). For streaming UX, this is the perceived responsiveness. I target under 500 ms for chat experiences.
- Tokens per second (TPS). The throughput once streaming starts. I target above 30 TPS for chat, higher for batch.
- End-to-end p95. The total time from request to last token. This is the SLO most users care about.
I also segment by prompt version, model, and route. A median latency spike across all traffic looks like a problem; the same spike on a single prompt version is a deployment regression.
In multi-step agentic workflows, latency is the sum of model calls plus tool calls plus retrieval. I track the latency contribution of each step separately. The slow step is almost always the place to optimise first.
A useful rule of thumb: if your latency dashboard does not let you segment by prompt version, you do not have an LLM latency dashboard.
Cost metrics and budget alerts
Cost is the metric that ends careers. Generative systems can quietly cost ten times more per month than expected, and the bill only arrives at the end of the month.
I instrument cost at three granularities:
- Per request, recorded as a metric and exposed in traces.
- Per tenant or per user, rolled up daily and weekly.
- Per route or feature, to identify expensive code paths.
Budget alerts should be set at multiple thresholds: a soft alert at fifty per cent of projected monthly spend, a hard alert at eighty per cent, and a circuit breaker at ninety-five per cent. The circuit breaker should be automated; do not rely on a human to throttle traffic at four in the morning.
A common pattern I now insist on is per-tenant cost ceilings enforced in the routing layer. If a single tenant exceeds its ceiling, requests are downgraded to a cheaper model or rejected with a clear error.
Quality metrics and hallucination tracking
Quality is the most important pillar and the hardest to measure. There are three approaches I use in combination:
- Automated evals. A fixed suite of prompts and expected outputs, scored with rule-based or LLM-judge methods, run continuously against production traffic.
- Inline judges. Lightweight LLM judges that score every request along a few axes (accuracy, helpfulness, safety).
- Reference comparison. For routes with deterministic answers, compare the model output to a known-good reference.
Hallucination rate is a special case. I usually define a hallucination as a factual claim not supported by the retrieved context, and I track it as a percentage of generations in RAG systems. The judge model that scores hallucination needs careful calibration against human labels; otherwise the metric drifts.
I aim for a quality dashboard that updates within minutes of a deployment. Catching a regression an hour after rollout is much better than catching it the next day.
User-satisfaction signals
User-facing signals are noisy but vital. The metrics I rely on:
- Explicit thumbs-up and thumbs-down rates per route.
- Escalation rate to a human agent or support channel.
- Conversation length, which often correlates with frustration.
- Retention or repeat usage of an AI-powered feature.
The trick with explicit feedback is that only a small fraction of users use it, and they are not representative. I treat thumbs as a leading indicator that something has shifted, then dig into traces to understand what.
User satisfaction is the only metric that ultimately matters, and it is the only one you cannot measure directly. Everything else is a proxy.
A good observability stack ties user feedback back to traces so the team can investigate the specific session that prompted a thumbs-down.
Top observability tools in 2026
The 2026 landscape has settled around a few clear leaders, each with a strong point of view.
- LangSmith. Mature tracing and evaluation, tight LangChain integration, strong prompt registry. My default for teams using LangChain.
- Arize Phoenix. Open-source, strong embeddings analysis, good drift detection, can be self-hosted.
- Helicone. Lightweight proxy with excellent cost analytics, easy to add to existing systems without code changes.
- Weights and Biases. The best choice for teams already invested in W&B, particularly for fine-tuning and evaluation workflows.
- Langfuse. Strong open-source alternative to LangSmith, good self-hosting story.
- Datadog LLM Observability. Solid choice for teams with an existing Datadog investment.
I treat the choice as a function of team familiarity, deployment posture (managed vs self-hosted) and integration depth. The best tool is the one your team will actually use day to day.
Choosing between LangSmith, Phoenix, Helicone and W&B
When clients ask me to pick one, I use a short decision tree.
| Constraint | Recommendation |
|---|---|
| Heavy LangChain use, managed acceptable | LangSmith |
| Need self-hosted open source | Phoenix or Langfuse |
| Want lightweight cost tracking with minimal code change | Helicone |
| Already on W&B for ML experiments | Weights and Biases |
| Need deep Datadog integration | Datadog LLM Observability |
| Need maximum control of data | Self-hosted Phoenix or Langfuse |
I generally advise against running multiple observability tools in parallel for too long. The cost of context switching across dashboards usually outweighs the perceived benefit. Pick one, integrate it deeply, and revisit in six months.
The exception is when one tool excels at a single capability (for example, Helicone for cost) and the team is comfortable maintaining a thin sidecar.
Dashboard design for LLM systems
The dashboard is where observability becomes operational. I design dashboards in three tiers.
- Health dashboard. Five-pillar overview, refreshed every minute, designed for on-call. Latency, cost, quality, error rate, satisfaction.
- Investigation dashboard. Drill-downs by prompt version, model, route, tenant. Used when an alert fires.
- Product dashboard. Trend graphs over weeks and months, used by product leadership.
Each tier needs a different colour, layout and refresh cadence. The health dashboard should be readable in under ten seconds. The investigation dashboard can be denser. The product dashboard prioritises trend lines over absolute values.
A practical tip: every chart on the health dashboard should answer a single question. If a chart needs a paragraph of explanation, it does not belong there.
Alert thresholds I actually use
Setting alert thresholds is half science and half taste. The numbers below are starting points I tune per client.
- p95 latency: page when above 1.5x the seven-day baseline for ten minutes.
- Cost: page when daily run-rate exceeds 1.25x the monthly budget divided by days.
- Quality eval pass rate: page when below baseline minus two standard deviations.
- Error rate: page when above five per cent for five minutes.
- Refusal rate: page when above ten per cent for ten minutes (route-specific).
- User thumbs-down rate: alert (not page) when above 1.5x baseline for thirty minutes.
I am deliberately conservative with paging thresholds. An on-call engineer woken at 3 am needs a real problem to debug, not a noisy signal that resolves itself.
The thresholds should be reviewed quarterly. As the system matures, baselines tighten and thresholds get more sensitive.
Integrating evaluation into observability
Evaluation and observability used to be separate concerns. In 2026 they have merged. The eval suite is a continuous process that produces metrics observable in the same dashboard as latency and cost.
The integration patterns I deploy:
- Run a fixed eval suite hourly against the production system, log scores as metrics.
- Sample a percentage of production traffic and score it with an LLM judge in real time.
- Tag every trace with the evaluation result so on-call engineers can filter by quality.
- Use the eval pass rate as a release gate in CI before promoting a new prompt.
The pattern that delivers the most value is the hourly smoke-test eval. It catches model drift, prompt regressions and provider outages in close to real time.
The eval suite itself needs an owner. Without one, it drifts, breaks and stops being trusted. I treat the eval engineer (or part-time evaluation responsibility) as a non-negotiable role.
Common observability mistakes
Three mistakes I see again and again.
- Treating logs as observability. Logs are necessary but not sufficient. Without structured traces, you cannot reconstruct a request.
- Monitoring only the model API. The model is one component of the system. Retrieval, tool calls and orchestration are equally likely to fail.
- Letting cost alerts arrive after the bill. Cost should be a real-time metric with active circuit breakers.
A fourth mistake worth calling out is the eval-once-and-forget pattern. Building an eval suite is the start of the work, not the end. Without continuous evaluation, the suite becomes a snapshot of the past and loses its value over time.
The strongest teams I work with treat observability as a product. They iterate on dashboards, refine alerts and own the platform with the same seriousness they bring to the application.
Building an observability roadmap
For a team starting from scratch in 2026, I recommend a sequenced roadmap.
- Instrument basic traces with prompt, completion, model and token counts.
- Add cost tracking per request and per tenant.
- Build a five-pillar health dashboard.
- Add an eval suite and run it hourly.
- Wire up alerts on the five pillars with the thresholds above.
- Add inline judges for quality scoring on production traffic.
- Integrate user feedback signals (thumbs, escalations).
- Add drift detection on embeddings and topics.
- Build a tenant-level dashboard for product leadership.
- Add circuit breakers for cost and quality.
Most teams can complete steps one to five in a quarter. Steps six to ten typically take another two to three quarters depending on team size.
The biggest determinant of success is whether the team treats observability as an ongoing investment rather than a one-time project.
