Devansh Kamlani
Latency in AI systems is unforgiving in a way that latency in traditional systems is not. When a web page takes an extra second to load, users grumble. When an AI assistant takes an extra second to start responding, users abandon. The threshold of what feels responsive is set by human attention, and the human attention budget is shorter than most engineers initially assume.
AI Latency Patterns: Caching, Streaming & Pre-fetching
I have spent a lot of time over the last few years measuring, modelling, and optimising latency in production AI systems. What I have learned is that AI latency is not a single number to be minimised. It is a profile to be shaped. Time to first token matters more than total completion time for chat interfaces. Tail latency matters more than median latency for retention. Streaming the right amount of content at the right moment matters more than absolute throughput.
In this article, I want to walk through the latency patterns I have shipped to production. I will cover the metrics that actually matter, latency budgets for different use cases, streaming and prompt caching as the foundational tools, semantic caching, speculative decoding, response pre-fetching, edge inference, and the monitoring discipline that lets you defend the experience over time. The goal is to give you a complete picture of how to think about AI latency, with the specifics to act on it.
Why AI Latency Demands Its Own Discipline
AI latency is fundamentally different from traditional service latency. A web service responds within tens or hundreds of milliseconds for almost every request. An AI service may take from a few hundred milliseconds to tens of seconds, with high variance, and the user experiences this duration interactively.
Three properties shape the AI latency discipline:
- Long, variable response times. Completion is not a moment, it is a duration during which the user is waiting.
- High variance per request. The same query asked twice can take dramatically different times depending on the prompt, the model state, and provider load.
- Streaming changes the user model. The user no longer waits for a complete response. They watch the response unfold and decide whether to keep waiting.
These properties mean that traditional latency thinking, which optimises a single end-to-end metric, leaves significant user experience on the table.
The right mental model for AI latency: shape the response timeline so that the user has a satisfying experience at every moment, not just at the end.
That shaping is what the patterns in this article are about.
The Metrics That Actually Matter
The latency metrics that matter for AI systems are not the ones I default to from web infrastructure work. The ones I track:
| Metric | What It Measures | Why It Matters |
|---|---|---|
| TTFT | Time to first token | User perception of responsiveness |
| TPS or TPOT | Tokens per second after first token | Pace of streaming |
| Total completion time | End to end | Total time the user spent waiting |
| Time to useful content | First semantically meaningful tokens | Practical responsiveness |
| End-to-end transaction time | Full workflow including retrieval and post-processing | Real user experience |
TTFT is the most important metric for interactive AI. A response that starts in 300ms feels instant even if it takes 10 seconds total. A response that starts in 3 seconds feels slow even if it completes in 5.
TPS matters because the user is reading as the response streams. If TPS is too low, the user finishes reading what is on screen and waits. If TPS is too high, it actually does not help, because reading is the bottleneck.
I track these metrics at P50, P95, and P99. The tail matters disproportionately because it is the experience that drives complaints and churn.
Latency Budgets by Use Case
Different AI use cases have different latency budgets. The budget I work with for common patterns:
- Real-time chat. TTFT under 500ms, TPS above 20. Users tolerate longer total durations if streaming starts fast.
- Voice assistant. TTFT under 200ms for the response to feel conversational. Lip-sync timing.
- In-product copilot. TTFT under 800ms, total completion under 5 seconds for typical tasks.
- Search and Q&A. First useful content under 1 second. Streaming helpful.
- Document analysis. Total completion under 30 seconds for short documents, with progress indicators for longer ones.
- Agentic workflows. Each step under 5 seconds, total workflow can be longer if intermediate progress is visible.
- Batch and offline. Throughput optimisation rather than latency.
The latency budget should be set before the architecture is designed, not discovered after launch. I have seen too many systems that work technically but feel slow because no one explicitly budgeted the latency.
A useful exercise: instrument an existing system the user finds fast and measure its actual latency profile. That gives you a concrete target rather than an abstract one.
Streaming Responses as the Default
Streaming should be the default for any user-facing AI interface in 2026. Non-streaming responses make the system feel slower than it actually is.
The architectural implications of streaming:
- The infrastructure layer must support server-sent events or WebSockets end to end.
- Intermediate components, gateways, load balancers, must not buffer responses.
- The application must render tokens as they arrive, not wait for completion.
- Error handling becomes more nuanced because the response is partially delivered.
- Cancellation needs to work cleanly when the user navigates away.
A pattern I have shipped: the streaming pipeline starts at the model provider, runs through a thin gateway that adds observability without buffering, and arrives at the client where the rendering layer paints tokens incrementally.
Streaming also enables UX patterns that improve perceived responsiveness. A typing indicator that animates as tokens arrive. A “thinking” skeleton that displays before the first token. Progressive disclosure of structured content as sections complete.
The model is generating, the user is reading, and the design job is to keep both pleasantly occupied.
Streaming is not optional for serious user-facing AI. Teams that ship non-streaming responses are leaving major perceptual responsiveness on the table.
Prompt Caching Across Providers
Prompt caching reduces both cost and latency by reusing pre-computed model state for stable prefixes. In 2026, Anthropic, OpenAI, and Google all offer prompt caching with different cache durations and pricing models.
The latency benefit is significant. A cached prefix can drop TTFT by 30 to 70 percent because the model does not have to re-process the prefix tokens. For systems with long stable contexts, retrieved documents, system instructions, examples, this is transformational.
The architectural pattern I use:
- Structure prompts with all stable content first, variable content last.
- Use the provider’s cache control mechanism to mark cacheable boundaries.
- Maintain cache key stability across requests that should hit the same cache.
- Monitor cache hit rates and TTFT impact in production.
I have shipped systems where the cache hit rate exceeded 90 percent, with corresponding latency improvements on most requests. The discipline is designing for cache reuse from the start, not retrofitting it later.
The trap to avoid is treating caching as a magic switch. Caches expire, and prompts that change frequently never warm up. The architecture has to be designed around the cache semantics of your chosen provider.
Semantic Caching for Sub-Second Responses
When prompt caching reduces the cost and latency of model calls, semantic caching eliminates the model call entirely for repeat queries. A semantic cache that hits returns a response in tens of milliseconds rather than seconds.
The pattern:
- Embed the incoming query.
- Look up similar past queries in a vector store.
- If similarity exceeds a threshold, return the cached response.
- If not, invoke the model and store the result for future hits.
The latency impact is dramatic when cache hits are common. A customer support system with 40 percent cache hit rate has a TTFT distribution that is bimodal: very fast for hits, normal for misses.
I deploy semantic caches behind a thin routing layer that can flag certain queries as uncacheable, for example queries that depend on real-time data or personal user context. The cache lookup itself should be fast, typically under 50ms, to make the optimisation worthwhile.
The architectural caveats:
- The vector index needs to be fast enough that the lookup is not itself a bottleneck.
- Cache invalidation needs an explicit policy. Stale answers degrade trust.
- The semantic similarity threshold needs tuning per use case.
When done well, semantic caching is one of the largest perceived-latency wins available.
Speculative Decoding and Draft Models
Speculative decoding is a technique where a small, fast model generates draft tokens and a larger model verifies them in parallel. When verification succeeds, the system accepts multiple tokens per forward pass, dramatically increasing tokens per second.
In 2026, speculative decoding is increasingly available as a provider-side optimisation rather than something you implement yourself. The user-visible effect is higher TPS without sacrificing quality.
When speculative decoding is available, I evaluate it explicitly. The TPS uplift is typically 2 to 4x for many workloads, which materially improves the streaming experience.
Related patterns I have used:
- Draft-and-confirm with explicit user feedback. Show a fast draft, then refine in place when the slower verification completes. Useful for low-stakes content.
- Cascaded streaming. Stream from a small model while a large model works in parallel, swap in the larger model’s output when ready.
- Predictive completion. For known patterns, start streaming the predictable parts of the response while the model generates the variable parts.
These techniques shift the latency profile by exploiting structure in the workload. They are not free, they require additional infrastructure or model calls, but the TPS improvements can be substantial.
Response Pre-fetching for Predictable Flows
When the next user action is predictable, you can begin generating the response before it is requested. This pattern, response pre-fetching, eliminates latency entirely for the pre-fetched action.
Cases where pre-fetching pays off:
- Onboarding flows where the next step is determined.
- Wizard-style interactions with known progression.
- Document analysis where the user is likely to ask follow-up questions on the same content.
- Search refinement where common refinement patterns can be anticipated.
The architectural pattern: a predictor identifies likely next actions, the system starts generating speculatively, and the result is cached for instant delivery when the user actually requests it.
The cost is real, because you pay for generations the user may never request. The trade-off is justified when the prediction accuracy is high enough that the perceived latency benefit exceeds the cost of wasted generations.
I have shipped pre-fetching in two patterns. In one, the system pre-fetches a small set of high-probability follow-ups and discards them if not used. In the other, the system generates an initial summary speculatively while the user is still reading the original content. Both improved perceived responsiveness significantly.
Edge Inference for Geographic Latency
Network latency from the client to the model provider is a real component of total latency, particularly for global products. A user in Sydney calling a US-hosted model adds 150 to 200ms of round-trip time before the model even starts working.
Edge inference patterns mitigate this:
- Regional model endpoints. Choose providers that offer regional endpoints in or near your users’ geographies.
- CDN-style caching. Cache semantic responses at edge locations close to users.
- Edge-hosted small models. Run small models on edge infrastructure for tasks that do not require frontier capability.
- On-device inference. For privacy-sensitive or latency-critical tasks, run inference on the user’s device.
The trade-offs are real. Edge inference adds operational complexity, and on-device models have capability limits. But for global products where geographic latency is a meaningful share of the total budget, edge patterns are worth the complexity.
I have shipped systems where a small classifier runs at the edge to handle simple queries instantly, with escalation to a centralised frontier model for complex ones. The user experience for the majority of queries is essentially zero perceived latency.
Parallel Execution Patterns
Many AI workflows are sequential by default. They can often be restructured to run in parallel, with significant latency benefits.
Patterns I use:
- Parallel retrieval. Retrieve from multiple sources simultaneously rather than sequentially.
- Parallel sub-task execution. Decompose a task into independent sub-tasks and dispatch them in parallel.
- Speculative tool calls. When the model is likely to call a specific tool, start the tool call in parallel with the model’s reasoning.
- Concurrent ranking and generation. Generate response candidates while ranking retrieved context, then combine.
The architectural pattern is to identify the dependency graph of the workflow and parallelise everything that does not have a strict dependency. The latency reduction can be substantial for workflows with many independent steps.
The cost is increased complexity in error handling and result aggregation. Parallel paths introduce more failure modes, and the orchestration layer needs to handle partial failures gracefully.
The cheapest millisecond is one you would have spent waiting for something that was not blocking the next step.
I treat parallelisation as the default for any multi-step workflow, and only fall back to sequential when there is a genuine dependency.
The First-Token Problem and How to Solve It
The single most impactful latency optimisation is reducing time to first token. Users perceive responsiveness almost entirely through this metric, and most other improvements have smaller perceptual impact.
The techniques I deploy to attack TTFT:
- Prompt caching to skip prefix re-processing.
- Shorter prompts through compression and context pruning.
- Provisioned throughput to eliminate queueing delays.
- Routing to faster models for queries where capability is not the bottleneck.
- Network optimisation to reduce client-to-provider round-trip time.
- Pre-warming for predictable spike patterns.
I treat TTFT as a P99 metric, not a P50 metric. The median TTFT is rarely the problem. The tail is where the bad experiences live, and reducing tail TTFT is what users notice.
A practical exercise: instrument your TTFT distribution and look for the long tail. The causes are usually identifiable: cold caches, queueing delays, network spikes, particularly long prompts. Each cause has a corresponding mitigation.
The teams that solve TTFT well typically do so by treating it as a top-three engineering metric, not a footnote. Once it is a measured target, the architecture follows.
Tail Latency and the P99 Discipline
The latency that defines user experience is not the median, it is the P95 and P99. A system with great median latency and terrible tail latency has unhappy users.
Sources of tail latency in AI systems:
- Provider variability. API providers have load-dependent response times. P99 can be 5 to 10x P50.
- Long prompts or completions. A user query with an unusually long context takes longer to process.
- Retry storms. Failed requests that retry add to the latency of the eventual successful response.
- Cold caches. First request to a route does not benefit from caches.
- Queueing delays. Provider rate limits or quota throttling produce occasional long waits.
The mitigations are mostly architectural:
- Bounded retries with circuit breakers to prevent retry storms.
- Timeout discipline that fails fast and falls over to a backup path.
- Cache warming for predictable cold-start scenarios.
- Multi-provider redundancy with race-to-respond patterns for latency-critical paths.
I track P99 TTFT and P99 total completion time as primary metrics. P50 is necessary but not sufficient. The P99 metric tells you what the worst-experienced users feel.
Latency SLOs and Monitoring
Latency SLOs are how you make latency a sustained discipline rather than a one-time tuning exercise. The SLOs I set for production AI systems:
- TTFT P95 within target budget.
- TTFT P99 within a wider budget.
- Streaming TPS above floor threshold.
- End-to-end completion P95 within target.
- Error rate within budget.
Each SLO has a corresponding error budget, and exceeding the error budget triggers a freeze on non-essential changes. This is the standard SRE discipline applied to AI workloads.
The dashboards I maintain:
- Real-time latency distributions per route and per provider.
- Burndown of error budget per SLO.
- Anomaly detection on latency spikes.
- Synthetic latency probes from multiple geographies.
- Comparison of internal latency to provider-reported latency to isolate where time is being spent.
The monitoring discipline is what keeps latency under control as the system evolves. New features, new prompts, and new providers all change the latency profile, and without monitoring the regressions go unnoticed until users complain.
Common Latency Anti-Patterns
A catalogue of latency mistakes I encounter often.
- Buffering instead of streaming. Intermediate components that buffer the full response before forwarding kill the streaming experience.
- Synchronous orchestration. Multi-step workflows that wait for each step to complete before starting the next, even when steps are independent.
- Sequential retrieval. RAG systems that query sources one at a time instead of in parallel.
- Cold cache by default. Architectures that do not pre-warm caches for predictable startup patterns.
- Ignoring tail latency. Optimising P50 while P99 is hours.
- No timeout discipline. Requests that hang waiting for a slow provider instead of failing over.
- Verbose prompts. Prompts that have grown unchecked, dragging TTFT up.
- One-size-fits-all routing. Sending every request to the same model, even when a faster model would serve.
The common thread is treating latency as a system property rather than an architectural design constraint. Teams that design for latency from the start avoid most of these patterns. Teams that retrofit latency later end up rebuilding more than they expected.
