Devansh Kamlani
The first time I presented an AI cost forecast to a finance team, I was off by a factor of four. Not high. Low. The pilot had been cheap, the production rollout was not, and the assumptions I had brought from web infrastructure economics did not survive contact with token-based pricing. That was the moment I started taking AI cost architecture seriously as its own discipline, distinct from cloud cost management.
AI Cost Architecture: Token Economics & Optimisation
In 2026, AI cost is one of the most consequential architectural decisions a team makes. The difference between a well-designed and a poorly designed cost architecture is not 10 or 20 percent. It is often 5x or 10x for the same end-user experience. I have seen organisations spend more on a single AI feature than on their entire web infrastructure. I have also seen teams cut their bills in half by changing nothing the user could see.
In this guide, I want to share the cost architecture frameworks I rely on. I will cover token economics across providers in 2026, the caching strategies that move the needle, batching and compression techniques, model cascading, the monitoring and budgeting discipline that keeps spend under control, and the cost-overrun patterns I keep running into. The goal is to give you a way of thinking about AI cost that holds up under the scrutiny of a CFO and the constraints of a real production system.
Why AI Cost Architecture Is Different
AI cost differs from traditional cloud cost in three ways that matter for architecture.
First, the unit of consumption is the token, not the request. A single user interaction can consume anywhere from a few hundred to several million tokens depending on context size and output length. The variance is much larger than for web requests.
Second, the marginal cost is non-trivial. A typical web request costs fractions of a cent. A frontier model request can cost tens of cents or more. The economics of “just one more call” are fundamentally different.
Third, the cost driver lives in user behaviour and prompt design, not infrastructure capacity. You cannot rightsize a cluster out of an AI cost problem. You have to architect around the workload itself.
The first principle of AI cost architecture: the cheapest token is the one you never send.
That principle drives every optimisation I will cover. Cache to avoid the call. Compress to shrink the call. Cascade to make the call cheaper when you must. Batch to amortise the overhead. None of these are clever tricks. They are foundational architectural choices that determine whether your AI system is economically viable at scale.
Token Economics in 2026: A Provider Snapshot
The pricing landscape in 2026 has stabilised in shape but continues to shift in magnitude. Models have become both more capable and cheaper per token, though the rate of decline has slowed compared to 2023 to 2024.
A simplified snapshot of indicative pricing tiers across providers. Actual rates depend on contracts, regions, and commitments.
| Tier | Indicative Input Cost | Indicative Output Cost | Use Case |
|---|---|---|---|
| Small fast | Sub-dollar per million | Low single digits per million | Classification, routing, simple Q&A |
| Mid-tier | Single digits per million | Mid single digits per million | Most production chat and RAG |
| Frontier | Double digits per million | High double digits per million | Complex reasoning, long context |
| Specialised | Variable | Variable | Embeddings, reranking, code, vision |
The cost ratios matter more than absolute numbers. A frontier model often costs 30 to 50x more per token than a small fast model. Output tokens typically cost 2 to 5x more than input tokens. Context caching, where supported, can drop input cost by 90 percent for cached prefixes.
I keep a current pricing sheet for every provider I use and update it monthly. The economics shift often enough that decisions made six months ago may no longer be optimal.
The Cost Equation Every Architect Should Know
The cost of an AI system is not “tokens times price”. It is the integral over every request of input tokens, output tokens, retries, cache misses, and orchestration overhead. The architecture choices that matter most affect each of these terms.
A simplified mental model I use:
Cost per user session = (input tokens input rate (1 - cache hit rate))
- (output tokens * output rate)
- (retries * full call cost)
- (orchestration calls * orchestration cost)
- (vector and storage costs * volume)
Each term suggests an optimisation. Input cost is attacked through caching and compression. Output cost is attacked through prompt design that constrains response length. Retry cost is attacked through reliability engineering. Orchestration cost is attacked through routing efficiency. Storage cost is attacked through embedding reuse and vector lifecycle policies.
The trap I see often is teams focusing on one term and ignoring the others. A team that obsesses over input token cost while ignoring orchestration overhead can end up worse off than they started.
Prompt Caching: The First Big Win
Prompt caching is the single highest-leverage cost optimisation available in 2026. Major providers including Anthropic, OpenAI, and Google offer prompt caching for stable prefixes, with discounts on cached tokens ranging from 50 to 90 percent.
The pattern that exploits caching well:
- Place all stable context at the beginning of the prompt: system instructions, retrieved documents, examples.
- Place variable content, primarily the user message, at the end.
- Mark the cacheable prefix explicitly where the provider’s API supports it.
- Reuse the cache key across requests that share the same prefix.
I have shipped systems where 80 percent of input tokens are cached, dropping effective input cost by 70 percent or more. The discipline required is treating the prompt structure as an architectural concern, not an afterthought.
Caching is not a feature you turn on. It is a way of designing prompts so the same prefix appears across many requests.
The trap to avoid is treating caching as a runtime optimisation rather than a design constraint. Caches expire, and prompts that change frequently never get cached. I design the caching strategy at the same time as the prompt structure, not after.
Semantic Caching for Repeat Queries
Semantic caching catches requests where the user’s question is functionally identical to a previous question even if the wording differs. The pattern: embed the incoming query, look up similar past queries in a vector store, and return the cached response if similarity exceeds a threshold.
This works exceptionally well for customer support, FAQ systems, and any workload where users ask the same things in different ways. I have seen semantic caches achieve 30 to 50 percent hit rates in production support systems, eliminating that share of model calls entirely.
The implementation considerations:
- Choose the similarity threshold carefully. Too aggressive and you serve wrong answers. Too conservative and the cache provides little value.
- Validate cached responses periodically. If the underlying knowledge has changed, the cache should expire.
- Distinguish between cacheable and uncacheable queries. Personalised queries that depend on user context should not be cached blindly.
- Monitor the false positive rate. A semantic cache that serves wrong answers undermines trust in the entire system.
Semantic caching is one of the few optimisations where the operational complexity is non-trivial but the cost benefit is large enough to justify the investment for high-volume systems.
Embedding Reuse and Vector Cost Management
Embeddings are cheap individually and expensive in aggregate. A typical RAG system at scale embeds millions of documents and queries per month, and the cost adds up.
The optimisations I deploy:
- Embed documents once and reuse the embedding across updates. Only re-embed when the source text changes.
- Cache query embeddings for repeat queries to skip re-embedding.
- Use smaller embedding models where the quality difference does not affect downstream performance.
- Tier embedding storage. Frequently accessed embeddings stay in hot vector stores; rarely accessed ones move to cold storage.
- Apply lifecycle policies that retire embeddings for documents no longer in the corpus.
Vector storage itself is non-trivial at scale. A managed vector store with hundreds of millions of embeddings can cost as much as the model inference. I treat vector storage as a tier-one cost concern, not a footnote.
The pattern that catches teams out is the implicit cost of dimensionality. A 3072-dimensional embedding is roughly 3x the storage of a 1024-dimensional embedding. The dimension choice has cascading cost implications that should be measured, not assumed.
Batching Strategies for Throughput Workloads
For non-interactive workloads, batching is the most underused cost optimisation I encounter. Provider batch APIs offer significant discounts, often 50 percent off standard rates, in exchange for asynchronous processing.
When batching applies:
- Nightly summarisation of new content.
- Bulk re-embedding after model updates.
- Periodic re-evaluation of historical content.
- Large-scale classification tasks.
- Offline analytics and reporting workloads.
The architectural pattern is to design batch and real-time paths explicitly. Real-time traffic goes to the synchronous API at standard rates. Batch traffic queues up and is dispatched to the batch API on a schedule. The cost saving is straightforward, and the latency trade-off is acceptable because batch workloads do not need immediate responses.
I have shipped systems where 60 percent of token volume runs through the batch path, cutting overall spend by close to 30 percent. The trick is identifying which workloads can tolerate the batch latency window and architecting them as batch from the start.
Prompt Compression and Context Pruning
Every token in the prompt is a token you pay for. Prompt compression and context pruning reduce the size of the input without proportionally degrading output quality.
Techniques I use:
- Context pruning. Retrieve more candidates than you need, then rerank and keep only the top results. The reranker is cheap; the model invocation on excessive context is not.
- Summarisation. Compress long-running conversation history into a running summary. Replace the verbatim history with the summary in the prompt.
- Structured prompts. Use compact formats like JSON or markdown tables rather than verbose prose for instructions and data.
- Reference rather than inclusion. Where the model has tool access, let it call a tool to fetch information rather than including everything in the prompt.
- Dedicated compression models. Specialised models like LLMLingua can compress prompts by 50 to 70 percent with modest quality loss.
The discipline is measuring quality impact, not just cost reduction. A compressed prompt that produces worse responses can cost more overall through retry loops and customer escalations.
The right way to think about compression: every token should earn its place in the prompt.
I run quarterly prompt reviews on production systems to look for accumulated cruft. Prompts grow organically as features are added, and pruning recovers cost without anyone noticing.
Model Cascading: Cheap First, Expensive Last
Model cascading attempts the cheap model first and only escalates to a more expensive model when the cheap one fails. The pattern is straightforward in principle, harder in practice because it requires reliable signals for when escalation is needed.
A cascade I run in production:
| Tier | Model | Share of Traffic | Cost per 1M Tokens (Indicative) |
|---|---|---|---|
| 1 | Small fast model | 65 percent | Low |
| 2 | Mid-tier model | 25 percent | Moderate |
| 3 | Frontier model | 10 percent | High |
The blended cost is dramatically lower than sending all traffic to the frontier model, while quality on hard queries remains high because they get escalated.
The signals that drive escalation:
- Confidence scores when the model provides them.
- Structured output validation failures.
- Response length exceeding expected bounds.
- Downstream task failures detected by a verifier.
- Explicit “I don’t know” or refusal patterns in the response.
The risk is escalation loops that cost more than going to the frontier model directly. I cap each cascade at a fixed budget per request to prevent this. A request that needs three escalations is usually better served by going to the frontier model immediately.
Choosing Between API and Provisioned Capacity
Most providers offer two billing models: pay-as-you-go API access and provisioned capacity with committed throughput. The cost trade-off is significant.
Pay-as-you-go is right when:
- Traffic is highly variable or unpredictable.
- Volume is too low to amortise a commitment.
- The use case is experimental and may be deprecated.
Provisioned capacity is right when:
- Traffic is sustained and predictable.
- Latency consistency is more important than per-token cost.
- Volume is high enough that the commitment discount is material.
The break-even point varies by provider and contract, but I generally consider provisioned capacity once a workload exceeds the equivalent of one or two reserved units of throughput continuously.
A pattern I have shipped: use provisioned capacity for the predictable baseline and burst to pay-as-you-go for spikes above the commitment. The mix gives the cost benefit of commitment with the flexibility of on-demand. The orchestration layer routes traffic between the two pools based on current utilisation.
Cost Monitoring and Budget Architecture
A cost architecture you cannot observe is one that will surprise you. The monitoring I put in place on every production AI system:
- Per-request cost attribution. Every request is tagged with workload, user tier, feature, and routing decision. The bill is broken down by these dimensions.
- Real-time cost dashboards. Spend per hour, per day, per week, with comparisons to historical baselines.
- Budget alerts. Hard budgets per workload that page when 50, 80, and 100 percent are reached.
- Anomaly detection. Automated alerts when token consumption deviates significantly from forecast.
- Per-user cost tracking. For multi-tenant systems, the cost contribution of each customer.
- Cost per outcome metrics. Spend per resolved support ticket, per generated document, per user session. The business-meaningful denominator.
The discipline I push hardest on is cost per outcome. A team focused on cost per request often optimises the wrong thing. A team focused on cost per successful business outcome makes better trade-offs.
I also enforce hard ceilings. Every workload has a daily spend cap. When the cap is reached, the workload throttles or fails over to a cheaper path rather than continuing to spend. This has prevented several incidents that would otherwise have been expensive.
Showback, Chargeback, and FinOps for AI
When AI becomes a significant share of an organisation’s cloud spend, the FinOps practices that worked for compute and storage need to evolve.
The patterns I have seen work:
- Showback. Every team sees the AI cost of their workloads in a dashboard, with breakdowns by feature, model, and user segment. Visibility alone drives behaviour change.
- Chargeback. Teams are billed internally for their AI consumption against budget. This forces engineering trade-offs to consider cost explicitly.
- Unit economics. AI cost is reported against business unit economics: cost per active user, cost per transaction, cost per conversion. The numerator alone is meaningless without the denominator.
- Forecast and variance. Each team forecasts its AI spend and reports variance. Persistent overruns trigger architectural review.
The cultural shift is treating AI cost as a first-class engineering metric, on par with latency and error rate. Teams that internalise this make different design choices, and the cost trajectory of the organisation bends.
I have seen the absence of these practices produce predictable failures: surprise quarterly bills, finance teams discovering AI spend after it has become unmanageable, executive sponsors questioning the ROI of AI initiatives because the cost narrative was never built.
Common Cost Overrun Patterns
A catalogue of cost overruns I have seen often enough to call out.
- The conversation loop. Chat applications that include the entire conversation history in every prompt grow quadratically in token cost. Always summarise long histories.
- The verbose system prompt. System prompts that accumulate over time, with new instructions stacked on old without pruning. Quarterly reviews are essential.
- The RAG over-retrieval. Retrieving 50 documents because “more context is better”. Reranking and pruning to the top 5 is almost always equivalent in quality and much cheaper.
- The retry storm. A flaky downstream service triggers retries that multiply token cost. Implement circuit breakers and bounded retry budgets.
- The agent free-roam. Agentic workflows without step budgets that decide they need fifteen iterations to answer a question that should take three.
- The forgotten endpoint. A pilot endpoint that was never deprecated continues serving low-volume traffic at frontier-model cost.
- The wrong default. A configuration default that routes all traffic to the most expensive model, often inherited from prototype code.
- The reasoning model on every request. Reasoning models cost dramatically more. Using them for simple tasks burns money without quality benefit.
The common thread is lack of monitoring. Every one of these patterns is visible in good cost telemetry and invisible without it.
The Sustainable Cost Architecture Playbook
Pulling everything together, the playbook I follow for sustainable AI cost architecture:
- Model token economics for every workload at design time. If the numbers do not work at projected scale, redesign.
- Cache aggressively. Prompt caching, semantic caching, embedding reuse.
- Cascade between models. Cheap first, expensive last.
- Batch where latency allows.
- Compress context. Every token earns its place.
- Tag every request for attribution. Cost without attribution is unactionable.
- Set hard budgets and enforce them.
- Report cost per outcome, not just cost per request.
- Review prompts quarterly for accumulated cruft.
- Renegotiate provider contracts annually based on actual usage patterns.
None of these are exotic. The discipline is treating them as ongoing operational practices, not one-time setup tasks.
A sustainable AI cost architecture is a habit, not a configuration.
The organisations that get this right end up with AI systems that scale economically. The ones that do not end up with AI initiatives that are killed by finance before they reach maturity.
