LLM FinOps: Token Costs, Monitoring, Caching, and Spend Forecasting

LLM API spend is unusual among cloud costs: the entire rate card is public, the unit prices span two orders of magnitude across model tiers, and three published discounts — cached input, batch processing, and model routing — can each cut a workload's bill by half or more without touching product behavior. The teams that control AI spend are not the ones negotiating hardest; they are the ones that attribute every token to a feature, exploit the pricing asymmetries deliberately, and forecast from per-feature token budgets instead of extrapolating the invoice.

This guide consolidates the whole discipline: the real unit economics of the three major LLM APIs, prompt caching mechanics and when they pay, the exact-versus-semantic caching decision, batch and model tiering, cost observability (tagging, budgets, alerts), and spend forecasting. Every price in it is quoted verbatim from the vendors' pricing pages as of August 20, 2026 — and one of the operational lessons of this piece is that you should assume those pages have changed by the time you read it, and re-price before you decide.

1. By the numbers

2. The rate card: what the three vendors actually charge

Every LLM FinOps decision starts from the same three-column structure: base input, cached input, and output, priced per million tokens (MTok). The asymmetries between those columns — not the headline prices — are what you optimize against. The table below shows representative models from each vendor's current catalog, quoted verbatim from the pricing pages.

Four structural asymmetries in that table drive nearly every optimization decision. Output costs 5x to 8x input. A feature that generates long answers from short prompts has completely different economics from one that reads long documents and returns a verdict — and controls that cap or compress output (structured formats, terse instructions, output token limits) attack the expensive column. Cached input costs one tenth of fresh input. Anthropic prices cache reads at 0.1x base input; OpenAI bills cached input tokens at 0.1x the uncached rate^[2][4]. Any architecture that re-sends a stable prefix without caching it is paying ten times the available price. A tier down is 5x to 25x cheaper. Claude Haiku 4.5 input is one fifth of Opus 5's; gpt-5.6-luna is 1/25th of gpt-5.6-sol's^[2][1]. Batch halves everything. All three vendors take 50% off asynchronous workloads^[2][1][3].

Three fine-print items matter more than they look. First, reasoning tokens bill as output — on models with extended thinking, the model's internal reasoning is metered at the output rate, which is the expensive rate, so a reasoning-heavy configuration can multiply a request's cost without changing the visible answer length (the economics are covered in depth in /guides/reasoning-models-enterprise-guide). Second, tokenizers are not stable across model generations: Anthropic notes that Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text"^[2] — meaning a model upgrade can silently reprice your entire workload even when the per-token rate is unchanged. Third, long-context and residency multipliers exist: Gemini 2.5 Pro input jumps from $1.25 to $2.50 per million tokens for prompts over 200k tokens^[3], while Anthropic prices its full 1M-token context window at standard rates on Claude 4.6 and later and applies a 1.1x multiplier for US-only inference routing^[2].

3. Prompt caching: the mechanics, and when it pays

Prompt caching is the highest-leverage optimization for any workload that re-sends a stable prefix — a system prompt, a tool catalog, a RAG knowledge block, or a growing conversation history. All three vendors implement it as exact prefix matching: the provider stores the processed state of the leading portion of your prompt and, when a subsequent request starts with byte-identical content, bills those tokens at the cached rate instead of the full input rate.

The vendors differ in mechanics and in whether writes cost extra. Anthropic prices the full lifecycle: a 5-minute cache write costs 1.25x base input, a 1-hour cache write costs 2x, and every read costs 0.1x^[2]. The break-even math is printed on the pricing page: "caching pays off after one cache read for the 5-minute duration (1.25x write), or after two cache reads for the 1-hour duration (2x write)"^[2]. The cache refreshes at no additional cost each time it is used within its lifetime, and minimum cacheable prompt lengths range from 512 to 4,096 tokens depending on model^[5]. OpenAI caching "works automatically for eligible requests, with no code changes required" above a 1,024-token minimum; cached input bills at 0.1x, and on GPT-5.6-generation models cache writes bill at 1.25x base input with a 30-minute TTL, where earlier models had no write fee and an inactivity-based lifetime of roughly 5 to 10 minutes up to one hour^[4]. Google enables implicit caching by default on Gemini 2.5 and newer models, with minimum prompt sizes of 4,096 tokens on the Gemini 3.x line (2,048 on Gemini 2.5), and also offers explicit caching where you create a cache object and pay a storage rate — currently $0.50 per million tokens per hour on Gemini 3.7/3.6 Flash — on top of a reduced per-token rate^[6][3].

Because matching is exact and positional, cache economics are an architecture property, not a billing setting. One byte of change near the top of the prompt invalidates everything after it: Anthropic's cache follows the hierarchy tools → system → messages, so editing a tool definition invalidates the entire cached prefix, and OpenAI states plainly that "cache hits are only possible for exact prefix matches within a prompt"^[5][4]. The design rule that follows: put stable content first (frozen system prompt, deterministic tool list, reference documents) and volatile content last (timestamps, request IDs, the user's question). The most common failure mode in practice is a silent invalidator — a `datetime.now()` interpolated into the system prompt, an unsorted JSON structure, a per-request session ID rendered early — that quietly zeroes the hit rate while everything appears to work. Prompt structure and caching discipline are two sides of the same coin; the prompt-side techniques are covered in /guides/enterprise-prompting-techniques.

Illustrative arithmetic (computed from the cited list prices, not a benchmark): a RAG assistant that prepends a 100,000-token knowledge pack to every call on Claude Sonnet 5 pays $0.20 per call uncached (100k × $2/MTok). With caching, the first call writes the 5-minute cache for $0.25 (100k × $2.50/MTok) and each subsequent call within the window reads it for $0.02 (100k × $0.20/MTok)^[2]. Ten calls inside the window cost $2.00 uncached versus $0.43 cached — roughly a 78% reduction on that prefix — and the saving compounds with traffic, because every additional hit costs a tenth of the uncached price. The same shape holds on OpenAI's 0.1x cached-input pricing^[4]. The corollary also matters: a prefix read fewer than once per cache lifetime never pays back its write premium, so low-traffic or highly personalized prompts are poor caching candidates.

4. Exact versus semantic caching: know which risk you are buying

Vendor prompt caching is exact and provider-side: it reduces the cost of tokens you still send, and it can never return a wrong answer, because the model still runs on the full prompt. A second, application-side family of techniques — response caching — avoids the API call entirely by returning a previously generated answer. Exact-match response caching (hash the normalized request, return the stored response) is safe but nearly useless for free-text inputs, since trivial phrasing differences miss. Semantic caching closes that gap by embedding each query, storing embeddings and responses in a vector index, and serving the cached response when a new query's similarity to a cached one exceeds a threshold. The approach is real and open-sourced — GPTCache, presented at the NLP-OSS workshop in 2023, reports that serving from cache "can increase response speed 2-10 times when the cache is hit" and insulates response time from network fluctuation^[7].

The honest framing for semantic caching is that it trades a cost problem for an accuracy problem. "Cancel my subscription" and "how do I cancel my subscription?" embed close together and should share an answer; "cancel my subscription" and "did you cancel my subscription?" also embed close together and must not. The similarity threshold is a product decision dressed up as a hyperparameter: set it high and the hit rate (and the savings) collapses; set it low and some fraction of users receives an answer to a question they did not ask. Widely circulated claims that semantic caching cuts LLM API calls by a fixed large percentage — 80% is the number that travels — do not trace to a citable primary source; hit rates are entirely workload-dependent, and you should measure your own. Beyond false hits, budget for staleness (cached answers outlive price lists, policies, and model upgrades), cache poisoning of shared caches, and privacy: a response generated for one user must never be served to another if it can embed personal context.

5. Batch processing and model tiering: the other two structural discounts

The batch discount is the simplest 50% most teams never fully collect. All three vendors price asynchronous processing at half the interactive rate — Anthropic's Batch API at a 50% discount on both input and output tokens (Claude Sonnet 5 falls to $1 in and $5 out; Haiku 4.5 to $0.50 and $2.50), OpenAI's Batch API at 50% off standard rates, and the Gemini API's batch tier at a 50% cost reduction^[2][1][3]. The organizational work is classification: audit every LLM workload for whether a human is waiting on the response. Nightly enrichment, document backfills, evaluation suites, embedding refreshes, content moderation sweeps, and report generation almost never need interactive latency — moving them is a routing change, not a redesign. Note the discounts compound: Anthropic states that Batch API and prompt caching discounts can be combined^[2].

Model tiering is the larger and riskier lever. The rate card rewards it heavily — within Anthropic's catalog, Haiku 4.5 input is one fifth the price of Opus 5 input and one tenth on output; within OpenAI's, the gap between gpt-5.6-luna and gpt-5.6-sol is 25x on both input and output^[2][1]. Illustrative arithmetic (from the cited prices): a classification pipeline processing 1M input and 200k output tokens per day costs $10.00/day on interactive Claude Opus 5 ($5.00 input + $5.00 output) and $1.00/day on batched Claude Haiku 4.5 ($0.50 + $0.50) — a 10x difference for a task that a small model may handle at equal quality^[2]. The discipline that keeps tiering honest is an evaluation gate: route a task down a tier only when an offline eval shows quality parity on your data, and keep the eval running so a model update or drift in the input mix cannot silently degrade a demoted workload. Routing by predicted difficulty — cheap model by default, escalate on low confidence or explicit triggers — captures most of the saving while capping the quality risk.

6. Cost observability: tagging, budgets, and alerts

None of the levers above can be operated on an invoice total. The unit of LLM cost management is the feature: cost per conversation, per document processed, per agent run — because that is the level at which caching, tiering, and batch decisions are made, and the level at which finance can ask whether a capability is worth what it costs. Getting there requires attribution machinery at three layers, and where model consumption flows through a cloud platform it should reuse the taxonomy your cloud FinOps practice already runs — AWS cost allocation tags, for example, group usage and costs by business categories such as cost centers and application names in billing reports and Cost Explorer^[8].

On top of attribution sit budgets and alerts, and the alerts should come in two kinds. Threshold alerts catch slow drift: OpenAI's platform supports a notification threshold that emails when usage passes a set level, and a hard spend limit that stops API traffic when tracked spend reaches it^[9]. Anomaly alerts catch fast failures — the retry loop that hammers an endpoint overnight, the test harness left running against production, the prompt change that doubled average output length — and for these, near-real-time granularity matters: Anthropic's usage endpoint supports 1-minute buckets, with data typically appearing within 5 minutes of request completion^[10]. A runaway agent burning tokens at midnight is found by an anomaly alert on token velocity per feature, not by a monthly budget review. Hard limits deserve care in production paths: stopping traffic is itself an outage, so reserve hard caps for non-critical keys and use paging alerts plus graceful degradation (queue, downgrade tier, shed load) for revenue-bearing ones.

7. Forecasting AI spend without inventing numbers

LLM spend forecasting has a bad reputation because teams try to forecast the invoice. Forecast the drivers instead. A per-feature token budget decomposes cleanly: monthly cost ≈ Σ over features of (calls × average fresh input tokens × input rate + calls × average cached tokens × cached rate + calls × average output tokens × output rate), with batch workloads priced at their discounted rates. Every term is observable from the metering layer above, and each term has a different owner: call volume moves with adoption (a product forecast), tokens per call move with prompt and retrieval design (an engineering variable), and rates move with vendor pricing and routing mix (a sourcing variable). Separating them is what turns a surprise overrun into a diagnosable variance: usage grew, calls got heavier, or the mix shifted to expensive paths.

Three driver behaviors deserve explicit modeling. First, per-call weight tends to grow: conversation histories lengthen, retrieval packs expand, agents add tool-call round-trips, and reasoning settings add output-billed thinking tokens — so a forecast that scales last quarter's cost-per-call by projected volume understates. Second, step changes dominate trend: a model upgrade can reprice the baseline overnight — Anthropic's newer tokenizer producing "approximately 30% more tokens for the same text" is a concrete, vendor-documented example of a silent repricing^[2] — and vendor price changes are sometimes literally scheduled, as with Gemini 3.7 Flash's published move from $0.75 to $1.50 input on January 1, 2027^[3]. Keep a small register of dated repricing events (model migrations, announced price changes, planned feature launches) and layer them on the volume trend. Third, optimizations are forecastable too: a planned caching rollout or batch migration has a computable effect on the rate terms, which is how a FinOps team commits to a cost-reduction number with a straight face. Add a variance buffer sized to your observed month-over-month volatility rather than a folklore percentage, and revisit the forecast on a monthly cadence with engineering and finance in the same review.

8. Honest objections

"Engineering time costs more than the tokens." Often true early, and the discipline should scale with the bill: at a few hundred dollars a month, provider dashboards and a weekly glance are the right amount of FinOps, and premature caching infrastructure is pure overhead. The counterpoint is that the cheap moves — key segmentation by feature, logging usage fields you already receive, prompt-prefix ordering — cost hours, not quarters, and they are much easier to install before volume arrives than to retrofit during the incident that follows it.

"Per-token prices keep falling, so optimization and forecasting are wasted effort." Rates have indeed trended down at the capability frontier, and Anthropic converting Sonnet 5's introductory $2/$10 into the standard price is a live example of a scheduled increase being canceled^[2]. But price per token is not spend: volume growth, heavier calls, and output-billed reasoning tokens routinely outrun per-token declines, and Google's scheduled 2x increase on Gemini 3.7 Flash shows list prices move in both directions^[3]. Falling unit prices are an argument for re-pricing your model quarterly — not for skipping the model.

"Caching layers add complexity and new failure modes." For semantic caching this objection is substantially correct, which is why the framework above treats it as a last-resort lever with an accuracy SLO. Vendor prompt caching is the opposite case: the provider carries the state, a miss simply bills at the standard rate, and correctness is unaffected by construction. The two should not be lumped together when a team debates "caching risk."

9. The read

LLM cost management is unusually tractable because the levers are published. The decision sequence that follows from the rate card: instrument attribution first (nothing else can be evaluated without it), collect the safe structural discounts second (prompt caching on stable prefixes, batch for asynchronous work — both provider-side, neither able to corrupt an answer), then take the higher-judgment levers (model tiering behind an evaluation gate, semantic caching behind an accuracy SLO), and forecast from per-feature drivers with a dated repricing register. A team that does the first two steps has usually cut unit costs substantially at near-zero risk; the second two are where FinOps starts trading cost against quality and needs product at the table. And because every number in this piece is a snapshot of a page that changes, the durable asset is not the prices — it is the machinery for re-pricing decisions quickly when they move.