LEARN / PROMPT CACHING

Prompt caching: the 90% discount most AI teams never claim

PUBLISHED 2026-07-10 · TOKENBURN INDEX

Every major LLM provider will sell you the same tokens at a fraction of the price if you just re-send them in the right order. It's called prompt caching, it's sitting in plain sight on the pricing page, and a remarkable number of teams are paying full fare anyway. This is how it works, what it saves, and the quiet ways it breaks.

Why input tokens dominate modern AI bills

Output gets the headlines (it's 4–8x pricier per token), but volume tells another story: production AI apps are input-heavy. A chatbot re-sends its system prompt and the conversation history on every turn. A RAG app re-sends retrieved documents. An agent re-reads its instructions and accumulated tool results on every step of its loop. In all three, the majority of billed tokens are the same bytes, re-read. That redundancy is exactly what prompt caching monetizes back to you.

What prompt caching actually is

When a request arrives, the provider checks whether its opening bytes — the prefix — match something processed recently. On a hit, the model skips re-processing that prefix and bills those input tokens at a heavy discount. On a miss, it processes normally and (depending on the provider) writes a cache entry for next time, sometimes at a small premium over the normal input price. Three properties define the mechanics everywhere: it's a prefix match (byte-exact, from position zero), entries have a time-to-live (typically minutes, extendable), and there's a minimum size below which nothing caches — silently.

The math that should get your attention

Example 1 — the system prompt. Your product sends a 2,000-token system prompt with every request, 1,000 times a day, on a $3/MTok-input model. Uncached, that prefix alone costs ~$6.00/day — $180/month for the same bytes, re-read daily like a goldfish. Cached at ~0.1x: ~$0.66/day, plus pennies of write premium. Same model, same prompts, ~90% off.

Example 2 — the RAG corpus. An internal assistant prepends the same 50,000-token policy handbook to every query, 200 queries/day. Uncached: ~$30/day ($900/month). Cached: ~$3.30/day. The bigger the shared context, the more absurd the uncached number — and 50K-token contexts are unremarkable in 2026.

The fine print: write premiums, TTLs, break-even

• — Writes can cost extra. Anthropic, for instance, bills cache writes at 1.25x normal input for the standard 5-minute TTL (2x for a 1-hour TTL). With those rates, the second request already breaks even; steady traffic is nearly all discounted reads.
• — TTL shapes everything. Bursty traffic inside the TTL window rides the cache; sparse traffic re-pays the write each time. If requests arrive hours apart, a longer (pricier-to-write) TTL or no caching at all may be the right call.
• — Minimums are silent. Prefixes below the provider's threshold (commonly ~1,000–4,000 tokens depending on model) simply don't cache. No error, no warning — just full-price billing.

Provider differences (mid-2026 snapshot)

• — OpenAI: automatic — repeated prefixes are detected and billed at the cached rate with no code changes (GPT-5.5: $0.50 cached vs $5.00 normal input per MTok, a 90% discount).
• — Anthropic: explicit — you place cache_control markers on the blocks you want cached; reads bill at ~0.1x. More work, more control over exactly what caches.
• — Google (Gemini): both implicit caching and an explicit cached-content API for pinning large contexts.

Rates and mechanics change; check your provider's pricing page. The strategy below is provider-independent.

The silent cache-breakers

Caching is a prefix match on exact bytes, so anything that varies early in the prompt disables it for everything after — without any error. The classics:

• — The timestamp at the top. “Current date: 2026-07-10 14:32:07” interpolated into the system prompt header changes bytes every request. The whole cache dies, every time, forever. Move volatile context to the end, or truncate to the day.
• — Non-deterministic serialization. JSON with unsorted keys, tool lists built from an unordered set — logically identical, byte-different, cache-hostile. Sort everything.
• — Per-user data early. A user ID or name in the first line guarantees zero cross-user reuse of your expensive shared prompt. Shared content first, personalization after.
• — Swapping models or tools mid-flow. Caches are scoped per model, and changing the tool set changes the prefix. “Route each request to a random model for load balancing” is also “never hit the cache.”
• — A/B testing the system prompt. Two variants = two caches = half the hit rate each. Fine if intentional; expensive if forgotten.

Trust nothing, verify the usage fields

Every provider reports cache activity in the API response usage data (fields like cached_tokens or cache_read_input_tokens). The audit takes five minutes: send two identical requests back-to-back and read the second one's usage. If cache reads are zero on repeated traffic, one of the breakers above is at work — diff the exact rendered prompts to find it. Teams that never look at these fields are, statistically, donating margin.

The checklist

• 1. Order every prompt stable-first: system prompt → tools → documents → history → the volatile bits last.
• 2. Freeze the stable parts — no timestamps, IDs, or feature flags interpolated early.
• 3. Serialize deterministically (sorted keys, stable tool order).
• 4. Check your provider's minimum cacheable size against your prefix.
• 5. Verify cache reads in usage data after deploying — and re-verify after prompt changes.
• 6. Re-run the math when traffic patterns change; sparse traffic may not be worth write premiums.

Know which kind of traffic you have

Caching is the highest-leverage optimization for anything that re-sends context — chatbots with long system prompts, agents re-reading repositories, RAG over a stable corpus. For one-off, ever-changing prompts it does exactly nothing. Find out which side you're on: measure a representative prompt in the Burnmeter, look at how much of it is fixed boilerplate, and multiply the repeated part by your real volume. That number is your caching opportunity.

Frequently asked questions