Prompt Caching Cuts Your LLM Bill by 80 Percent and Nobody Is Using It

Here is a number that should bother you: on most production LLM apps I have looked at, the prompt caching hit rate is close to zero, even though both Anthropic and OpenAI have shipped prompt caching and the feature is sitting there waiting. People hear “caching,” assume it is complicated, and move on. It is not complicated. The hard part is that your prompt is probably built in an order that makes a cache hit impossible, and no amount of enabling the feature fixes a badly ordered prompt.

This is not response caching. You are not storing answers and skipping the model. The model still runs on every request. What gets cached is the provider’s internal attention state (the KV cache) for the front of your prompt, so the model does not have to reprocess those tokens from scratch. You pay a fraction of the normal input price for the part that was already computed. For a long, stable prefix, that is the difference between paying for 50,000 tokens and paying for 5,000.

What actually gets cached

When you send a prompt, the model processes it token by token, building an attention state as it goes. That computation is the expensive part of input processing. Prompt caching lets the provider keep the attention state for a prefix of your prompt and reuse it on the next request that starts with the exact same bytes.

The key word is prefix. Caching is anchored to the start of the prompt and extends forward until the first byte that differs. The moment two requests diverge, the cache stops. Everything after the divergence point is reprocessed at full price.

Request A:  [ system prompt ][ tool defs ][ examples ][ user query A ]
Request B:  [ system prompt ][ tool defs ][ examples ][ user query B ]
            |<--------- identical prefix --------->|<-- differs -->|
            |<--------- served from cache ------->|<- full price ->|

If the identical span at the front is large and stable, you win big. If you put anything that changes per request near the front, you poison the cache for everything behind it. That single fact is what separates apps that benefit from apps that do not.

The naive prompt that never hits the cache

Here is a RAG prompt the way most apps build it. It looks reasonable and it caches nothing.

def build_prompt(user_query: str, retrieved_chunks: list[str]) -> str:
    # Retrieved context goes first because "context before question"
    # is the pattern everyone copies.
    context = "\n\n".join(retrieved_chunks)
    now = datetime.utcnow().isoformat()

    return f"""Current time: {now}
Retrieved context:
{context}

System: You are a support assistant for Acme Corp. Follow these rules...
(800 tokens of stable instructions)

User question: {user_query}
"""

Three things break caching here, in order of how early they appear:

1. Current time is at the very top. It changes every request, so the prefix diverges at token one. Nothing after it can ever be cached.
2. Retrieved context changes per query. Even if you fixed the timestamp, the context block differs on almost every request, so the stable system prompt sitting behind it never gets reused.
3. The stable, reusable 800-token system prompt is at the bottom, which is the worst place for it.

The prompt is upside down. The parts that never change are last, and the parts that change constantly are first. A cache that anchors to the prefix gets nothing to hold onto.

The cache-friendly reorder

The fix is to sort the prompt by how often each section changes. Most stable at the top, most volatile at the bottom.

[ MOST STABLE  ]  System prompt, persona, policies
[              ]  Tool / function definitions
[              ]  Few-shot examples
[              ]  Large static knowledge (docs, schema, catalog)
[              ]  -- cache breakpoint here --
[              ]  Retrieved RAG chunks (varies per query)
[ MOST VOLATILE ]  User message, timestamp, request-specific data

On Anthropic you mark the breakpoint explicitly. Everything up to and including the block tagged with cache_control becomes the cacheable prefix.

import anthropic

client = anthropic.Anthropic()

SYSTEM_BLOCKS = [
    {
        "type": "text",
        "text": STATIC_SYSTEM_PROMPT + TOOL_DEFS + FEW_SHOT + KNOWLEDGE_BASE,
        # Mark the end of the stable prefix. Everything up to here is cached.
        "cache_control": {"type": "ephemeral"},
    }
]

def answer(user_query: str, retrieved_chunks: list[str]) -> str:
    resp = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=512,
        system=SYSTEM_BLOCKS,  # stable, cached
        messages=[
            {
                "role": "user",
                # Volatile content lives here, after the breakpoint.
                "content": f"Context:\n{chr(10).join(retrieved_chunks)}\n\nQ: {user_query}",
            }
        ],
    )
    return resp.content[0].text

OpenAI does not need an explicit breakpoint. Its caching is automatic for prompts over 1024 tokens; it hashes the prefix in chunks and reuses whatever matches. But the same discipline applies: if a volatile value sits near the front, the automatic cache has nothing stable to latch onto. Automatic caching does not save you from a badly ordered prompt. It just means you do not get to pick the breakpoint.

The prefix rules that decide a hit

A cache hit is byte-exact on the prefix. These are the rules that trip people up.

The TTL rule is the one that quietly kills hit rates. A 5 minute sliding window means the cache only pays off when requests arrive close together. If your traffic is one request every few minutes, each one writes a fresh cache entry that expires before the next arrives, and you pay the write premium every time with no read benefit. Caching rewards bursty, concurrent traffic. It punishes a trickle.

Write price, read price, and the break-even

Caching is not free to populate. The first request that creates a cache entry pays a write premium, and later requests that hit it pay a steep discount. Rough numbers for Claude Sonnet at a $3.00 per million token base input price:

OpenAI is simpler and less generous: cached input tokens are billed at roughly half price (0.5x) with no separate write premium. Anthropic charges you more to write but far less to read, so it rewards high reuse; OpenAI’s 50 percent is automatic and lower-ceiling.

The break-even on Anthropic is easy to reason about. A cache write costs an extra 0.25x, and each read saves 0.9x. So a cached prefix pays for its write premium after a single additional hit, and everything past that is close to free. If a prefix is read even twice before it expires, caching already won.

Real numbers from a production RAG app

The 80 percent in the title is not marketing. It shows up when the stable prefix dominates the request, which is exactly the shape of a long-context retrieval or agent workload. Here are two scenarios on Claude Sonnet, output held at 500 tokens ($15 per million).

Scenario A, a typical short-prompt RAG request:

Stable prefix (system + tools + examples): 4,000 tokens
Volatile (retrieved chunks + query):       3,200 tokens
Output:                                       500 tokens

Scenario B, a long-context agent that loads a 50,000 token knowledge base and tool set into a stable prefix:

Stable prefix (knowledge base + instructions): 50,000 tokens
Volatile (query + small working context):         800 tokens
Output:                                            400 tokens

Scenario B is where the headline lives. When the stable part of the prompt is 60x larger than the volatile part, caching turns a 50,000 token input into a 5,000 token equivalent and the bill drops by the better part of 90 percent. The lesson is not that caching is magic. It is that caching pays in exact proportion to how much of your prompt is stable, and long-context apps are mostly stable prompt.

Measuring whether it works

Do not assume the cache is working. The API tells you. Anthropic returns cache_creation_input_tokens and cache_read_input_tokens in the usage block; OpenAI returns cached_tokens inside prompt_tokens_details. Track the ratio.

def log_cache_usage(resp):
    u = resp.usage
    written = getattr(u, "cache_creation_input_tokens", 0)
    read = getattr(u, "cache_read_input_tokens", 0)
    fresh = u.input_tokens  # non-cached input

    total_prefix = written + read
    hit_rate = read / total_prefix if total_prefix else 0.0

    # If hit_rate stays near 0, your prefix is unstable or traffic is too sparse.
    print(f"cache hit rate: {hit_rate:.0%}  read={read} written={written} fresh={fresh}")

A healthy hit rate for a well-ordered prefix under steady traffic is 70 to 95 percent. If you see 5 percent, you have one of two problems: something volatile crept into the prefix, or your requests are spaced too far apart for a 5 minute window. The fix for the first is to hunt down the moving value. The fix for the second is the 1 hour TTL, or accepting that caching is not for your traffic pattern.

The gotchas that eat the savings

A few things reliably break caching in ways that are hard to spot:

• Serialization drift. If you build the prefix from a dict and the key order is not stable, the serialized bytes change and the cache misses. Serialize with sorted keys, or hold the prefix as a fixed string.
• A timestamp or request ID in the system prompt. Extremely common. It changes one token near the front and invalidates everything.
• Per-user data high in the prompt. User name, account tier, or locale injected into the system block gives every user a separate cache entry. If the per-user part is small, push it below the breakpoint into the message body.
• Tool schema churn. Editing a tool definition invalidates the cache for every request that uses it. Expect a cold window after any deploy that touches tools or the system prompt.
• Too many breakpoints. Anthropic allows a small number of cache breakpoints. Scattering them does not help; one breakpoint at the end of the largest stable block is almost always right.

Honest assessment

What works: for long-context and agent workloads where a large stable prefix is reused under bursty traffic, prompt caching is the single highest-leverage cost lever available, and the 80 to 90 percent input savings is real. It is also nearly free to adopt on the Anthropic side (one cache_control tag) and free on OpenAI (nothing to do but order the prompt well).

What does not work: caching does little for short prompts, for sparse traffic where entries expire between requests, or for prompts where genuinely everything changes per call. The write premium means that if a prefix is read zero or one extra times before it expires, you lose money. And a badly ordered prompt gets no benefit no matter how much you enable the feature.

What to actually do, in order:

1. Reorder every prompt by volatility: most stable at the top, most volatile at the bottom. Do this even before you enable caching, because it is the whole game.
2. On Anthropic, put one cache_control breakpoint at the end of the largest stable block. On OpenAI, just trust the automatic cache once the order is right.
3. Log the cache read and write token counts and watch the hit rate. Treat a low hit rate as a bug to investigate, not a limitation to accept.
4. Hunt down anything volatile that sits above the breakpoint. Timestamps, request IDs, and per-user fields are the usual culprits.
5. Only reach for the 1 hour TTL if your traffic is too sparse for the default window and the prefix is large enough to justify the write premium.

The feature is sitting in the API, it costs almost nothing to adopt, and it is the rare optimization that gets better the larger your context grows. The reason nobody is using it is not difficulty. It is that their prompts are built in the wrong order, and they never looked at the hit rate to find out.