Lab 5 — Prompt caching and cost

Time: 35 minutes · Prerequisites: Labs 1–2

Why this matters

This lab is the difference between the project shipping and not shipping.

The arithmetic is not subtle. Priya's budget is roughly $4,000 a month. Peak week is 11,300 tickets, about 45,000 in a peak month. That is just under nine cents per ticket for everything — triage, resolution, and a drafted reply.

The cached prefix — role instructions plus the full handbook — measures about 3,400 tokens, and it goes out on every request, because legal changes it weekly and it cannot be baked into a prompt. Priced at full input rate on a peak month that is about $765 in handbook tokens alone, before a single word of output. Caching drops the same line to roughly $75.

Be honest about what that does and does not prove. $765 is not the whole $4,000, and Lab 7 will measure the entire flagship pipeline at about $137 a month — this system is not close to its budget ceiling, and anyone who tells you caching is what makes it ship is selling something. What caching buys here is a fivefold cost headroom on the largest single line item, which is the difference between "we can afford to send the handbook on every request" and "we start trimming the handbook to save money" — and a trimmed handbook is an accuracy problem, not a cost one.

The failure mode is the real reason this deserves a whole lab: every way of breaking the cache succeeds silently. HTTP 200, correct answer, and a prefix line item that jumps 10×. Nobody notices until finance does.

---

Try this

Spot the cache bug

Four variants of the same request. One caches. Three do not — and every one of those three returns HTTP 200 with a correct answer. Pick a variant and read the second-call usage block.

src/prompts.ts

system: [
  {
    type: "text",
    text: `${ROLE}\n\n${POLICY_HANDBOOK}`,
    cache_control: { type: "ephemeral" },
  },
  { type: "text", text: volatileContext(opts) },
]

usage on the 2nd identical call

Does this cache? Every variant returns HTTP 200 with a correct answer, so the response body will not tell you.

The only field that distinguishes these four is cache_read_input_tokens. That is why it is the thing to alert on, and why a caching change cannot be validated by reading the diff.

Open full playground →

Try this

Priya's budget, interactively

Volume × caching × model tier against a ~$4,000 monthly ceiling. This is the arithmetic behind Steps 1 and 2.

Route

One call. Classify and route.

Model

Token counts were measured on this model.

Monthly tickets

45,000 tickets a month

Prompt caching

On

The 3,358-token prefix is role instructions plus the full policy handbook.

Per request

$0.02

Per month

$1,120

Without caching

$932

Inside the $4,000 monthly budget, at 28% of it.

Where the money goes, per request

• Prefix (handbook + role) $0.02
• Variable input $0.00056
• Output $0.00335

Caching moves the prefix from full rate to 0.1x on every request after the first. Turn caching off and watch the blue segment take over the bar.

Open full playground →

Objectives

• Get a cache hit, and prove it with cache_read_input_tokens
• Break the cache four different ways and recognize each signature
• Decide where a breakpoint belongs
• Reason about when caching loses money

Silent failure

A broken cache does not look broken. The tell is a single usage field on the second identical call — not the status code, not the answer text.

---

Step 1 — a cold call and a warm one

npm run smoke 2>&1 | grep -A12 "call 2"

The second call should report cache_hit: true with several thousand cache_read_input_tokens.

Usage receipt

Warm triage call — representative

After the prefix is written. A miss looks the same in the response body; only these fields change.

input_tokens112

output_tokens134

cache_creation_input_tokens0

cache_read_input_tokens3358

cost (list)$0.0039

Q1. The first call shows cache_creation_input_tokens and the second shows cache_read_input_tokens, both large, while input_tokens stays small on both. Explain what each field is counting, and compute the cost difference.

Step 2 — break it (four ways)

Read src/prompts.ts. Then break the cache deliberately, one change at a time, running npm run smoke after each and recording whether cache_hit survives.

Break A — a timestamp in the prefix. In buildSystem, change the frozen block to:

text: `Generated at ${new Date().toISOString()}\n${roleText}\n\n---\n\n${POLICY_HANDBOOK}`,

Break B — move the breakpoint. Put cache_control on the volatile block instead of the frozen one.

Break C — drop under the floor. Replace the whole frozen block's text with POLICY_HANDBOOK.slice(0, 400) — roughly 110 tokens, and note that you have to drop the role text too. The role instructions alone measure ~554 tokens, which already clears Opus 5's 512-token minimum on their own; trimming only the handbook would leave the prefix cacheable and this break would quietly demonstrate nothing. Getting a "below the minimum" repro is fiddlier than it looks, which is itself the point.

Break D — reorder tools. In src/tools/index.ts, return the tools array reversed, then hit /v1/resolve twice.

Q2. For each break, record: does cache_hit go false? Is there an error? Which is the most dangerous in production, and why?

Restore everything.

The signature of a cache bug is silence. Every break above succeeds with HTTP 200 and a correct answer. The only symptom is money: the handbook line goes to 10× (a cache read is 90% off, so losing it multiplies that line by ten), and the request as a whole to roughly 5×, because output tokens are never cached and they dominate a small request. Run the numbers on the receipt above before you quote either figure — "10× the bill" is the version of this that gets repeated and it is wrong. cache_read_input_tokens is your only detector — alert on it.

Check1 of 2

Someone adds `Today is ${new Date().toISOString()}` to the top of the cached system block. What breaks?

Check2 of 2

You cache a 3,400-token prefix for a tenant that sends one ticket a day. What happens to cost?

Step 3 — the floor, and why it is not a number you can memorize

curl -s localhost:8787/v1/estimate -H 'content-type: application/json' \
  -d '{"message":"test","role":"triage"}' | jq .tokens

Note cache_minimum_tokens in that output. It is read from MODEL_CATALOG in src/config.ts for the configured model, not written as a literal, because the minimum is a per-model property:

model	shortest cacheable prefix
claude-opus-5	512
claude-sonnet-5	1024
claude-haiku-4-5	4096

Two things about that table. First, it is not monotonic — the cheap tier requires the longest prefix, eight times the flagship's. Any intuition of the form "smaller model, smaller everything" gets this exactly backwards. Second, this service's prefix is ~3,400 tokens, which sits between the two: it caches on Opus 5 and Sonnet 5 and does not cache on Haiku 4.5. You will meet the consequence of that in Lab 7, where it is hiding inside a cost table.

Q3. prefix_meets_cache_minimum is computed against the configured model. What happens if you set a breakpoint on a 400-token prefix — error, warning, or silence? Now the harder version: what happens to a prefix that has always been comfortably over the line when someone changes TRIAGE_MODEL? What does that imply about how you validate a caching change before shipping it?

Step 4 — where does the breakpoint go?

Render order is tools → system → messages. You have four content categories:

1. A 40K-token product catalog, identical for all users
2. A 3K-token per-tenant policy override, stable within a tenant
3. Conversation history, growing each turn
4. The current user message

Q4. Sketch the ordering and place up to 4 breakpoints. Which category must come last, and why does putting the growing history before the per-tenant block cost you money?

Step 5 — when caching loses

curl -s localhost:8787/v1/estimate -H 'content-type: application/json' -d '{
  "message":"Where is my order?",
  "monthly_volume":10000
}' | jq .monthly_projection_usd

Cache writes cost ~1.25× fresh tokens; reads cost ~0.1×. The default TTL is 5 minutes.

Q5. At what request rate does caching start losing money? Derive the break-even in terms of requests-per-TTL-window, then name a real traffic pattern in this support domain where you would deliberately not cache.

Step 6 — spend the savings

src/config.ts sets EFFORT.triage = "low". Change it to "high" and run:

npm run eval 2>&1 | tail -12

Q6. Compare accuracy and total cost against the low baseline. If accuracy is unchanged, what have you learned about this task — and what would you need to see before spending the extra tokens?

Step 7 — re-run the scoreboard

npm run eval:quick

This one matters. If you left Break C in place, the cached prefix is broken and you may see cost climb without accuracy moving at all — which is exactly the production failure the lab is about. git diff before you conclude anything.

---

Checkpoint

• What is the one field that proves caching is working?
• Name three silent invalidators.
• What is the minimum cacheable prefix for the model you configured, and what happens below it?
• When is caching a net loss?
• Scoreboard re-run; you can say why it did or did not move

---

Extension

Add a cache_hit_rate counter to the service and expose it at GET /metrics. Then write the alert rule you would page on. (Hint: the rule is not "hit rate < 100%" — cold starts are legitimate. What is the actual signal?)

Answers: ../solutions/lab-5.md