Skip to main content

Lab 9 — Shipping it

Time: 60 minutes · Prerequisites: Lab 0, Lab 3, Lab 5, Lab 7

Why this matters​

Northwind's queue is 4,100 tickets a week that nobody reads in real time. The script this repo has been using for it calls a synchronous endpoint in a loop, one ticket at a time, because that was the endpoint that already existed.

Everyone who reads that sentence reaches the same conclusion: use the Batches API, it's half price. This lab is going to have you measure that, and the measurement disagrees.

That is the shape of most production work on these systems. The received wisdom is usually directionally right and quantitatively wrong for your specific workload, and the difference between a team that knows this and one that doesn't is a single command they were willing to run.

The rest of the lab is the other things that separate a demo from a service: knowing your rate limits before they bite, surviving a model that changes underneath you, and publishing your tools to clients you did not write.

Objectives​

By the end you can:

  • Run the same workload three ways and explain which is cheapest, and why
  • Read the rate-limit headers that come back on every response
  • Say when to pin a model id and when to let it float
  • Publish a tool surface over MCP, and tell a tool from a resource

Step 1 — three ways to do the same work​

npm run triage:queue
npm run triage:queue -- --concurrency 8
npm run triage:queue:batch

The measured result on this repo:

modewall clockcostcache hits
serial91s$0.164520/20
concurrent (8)60s$0.175120/20
Batches API163–224s$0.201811/20

Batch was the slowest and the most expensive. On a workload that is the textbook case for it.

Q1. Before reading on: the batch rate is half the synchronous rate. Explain how the bill came out 23% higher.

Step 2 — the two discounts compete​

The batch discount is 50% off. A prompt-cache read is 90% off. They apply to the same tokens, and this workload sends a ~3,400-token policy handbook on every request.

The synchronous runs hit that cache 20 times out of 20 — each request follows close behind the last, and the prefix stays warm. The batch ran 11 out of 20. Batch requests are executed on the provider's schedule, in parallel, and a prefix that is not warm when a request lands is one that request pays full rate for.

Losing a 90% discount to gain a 50% one is a net loss. The arithmetic is not close.

Q2. For what workload does batch clearly win? Describe it in terms of prefix size and request spacing, not in terms of ticket volume.

Q3. Northwind's real backfill is 400,000 archived tickets, run once. Does your answer to Q1 apply to it? What would you measure before committing?

Check1 of 2

Your workload sends a large cached prefix on every request. You move it to the Batches API for the 50% discount. What is the risk?

Check2 of 2

You raise concurrency from 1 to 8 on a synchronous workload. What happens to cost per ticket?

Step 3 — see your rate limits​

npm run dev
curl -s localhost:8787/v1/triage -H 'content-type: application/json' \
  -d '{"message":"zipper broke on NW-48211"}' > /dev/null && \
  curl -s localhost:8787/v1/limits | jq

Every response carries a full accounting of your remaining headroom, and almost nobody reads it. The usual first encounter with a rate limit is a 429 during a spike, at which point you are answering a capacity question with no history of your own capacity.

Read src/anthropic.ts for where the headers get captured. It is not at the call sites.

Q4. The snapshot is recorded by wrapping the client's fetch rather than using .withResponse() at each call site. Give two reasons, one of which is about coverage and one of which is about a bug that .withResponse() would have caused here.

Step 4 — back off, don't retry​

Read AdaptiveGate in src/lib/limits.ts. On a 429 it halves in-flight concurrency, waits out retry-after, and re-runs.

Note what it does not do:

// src/lib/limits.ts
this.throttleEvents++;
this.width = Math.max(1, Math.floor(this.width / 2));

Q5. The SDK already retries 429s three times, honouring retry-after. Explain why adding a second retry layer here would be a mistake, and what this class does instead.

Step 5 — pinned versus floating​

claude-opus-5 is an alias. It improves without you doing anything, and it changes without telling you.

Read MODEL_PINS in src/config.ts, then look at the model-upgrade job in .github/workflows/ci.yml. It runs the tier matrix weekly and posts the table to the job summary.

Q6. Your eval drops two points on a Tuesday. Nothing was deployed. Walk through how you would establish whether the model changed, and what you would have needed to have in place beforehand.

Step 6 — publish the tools over MCP​

npm run mcp

The same three tools, over the Model Context Protocol, for clients you did not write. Connect Claude Desktop or Claude Code to it and ask a policy question.

Read src/mcp/server.ts and notice what is absent: no descriptions, no schemas, no business logic. It maps over TOOL_DEFS from src/tools/definitions.ts, which is the same array /v1/resolve wraps.

The handbook is published as a resource, not a tool.

Q7. search_policy is a tool and the handbook is a resource. State the rule you would use to decide, and give one example from your own work of something currently modelled as a tool that should be a resource.

Step 7 — name what you built​

You have written four workflow patterns without calling them that. Label them:

  • /v1/triage classifying a ticket so it reaches the right queue
  • pickModel and ?escalate=true from Lab 7
  • triage → resolve → draft
  • the judge in evals/lib/judge.ts critiquing the drafter

Three of them are routing, prompt chaining, and evaluator-optimizer. The fourth named pattern — orchestrator-workers, where a model decomposes a task and farms out subtasks — appears nowhere in this repo.

Q8. Make the case for adding an orchestrator to /v1/resolve. Then make the case against. Which would you ship, and what would have to be true about Northwind's tickets to change your answer?

Checkpoint​

You should be able to answer, without looking anything up:

  • Why did batch cost more than synchronous here?
  • What does concurrency buy, and what does it never buy?
  • When do you pin a model id?
  • What distinguishes an MCP tool from an MCP resource?

Extension​

Wire AdaptiveGate into evals/compare-models.ts in place of the plain pool, then force a 429 by running the matrix at --concurrency 40. Watch throttleEvents climb and the width halve. Then answer the harder question: the gate recovers by one after a clean batch, and halves on failure. What happens to throughput if the true limit sits just below your starting width — and what would you change?

Answers: ../solutions/lab-9.md