Batch planner

The Batches API bills at half rate, which makes it the obvious tool for a queue nobody reads in real time. Run against this repo’s twenty tickets, it was the slowest option and the most expensive — $0.2018 against $0.1645 synchronous.

The reason is that a prompt-cache read is 90% off and the batch discount is 50% off, and they apply to the same tokens. The synchronous runs hit the cache 20 times out of 20; the batch managed 11. Two discounts on the same tokens compete — they do not compose.

Cached prefix

3,400 tokens

This repo’s policy handbook is ~3,400. Below ~1,024 the API declines to cache at all.

Batch cache-hit rate

55%

Measured at 55% (11/20) on the twenty-ticket sample. Synchronous hit 100%.

Tickets per week

4,100

Northwind runs 4,100.

Synchronous

$23.00

per week · $0.0056 each

100% cache hits

Batches API

$29.54

per week · $0.0072 each

55% cache hits · half rate

Batch costs $6.53 more a week (28%) — the 50% batch discount does not cover the 90% cache discount it costs you.

Break-even is a batch cache-hit rate of 72%. Below that, synchronous wins. Measure yours with a pilot before committing — npm run triage:queue:batch reports it.

Run the comparison yourself, and read the rate-limit headers you have been ignoring, in Lab 9. The per-request cost model is in the cost explorer.