Model matrix
The same twelve hand-labelled cases, run against three model tiers. Every number comes from an actual run of this repo, emitted by npm run eval:models -- --emit-site rather than typed in.
Accuracy is the column people quote and the least useful one here. Read the calibration gap first — it decides whether you can build a confidence threshold on top of a model at all — and then the grid, which tells you which cases each tier loses. Those two readings disagree with the accuracy ranking more often than you would expect.
npm run eval:models -- --emit-site · judge opus-5 (pinned, prompt 47ddcecf)One run. This set moves by up to two cases run-to-run with nothing changed, so read the disagreement matrix and the calibration gap, not the accuracy column.
output_config.effort — ran with the parameter dropped, not at low effort| case | opus-5 | sonnet-5 | haiku-4-5 |
|---|---|---|---|
| eval-01 | ·0.90 | ·0.85 | ·0.95 |
| eval-02 | ·0.93 | ·0.90 | ·0.95 |
| eval-03 | ·0.72 | X0.65 | X0.92 |
| eval-04 | ·0.95 | ·0.95 | X0.95 |
| eval-05 | ·0.95 | ·0.90 | ·0.98 |
| eval-06 | ·0.72 | X0.00 | X0.00 |
| eval-07 | ·0.90 | ·0.90 | ·0.95 |
| eval-08 | ·0.95 | X0.90 | X0.98 |
| eval-09 | ·0.92 | ·0.85 | X0.92 |
| eval-10 | ·0.82 | X0.60 | X0.82 |
| eval-11 | X0.45 | ·0.60 | ·0.72 |
| eval-12 | ·0.66 | X0.60 | ·0.82 |
Run it yourself, and build the escalation route that acts on it, in Lab 7. The cost model behind the monthly projection is in the cost explorer.