diff --git a/CHANGELOG.md b/CHANGELOG.md
index 156e60f..1bef957 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,21 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+### Measured
+- **An NLI bake-off** (`scripts/eval_nli_bakeoff.py`, report in
+  `docs/eval/2026-10-01-nli-bakeoff.md`). It compares three NLI models, three
+  values of k and three aggregation rules, 27 combinations in all. Each combination
+  is calibrated on SciFact train and scored on SciFact dev and a frozen AVeriTeC
+  snapshot. The winner is picked by a rule written down before any number was
+  seen.
+  - **Winner:** `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` with
+    k=2. SciFact dev macro-F1 is 0.697, against 0.580 for today's default and 0.608
+    for the best configuration of the default model.
+  - **AVeriTeC** stays below its majority baseline: 0.475 on claims with a readable
+    source, against 0.708.
+  - The report lists its own limits. The main one is that dev both chose the winner
+    and reports its score.
+
 ## [0.4.9] - 2026-09-30
 
 Wikipedia can be read again, and a verdict now says more about how much it rests
diff --git a/docs/eval/2026-10-01-nli-bakeoff.md b/docs/eval/2026-10-01-nli-bakeoff.md
new file mode 100644
index 0000000..e7fcd77
--- /dev/null
+++ b/docs/eval/2026-10-01-nli-bakeoff.md
@@ -0,0 +1,158 @@
+# NLI bake-off — 2026-10-01
+
+- spec: `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`
+- calibrated on SciFact train (919 pairs); reported on SciFact dev (340 pairs) and AVeriTeC dev (100 claims, 89 answerable (3-way gold label), 80 of those with a readable source; frozen snapshot, browser not permitted)
+- embedder `BAAI/bge-small-en-v1.5`, top 5 passages per item, numeric layer on
+- machine: Darwin arm64
+- coverage: all candidate models were scored
+
+## Decision rule (fixed before any number was seen)
+
+The steps run in this order. Each one filters the rows the previous step left.
+
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Beats the baseline:** a row stays only if its SciFact dev macro-F1 beats
+   `base` × k=1 × `max` by **≥ 0.01**. If no row stays, the result is "no change". It is
+   written up as such, and the next package (B, coverage) starts.
+3. **Size gate:** a `large*` row stays only if it beats the highest macro-F1 of any
+   eligible `base` row by **≥ 0.03**. That is the strongest eligible `base` row, not the
+   one the tie rule would pick.
+4. **Winner:** the highest SciFact dev macro-F1 among the rows left.
+5. **Tie:** rows left within 0.01 of the winner go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`.
+
+## Result
+
+**Winner:** `large-fever` × k=2 × `max` is +0.117 macro-F1 over `base` × k=1 × `max`.
+
+Over the best eligible `base` row, `base` × k=2 × `margin` (0.608), the winner is +0.089 macro-F1.
+
+## Models
+
+| model | repo | size (MB) | ms/pair |
+|---|---|---|---|
+| base | `cross-encoder/nli-deberta-v3-base` | 244 | 19 |
+| large | `cross-encoder/nli-deberta-v3-large` | 643 | 55 |
+| large-fever | `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` | 643 | 54 |
+
+## Every combination
+
+AVeriTeC majority baseline: 0.708. The 2026-09-16 run reported 0.270 (all) and 0.361 (readable), but on a different snapshot with different cuts, over 89 answerable / 61 readable claims; this run counts 89 answerable / 80 readable. A gap between the two runs is partly a coverage change, not only a model gain.
+
+| model | k | aggregation | decide | medium | high | dev acc | dev macro-F1 | F1 S / R / NEI | rationale F1 | AVeriTeC all | AVeriTeC readable | w/o passage |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| base | 1 | max | 0.20 | 0.915356 | 1.000000 | 0.591 | 0.580 | 0.54 / 0.57 / 0.63 | 0.299 | 0.404 | 0.438 | 0 |
+| base | 1 | max_nei | 0.40 | 0.915356 | 1.000000 | 0.594 | 0.578 | 0.52 / 0.57 / 0.65 | 0.291 | 0.360 | 0.388 | 0 |
+| base | 1 | margin | 0.20 | 0.878350 | 1.000000 | 0.591 | 0.579 | 0.54 / 0.57 / 0.63 | 0.296 | 0.393 | 0.425 | 0 |
+| base | 2 | max | 0.20 | 0.999791 | 1.000000 | 0.609 | 0.606 | 0.61 / 0.59 / 0.62 | 0.324 | 0.494 | 0.537 | 0 |
+| base | 2 | max_nei | 0.40 | 0.999791 | 1.000000 | 0.594 | 0.589 | 0.57 / 0.58 / 0.62 | 0.316 | 0.472 | 0.512 | 0 |
+| base | 2 | margin | 0.15 | 0.992588 | 1.000000 | 0.609 | 0.608 | 0.60 / 0.61 / 0.62 | 0.326 | 0.483 | 0.525 | 0 |
+| base | 3 | max | 0.80 | 1.000000 | 1.000000 | 0.594 | 0.588 | 0.59 / 0.55 / 0.63 | 0.339 | 0.494 | 0.537 | 0 |
+| base | 3 | max_nei | 0.80 | 1.000000 | 1.000000 | 0.594 | 0.588 | 0.59 / 0.55 / 0.63 | 0.339 | 0.494 | 0.537 | 0 |
+| base | 3 | margin | 0.05 (grid floor) | 1.000000 | 1.000000 | 0.562 | 0.563 | 0.61 / 0.54 / 0.54 | 0.330 | 0.551 | 0.600 | 0 |
+| large | 1 | max | 0.70 | 0.924491 | 1.000000 | 0.582 | 0.572 | 0.49 / 0.61 / 0.62 | 0.292 | 0.393 | 0.425 | 0 |
+| large | 1 | max_nei | 0.70 | 0.924491 | 1.000000 | 0.582 | 0.572 | 0.49 / 0.61 / 0.62 | 0.292 | 0.393 | 0.425 | 0 |
+| large | 1 | margin | 0.70 | 0.920455 | 1.000000 | 0.582 | 0.573 | 0.49 / 0.61 / 0.62 | 0.288 | 0.393 | 0.425 | 0 |
+| large | 2 | max | 0.80 | 0.999586 | 1.000000 | 0.609 | 0.607 | 0.54 / 0.65 / 0.63 | 0.338 | 0.427 | 0.463 | 0 |
+| large | 2 | max_nei | 0.80 | 0.999586 | 1.000000 | 0.609 | 0.607 | 0.54 / 0.65 / 0.63 | 0.338 | 0.427 | 0.463 | 0 |
+| large | 2 | margin | 0.70 | 0.986530 | 1.000000 | 0.597 | 0.597 | 0.52 / 0.66 / 0.61 | 0.331 | 0.416 | 0.450 | 0 |
+| large | 3 | max | 0.90 | 0.999698 | 1.000000 | 0.597 | 0.595 | 0.57 / 0.60 / 0.61 | 0.343 | 0.494 | 0.537 | 0 |
+| large | 3 | max_nei | 0.90 | 0.999698 | 1.000000 | 0.597 | 0.595 | 0.57 / 0.60 / 0.61 | 0.343 | 0.494 | 0.537 | 0 |
+| large | 3 | margin | 0.90 | 0.999396 | 1.000000 | 0.588 | 0.585 | 0.52 / 0.63 / 0.61 | 0.334 | 0.449 | 0.487 | 0 |
+| large-fever | 1 | max | 0.05 (grid floor) | 0.074872 | 0.997081 | 0.647 | 0.642 | 0.61 / 0.64 / 0.67 | 0.339 | 0.472 | 0.512 | 0 |
+| large-fever | 1 | max_nei | 0.40 | 0.440642 | 0.997081 | 0.632 | 0.624 | 0.53 / 0.67 / 0.67 | 0.313 | 0.360 | 0.388 | 0 |
+| large-fever | 1 | margin | 0.05 (grid floor) | 0.064411 | 1.000000 | 0.638 | 0.633 | 0.59 / 0.64 / 0.66 | 0.338 | 0.472 | 0.512 | 0 |
+| large-fever | 2 | max | 0.25 | 0.252721 | 1.000000 | 0.697 | 0.697 | 0.66 / 0.72 / 0.71 | 0.383 | 0.438 | 0.475 | 0 |
+| large-fever | 2 | max_nei | 0.45 | 0.465651 | 1.000000 | 0.679 | 0.680 | 0.64 / 0.71 / 0.69 | 0.372 | 0.416 | 0.450 | 0 |
+| large-fever | 2 | margin | 0.20 | 0.232304 | 1.000000 | 0.688 | 0.690 | 0.65 / 0.73 / 0.69 | 0.373 | 0.438 | 0.475 | 0 |
+| large-fever | 3 | max | 0.45 | 0.552317 | 1.000000 | 0.674 | 0.672 | 0.65 / 0.68 / 0.69 | 0.356 | 0.472 | 0.512 | 0 |
+| large-fever | 3 | max_nei | 0.45 | 0.552317 | 1.000000 | 0.671 | 0.669 | 0.64 / 0.68 / 0.68 | 0.356 | 0.472 | 0.512 | 0 |
+| large-fever | 3 | margin | 0.20 | 0.438075 | 1.000000 | 0.674 | 0.675 | 0.65 / 0.70 / 0.67 | 0.361 | 0.483 | 0.525 | 0 |
+
+## AVeriTeC per label
+
+Cells are correct/n per gold label. `n (not scored)` marks a gold label with no 3-way verdict; it is not a model failure.
+
+| model | k | aggregation | Conflicting Evidence/Cherrypicking | Not Enough Evidence | Refuted | Supported |
+| --- | --- | --- | --- | --- | --- | --- |
+| base | 1 | max | 11 (not scored) | 5/7 | 30/63 | 1/19 |
+| base | 1 | max_nei | 11 (not scored) | 5/7 | 27/63 | 0/19 |
+| base | 1 | margin | 11 (not scored) | 5/7 | 29/63 | 1/19 |
+| base | 2 | max | 11 (not scored) | 2/7 | 41/63 | 1/19 |
+| base | 2 | max_nei | 11 (not scored) | 2/7 | 39/63 | 1/19 |
+| base | 2 | margin | 11 (not scored) | 2/7 | 40/63 | 1/19 |
+| base | 3 | max | 11 (not scored) | 3/7 | 40/63 | 1/19 |
+| base | 3 | max_nei | 11 (not scored) | 3/7 | 40/63 | 1/19 |
+| base | 3 | margin | 11 (not scored) | 2/7 | 46/63 | 1/19 |
+| large | 1 | max | 11 (not scored) | 5/7 | 27/63 | 3/19 |
+| large | 1 | max_nei | 11 (not scored) | 5/7 | 27/63 | 3/19 |
+| large | 1 | margin | 11 (not scored) | 5/7 | 27/63 | 3/19 |
+| large | 2 | max | 11 (not scored) | 3/7 | 31/63 | 4/19 |
+| large | 2 | max_nei | 11 (not scored) | 3/7 | 31/63 | 4/19 |
+| large | 2 | margin | 11 (not scored) | 3/7 | 31/63 | 3/19 |
+| large | 3 | max | 11 (not scored) | 2/7 | 38/63 | 4/19 |
+| large | 3 | max_nei | 11 (not scored) | 2/7 | 38/63 | 4/19 |
+| large | 3 | margin | 11 (not scored) | 2/7 | 35/63 | 3/19 |
+| large-fever | 1 | max | 11 (not scored) | 5/7 | 34/63 | 3/19 |
+| large-fever | 1 | max_nei | 11 (not scored) | 6/7 | 24/63 | 2/19 |
+| large-fever | 1 | margin | 11 (not scored) | 5/7 | 34/63 | 3/19 |
+| large-fever | 2 | max | 11 (not scored) | 4/7 | 33/63 | 2/19 |
+| large-fever | 2 | max_nei | 11 (not scored) | 4/7 | 31/63 | 2/19 |
+| large-fever | 2 | margin | 11 (not scored) | 4/7 | 33/63 | 2/19 |
+| large-fever | 3 | max | 11 (not scored) | 4/7 | 35/63 | 3/19 |
+| large-fever | 3 | max_nei | 11 (not scored) | 4/7 | 35/63 | 3/19 |
+| large-fever | 3 | margin | 11 (not scored) | 4/7 | 36/63 | 3/19 |
+
+## Tiers on SciFact dev
+
+Cells are n · precision (95 % Wilson interval); n below 20 is too few to judge.
+
+| model | k | aggregation | high | medium | low |
+|---|---|---|---|---|---|
+| base | 1 | max | 1 · too few to judge | 109 · 0.725 (Wilson 0.63–0.80) | 33 · 0.515 (Wilson 0.35–0.67) |
+| base | 1 | max_nei | 1 · too few to judge | 109 · 0.725 (Wilson 0.63–0.80) | 18 · too few to judge |
+| base | 1 | margin | 1 · too few to judge | 112 · 0.732 (Wilson 0.64–0.81) | 27 · 0.481 (Wilson 0.31–0.66) |
+| base | 2 | max | 1 · too few to judge | 13 · too few to judge | 175 · 0.611 (Wilson 0.54–0.68) |
+| base | 2 | max_nei | 1 · too few to judge | 13 · too few to judge | 158 · 0.614 (Wilson 0.54–0.69) |
+| base | 2 | margin | 1 · too few to judge | 75 · 0.800 (Wilson 0.70–0.87) | 110 · 0.527 (Wilson 0.43–0.62) |
+| base | 3 | max | 1 · too few to judge | 0 · too few to judge | 181 · 0.608 (Wilson 0.54–0.68) |
+| base | 3 | max_nei | 1 · too few to judge | 0 · too few to judge | 181 · 0.608 (Wilson 0.54–0.68) |
+| base | 3 | margin | 1 · too few to judge | 0 · too few to judge | 220 · 0.559 (Wilson 0.49–0.62) |
+| large | 1 | max | 1 · too few to judge | 112 · 0.759 (Wilson 0.67–0.83) | 8 · too few to judge |
+| large | 1 | max_nei | 1 · too few to judge | 112 · 0.759 (Wilson 0.67–0.83) | 8 · too few to judge |
+| large | 1 | margin | 1 · too few to judge | 112 · 0.750 (Wilson 0.66–0.82) | 7 · too few to judge |
+| large | 2 | max | 1 · too few to judge | 31 · 0.935 (Wilson 0.79–0.98) | 117 · 0.650 (Wilson 0.56–0.73) |
+| large | 2 | max_nei | 1 · too few to judge | 31 · 0.935 (Wilson 0.79–0.98) | 117 · 0.650 (Wilson 0.56–0.73) |
+| large | 2 | margin | 1 · too few to judge | 93 · 0.828 (Wilson 0.74–0.89) | 48 · 0.500 (Wilson 0.36–0.64) |
+| large | 3 | max | 1 · too few to judge | 34 · 0.882 (Wilson 0.73–0.95) | 129 · 0.605 (Wilson 0.52–0.68) |
+| large | 3 | max_nei | 1 · too few to judge | 34 · 0.882 (Wilson 0.73–0.95) | 129 · 0.605 (Wilson 0.52–0.68) |
+| large | 3 | margin | 1 · too few to judge | 28 · 0.929 (Wilson 0.77–0.98) | 114 · 0.640 (Wilson 0.55–0.72) |
+| large-fever | 1 | max | 27 · 0.963 (Wilson 0.82–0.99) | 121 · 0.678 (Wilson 0.59–0.75) | 5 · too few to judge |
+| large-fever | 1 | max_nei | 27 · 0.963 (Wilson 0.82–0.99) | 93 · 0.763 (Wilson 0.67–0.84) | 0 · too few to judge |
+| large-fever | 1 | margin | 1 · too few to judge | 147 · 0.735 (Wilson 0.66–0.80) | 1 · too few to judge |
+| large-fever | 2 | max | 1 · too few to judge | 165 · 0.776 (Wilson 0.71–0.83) | 0 · too few to judge |
+| large-fever | 2 | max_nei | 1 · too few to judge | 155 · 0.781 (Wilson 0.71–0.84) | 0 · too few to judge |
+| large-fever | 2 | margin | 1 · too few to judge | 161 · 0.783 (Wilson 0.71–0.84) | 3 · too few to judge |
+| large-fever | 3 | max | 1 · too few to judge | 167 · 0.743 (Wilson 0.67–0.80) | 6 · too few to judge |
+| large-fever | 3 | max_nei | 1 · too few to judge | 167 · 0.743 (Wilson 0.67–0.80) | 5 · too few to judge |
+| large-fever | 3 | margin | 1 · too few to judge | 161 · 0.758 (Wilson 0.69–0.82) | 21 · 0.429 (Wilson 0.24–0.63) |
+
+## Numeric layer
+
+The numeric layer (spec §10) runs on the same top-k passages before any model. A mismatch decides REFUTED by rule at score 1.0, and no model is credited with it. It depends on k only, so one row per k covers every model and aggregation. A firing is correct when the gold label is REFUTED.
+
+| k | SciFact train fired | train correct | SciFact dev fired | dev correct |
+|---|---|---|---|---|
+| 1 | 4 | 0 | 1 | 1 |
+| 2 | 5 | 0 | 1 | 1 |
+| 3 | 6 | 0 | 1 | 1 |
+
+Rule-decided rows score exactly 1.0, the top of the scale, so they are among the first rows the `high` cut's walk down SciFact train meets: a wrong firing lowers the precision every lower cut is measured with, and a `high` cut of 1.000000 means no score below 1.0 held 0.85 precision, so only a score of 1.0 reaches the tier. The `high` cut is 1.000000 in 25 of 27 combinations. In 25 of 27, every SciFact dev `high` verdict is a rule firing (the high-tier n equals the dev firings at that k): there the high tier on dev comes from the rule, not the model.
+
+## Limits
+
+- **Dev chooses and reports.** SciFact dev both picks the winner and reports it. The winner's 0.697 macro-F1 comes from the split that chose it, out of 27 combinations whose highest dev macro-F1 is 0.697, so it is optimistic; a held-out split would likely read lower.
+- **int8 scores depend on the batch.** The candidate files are dynamically quantised to int8, so a pair's probabilities depend on what else is in its batch. Re-scoring 48 SciFact dev top-1 pairs one at a time moved probabilities by up to 0.197 (mean 0.008) and flipped 1 argmax (measured 2026-10-01 by the final review). That likely explains why `base` × k=1 × `max` reaches dev accuracy 0.591 here (0.597 even at decide 0.45) against the 0.609 published on 2026-09-12 with dev-fitted cuts.
+- **The AVeriTeC snapshot is new.** It is not the 2026-09-16 run's: coverage improved since (then 61 of 89 answerable claims were readable, now 80 of 89), so AVeriTeC numbers here are not comparable with that run's.
+- **Grid floor.** 3 of 27 combinations put `decide` on the grid floor (0.05), marked `(grid floor)` above: SciFact train accuracy was best at the lowest cut tried, so the best cut may lie below the grid.
diff --git a/docs/superpowers/OPEN-ITEMS.md b/docs/superpowers/OPEN-ITEMS.md
index 727b2a1..91955b5 100644
--- a/docs/superpowers/OPEN-ITEMS.md
+++ b/docs/superpowers/OPEN-ITEMS.md
@@ -578,3 +578,34 @@ third needed a design decision and is now decided and implemented (19.1).
 | 19.1 | ~~Underspecified claims pass too easily~~ **Decided and implemented 2026-09-30** | "openai won" was `SUPPORTED (found by proofpath)`, medium, on a single vanity-award page ("OpenAI's recognition with a 2026 Global Recognition Award"). The passage does entail the claim, so rules 1–3 hold; but a claim with no object ("won *what*?") is satisfied by any win. **Decided (the user): rule-based, note only.** `claims.is_underspecified` counts content words in the text the models check (`claim.hypothesis or claim.text`, so a translated claim is counted in English): hashtags, handles, punctuation and function words (the sentence openers plus auxiliaries, negation, prepositions, pronouns) are dropped; a number is one term ("12.5") and so is a run of capitalised words, a name ("Elon Musk"). **≤ 2 content terms is underspecified** (`UNDERSPECIFIED_MAX_WORDS`), **unless a linking verb stands between two content terms** — a complete predication ("The Earth is flat", "Revenue was 12.5%"). "openai won", "Elon Musk won", "OpenAI failed" and "It is true" are underspecified; "OpenAI won the award" is not. Every verdict finding on such a claim (all kinds, cited or found) carries `report.UNDERSPECIFIED` in its detail, after `ABSTRACT_BASIS` and before `FOUND_BY_PROOFPATH`; the SARIF message appends it the same way. No judge, no new dependency, and kind, level, tier and exit code are unchanged. A supported *cited* claim still produces no finding, so it gets no note. Known limits: a passive with a linking verb ("OpenAI was acquired") counts as complete, and a cited Turkish sentence is counted untranslated, so it reads slightly shorter. In title case or all caps (every content word capitalised) no names are joined, so a verb is never swallowed into one. Known small misses, documented and left: a sentence-initial adverb joins the name after it ("Yesterday Elon Musk won" is flagged), a dotted abbreviation splits ("U.S. won" is not flagged), a "BREAKING:" prefix counts as a content word, and a two-word name in title case ("Elon Musk Won") is three terms, so it is not flagged. The note is worded for every verdict kind (`underspecified claim: it names no object, time or scope, so many different events match it; check that the passage is about the one meant`). Source credibility (vanity awards, content farms) stays out of scope (§3). Evidence-search spec §2 records the rule |
 | 19.2 | A bot wall inside a site's full template can read as `ok` | challenge *phrases* now count only on pages under `fetch.CHALLENGE_MAX_WORDS` (300) extracted words, so a long real article that says "CAPTCHA" is not blocked. A wall served inside the site's own layout, with menus not in `<nav>`, can exceed 300 words and be read as content (review, 2026-09-30: a 364-word probe). A title/h1 signal was considered and rejected: it would block real pages titled "CAPTCHA". `cf-chl` stays unconditional. Cloudflare's underscore traces (`_cf_chl_opt`, `__cf_chl_f_tk`) are not matched by `cf-chl` — a candidate structural marker |
 | 19.3 | The markdown `## Checked` table does not mark abstract-grade support | a cited claim SUPPORTED by an abstract produces no finding (by design), so its row in `report.py`'s Checked table is the only place it appears, and it carries no abstract mark. The source's own `LOW CONFIDENCE (abstract only)` finding and the coverage line still state it; a per-row mark would make the table honest on its own (review, 2026-09-30) |
+
+## 20. What to improve next — 2026-10-01
+
+Two read-only sweeps (OPEN-ITEMS plus `docs/eval`, and the pipeline code) were turned into
+five packages. The user's decisions:
+
+- **A starts first.** Its first plan was "k=1 → k=3" plus retrieval tweaks. That plan was
+  dropped once the 2026-09-12 sweep showed k=3 does not raise accuracy (0.603 against
+  0.609). A became a measured NLI bake-off with a decision rule fixed in advance
+  (`docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`). The candidates are the
+  current base model and the two large int8 exports, cross-encoder and MoritzLaurer
+  FEVER/ANLI, downloaded for the eval only.
+- **Calibration** is fitted on SciFact **train** and reported on dev.
+- **Wayback** is tried after 429/5xx only. A URL that robots.txt disallows is never read
+  from the archive.
+
+### Backlog (packages B–E)
+
+| # | Item | Note |
+|---|---|---|
+| 20.1 | Unpaywall never runs by default | `oa.py` skips Unpaywall without a contact email, and the default is `""` (`config.py`). Add a `/config` "Contact email" row, in the same pattern as the Tavily row |
+| 20.2 | Optional Semantic Scholar API key | lifts the 1.1 s/request throttle; same as 7.9 and 8.21 |
+| 20.3 | Web pages use the paper full-text bar | `FULLTEXT_MIN_WORDS=1500` labels a normal news article "abstract only". Web needs its own lower bar (see 8.16) |
+| 20.4 | Wayback after 429/5xx | `fetch.py` skips the archive on rate limits and server errors. Pick the snapshot nearest the citation year, not the newest |
+| 20.5 | Crossref `/works/{doi}` is fetched three times per source | in resolve, the retraction check and the OA chain; add an in-memory memo |
+| 20.6 | Concurrent cold resolve/OA | thread pool over the thread-safe `polite.py` throttle. Needs per-thread cache connections first (10.11). Same as 11.10 |
+| 20.7 | Skip model loads on a fully cached re-run | same as 10.2 |
+| 20.8 | Bibliography parsing that was measured but never shipped | two-column rejoin (9.10/12.1), multi-block headless fallback (11.11), more headings (16.5), `[1]-[3]` ranges (9.4) |
+| 20.9 | The judge's quote is never checked | `judge._collect` does not verify that the quoted span occurs in the passage. Checking it is a cheap way to enforce product rule 1 |
+| 20.10 | Eval honesty | add a `--judge` flag to `scripts/eval_averitec.py`; do one run with the browser allowed; show the 0.404 search score in the README "Measured" table |
+| 20.11 | NLI bake-off result | **Winner: `large-fever` (MoritzLaurer DeBERTa-v3-large mnli-fever-anli-ling-wanli) × k=2 × `max`.** SciFact dev macro-F1 0.697, against 0.580 for the `base` × k=1 × `max` baseline and 0.608 for the best `base` row. Accuracy 0.697. AVeriTeC 3-way 0.438, or 0.475 on the readable subset, against a 0.708 majority baseline. Cost: 643 MB and 54 ms/pair, against 244 MB and 19 ms/pair. Full report: `docs/eval/2026-10-01-nli-bakeoff.md`. Next: a follow-up spec for an opt-in "accurate" NLI model (consent download, spec §7.1). It must settle three open points. (a) Exclude rule-decided rows from the tier walk: the numeric layer fires 4–6 times on train with 0 correct, so it pins `high` at 1.000000. (b) The winner's `medium` cut sits on `decide`, so the report shows only two tiers. (c) On AVeriTeC the winner is below `base` × k=2 (0.475 against 0.537), and `Supported` is still 2/19 |
diff --git a/docs/superpowers/plans/2026-10-01-nli-bakeoff.md b/docs/superpowers/plans/2026-10-01-nli-bakeoff.md
new file mode 100644
index 0000000..0874a57
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-01-nli-bakeoff.md
@@ -0,0 +1,2006 @@
+# NLI Bake-off Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Measure three NLI models × k ∈ {1,2,3} × three aggregation rules on SciFact (calibrated on train, reported on dev) and a frozen AVeriTeC snapshot, and pick a winner by a rule fixed in advance.
+
+**Architecture:** The pure logic lives in two new modules under `src/proofpath/eval/`:
+- `bakeoff_data.py`: stored inputs and their I/O;
+- `bakeoff.py`: aggregation, calibration, evaluation and the decision rule.
+
+Both are covered by `mypy --strict` and unit tests. The dev-only script `scripts/eval_nli_bakeoff.py` does the network and model work (`snapshot`, `score`) and writes the markdown (`report`). Nothing in the product pipeline changes.
+
+**Tech Stack:** Python 3.10+, numpy, onnxruntime through `proofpath.entailment.OnnxNli`, fastembed through `proofpath.retrieval.FastEmbedder`, pytest.
+
+**Spec:** `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`
+
+## Global Constraints
+
+- Python 3.10+. Every function signature carries type annotations. `ruff check`, `ruff format --check` and `mypy` (strict, `src/`) are clean.
+- `pathlib` everywhere. Every text read and write passes `encoding="utf-8"`.
+- Unit tests make no network calls.
+- Code, identifiers and comments are in English.
+- Do not change product code (`pipeline.py`, `verify.py`, `entailment.py`, defaults). The bake-off only reads it.
+- Model revisions are pinned to full 40-character SHAs:
+  - base `6c749ce3425cd33b46d187e45b92bbf96ee12ec7`
+  - large `bab4bc7178836f731dcfd18c06ca9def0a137712`
+  - large-fever `b3546ea6b0346eb6f8d5d68b13c7dc6d0376b3d7`
+- The AVeriTeC snapshot is fetched with the browser **not permitted** and `interactive=False`.
+- Third-party page text (the snapshot) stays under `<cache>/datasets/` and never goes into the repo.
+- The decision rule is the spec's, verbatim. Numbers: TIE 0.01, LARGE 0.03, NO-CHANGE 0.01, MIN_TIER_N 20, precision targets 0.85 / 0.70.
+- Subagents never commit. The controller stages and commits on `feat/nli-bakeoff` after a clean review.
+- The code blocks in this plan are not pre-formatted. Run `uv run ruff format <new or changed files>` before every `--check`.
+- When a step says "append" to a test file, put its `import` lines in the file's top import block. A mid-file import fails ruff's E402.
+
+**Drift from the spec (binding):** the spec put the aggregation variants "in the script". They live in `src/proofpath/eval/bakeoff.py` instead, beside `metrics.py` and `scifact.py`. That way they get `mypy --strict` and clean imports in tests. That package is eval tooling, not the product pipeline, so "no product change" still holds.
+
+## File Structure
+
+| File | Responsibility |
+|---|---|
+| Create `src/proofpath/eval/bakeoff_data.py` | `SnapshotSource`, `SnapshotClaim`, `Item`; snapshot/items JSON and probability npz I/O; `pending`, `source_from_fetched`, `items_fingerprint` |
+| Create `src/proofpath/eval/bakeoff.py` | `Signal`, three aggregators, `signal_for`, `decide`, `wilson`, `calibrate`, `calibration_rows`, `evaluate_scifact`, `evaluate_averitec`, `Combo`/`Outcome`/`Choice`, `choose`, `DECISION_RULE` |
+| Create `scripts/eval_nli_bakeoff.py` | `snapshot`, `score` and `report` subcommands; `CANDIDATES`; `render_report` |
+| Create `tests/test_bakeoff_data.py` | I/O round trips, resume, stale-probability guards |
+| Create `tests/test_bakeoff.py` | aggregation, Wilson, calibration split guard, evaluation, decision rule |
+| Create `tests/test_eval_nli_bakeoff.py` | the script imports without side effects; candidate pins; `render_report` |
+| Create (Task 7) `docs/eval/<date>-nli-bakeoff.md` | the result |
+| Modify (Task 7) `docs/superpowers/OPEN-ITEMS.md` | §20: the outcome line |
+
+---
+
+### Task 1: Stored inputs — `bakeoff_data.py`
+
+**Files:**
+- Create: `src/proofpath/eval/bakeoff_data.py`
+- Test: `tests/test_bakeoff_data.py`
+
+**Interfaces:**
+- Consumes:
+  - `proofpath.models.Passage(text: str, source_id: str, index: int)`
+  - `proofpath.fetch.Fetched` (`.ok`, `.text`, `.outcome.value`)
+  - `proofpath.fetch.Outcome`
+  - `proofpath.providers.NO_TEXT: str`
+  - `proofpath.eval.averitec.Claim(id, text, label, source_urls, non_urls)`
+- Produces:
+  - `DATASETS: tuple[str, ...] = ("scifact-train", "scifact-dev", "averitec")`
+  - `SnapshotSource(url: str, state: str, text: str)`
+  - `SnapshotClaim(claim_id: int, claim: str, gold: str, sources: tuple[SnapshotSource, ...], non_urls: int)`
+  - `Item(key: str, dataset: str, group: str, claim: str, gold: str, passages: tuple[Passage, ...], rationale: frozenset[int] = frozenset())`
+  - `source_from_fetched(url: str, fetched: Fetched) -> SnapshotSource`
+  - `pending(claims: Sequence[averitec.Claim], done: Sequence[SnapshotClaim]) -> list[averitec.Claim]`
+  - `save_snapshot(path: Path, claims: Sequence[SnapshotClaim]) -> None` / `load_snapshot(path: Path) -> list[SnapshotClaim]`
+  - `save_items(path: Path, items: Sequence[Item]) -> None` / `load_items(path: Path) -> list[Item]`
+  - `items_fingerprint(items: Sequence[Item]) -> str`
+  - `save_probs(path: Path, items: Sequence[Item], probs: Sequence[np.ndarray], *, ms_per_pair: float) -> None`
+  - `load_probs(path: Path, items: Sequence[Item]) -> tuple[list[np.ndarray], float]`
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+"""Offline tests for the bake-off's stored inputs (no network, no model)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from proofpath.eval import averitec
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    SnapshotSource,
+    items_fingerprint,
+    load_items,
+    load_probs,
+    load_snapshot,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+)
+from proofpath.fetch import Fetched, Outcome
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+
+def _fetched(outcome: Outcome, text: str) -> Fetched:
+    return Fetched(
+        url="https://example.org/a",
+        final_url="https://example.org/a",
+        step=1,
+        outcome=outcome,
+        status=200 if outcome is Outcome.OK else 403,
+        content_type="text/html",
+        kind="html",
+        body=b"",
+        text=text,
+        notes=[],
+    )
+
+
+def _item(key: str, n: int = 2, dataset: str = "scifact-dev") -> Item:
+    passages = tuple(Passage(f"sentence {i} of {key}", "doc", i) for i in range(n))
+    return Item(key, dataset, key, "a claim", "SUPPORTED", passages, frozenset({0}))
+
+
+def test_a_read_page_keeps_its_text_and_ok_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "Some text."))
+    assert source == SnapshotSource("https://example.org/a", "ok", "Some text.")
+
+
+def test_a_reached_page_with_no_text_gets_the_no_text_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "   "))
+    assert source.state == NO_TEXT
+    assert source.text == ""
+
+
+def test_a_blocked_page_keeps_its_state_and_no_text() -> None:
+    blocked = _fetched(Outcome.BLOCKED_NO_BROWSER, "challenge page")
+    source = source_from_fetched("https://example.org/a", blocked)
+    assert source.state == Outcome.BLOCKED_NO_BROWSER.value
+    assert source.text == ""
+
+
+def test_snapshot_round_trips(tmp_path: Path) -> None:
+    claims = [
+        SnapshotClaim(
+            7, "c", "Refuted", (SnapshotSource("u", "ok", "t"), SnapshotSource("v", "x", "")), 1
+        )
+    ]
+    path = tmp_path / "snap.json"
+    save_snapshot(path, claims)
+    assert load_snapshot(path) == claims
+
+
+def test_a_missing_snapshot_loads_as_empty(tmp_path: Path) -> None:
+    assert load_snapshot(tmp_path / "absent.json") == []
+
+
+def test_pending_skips_claims_already_in_the_snapshot() -> None:
+    claims = [averitec.Claim(i, f"claim {i}", "Refuted", ("https://e.org",)) for i in range(3)]
+    done = [SnapshotClaim(1, "claim 1", "Refuted", (), 0)]
+    assert [c.id for c in pending(claims, done)] == [0, 2]
+
+
+def test_items_round_trip(tmp_path: Path) -> None:
+    items = [_item("a"), _item("b", 1, "averitec")]
+    path = tmp_path / "items.json"
+    save_items(path, items)
+    assert load_items(path) == items
+
+
+def test_an_unknown_dataset_is_refused() -> None:
+    with pytest.raises(ValueError, match="unknown dataset"):
+        _item("a", dataset="scifact-test")
+
+
+def test_probs_round_trip(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 3)]
+    probs = [np.full((2, 3), 0.25, np.float32), np.full((3, 3), 0.5, np.float32)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, probs, ms_per_pair=12.5)
+    loaded, ms = load_probs(path, items)
+    assert ms == pytest.approx(12.5)
+    assert [p.shape for p in loaded] == [(2, 3), (3, 3)]
+    assert np.allclose(loaded[1], 0.5)
+
+
+def test_probs_scored_on_other_items_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, [np.zeros((2, 3), np.float32)], ms_per_pair=1.0)
+    changed = [_item("z", 2)]
+    with pytest.raises(ValueError, match="other items"):
+        load_probs(path, changed)
+
+
+def test_probs_with_the_wrong_shape_are_refused(tmp_path: Path) -> None:
+    with pytest.raises(ValueError, match="shape"):
+        save_probs(tmp_path / "x.npz", [_item("a", 2)], [np.zeros((3, 3))], ms_per_pair=1.0)
+
+
+def test_the_fingerprint_changes_with_a_passage_text() -> None:
+    a = _item("a")
+    b = Item(a.key, a.dataset, a.group, a.claim, a.gold, (Passage("other", "doc", 0),))
+    assert items_fingerprint([a]) != items_fingerprint([b])
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff_data.py -q`
+Expected: FAIL with `ModuleNotFoundError: No module named 'proofpath.eval.bakeoff_data'`
+
+- [ ] **Step 3: Write the implementation**
+
+```python
+"""Stored inputs of the NLI bake-off (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Three files under ``<cache>/datasets``, written once and read by every later step:
+the frozen AVeriTeC snapshot (page text per source URL), the retrieved items (top
+passages per claim, shared by every model), and one npz of raw NLI probabilities per
+model. Nothing here touches the network or a model.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from proofpath.eval import averitec
+from proofpath.fetch import Fetched
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+DATASETS: tuple[str, ...] = ("scifact-train", "scifact-dev", "averitec")
+
+
+@dataclass(frozen=True)
+class SnapshotSource:
+    """One source URL as the snapshot run found it. ``text`` is "" unless it was read."""
+
+    url: str
+    state: str
+    text: str
+
+
+@dataclass(frozen=True)
+class SnapshotClaim:
+    claim_id: int
+    claim: str
+    gold: str  # the AVeriTeC label, verbatim
+    sources: tuple[SnapshotSource, ...]
+    non_urls: int
+
+
+@dataclass(frozen=True)
+class Item:
+    """One (claim, source) unit with its top retrieved passages, best first.
+
+    ``group`` is the claim an item belongs to: a SciFact pair is its own group, an
+    AVeriTeC claim groups every readable source it has.
+    """
+
+    key: str
+    dataset: str
+    group: str
+    claim: str
+    gold: str
+    passages: tuple[Passage, ...]
+    rationale: frozenset[int] = frozenset()
+
+    def __post_init__(self) -> None:
+        if self.dataset not in DATASETS:
+            raise ValueError(f"unknown dataset {self.dataset!r}")
+
+
+def source_from_fetched(url: str, fetched: Fetched) -> SnapshotSource:
+    """The state and text one fetch earned, worded as ``eval_averitec.state_for`` does.
+
+    A page that was reached but yielded no text is not ``ok`` (product rule 6), and
+    only a read page keeps its text: a challenge page's words are not the source.
+    """
+    if fetched.ok and fetched.text.strip():
+        return SnapshotSource(url, fetched.outcome.value, fetched.text)
+    if fetched.ok:
+        return SnapshotSource(url, NO_TEXT, "")
+    return SnapshotSource(url, fetched.outcome.value, "")
+
+
+def pending(
+    claims: Sequence[averitec.Claim], done: Sequence[SnapshotClaim]
+) -> list[averitec.Claim]:
+    """The claims a resumed snapshot run still has to fetch, in dataset order."""
+    finished = {claim.claim_id for claim in done}
+    return [claim for claim in claims if claim.id not in finished]
+
+
+def _write_atomic(path: Path, payload: str) -> None:
+    """Write a whole file at once, so a Ctrl-C never leaves half of one."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    temporary.write_text(payload, encoding="utf-8")
+    temporary.replace(path)
+
+
+def save_snapshot(path: Path, claims: Sequence[SnapshotClaim]) -> None:
+    payload = [
+        {
+            "claim_id": claim.claim_id,
+            "claim": claim.claim,
+            "gold": claim.gold,
+            "non_urls": claim.non_urls,
+            "sources": [{"url": s.url, "state": s.state, "text": s.text} for s in claim.sources],
+        }
+        for claim in claims
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_snapshot(path: Path) -> list[SnapshotClaim]:
+    if not path.exists():
+        return []
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        SnapshotClaim(
+            claim_id=int(row["claim_id"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            sources=tuple(
+                SnapshotSource(str(s["url"]), str(s["state"]), str(s["text"]))
+                for s in row["sources"]
+            ),
+            non_urls=int(row["non_urls"]),
+        )
+        for row in raw
+    ]
+
+
+def save_items(path: Path, items: Sequence[Item]) -> None:
+    payload = [
+        {
+            "key": item.key,
+            "dataset": item.dataset,
+            "group": item.group,
+            "claim": item.claim,
+            "gold": item.gold,
+            "passages": [[p.text, p.source_id, p.index] for p in item.passages],
+            "rationale": sorted(item.rationale),
+        }
+        for item in items
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_items(path: Path) -> list[Item]:
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        Item(
+            key=str(row["key"]),
+            dataset=str(row["dataset"]),
+            group=str(row["group"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            passages=tuple(Passage(str(t), str(s), int(i)) for t, s, i in row["passages"]),
+            rationale=frozenset(int(i) for i in row["rationale"]),
+        )
+        for row in raw
+    ]
+
+
+def items_fingerprint(items: Sequence[Item]) -> str:
+    """A digest of what was scored: keys, claims and passage texts, in order.
+
+    Stored beside the probabilities so a rebuilt item file can never be paired with
+    probabilities computed on different passages.
+    """
+    digest = hashlib.sha256()
+    for item in items:
+        digest.update(item.key.encode("utf-8") + b"\0" + item.claim.encode("utf-8") + b"\0")
+        for passage in item.passages:
+            digest.update(passage.text.encode("utf-8") + b"\0")
+    return digest.hexdigest()
+
+
+def save_probs(
+    path: Path, items: Sequence[Item], probs: Sequence[np.ndarray], *, ms_per_pair: float
+) -> None:
+    """Store one model's ``(n_passages, 3)`` probabilities per item, flattened."""
+    if len(items) != len(probs):
+        raise ValueError("one probability block per item is required")
+    for item, block in zip(items, probs, strict=True):
+        if np.asarray(block).shape != (len(item.passages), 3):
+            raise ValueError(f"probability block for {item.key} has the wrong shape")
+    offsets = np.cumsum([0, *(len(item.passages) for item in items)], dtype=np.int64)
+    flat = (
+        np.concatenate([np.asarray(b, dtype=np.float32) for b in probs])
+        if probs
+        else np.zeros((0, 3), dtype=np.float32)
+    )
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    with temporary.open("wb") as handle:
+        np.savez(
+            handle,
+            fingerprint=np.array(items_fingerprint(items)),
+            offsets=offsets,
+            probs=flat,
+            ms_per_pair=np.array(ms_per_pair, dtype=np.float64),
+        )
+    temporary.replace(path)
+
+
+def load_probs(path: Path, items: Sequence[Item]) -> tuple[list[np.ndarray], float]:
+    """The stored probabilities, one block per item, and the measured ms per pair."""
+    with np.load(path, allow_pickle=False) as data:
+        fingerprint = str(data["fingerprint"])
+        offsets = data["offsets"]
+        flat = data["probs"]
+        ms_per_pair = float(data["ms_per_pair"])
+    if fingerprint != items_fingerprint(items):
+        raise ValueError(f"{path.name} was scored on other items; run `score` again")
+    return [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))], ms_per_pair
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff_data.py -q`
+Expected: 12 passed
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src/proofpath/eval/bakeoff_data.py tests/test_bakeoff_data.py && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 2: Signals and aggregation — `bakeoff.py` part 1
+
+**Files:**
+- Create: `src/proofpath/eval/bakeoff.py`
+- Test: `tests/test_bakeoff.py`
+
+**Interfaces:**
+- Consumes:
+  - `proofpath.numerics.check(claim: str, passages: Sequence[Passage]) -> NumericResult | None` (`.mismatch`, `.passage`)
+  - `proofpath.entailment.LABEL_ORDER = (SUPPORTED, REFUTED, NEI)`
+  - `proofpath.pipeline.Thresholds(decide, high, medium)` with `.tier(score)`
+- Produces:
+  - `AGGREGATIONS = ("max", "max_nei", "margin")`
+  - `KS = (1, 2, 3)`
+  - `TOP_N = 5`
+  - `Signal(label: Label, score: float, passage: Passage | None)`
+  - `NOTHING: Signal`
+  - `aggregate_max` / `aggregate_max_nei` / `aggregate_margin(passages: Sequence[Passage], probs: np.ndarray) -> Signal`
+  - `AGGREGATORS: Mapping[str, Aggregator]`
+  - `signal_for(claim: str, passages: Sequence[Passage], probs: np.ndarray, *, k: int, aggregation: str) -> Signal`
+  - `decide(signal: Signal, thresholds: Thresholds) -> Label`
+  - `wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]`
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+"""Offline tests for the bake-off's aggregation, calibration and decision rule."""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+from proofpath.eval import bakeoff
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+
+# Rows are (SUPPORTED, REFUTED, NEI), the order of entailment.LABEL_ORDER.
+P = [Passage(f"passage {i}", "doc", i) for i in range(3)]
+
+
+def _probs(*rows: tuple[float, float, float]) -> np.ndarray:
+    return np.array(rows, dtype=np.float32)
+
+
+def test_max_takes_the_loudest_assertion_over_the_hits() -> None:
+    probs = _probs((0.6, 0.1, 0.3), (0.1, 0.8, 0.1))
+    signal = bakeoff.aggregate_max(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.REFUTED, P[1])
+    assert signal.score == pytest.approx(0.8)
+
+
+def test_max_nei_ignores_an_assertion_its_own_hit_calls_neutral() -> None:
+    probs = _probs((0.45, 0.05, 0.50), (0.40, 0.05, 0.30))
+    signal = bakeoff.aggregate_max_nei(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[1])
+    assert signal.score == pytest.approx(0.40)
+
+
+def test_max_nei_proposes_nothing_when_every_hit_is_neutral() -> None:
+    probs = _probs((0.2, 0.1, 0.7))
+    assert bakeoff.aggregate_max_nei(P[:1], probs) == bakeoff.NOTHING
+
+
+def test_margin_lets_a_contradicting_passage_cancel_a_supporting_one() -> None:
+    probs = _probs((0.90, 0.05, 0.05), (0.05, 0.85, 0.10))
+    signal = bakeoff.aggregate_margin(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[0])
+    assert signal.score == pytest.approx(0.05)
+
+
+def test_margin_with_an_exact_tie_proposes_nothing() -> None:
+    probs = _probs((0.5, 0.0, 0.5), (0.0, 0.5, 0.5))
+    assert bakeoff.aggregate_margin(P[:2], probs) == bakeoff.NOTHING
+
+
+@pytest.mark.parametrize("name", bakeoff.AGGREGATIONS)
+def test_no_hits_propose_nothing(name: str) -> None:
+    assert bakeoff.AGGREGATORS[name]([], np.zeros((0, 3), np.float32)) == bakeoff.NOTHING
+
+
+def test_signal_for_reads_only_the_first_k_hits() -> None:
+    probs = _probs((0.2, 0.1, 0.7), (0.95, 0.0, 0.05))
+    first = bakeoff.signal_for("a claim", P[:2], probs, k=1, aggregation="max")
+    assert first.passage == P[0]
+    both = bakeoff.signal_for("a claim", P[:2], probs, k=2, aggregation="max")
+    assert both.passage == P[1]
+
+
+def test_a_numeric_mismatch_decides_before_any_aggregation() -> None:
+    passages = [Passage("Mortality fell by 12% in the treated group.", "doc", 0)]
+    probs = _probs((0.99, 0.0, 0.01))
+    signal = bakeoff.signal_for(
+        "Mortality fell by 40% in the treated group.", passages, probs, k=1, aggregation="max"
+    )
+    assert signal == bakeoff.Signal(Label.REFUTED, 1.0, passages[0])
+
+
+def test_decide_needs_the_cut_and_a_passage() -> None:
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.6, P[0]), cuts) is Label.SUPPORTED
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.4, P[0]), cuts) is Label.NEI
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.9, None), cuts) is Label.NEI
+
+
+def test_wilson_interval_matches_known_values() -> None:
+    low, high = bakeoff.wilson(0, 10)
+    assert low == pytest.approx(0.0)
+    assert high == pytest.approx(0.2775, abs=1e-4)
+    low, high = bakeoff.wilson(10, 10)
+    assert low == pytest.approx(0.7225, abs=1e-4)
+    assert high == pytest.approx(1.0)
+
+
+def test_wilson_refuses_an_empty_sample() -> None:
+    with pytest.raises(ValueError):
+        bakeoff.wilson(0, 0)
+```
+
+The numeric test relies on `numerics.check` refuting 40 % against a single 12 %. If the layer stays silent on this pair, change the sentences to a pair that `tests/test_numerics.py` already shows mismatching, and copy that pair verbatim.
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: FAIL with `ImportError: cannot import name 'bakeoff'`
+
+- [ ] **Step 3: Write the implementation**
+
+```python
+"""NLI bake-off: aggregation variants, calibration, evaluation and the decision rule.
+
+Pure functions over stored NLI probabilities; nothing here touches the network or a
+model (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md). A variant moves into
+``pipeline.py`` only if the bake-off picks it.
+"""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Callable, Mapping, Sequence
+from dataclasses import dataclass
+
+import numpy as np
+
+from proofpath import numerics
+from proofpath.entailment import LABEL_ORDER
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+
+_S = LABEL_ORDER.index(Label.SUPPORTED)
+_R = LABEL_ORDER.index(Label.REFUTED)
+_N = LABEL_ORDER.index(Label.NEI)
+
+# Simplest first: this order is also the tie order of the decision rule.
+AGGREGATIONS: tuple[str, ...] = ("max", "max_nei", "margin")
+KS: tuple[int, ...] = (1, 2, 3)
+# Passages retrieved and scored per item; every k is a prefix of these.
+TOP_N = 5
+
+
+@dataclass(frozen=True)
+class Signal:
+    """The strongest assertion a combination proposes for one item, before ``decide``.
+
+    ``label`` is SUPPORTED or REFUTED, or NEI when the aggregation proposes nothing;
+    ``passage`` is what the assertion rests on.
+    """
+
+    label: Label
+    score: float
+    passage: Passage | None
+
+
+NOTHING = Signal(Label.NEI, 0.0, None)
+
+Aggregator = Callable[[Sequence[Passage], np.ndarray], Signal]
+
+
+def aggregate_max(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """Today's ``pipeline.aggregate``: the loudest SUPPORTED or REFUTED over the hits."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_max_nei(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """As ``max``, but a hit only counts when its assertion beats its own p(NEI)."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        neutral = float(row[_N])
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > neutral and value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_margin(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """max p(S) − max p(R): a supporting and a contradicting passage cancel out."""
+    if len(passages) == 0:
+        return NOTHING
+    s_at = int(np.argmax(probs[:, _S]))
+    r_at = int(np.argmax(probs[:, _R]))
+    supported = float(probs[s_at, _S])
+    refuted = float(probs[r_at, _R])
+    if supported > refuted:
+        return Signal(Label.SUPPORTED, supported - refuted, passages[s_at])
+    if refuted > supported:
+        return Signal(Label.REFUTED, refuted - supported, passages[r_at])
+    return NOTHING
+
+
+AGGREGATORS: Mapping[str, Aggregator] = {
+    "max": aggregate_max,
+    "max_nei": aggregate_max_nei,
+    "margin": aggregate_margin,
+}
+
+
+def signal_for(
+    claim: str, passages: Sequence[Passage], probs: np.ndarray, *, k: int, aggregation: str
+) -> Signal:
+    """What one combination proposes for one item, numeric layer first (spec §10).
+
+    A numeric mismatch decides by rule at score 1.0, as ``pipeline.decide_indexed``
+    does, so no candidate model is credited with what the rule decided.
+    """
+    top = list(passages[:k])
+    numeric = numerics.check(claim, top)
+    if numeric is not None and numeric.mismatch:
+        return Signal(Label.REFUTED, 1.0, numeric.passage)
+    return AGGREGATORS[aggregation](top, probs[:k])
+
+
+def decide(signal: Signal, thresholds: Thresholds) -> Label:
+    """The verdict a signal earns: below the cut, or with no passage, it is NEI."""
+    if signal.label is Label.NEI or signal.passage is None:
+        return Label.NEI
+    return signal.label if signal.score >= thresholds.decide else Label.NEI
+
+
+def wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]:
+    """95 % Wilson score interval for ``correct`` successes out of ``n``."""
+    if n <= 0:
+        raise ValueError("a Wilson interval needs at least one observation")
+    p = correct / n
+    denominator = 1 + z * z / n
+    centre = (p + z * z / (2 * n)) / denominator
+    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator
+    return max(0.0, centre - half), min(1.0, centre + half)
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: 13 passed (11 test functions; the parametrised one runs three times)
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src tests && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 3: Calibration and evaluation — `bakeoff.py` part 2
+
+**Files:**
+- Modify: `src/proofpath/eval/bakeoff.py` (append)
+- Test: `tests/test_bakeoff.py` (append)
+
+**Interfaces:**
+- Consumes:
+  - Task 1: `Item`, `SnapshotClaim`
+  - Task 2: `signal_for`, `decide`, `wilson`
+  - `proofpath.eval.metrics`: `Row`, `LABELS`, `accuracy`, `macro_f1`, `rationale_f1`, `sweep_decide`, `tier_cutpoints`
+  - `proofpath.eval.averitec.to_label(str) -> Label | None`
+- Produces:
+  - `GRID: tuple[float, ...]` (0.05 … 0.95)
+  - `HIGH_TARGET = 0.85`, `MEDIUM_TARGET = 0.70`, `MIN_TIER_N = 20`
+  - `Scored(item: Item, probs: np.ndarray)`
+  - `ship_cut(value: float) -> float`
+  - `calibration_rows(scored: Sequence[Scored], *, k: int, aggregation: str) -> list[metrics.Row]`, which raises `ValueError` on any item that is not `scifact-train`
+  - `calibrate(rows: Sequence[metrics.Row]) -> Thresholds`
+  - `TierStat(tier: str, n: int, correct: int)` with `.precision -> float | None` (None below `MIN_TIER_N`) and `.interval -> tuple[float, float] | None`
+  - `SciFactResult(n, accuracy, macro_f1, f1: dict[Label, float], rationale_f1, tiers: tuple[TierStat, ...], asserted_without_passage)`
+  - `evaluate_scifact(scored, thresholds, *, k, aggregation) -> SciFactResult`, which accepts only `scifact-dev`
+  - `AveritecResult(n, counted, accuracy_all, readable, accuracy_readable, majority_baseline, per_label: dict[str, tuple[int, int]], asserted_without_passage)`
+  - `evaluate_averitec(claims: Sequence[SnapshotClaim], scored, thresholds, *, k, aggregation) -> AveritecResult`, which accepts only `averitec`
+
+- [ ] **Step 1: Write the failing tests (append to `tests/test_bakeoff.py`)**
+
+```python
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim, SnapshotSource
+
+
+def _scored(
+    key: str,
+    dataset: str,
+    gold: str,
+    row: tuple[float, float, float],
+    *,
+    group: str | None = None,
+    rationale: frozenset[int] = frozenset({0}),
+) -> bakeoff.Scored:
+    item = Item(key, dataset, group or key, "a claim", gold, (Passage("p", key, 0),), rationale)
+    return bakeoff.Scored(item, _probs(row))
+
+
+def test_calibration_refuses_anything_but_scifact_train() -> None:
+    dev = [_scored("d", "scifact-dev", "SUPPORTED", (0.9, 0.0, 0.1))]
+    with pytest.raises(ValueError, match="scifact-train"):
+        bakeoff.calibration_rows(dev, k=1, aggregation="max")
+
+
+def test_evaluation_refuses_the_calibration_split() -> None:
+    train = [_scored("t", "scifact-train", "SUPPORTED", (0.9, 0.0, 0.1))]
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    with pytest.raises(ValueError, match="scifact-dev"):
+        bakeoff.evaluate_scifact(train, cuts, k=1, aggregation="max")
+
+
+def test_calibrate_picks_a_cut_that_separates_right_from_wrong() -> None:
+    rows = [(Label.SUPPORTED, Label.SUPPORTED, 0.9)] * 5 + [(Label.NEI, Label.SUPPORTED, 0.3)] * 5
+    cuts = bakeoff.calibrate(rows)
+    assert 0.3 < cuts.decide <= 0.9
+    assert cuts.decide <= cuts.medium <= cuts.high
+
+
+def test_tier_stats_hide_precision_below_the_minimum_n() -> None:
+    assert bakeoff.TierStat("high", 19, 19).precision is None
+    assert bakeoff.TierStat("high", 19, 19).interval is None
+    assert bakeoff.TierStat("high", 20, 15).precision == pytest.approx(0.75)
+
+
+def test_evaluate_scifact_counts_verdicts_tiers_and_rationale() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    scored = [
+        _scored("a", "scifact-dev", "SUPPORTED", (0.97, 0.01, 0.02)),  # high, right
+        _scored("b", "scifact-dev", "REFUTED", (0.80, 0.10, 0.10)),  # medium, wrong
+        _scored("c", "scifact-dev", "NEI", (0.30, 0.10, 0.60)),  # below decide
+    ]
+    result = bakeoff.evaluate_scifact(scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.accuracy == pytest.approx(2 / 3)
+    assert {t.tier: (t.n, t.correct) for t in result.tiers} == {
+        "high": (1, 1),
+        "medium": (1, 0),
+        "low": (0, 0),
+    }
+    assert result.asserted_without_passage == 0
+    assert 0.0 < result.rationale_f1 <= 1.0
+
+
+def test_evaluate_averitec_keeps_the_strongest_source_and_both_denominators() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [
+        SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u1", "ok", "t"),), 0),
+        SnapshotClaim(2, "c2", "Supported", (SnapshotSource("u2", "UNVERIFIED (x)", ""),), 0),
+        SnapshotClaim(3, "c3", "Conflicting Evidence/Cherrypicking", (), 0),
+    ]
+    scored = [
+        _scored("averitec:1:0", "averitec", "Refuted", (0.6, 0.1, 0.3), group="averitec:1"),
+        _scored("averitec:1:1", "averitec", "Refuted", (0.0, 0.9, 0.1), group="averitec:1"),
+    ]
+    result = bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.counted == 2  # the Conflicting claim has no 3-way label
+    assert result.accuracy_all == pytest.approx(0.5)  # claim 1 right, claim 2 unread → NEI
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(1.0)
+    assert result.majority_baseline == pytest.approx(0.5)
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: FAIL with `AttributeError: module 'proofpath.eval.bakeoff' has no attribute 'Scored'`
+
+- [ ] **Step 3: Write the implementation (append to `bakeoff.py`; merge the imports into the top block)**
+
+```python
+from collections import Counter
+
+from proofpath.eval import averitec, metrics
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim
+
+# Wider than eval_scifact's 0.30–0.95 grid: a ``margin`` score is a difference of two
+# probabilities and lives much nearer 0. Every combination is swept on the same grid.
+GRID: tuple[float, ...] = tuple(round(0.05 * i, 2) for i in range(1, 20))
+HIGH_TARGET = 0.85
+MEDIUM_TARGET = 0.70
+# Below this many dev verdicts a tier's precision is reported as "too few to judge".
+MIN_TIER_N = 20
+CUT_DECIMALS = 6
+
+
+@dataclass(frozen=True)
+class Scored:
+    item: Item
+    probs: np.ndarray  # (len(item.passages), 3) in LABEL_ORDER
+
+
+def _require(scored: Sequence[Scored], dataset: str) -> None:
+    wrong = sorted({s.item.dataset for s in scored if s.item.dataset != dataset})
+    if wrong:
+        raise ValueError(f"expected only {dataset} items, got {', '.join(wrong)}")
+
+
+def ship_cut(value: float) -> float:
+    """Round a cut up to ``CUT_DECIMALS`` — the rule of ``eval_scifact.ship_cut``.
+
+    Up, never down: a cut rounded down would hand a tier to scores the calibration
+    split never showed were that good.
+    """
+    scale = 10**CUT_DECIMALS
+    return min(1.0, math.ceil(value * scale) / scale)
+
+
+def calibration_rows(scored: Sequence[Scored], *, k: int, aggregation: str) -> list[metrics.Row]:
+    """``(gold, proposed label, score)`` rows for calibration, SciFact train only."""
+    _require(scored, "scifact-train")
+    rows: list[metrics.Row] = []
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        rows.append((Label(s.item.gold), signal.label, signal.score))
+    return rows
+
+
+def calibrate(rows: Sequence[metrics.Row]) -> Thresholds:
+    """Best-accuracy ``decide``, then the 0.85 / 0.70 precision cuts above it."""
+    best = metrics.sweep_decide(rows, grid=GRID)
+    raw_high, raw_medium = metrics.tier_cutpoints(
+        rows,
+        decide=best.threshold,
+        high_precision=HIGH_TARGET,
+        medium_precision=MEDIUM_TARGET,
+    )
+    high, medium = ship_cut(raw_high), ship_cut(raw_medium)
+    return Thresholds(decide=best.threshold, high=max(high, medium), medium=medium)
+
+
+@dataclass(frozen=True)
+class TierStat:
+    tier: str
+    n: int
+    correct: int
+
+    @property
+    def precision(self) -> float | None:
+        return self.correct / self.n if self.n >= MIN_TIER_N else None
+
+    @property
+    def interval(self) -> tuple[float, float] | None:
+        return wilson(self.correct, self.n) if self.n >= MIN_TIER_N else None
+
+
+@dataclass(frozen=True)
+class SciFactResult:
+    n: int
+    accuracy: float
+    macro_f1: float
+    f1: dict[Label, float]
+    rationale_f1: float
+    tiers: tuple[TierStat, ...]  # high, medium, low
+    asserted_without_passage: int
+
+
+def per_label_f1(gold: Sequence[Label], pred: Sequence[Label]) -> dict[Label, float]:
+    scores: dict[Label, float] = {}
+    for label in metrics.LABELS:
+        tp = sum(g is label and p is label for g, p in zip(gold, pred, strict=True))
+        fp = sum(g is not label and p is label for g, p in zip(gold, pred, strict=True))
+        fn = sum(g is label and p is not label for g, p in zip(gold, pred, strict=True))
+        scores[label] = 2 * tp / (2 * tp + fp + fn) if tp else 0.0
+    return scores
+
+
+def _unbacked(signal: Signal, thresholds: Thresholds) -> bool:
+    """An assertion over the cut with no passage under it (product rule 1)."""
+    return (
+        signal.label is not Label.NEI
+        and signal.score >= thresholds.decide
+        and signal.passage is None
+    )
+
+
+def evaluate_scifact(
+    scored: Sequence[Scored], thresholds: Thresholds, *, k: int, aggregation: str
+) -> SciFactResult:
+    _require(scored, "scifact-dev")
+    gold: list[Label] = []
+    pred: list[Label] = []
+    gold_rationale: list[frozenset[int]] = []
+    pred_rationale: list[frozenset[int]] = []
+    tiers = {"high": [0, 0], "medium": [0, 0], "low": [0, 0]}
+    unbacked = 0
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        unbacked += _unbacked(signal, thresholds)
+        label = decide(signal, thresholds)
+        truth = Label(s.item.gold)
+        gold.append(truth)
+        pred.append(label)
+        gold_rationale.append(s.item.rationale)
+        if label is not Label.NEI and signal.passage is not None:
+            pred_rationale.append(frozenset({signal.passage.index}))
+            counts = tiers[thresholds.tier(signal.score)]
+            counts[0] += 1
+            counts[1] += label is truth
+        else:
+            pred_rationale.append(frozenset())
+    return SciFactResult(
+        n=len(scored),
+        accuracy=metrics.accuracy(gold, pred),
+        macro_f1=metrics.macro_f1(gold, pred),
+        f1=per_label_f1(gold, pred),
+        rationale_f1=metrics.rationale_f1(gold_rationale, pred_rationale),
+        tiers=tuple(TierStat(name, n, c) for name, (n, c) in tiers.items()),
+        asserted_without_passage=unbacked,
+    )
+
+
+@dataclass(frozen=True)
+class AveritecResult:
+    n: int
+    counted: int  # claims with a 3-way gold label
+    accuracy_all: float
+    readable: int  # counted claims with at least one readable source
+    accuracy_readable: float
+    majority_baseline: float
+    per_label: dict[str, tuple[int, int]]  # gold label -> (n, correct)
+    asserted_without_passage: int
+
+
+def evaluate_averitec(
+    claims: Sequence[SnapshotClaim],
+    scored: Sequence[Scored],
+    thresholds: Thresholds,
+    *,
+    k: int,
+    aggregation: str,
+) -> AveritecResult:
+    """Per claim, the rule of ``eval_averitec._decide_claim``.
+
+    The strongest decided non-NEI verdict across the claim's sources wins. A claim
+    with a readable source and no assertion is NEI. A claim with nothing readable is
+    unanswered, and is scored as NEI.
+    """
+    _require(scored, "averitec")
+    by_group: dict[str, list[Scored]] = {}
+    for s in scored:
+        by_group.setdefault(s.item.group, []).append(s)
+    per_label: dict[str, tuple[int, int]] = {}
+    golds: list[Label] = []
+    correct_all = readable = correct_readable = unbacked = 0
+    for claim in claims:
+        group = by_group.get(f"averitec:{claim.claim_id}", [])
+        best: Signal | None = None
+        for s in group:
+            signal = signal_for(
+                s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation
+            )
+            unbacked += _unbacked(signal, thresholds)
+            if decide(signal, thresholds) is Label.NEI:
+                continue
+            if best is None or signal.score > best.score:
+                best = signal
+        predicted = best.label if best is not None else Label.NEI
+        gold = averitec.to_label(claim.gold)
+        correct = gold is not None and predicted is gold
+        seen, right = per_label.get(claim.gold, (0, 0))
+        per_label[claim.gold] = (seen + 1, right + int(correct))
+        if gold is None:
+            continue
+        golds.append(gold)
+        correct_all += correct
+        if group:
+            readable += 1
+            correct_readable += correct
+    majority = max(Counter(golds).values(), default=0)
+    return AveritecResult(
+        n=len(claims),
+        counted=len(golds),
+        accuracy_all=correct_all / len(golds) if golds else 0.0,
+        readable=readable,
+        accuracy_readable=correct_readable / readable if readable else 0.0,
+        majority_baseline=majority / len(golds) if golds else 0.0,
+        per_label=per_label,
+        asserted_without_passage=unbacked,
+    )
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: all passed
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src tests && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 4: The decision rule — `bakeoff.py` part 3
+
+**Files:**
+- Modify: `src/proofpath/eval/bakeoff.py` (append)
+- Test: `tests/test_bakeoff.py` (append)
+
+**Interfaces:**
+- Consumes: `AGGREGATIONS` from Task 2.
+- Produces:
+  - `TIE_MARGIN = 0.01`, `LARGE_MARGIN = 0.03`, `NO_CHANGE_MARGIN = 0.01`
+  - `DECISION_RULE: str`
+  - `Combo(model: str, k: int, aggregation: str)` with `.label() -> str`
+  - `Outcome(combo: Combo, large: bool, macro_f1: float, averitec_readable: float, asserted_without_passage: int)`
+  - `Choice(winner: Combo | None, reason: str)`
+  - `choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice`
+
+- [ ] **Step 1: Write the failing tests (append)**
+
+```python
+BASE = bakeoff.Combo("base", 1, "max")
+
+
+def _o(
+    model: str, k: int, agg: str, f1: float, *, readable: float = 0.40, unbacked: int = 0
+) -> bakeoff.Outcome:
+    return bakeoff.Outcome(bakeoff.Combo(model, k, agg), model != "base", f1, readable, unbacked)
+
+
+def test_no_change_when_nothing_beats_the_baseline_by_a_hundredth() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.604)], baseline=BASE
+    )
+    assert choice.winner is None
+    assert "no change" in choice.reason
+
+
+def test_a_base_row_that_clears_the_margin_wins() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 3, "margin", 0.630)], baseline=BASE
+    )
+    assert choice.winner == bakeoff.Combo("base", 3, "margin")
+
+
+def test_a_large_row_needs_three_hundredths_over_the_best_base_row() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max_nei", 0.620),
+        _o("large", 1, "max", 0.645),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+    rows.append(_o("large-fever", 1, "max", 0.651))
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large-fever", 1, "max")
+
+
+def test_ties_go_to_the_smaller_k_then_the_simpler_aggregation() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.640),
+        _o("base", 2, "margin", 0.635),
+        _o("base", 2, "max_nei", 0.632),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+
+
+def test_a_row_that_asserts_without_a_passage_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.700, unbacked=1)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_a_row_worse_on_averitec_readable_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597, readable=0.40), _o("base", 2, "max", 0.700, readable=0.39)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_the_baseline_must_be_among_the_outcomes() -> None:
+    with pytest.raises(ValueError, match="baseline"):
+        bakeoff.choose([_o("base", 2, "max", 0.6)], baseline=BASE)
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: FAIL with `AttributeError: module 'proofpath.eval.bakeoff' has no attribute 'Combo'`
+
+- [ ] **Step 3: Write the implementation (append)**
+
+```python
+TIE_MARGIN = 0.01
+LARGE_MARGIN = 0.03
+NO_CHANGE_MARGIN = 0.01
+# Float slack, so a difference printed as 0.030 counts as 0.03.
+_EPS = 1e-9
+
+DECISION_RULE = """\
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Winner:** the highest SciFact dev macro-F1 among eligible rows.
+3. **Size gate:** a `large*` row wins only if it beats the best eligible `base` row by
+   **≥ 0.03** macro-F1. Otherwise the best `base` row wins.
+4. **Tie:** two rows within 0.01 of each other go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`.
+5. If nothing beats `base` × k=1 × `max` by ≥ 0.01 macro-F1, the result is "no change".
+   It is written up as such, and the next package (B, coverage) starts."""
+
+
+@dataclass(frozen=True)
+class Combo:
+    model: str
+    k: int
+    aggregation: str
+
+    def label(self) -> str:
+        return f"`{self.model}` × k={self.k} × `{self.aggregation}`"
+
+
+@dataclass(frozen=True)
+class Outcome:
+    combo: Combo
+    large: bool
+    macro_f1: float  # SciFact dev
+    averitec_readable: float
+    asserted_without_passage: int
+
+
+@dataclass(frozen=True)
+class Choice:
+    winner: Combo | None  # None means "no change"
+    reason: str
+
+
+def _pick(rows: Sequence[Outcome]) -> Outcome:
+    top = max(row.macro_f1 for row in rows)
+    contenders = [row for row in rows if top - row.macro_f1 <= TIE_MARGIN + _EPS]
+    return min(
+        contenders,
+        key=lambda row: (
+            row.combo.k,
+            AGGREGATIONS.index(row.combo.aggregation),
+            -row.macro_f1,
+        ),
+    )
+
+
+def choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
+    """Apply ``DECISION_RULE`` to the measured rows."""
+    by_combo = {row.combo: row for row in outcomes}
+    if baseline not in by_combo:
+        raise ValueError(f"the baseline {baseline.label()} is not among the outcomes")
+    base = by_combo[baseline]
+    eligible = [
+        row
+        for row in outcomes
+        if row.asserted_without_passage == 0
+        and row.averitec_readable >= base.averitec_readable - _EPS
+    ]
+    small = [row for row in eligible if not row.large]
+    large = [row for row in eligible if row.large]
+    if not small:
+        return Choice(None, "no change: no eligible base row (the baseline itself failed rule 1)")
+    candidate = _pick(small)
+    if large:
+        best_large = _pick(large)
+        if best_large.macro_f1 - candidate.macro_f1 >= LARGE_MARGIN - _EPS:
+            candidate = best_large
+    gain = candidate.macro_f1 - base.macro_f1
+    if gain < NO_CHANGE_MARGIN - _EPS:
+        return Choice(
+            None,
+            f"no change: the best eligible row, {candidate.combo.label()}, is "
+            f"{gain:+.3f} macro-F1 over the baseline, below {NO_CHANGE_MARGIN}",
+        )
+    return Choice(
+        candidate.combo,
+        f"{candidate.combo.label()} is {gain:+.3f} macro-F1 over {baseline.label()}",
+    )
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: all passed
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src tests && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 5: The script's `snapshot` and `score` subcommands
+
+**Files:**
+- Create: `scripts/eval_nli_bakeoff.py`
+- Test: `tests/test_eval_nli_bakeoff.py`
+
+**Interfaces:**
+- Consumes:
+  - Tasks 1–4
+  - `proofpath.retrieval.rank`, `split_sentences`, `FastEmbedder`
+  - `proofpath.entailment.OnnxNli(cache_dir=, repo_id=, revision=, onnx_file=, providers=)` and `pick_onnx_file`
+  - `proofpath.verify.Engine.default(config, interactive=False, browser=False)`, then `.fetcher.fetch(url)` and `.close()`
+  - `proofpath.eval.scifact.ensure_downloaded/load` and `proofpath.eval.averitec.ensure_downloaded/load`
+- Produces:
+  - `Candidate(repo, revision, onnx_file: Callable[[str], str], large: bool)`
+  - `CANDIDATES: dict[str, Candidate]`
+  - `BASELINE = bakeoff.Combo("base", 1, "max")`
+  - `bakeoff_dir() -> Path`, `snapshot_path() -> Path`, `items_path() -> Path`
+  - `nli_providers() -> list[str]`
+  - `build_items(embedder: retrieval.Embedder) -> list[Item]`
+  - `main(argv: list[str] | None = None) -> int`, with subcommands `snapshot`, `score` and `report`. `report` is a stub that returns 2 until Task 6.
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+"""Offline tests for scripts/eval_nli_bakeoff.py's importable parts."""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+from types import ModuleType
+
+SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "eval_nli_bakeoff.py"
+
+
+def _load_script() -> ModuleType:
+    spec = importlib.util.spec_from_file_location("eval_nli_bakeoff", SCRIPT_PATH)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+script = _load_script()
+
+
+def test_every_candidate_is_pinned_to_a_full_revision() -> None:
+    assert set(script.CANDIDATES) == {"base", "large", "large-fever"}
+    for candidate in script.CANDIDATES.values():
+        assert len(candidate.revision) == 40
+        int(candidate.revision, 16)
+
+
+def test_only_the_large_models_are_marked_large() -> None:
+    assert {name for name, c in script.CANDIDATES.items() if c.large} == {"large", "large-fever"}
+
+
+def test_each_candidate_names_an_onnx_file_per_machine() -> None:
+    for machine in ("arm64", "x86_64"):
+        for candidate in script.CANDIDATES.values():
+            assert candidate.onnx_file(machine).startswith("onnx/")
+            assert candidate.onnx_file(machine).endswith(".onnx")
+
+
+def test_the_baseline_is_base_k1_max() -> None:
+    assert (script.BASELINE.model, script.BASELINE.k, script.BASELINE.aggregation) == (
+        "base",
+        1,
+        "max",
+    )
+
+
+def test_coreml_is_never_handed_to_the_int8_models() -> None:
+    assert "CoreMLExecutionProvider" not in script.nli_providers()
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: FAIL with `FileNotFoundError` (the script does not exist yet)
+
+- [ ] **Step 3: Write the script**
+
+```python
+"""NLI bake-off harness (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Usage:
+    uv run python scripts/eval_nli_bakeoff.py snapshot [--limit 100] [--sleep 1.0] [--resume]
+    uv run python scripts/eval_nli_bakeoff.py score --model base|large|large-fever
+        [--rebuild-items]
+    uv run python scripts/eval_nli_bakeoff.py report [--out docs/eval/<date>-nli-bakeoff.md]
+
+``snapshot`` is the only network step, apart from the one-time dataset and model
+downloads. ``score`` runs one NLI model over every stored (passage, claim) pair.
+``report`` is offline. Importing this module touches neither the network nor a model.
+"""
+
+from __future__ import annotations
+
+import argparse
+import platform
+import sys
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from proofpath import retrieval
+from proofpath.entailment import pick_onnx_file
+from proofpath.eval import averitec, bakeoff, scifact
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    load_items,
+    load_snapshot,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+)
+from proofpath.models import Passage
+from proofpath.paths import cache_dir, models_dir
+
+EMBEDDER = "BAAI/bge-small-en-v1.5"
+
+
+@dataclass(frozen=True)
+class Candidate:
+    repo: str
+    revision: str
+    onnx_file: Callable[[str], str]  # platform.machine() -> file inside the repo
+    large: bool
+
+
+def _fever_file(_machine: str) -> str:
+    # The repo ships one dynamic-int8 export for every CPU.
+    return "onnx/model_quantized.onnx"
+
+
+CANDIDATES: dict[str, Candidate] = {
+    "base": Candidate(
+        "cross-encoder/nli-deberta-v3-base",
+        "6c749ce3425cd33b46d187e45b92bbf96ee12ec7",
+        pick_onnx_file,
+        large=False,
+    ),
+    "large": Candidate(
+        "cross-encoder/nli-deberta-v3-large",
+        "bab4bc7178836f731dcfd18c06ca9def0a137712",
+        pick_onnx_file,
+        large=True,
+    ),
+    "large-fever": Candidate(
+        "MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
+        "b3546ea6b0346eb6f8d5d68b13c7dc6d0376b3d7",
+        _fever_file,
+        large=True,
+    ),
+}
+BASELINE = bakeoff.Combo("base", 1, "max")
+# Pairs handed to the scorer per call: several of its 16-pair batches, so progress
+# can be printed without starving the batches.
+SCORE_CHUNK = 400
+
+
+def bakeoff_dir() -> Path:
+    return cache_dir() / "datasets" / "bakeoff"
+
+
+def snapshot_path() -> Path:
+    return cache_dir() / "datasets" / "averitec_snapshot.json"
+
+
+def items_path() -> Path:
+    return bakeoff_dir() / "items.json"
+
+
+def nli_providers() -> list[str]:
+    """CUDA first where present, then CPU.
+
+    CoreML is left out for every candidate. All three files are int8, and the
+    product leaves CoreML out for int8 too (``entailment.providers_for``).
+    """
+    from proofpath.device import onnx_providers
+
+    return [p for p in onnx_providers() if p != "CoreMLExecutionProvider"]
+
+
+def _ranked(
+    claim: str, passages: list[Passage], embedder: retrieval.Embedder
+) -> tuple[Passage, ...]:
+    hits = retrieval.rank(claim, passages, embedder, k=bakeoff.TOP_N)
+    return tuple(hit.passage for hit in hits)
+
+
+def build_items(embedder: retrieval.Embedder) -> list[Item]:
+    """SciFact train and dev pairs, then every readable AVeriTeC source, top 5 each.
+
+    Passages are split exactly as the product splits them: SciFact ships its
+    sentences, and a page goes through ``retrieval.split_sentences`` as in
+    ``eval_averitec._decide_claim``.
+    """
+    items: list[Item] = []
+    tarball = scifact.ensure_downloaded(cache_dir())
+    for split in ("train", "dev"):
+        data = scifact.load(tarball, split=split)
+        for pair in data.pairs():
+            doc = data.corpus[pair.doc_id]
+            passages = [Passage(t, str(doc.doc_id), i) for i, t in enumerate(doc.sentences)]
+            key = f"scifact-{split}:{pair.claim_id}:{pair.doc_id}"
+            items.append(
+                Item(
+                    key,
+                    f"scifact-{split}",
+                    key,
+                    pair.claim,
+                    pair.label.value,
+                    _ranked(pair.claim, passages, embedder),
+                    pair.rationale,
+                )
+            )
+    for claim in load_snapshot(snapshot_path()):
+        for position, source in enumerate(claim.sources):
+            sentences = retrieval.split_sentences(source.text) if source.text.strip() else []
+            if not sentences:
+                continue
+            passages = [Passage(s, source.url, i) for i, s in enumerate(sentences)]
+            items.append(
+                Item(
+                    f"averitec:{claim.claim_id}:{position}",
+                    "averitec",
+                    f"averitec:{claim.claim_id}",
+                    claim.claim,
+                    claim.gold,
+                    _ranked(claim.claim, passages, embedder),
+                )
+            )
+    return items
+
+
+def cmd_snapshot(args: argparse.Namespace) -> int:
+    from proofpath.config import load_config
+    from proofpath.verify import Engine
+
+    path = snapshot_path()
+    if path.exists() and not args.resume:
+        print(
+            f"error     {path} already holds a snapshot.\n"
+            "          Pass --resume to continue it, or delete the file to start over.",
+            file=sys.stderr,
+        )
+        return 2
+    claims = averitec.load(averitec.ensure_downloaded(cache_dir()), limit=args.limit or None)
+    stored = load_snapshot(path) if args.resume else []
+    todo = pending(claims, stored)
+    print(f"snapshot  {len(claims)} claims, {len(stored)} done, {len(todo)} to fetch")
+    # Browser not permitted and no prompts: the default install, as in the published
+    # AVeriTeC number (spec: Data).
+    engine = Engine.default(load_config(), interactive=False, browser=False)
+    try:
+        for position, claim in enumerate(todo, start=1):
+            sources = []
+            for url in claim.source_urls:
+                if args.sleep:
+                    time.sleep(args.sleep)
+                sources.append(source_from_fetched(url, engine.fetcher.fetch(url)))
+            stored.append(
+                SnapshotClaim(claim.id, claim.text, claim.label, tuple(sources), claim.non_urls)
+            )
+            save_snapshot(path, stored)
+            read = sum(bool(s.text) for s in sources)
+            print(
+                f"  {position}/{len(todo)}  claim {claim.id}  read {read}/{len(sources)}",
+                file=sys.stderr,
+            )
+    except KeyboardInterrupt:
+        print(f"\ninterrupted; {path} is usable with --resume", file=sys.stderr)
+    finally:
+        engine.close()
+    return 0
+
+
+def _items(rebuild: bool) -> list[Item]:
+    if items_path().exists() and not rebuild:
+        return load_items(items_path())
+    embedder = retrieval.FastEmbedder(EMBEDDER, cache_dir=models_dir())
+    try:
+        items = build_items(embedder)
+    finally:
+        embedder.close()
+    save_items(items_path(), items)
+    print(f"items     {len(items)} written to {items_path()}")
+    return items
+
+
+def cmd_score(args: argparse.Namespace) -> int:
+    from proofpath.entailment import OnnxNli
+
+    if not snapshot_path().exists():
+        print("error     no AVeriTeC snapshot; run `snapshot` first", file=sys.stderr)
+        return 2
+    items = _items(args.rebuild_items)
+    candidate = CANDIDATES[args.model]
+    scorer = OnnxNli(
+        cache_dir=models_dir(),
+        repo_id=candidate.repo,
+        revision=candidate.revision,
+        onnx_file=candidate.onnx_file(platform.machine()),
+        providers=nli_providers(),
+    )
+    print(f"nli       {scorer.name}  providers={scorer.providers}")
+    pairs = [(p.text, item.claim) for item in items for p in item.passages]
+    try:
+        started = time.perf_counter()
+        blocks: list[np.ndarray] = []
+        for start in range(0, len(pairs), SCORE_CHUNK):
+            blocks.append(scorer.score(pairs[start : start + SCORE_CHUNK]))
+            done = min(start + SCORE_CHUNK, len(pairs))
+            print(
+                f"  {done}/{len(pairs)} pairs  {time.perf_counter() - started:.0f}s",
+                file=sys.stderr,
+            )
+        elapsed = time.perf_counter() - started
+    finally:
+        scorer.close()
+    flat = np.concatenate(blocks) if blocks else np.zeros((0, 3), dtype=np.float32)
+    offsets = np.cumsum([0, *(len(item.passages) for item in items)])
+    probs = [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))]
+    ms_per_pair = elapsed / max(len(pairs), 1) * 1000
+    out = bakeoff_dir() / f"{args.model}.npz"
+    save_probs(out, items, probs, ms_per_pair=ms_per_pair)
+    print(f"scored    {len(pairs)} pairs, {ms_per_pair:.0f} ms/pair -> {out}")
+    return 0
+
+
+def cmd_report(args: argparse.Namespace) -> int:
+    print("error     report is implemented in Task 6", file=sys.stderr)
+    return 2
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+    snap = sub.add_parser("snapshot", help="fetch AVeriTeC gold sources once (network)")
+    snap.add_argument("--limit", type=int, default=100, help="first N dev claims")
+    snap.add_argument("--sleep", type=float, default=1.0, help="seconds between fetches")
+    snap.add_argument("--resume", action="store_true", help="skip claims already stored")
+    score = sub.add_parser("score", help="run one NLI model over every stored pair")
+    score.add_argument("--model", required=True, choices=sorted(CANDIDATES))
+    score.add_argument("--rebuild-items", action="store_true", help="re-run retrieval")
+    report = sub.add_parser("report", help="calibrate, evaluate and choose (offline)")
+    report.add_argument("--out", default="")
+    args = parser.parse_args(argv)
+    commands = {"snapshot": cmd_snapshot, "score": cmd_score, "report": cmd_report}
+    return commands[args.command](args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: 5 passed
+
+- [ ] **Step 5: Lint**
+
+Run: `uv run ruff check scripts/eval_nli_bakeoff.py tests && uv run ruff format --check scripts tests`
+Expected: no errors. Wrap any line over 100 characters as ruff formats it.
+
+- [ ] **Step 6: Smoke-test that each model loads (controller-run, network, ~1.3 GB once)**
+
+Run:
+```bash
+uv run python - <<'EOF'
+import platform, importlib.util, sys
+spec = importlib.util.spec_from_file_location("b", "scripts/eval_nli_bakeoff.py")
+m = importlib.util.module_from_spec(spec); sys.modules["b"] = m; spec.loader.exec_module(m)
+from proofpath.entailment import OnnxNli
+from proofpath.paths import models_dir
+pairs = [("The cat sat on the mat.", "A cat is sitting."), ("The cat sat on the mat.", "No animal is present.")]
+for name, c in m.CANDIDATES.items():
+    s = OnnxNli(cache_dir=models_dir(), repo_id=c.repo, revision=c.revision,
+                onnx_file=c.onnx_file(platform.machine()), providers=m.nli_providers())
+    print(name, s.score(pairs).round(3).tolist()); s.close()
+EOF
+```
+Expected: for every model, row 0 has its largest value in column 0 (SUPPORTED) and row 1 in column 1 (REFUTED). If `large-fever` fails to load on arm64 (no int8 kernel), stop and report back; do not substitute a different file without asking.
+
+---
+
+### Task 6: The `report` subcommand and `render_report`
+
+**Files:**
+- Modify: `scripts/eval_nli_bakeoff.py`. Replace `cmd_report` and add `ReportRow`, `model_size_mb` and `render_report`.
+- Test: `tests/test_eval_nli_bakeoff.py` (append)
+
+**Interfaces:**
+- Consumes:
+  - Tasks 1–5
+  - `huggingface_hub.try_to_load_from_cache(repo_id, filename, cache_dir=, revision=)`
+- Produces:
+  - `ReportRow(combo: bakeoff.Combo, large: bool, thresholds: Thresholds, scifact: bakeoff.SciFactResult, averitec: bakeoff.AveritecResult)`
+  - `render_report(rows, choice, *, speed: Mapping[str, float], sizes: Mapping[str, float | None], counts: Mapping[str, int], today: str, machine: str) -> str`
+
+- [ ] **Step 1: Write the failing tests (append)**
+
+```python
+from proofpath.eval import bakeoff
+from proofpath.models import Label
+from proofpath.pipeline import Thresholds
+
+
+def _row(model: str, k: int, agg: str, f1: float, tiers: tuple[int, int, int]) -> object:
+    sci = bakeoff.SciFactResult(
+        n=340,
+        accuracy=f1 + 0.01,
+        macro_f1=f1,
+        f1={Label.SUPPORTED: 0.7, Label.REFUTED: 0.5, Label.NEI: 0.6},
+        rationale_f1=0.3,
+        tiers=(
+            bakeoff.TierStat("high", tiers[0], tiers[0]),
+            bakeoff.TierStat("medium", tiers[1], tiers[1] // 2),
+            bakeoff.TierStat("low", tiers[2], 0),
+        ),
+        asserted_without_passage=0,
+    )
+    ave = bakeoff.AveritecResult(
+        n=100,
+        counted=89,
+        accuracy_all=0.27,
+        readable=61,
+        accuracy_readable=0.36,
+        majority_baseline=0.708,
+        per_label={"Refuted": (63, 20)},
+        asserted_without_passage=0,
+    )
+    return script.ReportRow(
+        bakeoff.Combo(model, k, agg),
+        model != "base",
+        Thresholds(decide=0.45, high=0.99933, medium=0.457948),
+        sci,
+        ave,
+    )
+
+
+def _render(choice: bakeoff.Choice) -> str:
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("large", 2, "margin", 0.650, (30, 40, 25)),
+    ]
+    return script.render_report(
+        rows,
+        choice,
+        speed={"base": 185.0, "large": 540.0},
+        sizes={"base": 244.4, "large": None},
+        counts={"scifact-train": 919, "scifact-dev": 340, "averitec": 100},
+        today="2026-10-02",
+        machine="Darwin arm64",
+    )
+
+
+def test_the_report_prints_the_rule_verbatim_and_the_winner() -> None:
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"))
+    assert bakeoff.DECISION_RULE in text
+    assert "**Winner:** `large` × k=2 × `margin`" in text
+    assert "| base | 1 | max | 0.45 |" in text
+    assert "| large | 2 | margin | 0.45 |" in text
+
+
+def test_the_report_says_no_change_in_words() -> None:
+    text = _render(bakeoff.Choice(None, "no change: below 0.01"))
+    assert "**No change.**" in text
+    assert "no change: below 0.01" in text
+
+
+def test_a_tier_below_the_minimum_n_is_too_few_to_judge() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "too few to judge" in text
+
+
+def test_an_unknown_model_size_is_a_dash_not_zero() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "| large | `cross-encoder/nli-deberta-v3-large` | — |" in text
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: FAIL with `AttributeError: module 'eval_nli_bakeoff' has no attribute 'ReportRow'`
+
+- [ ] **Step 3: Write the implementation**
+
+Add `from collections.abc import Callable, Mapping, Sequence`, `from datetime import date`, `from proofpath.eval.bakeoff_data import load_probs` and `from proofpath.pipeline import Thresholds` to the imports. Then replace `cmd_report`:
+
+```python
+@dataclass(frozen=True)
+class ReportRow:
+    combo: bakeoff.Combo
+    large: bool
+    thresholds: Thresholds
+    scifact: bakeoff.SciFactResult
+    averitec: bakeoff.AveritecResult
+
+
+def model_size_mb(model: str) -> float | None:
+    """Size of the ONNX file the run used, or None when it is not in the cache."""
+    from huggingface_hub import try_to_load_from_cache
+
+    candidate = CANDIDATES[model]
+    found = try_to_load_from_cache(
+        candidate.repo,
+        candidate.onnx_file(platform.machine()),
+        cache_dir=str(models_dir()),
+        revision=candidate.revision,
+    )
+    return Path(found).stat().st_size / 1e6 if isinstance(found, str) else None
+
+
+def _md(*cells: object) -> str:
+    return "| " + " | ".join(str(c) for c in cells) + " |"
+
+
+def _tier_lines(row: ReportRow) -> list[str]:
+    lines = [
+        f"### {row.combo.label()}  (decide {row.thresholds.decide:.2f}, "
+        f"medium {row.thresholds.medium:.6f}, high {row.thresholds.high:.6f})",
+        "",
+        "| tier | n | precision | 95 % Wilson |",
+        "|---|---|---|---|",
+    ]
+    for tier in row.scifact.tiers:
+        if tier.precision is None or tier.interval is None:
+            lines.append(_md(tier.tier, tier.n, "too few to judge", "—"))
+        else:
+            low, high = tier.interval
+            lines.append(_md(tier.tier, tier.n, f"{tier.precision:.3f}", f"{low:.2f}–{high:.2f}"))
+    lines.append("")
+    return lines
+
+
+def render_report(
+    rows: Sequence[ReportRow],
+    choice: bakeoff.Choice,
+    *,
+    speed: Mapping[str, float],
+    sizes: Mapping[str, float | None],
+    counts: Mapping[str, int],
+    today: str,
+    machine: str,
+) -> str:
+    readable = rows[0].averitec.readable if rows else 0
+    baseline_ave = rows[0].averitec.majority_baseline if rows else 0.0
+    lines = [
+        f"# NLI bake-off — {today}",
+        "",
+        "- spec: `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`",
+        f"- calibrated on SciFact train ({counts.get('scifact-train', 0)} pairs); reported on "
+        f"SciFact dev ({counts.get('scifact-dev', 0)} pairs) and AVeriTeC dev "
+        f"({counts.get('averitec', 0)} claims, {readable} answerable with a readable source; "
+        "frozen snapshot, browser not permitted)",
+        f"- embedder `{EMBEDDER}`, top {bakeoff.TOP_N} passages per item, numeric layer on",
+        f"- machine: {machine}",
+        "",
+        "## Decision rule (fixed before any number was seen)",
+        "",
+        bakeoff.DECISION_RULE,
+        "",
+        "## Result",
+        "",
+    ]
+    if choice.winner is None:
+        lines.append(f"**No change.** {choice.reason}")
+    else:
+        lines.append(f"**Winner:** {choice.winner.label()} — {choice.reason}")
+    lines.extend(
+        ["", "## Models", "", "| model | repo | size (MB) | ms/pair |", "|---|---|---|---|"]
+    )
+    for model in dict.fromkeys(row.combo.model for row in rows):
+        size = sizes.get(model)
+        lines.append(
+            _md(
+                model,
+                f"`{CANDIDATES[model].repo}`",
+                "—" if size is None else f"{size:.0f}",
+                f"{speed.get(model, 0.0):.0f}",
+            )
+        )
+    lines.extend(
+        [
+            "",
+            "## Every combination",
+            "",
+            f"AVeriTeC majority baseline: {baseline_ave:.3f}. Published numbers to compare: "
+            "0.270 (all) and 0.361 (readable), 2026-09-16.",
+            "",
+            "| model | k | aggregation | decide | dev acc | dev macro-F1 | F1 S / R / NEI "
+            "| rationale F1 | AVeriTeC all | AVeriTeC readable | w/o passage |",
+            "|---|---|---|---|---|---|---|---|---|---|---|",
+        ]
+    )
+    for row in rows:
+        f1 = row.scifact.f1
+        lines.append(
+            _md(
+                row.combo.model,
+                row.combo.k,
+                row.combo.aggregation,
+                f"{row.thresholds.decide:.2f}",
+                f"{row.scifact.accuracy:.3f}",
+                f"{row.scifact.macro_f1:.3f}",
+                f"{f1[Label.SUPPORTED]:.2f} / {f1[Label.REFUTED]:.2f} / {f1[Label.NEI]:.2f}",
+                f"{row.scifact.rationale_f1:.3f}",
+                f"{row.averitec.accuracy_all:.3f}",
+                f"{row.averitec.accuracy_readable:.3f}",
+                row.scifact.asserted_without_passage + row.averitec.asserted_without_passage,
+            )
+        )
+    lines.extend(["", "## Tiers on SciFact dev", ""])
+    shown = [BASELINE] + ([choice.winner] if choice.winner and choice.winner != BASELINE else [])
+    for combo in shown:
+        lines.extend(_tier_lines(next(r for r in rows if r.combo == combo)))
+    return "\n".join(lines) + "\n"
+
+
+def cmd_report(args: argparse.Namespace) -> int:
+    if not items_path().exists() or not snapshot_path().exists():
+        print("error     run `snapshot` and `score --model base` first", file=sys.stderr)
+        return 2
+    items = load_items(items_path())
+    snapshot = load_snapshot(snapshot_path())
+    rows: list[ReportRow] = []
+    speed: dict[str, float] = {}
+    for model, candidate in CANDIDATES.items():
+        path = bakeoff_dir() / f"{model}.npz"
+        if not path.exists():
+            print(f"skip      {model}: not scored", file=sys.stderr)
+            continue
+        probs, speed[model] = load_probs(path, items)
+        scored = [bakeoff.Scored(i, p) for i, p in zip(items, probs, strict=True)]
+        train = [s for s in scored if s.item.dataset == "scifact-train"]
+        dev = [s for s in scored if s.item.dataset == "scifact-dev"]
+        ave = [s for s in scored if s.item.dataset == "averitec"]
+        for k in bakeoff.KS:
+            for aggregation in bakeoff.AGGREGATIONS:
+                thresholds = bakeoff.calibrate(
+                    bakeoff.calibration_rows(train, k=k, aggregation=aggregation)
+                )
+                rows.append(
+                    ReportRow(
+                        bakeoff.Combo(model, k, aggregation),
+                        candidate.large,
+                        thresholds,
+                        bakeoff.evaluate_scifact(dev, thresholds, k=k, aggregation=aggregation),
+                        bakeoff.evaluate_averitec(
+                            snapshot, ave, thresholds, k=k, aggregation=aggregation
+                        ),
+                    )
+                )
+    if "base" not in speed:
+        print(
+            "error     the baseline model is not scored; run `score --model base`", file=sys.stderr
+        )
+        return 2
+    outcomes = [
+        bakeoff.Outcome(
+            r.combo,
+            r.large,
+            r.scifact.macro_f1,
+            r.averitec.accuracy_readable,
+            r.scifact.asserted_without_passage + r.averitec.asserted_without_passage,
+        )
+        for r in rows
+    ]
+    choice = bakeoff.choose(outcomes, baseline=BASELINE)
+    counts = {
+        "scifact-train": sum(i.dataset == "scifact-train" for i in items),
+        "scifact-dev": sum(i.dataset == "scifact-dev" for i in items),
+        "averitec": len(snapshot),
+    }
+    report = render_report(
+        rows,
+        choice,
+        speed=speed,
+        sizes={model: model_size_mb(model) for model in speed},
+        counts=counts,
+        today=date.today().isoformat(),
+        machine=f"{platform.system()} {platform.machine()}",
+    )
+    out = (
+        Path(args.out)
+        if args.out
+        else Path("docs/eval") / f"{date.today().isoformat()}-nli-bakeoff.md"
+    )
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(report, encoding="utf-8")
+    print(report)
+    print(f"written   {out}")
+    return 0
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: 9 passed
+
+- [ ] **Step 5: Full gate**
+
+Run: `uv run ruff check src tests scripts && uv run ruff format --check . && uv run mypy && uv run pytest -q`
+Expected: no lint or type errors; the whole suite passes.
+
+---
+
+### Task 7: Run the bake-off and write it up (controller)
+
+This task is long-running measurement, not code. The controller runs it, and a reviewer checks the write-up against the numbers.
+
+- [ ] **Step 1: Snapshot (network, about 30–60 min at `--sleep 1.0`)**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py snapshot --limit 100 --sleep 1.0 --resume`
+Expected: `averitec_snapshot.json` holds 100 claims. If it is interrupted, re-run the same command.
+
+- [ ] **Step 2: Score the baseline first**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py score --model base`
+Expected: `bakeoff/base.npz` is written. The ms/pair for base should be the same order as the 185 ms/pair measured on 2026-09-12; it may be lower, because batches are now filled.
+
+- [ ] **Step 3: Sanity-check before spending hours on the large models**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py report --out /tmp/bakeoff-base.md`
+Expected: the `base` × k=1 × `max` row has a dev accuracy near 0.609. It will not match exactly: the calibration now comes from train, not dev. If it is off by more than 0.05, stop and investigate with superpowers:systematic-debugging before going on.
+
+- [ ] **Step 4: Score both large models (about 1 hour each on Apple Silicon CPU)**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py score --model large && uv run python scripts/eval_nli_bakeoff.py score --model large-fever`
+
+- [ ] **Step 5: Write the report**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py report`
+Expected: `docs/eval/<date>-nli-bakeoff.md`, with the winner or "No change" stated by the rule.
+
+- [ ] **Step 6: Record the outcome**
+
+Append one row to the backlog table in OPEN-ITEMS §20:
+
+`| 20.11 | NLI bake-off result | <winner or "no change">, <one line of numbers>; docs/eval/<date>-nli-bakeoff.md. Next: <follow-up spec, or package B> |`
+
+Change no README, spec §14 or product numbers here. That belongs to the follow-up spec.
+
+- [ ] **Step 7: Final gate and commit**
+
+Run: `uv run ruff check . && uv run ruff format --check . && uv run mypy && uv run pytest -q`
+Then the controller commits:
+- the script, the two modules, the tests, the report and OPEN-ITEMS;
+- on `feat/nli-bakeoff`, with the message `feat(eval): add the NLI bake-off harness and its first result`.
+
+Push and merge only with the user's go.
diff --git a/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md b/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md
new file mode 100644
index 0000000..277b8f6
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md
@@ -0,0 +1,162 @@
+# NLI bake-off — design (2026-10-01)
+
+## Why
+
+The entailment step is where proofpath loses most of its accuracy, and the obvious fix
+has already been measured and does not work:
+
+- SciFact dev retrieval recall goes from 0.560 at k=1 to 0.852 at k=3, but accuracy does
+  not follow: k=1 0.609, k=2 0.609, k=3 0.603 (`docs/eval/2026-09-12-scifact-dev.md`).
+  Taking the strongest of more passages adds as many wrong assertions as right ones.
+- The model is overconfident: 116 verdicts score in [0.90, 1.00] and only 71.6 % of them
+  are right. The `medium` cut (0.457948) sits on `decide` (0.45), so the report really
+  has two tiers, and every cut was fitted and measured on the same split.
+- On AVeriTeC every one of the 19 `Supported` claims was missed, and readable-source
+  accuracy is 0.361 against a 0.708 baseline (`docs/eval/2026-09-16-averitec.md`).
+
+So the next step is not a code change to the product. It is a measured comparison of
+NLI models and aggregation rules, with the winner chosen by a rule fixed in advance.
+Product integration is a separate spec written after the result.
+
+## Candidates
+
+| id | repo | export | size | trained on |
+|---|---|---|---|---|
+| `base` (current) | `cross-encoder/nli-deberta-v3-base` @ `6c749ce` | int8 per CPU | 244 MB | SNLI + MNLI |
+| `large` | `cross-encoder/nli-deberta-v3-large` @ `bab4bc7` | int8 per CPU | 643 MB | SNLI + MNLI |
+| `large-fever` | `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` @ `b3546ea` | `onnx/model_quantized.onnx` | 643 MB | MNLI, FEVER-NLI, ANLI, LingNLI, WANLI |
+
+All three load through the existing `entailment.OnnxNli(repo_id=, revision=, onnx_file=)`;
+`label_permutation` already maps each model's `id2label`. Revisions are pinned to full
+SHAs in the script. The downloads land on the developer's machine for this eval only;
+nothing in the product downloads them (product rule 5).
+
+## Data
+
+- **SciFact train** — calibration only (`eval.scifact.load(split="train")`).
+- **SciFact dev** — the reported, held-out split. It is the same 340 pairs as before, so
+  the numbers compare directly with 2026-09-12.
+- **AVeriTeC dev, first 100 claims** — a frozen snapshot. The 2026-09-16 run's fetched
+  text has expired from the cache (7-day TTL), and re-fetching for each candidate would
+  compare models on different pages. One network pass with the current ladder, browser
+  not permitted (the default install, as in the published number), writes
+  `<cache>/datasets/averitec_snapshot.json`: per claim, the gold label, and per source
+  URL the fetch state and the extracted text. The snapshot stays outside the repo
+  (third-party page text). Every candidate is scored from that one file.
+
+## Method
+
+`scripts/eval_nli_bakeoff.py`, dev-only, three subcommands:
+
+1. `snapshot` — builds the AVeriTeC snapshot (network, `--sleep`, `--resume`). It reuses
+   `eval_averitec.py`'s engine construction, `state_for` and per-URL fetch loop.
+2. `score --model <id>` — offline. Splits each source into sentences with
+   `retrieval.split_sentences`, as the product does, and retrieves the top 5 with
+   `bge-small-en-v1.5` once; the retrieval is shared by every model and stored. Runs the
+   model's NLI once over every (passage, claim) pair in that top 5 and stores the raw
+   probabilities in `<cache>/datasets/bakeoff/<id>.npz`, with ms/pair. The numeric layer
+   runs unchanged on the same top-k (`numerics.check`), so a candidate is never credited
+   with what the rule decided.
+3. `report` — offline. From the stored probabilities it computes every combination of
+   model × k ∈ {1, 2, 3} × aggregation, calibrates each on SciFact train, scores it on
+   SciFact dev and on AVeriTeC, and writes `docs/eval/<date>-nli-bakeoff.md`.
+
+### Aggregation variants
+
+Pure functions over `(hits, probs)` that return a `Verdict`. They live in the script; a
+variant moves into `pipeline.py` only if it wins.
+
+- `max` — today's `pipeline.aggregate`: the strongest p(SUPPORTED) or p(REFUTED) over
+  the hits decides.
+- `max_nei` — as `max`, but a hit only counts when its asserted label beats its own
+  p(NEI).
+- `margin` — score = max p(S) − max p(R) over the hits. The sign gives the label and the
+  absolute value is the score, so a supporting passage and a contradicting passage on
+  the same source cancel out instead of the louder one winning.
+
+### Calibration
+
+For each combination, on SciFact **train**:
+
+- `decide` is the best-accuracy cut from `eval.metrics.sweep_decide`.
+- `high` and `medium` come from `eval.metrics.tier_cutpoints` at the existing 0.85 and
+  0.70 precision targets.
+
+Those cuts are then applied unchanged to dev and AVeriTeC. Each tier's dev precision is
+reported with its n and a 95 % Wilson interval. A tier that holds fewer than 20 dev
+verdicts is reported as "too few to judge" rather than given a precision.
+
+### AVeriTeC scoring
+
+Per claim, the same rule as `eval_averitec._decide_claim`:
+
+- the strongest non-NEI verdict that carries a passage, across the claim's sources, wins;
+- a claim with a readable source but no assertion is `NEI`;
+- a claim with no readable source is unanswered.
+
+Two numbers are reported: 3-way accuracy over all answerable claims (comparable to
+0.270), and over claims with at least one readable source (comparable to 0.361). The
+0.708 majority baseline is printed beside both.
+
+## Metrics in the report
+
+For every combination:
+
+- SciFact dev: accuracy, macro-F1, per-label F1, rationale F1, and the tier table;
+- AVeriTeC: both 3-way accuracies and the per-label counts;
+- for each model: ms/pair (Apple Silicon CPU, int8) and download size;
+- "asserted without a passage", which must be 0 in every row (product rule 1).
+
+## Decision rule (fixed before any number is seen)
+
+The steps run in this order. Each one filters the rows the previous step left.
+
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Beats the baseline:** a row stays only if its SciFact dev macro-F1 beats
+   `base` × k=1 × `max` by **≥ 0.01**. If no row stays, the result is "no change". It is
+   written up as such, and the next package (B, coverage) starts.
+3. **Size gate:** a `large*` row stays only if it beats the highest macro-F1 of any
+   eligible `base` row by **≥ 0.03**. That is the strongest eligible `base` row, not the
+   one the tie rule would pick.
+4. **Winner:** the highest SciFact dev macro-F1 among the rows left.
+5. **Tie:** rows left within 0.01 of the winner go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`.
+
+Clarified on 2026-10-01, before any number was seen. The first wording did not fix the
+order of the steps. Read as "tie rule first", it let a `large` row through on a 0.022
+lead, because it was measured against a simpler but weaker `base` row.
+
+The rule is copied into the report verbatim, next to the numbers it chose.
+
+## What happens after
+
+A follow-up spec, written after the report:
+
+- **`base` wins with a new k or aggregation:** change `pipeline.aggregate` and
+  `DEFAULT_THRESHOLDS`, and update `cache.model_id` keying. Old verdicts invalidate
+  through the threshold key that already exists.
+- **A `large*` row wins:** add an opt-in "accurate" NLI model. It is downloaded only with
+  consent (product rule 5, spec §7.1) and is never the default, because the default
+  install must stay small. It carries its own thresholds.
+
+Either way, the README "Measured" table, spec §14 and the tier note are updated with the
+new numbers.
+
+## Tests (no network)
+
+- The aggregation variants are tested on hand-built probability rows: the `margin`
+  cancellation, the NEI veto in `max_nei`, and no passage → no assertion.
+- The train/dev split guard: calibration must never see a dev pair id.
+- Wilson interval values, and the "too few to judge" cut-off.
+- The snapshot round trip: write, load, and `--resume` skipping finished ids, against a
+  fixture `Fetched`.
+- The decision rule as a function over synthetic result rows, covering the size gate,
+  the tie order and "no change".
+
+## Out of scope
+
+- Changing the embedder or chunking (OPEN-ITEMS 3.6 stays open).
+- Fetch coverage, including running with the browser allowed; package B covers that.
+- Any change to product code or defaults; that is the follow-up spec.
+- A FEVER-trained `base` model: it has no ONNX export, and exporting one needs torch.
diff --git a/scripts/eval_nli_bakeoff.py b/scripts/eval_nli_bakeoff.py
new file mode 100644
index 0000000..07b7e2f
--- /dev/null
+++ b/scripts/eval_nli_bakeoff.py
@@ -0,0 +1,711 @@
+"""NLI bake-off harness (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Usage:
+    uv run python scripts/eval_nli_bakeoff.py snapshot [--limit 100] [--sleep 1.0] [--resume]
+    uv run python scripts/eval_nli_bakeoff.py score --model base|large|large-fever
+        [--rebuild-items]
+    uv run python scripts/eval_nli_bakeoff.py report [--out docs/eval/<date>-nli-bakeoff.md]
+
+``snapshot`` is the only network step, apart from the one-time dataset and model
+downloads. ``score`` runs one NLI model over every stored (passage, claim) pair.
+``report`` is offline. Importing this module touches neither the network nor a model.
+"""
+
+from __future__ import annotations
+
+import argparse
+import platform
+import sys
+import time
+from collections.abc import Callable, Mapping, Sequence
+from dataclasses import dataclass
+from datetime import date
+from pathlib import Path
+
+import numpy as np
+
+from proofpath import retrieval
+from proofpath.entailment import pick_onnx_file
+from proofpath.eval import averitec, bakeoff, scifact
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    load_items,
+    load_probs,
+    load_snapshot,
+    passage_offsets,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+    split_at,
+)
+from proofpath.models import Label, Passage
+from proofpath.paths import cache_dir, models_dir
+from proofpath.pipeline import Thresholds
+
+EMBEDDER = "BAAI/bge-small-en-v1.5"
+
+
+@dataclass(frozen=True)
+class Candidate:
+    repo: str
+    revision: str
+    onnx_file: Callable[[str], str]  # platform.machine() -> file inside the repo
+    large: bool
+
+
+def _fever_file(_machine: str) -> str:
+    # The repo ships one dynamic-int8 export for every CPU.
+    return "onnx/model_quantized.onnx"
+
+
+CANDIDATES: dict[str, Candidate] = {
+    "base": Candidate(
+        "cross-encoder/nli-deberta-v3-base",
+        "6c749ce3425cd33b46d187e45b92bbf96ee12ec7",
+        pick_onnx_file,
+        large=False,
+    ),
+    "large": Candidate(
+        "cross-encoder/nli-deberta-v3-large",
+        "bab4bc7178836f731dcfd18c06ca9def0a137712",
+        pick_onnx_file,
+        large=True,
+    ),
+    "large-fever": Candidate(
+        "MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
+        "b3546ea6b0346eb6f8d5d68b13c7dc6d0376b3d7",
+        _fever_file,
+        large=True,
+    ),
+}
+BASELINE = bakeoff.Combo("base", 1, "max")
+# Pairs handed to the scorer per call: several of its 16-pair batches, so progress
+# can be printed without starving the batches.
+SCORE_CHUNK = 400
+
+
+def bakeoff_dir() -> Path:
+    return cache_dir() / "datasets" / "bakeoff"
+
+
+def snapshot_path() -> Path:
+    return cache_dir() / "datasets" / "averitec_snapshot.json"
+
+
+def items_path() -> Path:
+    return bakeoff_dir() / "items.json"
+
+
+def nli_providers() -> list[str]:
+    """CUDA first where present, then CPU.
+
+    CoreML is left out for every candidate. All three files are int8, and the
+    product leaves CoreML out for int8 too (``entailment.providers_for``).
+    """
+    from proofpath.device import onnx_providers
+
+    return [p for p in onnx_providers() if p != "CoreMLExecutionProvider"]
+
+
+def _ranked(
+    claim: str, passages: list[Passage], embedder: retrieval.Embedder
+) -> tuple[Passage, ...]:
+    hits = retrieval.rank(claim, passages, embedder, k=bakeoff.TOP_N)
+    return tuple(hit.passage for hit in hits)
+
+
+def build_items(embedder: retrieval.Embedder) -> list[Item]:
+    """SciFact train and dev pairs, then every readable AVeriTeC source, top 5 each.
+
+    Passages are split exactly as the product splits them: SciFact ships its
+    sentences, and a page goes through ``retrieval.split_sentences`` as in
+    ``eval_averitec._decide_claim``.
+    """
+    items: list[Item] = []
+    tarball = scifact.ensure_downloaded(cache_dir())
+    for split in ("train", "dev"):
+        data = scifact.load(tarball, split=split)
+        for pair in data.pairs():
+            doc = data.corpus[pair.doc_id]
+            passages = [Passage(t, str(doc.doc_id), i) for i, t in enumerate(doc.sentences)]
+            # Keys are not unique (dev repeats the pair 1245:7662395); consumers are positional.
+            key = f"scifact-{split}:{pair.claim_id}:{pair.doc_id}"
+            items.append(
+                Item(
+                    key,
+                    f"scifact-{split}",
+                    key,
+                    pair.claim,
+                    pair.label.value,
+                    _ranked(pair.claim, passages, embedder),
+                    pair.rationale,
+                )
+            )
+    items.extend(averitec_items(load_snapshot(snapshot_path()), embedder))
+    return items
+
+
+def averitec_items(snapshot: Sequence[SnapshotClaim], embedder: retrieval.Embedder) -> list[Item]:
+    """One item per readable source; the key carries the source's position in its claim."""
+    items: list[Item] = []
+    for claim in snapshot:
+        for position, source in enumerate(claim.sources):
+            sentences = retrieval.split_sentences(source.text) if source.text.strip() else []
+            if not sentences:
+                continue
+            passages = [Passage(s, source.url, i) for i, s in enumerate(sentences)]
+            items.append(
+                Item(
+                    f"averitec:{claim.claim_id}:{position}",
+                    "averitec",
+                    f"averitec:{claim.claim_id}",
+                    claim.claim,
+                    claim.gold,
+                    _ranked(claim.claim, passages, embedder),
+                )
+            )
+    return items
+
+
+def split_blocks(flat: np.ndarray, items: Sequence[Item]) -> list[np.ndarray]:
+    """Cut the flat ``(n_pairs, 3)`` scores into one block per item, in item order."""
+    offsets = passage_offsets(items)
+    if offsets[-1] != len(flat):
+        raise ValueError(f"{len(flat)} score rows for {offsets[-1]} passages")
+    return split_at(flat, offsets)
+
+
+def cmd_snapshot(args: argparse.Namespace) -> int:
+    from proofpath.config import load_config
+    from proofpath.verify import Engine
+
+    path = snapshot_path()
+    if path.exists() and not args.resume:
+        print(
+            f"error     {path} already holds a snapshot.\n"
+            "          Pass --resume to continue it, or delete the file to start over.",
+            file=sys.stderr,
+        )
+        return 2
+    claims = averitec.load(averitec.ensure_downloaded(cache_dir()), limit=args.limit or None)
+    stored = load_snapshot(path) if args.resume else []
+    todo = pending(claims, stored)
+    print(f"snapshot  {len(claims)} claims, {len(stored)} done, {len(todo)} to fetch")
+    # Browser not permitted and no prompts: the default install, as in the published
+    # AVeriTeC number (spec: Data).
+    engine = Engine.default(load_config(), interactive=False, browser=False)
+    try:
+        for position, claim in enumerate(todo, start=1):
+            sources = []
+            for url in claim.source_urls:
+                if args.sleep:
+                    time.sleep(args.sleep)
+                sources.append(source_from_fetched(url, engine.fetcher.fetch(url)))
+            stored.append(
+                SnapshotClaim(claim.id, claim.text, claim.label, tuple(sources), claim.non_urls)
+            )
+            save_snapshot(path, stored)
+            read = sum(bool(s.text) for s in sources)
+            print(
+                f"  {position}/{len(todo)}  claim {claim.id}  read {read}/{len(sources)}",
+                file=sys.stderr,
+            )
+    except KeyboardInterrupt:
+        print(f"\ninterrupted; {path} is usable with --resume", file=sys.stderr)
+        return 130
+    finally:
+        engine.close()
+    return 0
+
+
+def items_stale(items: Path, snapshot: Path) -> bool:
+    """True when the snapshot was written after ``items`` was built."""
+    return snapshot.stat().st_mtime > items.stat().st_mtime
+
+
+def _items(rebuild: bool) -> list[Item]:
+    if items_path().exists() and not rebuild:
+        if not items_stale(items_path(), snapshot_path()):
+            return load_items(items_path())
+        print("items     the snapshot is newer than items.json; rebuilding")
+    embedder = retrieval.FastEmbedder(EMBEDDER, cache_dir=models_dir())
+    try:
+        items = build_items(embedder)
+    finally:
+        embedder.close()
+    save_items(items_path(), items)
+    print(f"items     {len(items)} written to {items_path()}")
+    return items
+
+
+def cmd_score(args: argparse.Namespace) -> int:
+    from proofpath.entailment import OnnxNli
+
+    if not snapshot_path().exists():
+        print("error     no AVeriTeC snapshot; run `snapshot` first", file=sys.stderr)
+        return 2
+    items = _items(args.rebuild_items)
+    candidate = CANDIDATES[args.model]
+    scorer = OnnxNli(
+        cache_dir=models_dir(),
+        repo_id=candidate.repo,
+        revision=candidate.revision,
+        onnx_file=candidate.onnx_file(platform.machine()),
+        providers=nli_providers(),
+    )
+    print(f"nli       {scorer.name}  providers={scorer.providers}")
+    pairs = [(p.text, item.claim) for item in items for p in item.passages]
+    try:
+        started = time.perf_counter()
+        blocks: list[np.ndarray] = []
+        for start in range(0, len(pairs), SCORE_CHUNK):
+            blocks.append(scorer.score(pairs[start : start + SCORE_CHUNK]))
+            done = min(start + SCORE_CHUNK, len(pairs))
+            print(
+                f"  {done}/{len(pairs)} pairs  {time.perf_counter() - started:.0f}s",
+                file=sys.stderr,
+            )
+        elapsed = time.perf_counter() - started
+    finally:
+        scorer.close()
+    flat = np.concatenate(blocks) if blocks else np.zeros((0, 3), dtype=np.float32)
+    probs = split_blocks(flat, items)
+    ms_per_pair = elapsed / max(len(pairs), 1) * 1000
+    out = bakeoff_dir() / f"{args.model}.npz"
+    save_probs(out, items, probs, ms_per_pair=ms_per_pair)
+    print(f"scored    {len(pairs)} pairs, {ms_per_pair:.0f} ms/pair -> {out}")
+    return 0
+
+
+@dataclass(frozen=True)
+class ReportRow:
+    combo: bakeoff.Combo
+    large: bool
+    thresholds: Thresholds
+    scifact: bakeoff.SciFactResult
+    averitec: bakeoff.AveritecResult
+
+
+@dataclass(frozen=True)
+class NumericLayer:
+    """Numeric-layer firings at one k, as ``(fired, correct)``; model-independent."""
+
+    train: tuple[int, int]
+    dev: tuple[int, int]
+
+
+def outcomes_for(rows: Sequence[ReportRow]) -> list[bakeoff.Outcome]:
+    """The numbers the decision rule reads, one per report row."""
+    return [
+        bakeoff.Outcome(
+            r.combo,
+            r.large,
+            r.scifact.macro_f1,
+            r.averitec.accuracy_readable,
+            r.scifact.asserted_without_passage + r.averitec.asserted_without_passage,
+        )
+        for r in rows
+    ]
+
+
+def model_size_mb(model: str) -> float | None:
+    """Size of the ONNX file the run used, or None when it is not in the cache."""
+    from huggingface_hub import try_to_load_from_cache
+
+    candidate = CANDIDATES[model]
+    found = try_to_load_from_cache(
+        candidate.repo,
+        candidate.onnx_file(platform.machine()),
+        cache_dir=str(models_dir()),
+        revision=candidate.revision,
+    )
+    return Path(found).stat().st_size / 1e6 if isinstance(found, str) else None
+
+
+def _md(*cells: object) -> str:
+    return "| " + " | ".join(str(c) for c in cells) + " |"
+
+
+def _tier_cell(tier: bakeoff.TierStat) -> str:
+    if tier.precision is None or tier.interval is None:
+        return f"{tier.n} · too few to judge"
+    low, high = tier.interval
+    return f"{tier.n} · {tier.precision:.3f} (Wilson {low:.2f}–{high:.2f})"  # noqa: RUF001
+
+
+def _without_passage_cell(count: int) -> str:
+    return f"**{count} — rule 1 violated**" if count else "0"
+
+
+def on_grid_floor(decide: float) -> bool:
+    """True when ``decide`` is the lowest cut the sweep tried: the best may lie below."""
+    return decide == bakeoff.GRID[0]
+
+
+def _decide_cell(decide: float) -> str:
+    return f"{decide:.2f} (grid floor)" if on_grid_floor(decide) else f"{decide:.2f}"
+
+
+# The last AVeriTeC run before this one (docs/eval/2026-09-16-averitec.md): its snapshot,
+# its cuts and its denominators.
+PUBLISHED_AVERITEC = "2026-09-16"
+PUBLISHED_ANSWERABLE = 89
+PUBLISHED_READABLE = 61
+
+
+def _result_lines(rows: Sequence[ReportRow], choice: bakeoff.Choice) -> list[str]:
+    """The verdict of the rule, the winner named once, and its gain over the best base row."""
+    if choice.winner is None:
+        return [f"**No change.** {choice.reason}"]
+    label = choice.winner.label()
+    if choice.reason.startswith(label):
+        lines = [f"**Winner:** {choice.reason}."]
+    else:
+        lines = [f"**Winner:** {label} — {choice.reason}."]
+    outcomes = outcomes_for(rows)
+    winner = next(row for row in outcomes if row.combo == choice.winner)
+    best = bakeoff.best_eligible_base(outcomes, baseline=BASELINE)
+    if best is None:
+        lines.append("No `base` row is eligible under step 1, so the winner has no base bar.")
+    elif best.combo == winner.combo:
+        lines.append("The winner is itself the best eligible `base` row.")
+    else:
+        lines.append(
+            f"Over the best eligible `base` row, {best.combo.label()} "
+            f"({best.macro_f1:.3f}), the winner is {winner.macro_f1 - best.macro_f1:+.3f} "
+            "macro-F1."
+        )
+    return [lines[0], "", *lines[1:]]
+
+
+def _numeric_lines(rows: Sequence[ReportRow], numeric: Mapping[int, NumericLayer]) -> list[str]:
+    """Per-k firings, and how much of the `high` tier the rule rather than a model holds."""
+    lines = [
+        "## Numeric layer",
+        "",
+        "The numeric layer (spec §10) runs on the same top-k passages before any model. A "
+        "mismatch decides REFUTED by rule at score 1.0, and no model is credited with it. "
+        "It depends on k only, so one row per k covers every model and aggregation. A "
+        "firing is correct when the gold label is REFUTED.",
+        "",
+        "| k | SciFact train fired | train correct | SciFact dev fired | dev correct |",
+        "|---|---|---|---|---|",
+    ]
+    for k in sorted(numeric):
+        layer = numeric[k]
+        lines.append(_md(k, layer.train[0], layer.train[1], layer.dev[0], layer.dev[1]))
+    at_one = sum(row.thresholds.high >= 1.0 for row in rows)
+    rule_only = 0
+    for row in rows:
+        high = next(t for t in row.scifact.tiers if t.tier == "high")
+        fired = numeric[row.combo.k].dev[0] if row.combo.k in numeric else -1
+        rule_only += high.n > 0 and high.n == fired
+    lines.extend(
+        [
+            "",
+            "Rule-decided rows score exactly 1.0, the top of the scale, so they are among "
+            "the first rows the `high` cut's walk down SciFact train meets: a wrong firing "
+            "lowers the precision every lower cut is measured with, and a `high` cut of "
+            "1.000000 means no score below 1.0 held "
+            f"{bakeoff.HIGH_TARGET:.2f} precision, so only a score of 1.0 reaches the tier. "
+            f"The `high` cut is 1.000000 in {at_one} of {len(rows)} combinations. In "
+            f"{rule_only} of {len(rows)}, every SciFact dev `high` verdict is a rule firing "
+            "(the high-tier n equals the dev firings at that k): there the high tier on "
+            "dev comes from the rule, not the model.",
+        ]
+    )
+    return lines
+
+
+def _limits_lines(rows: Sequence[ReportRow], choice: bakeoff.Choice) -> list[str]:
+    outcomes = outcomes_for(rows)
+    top = max((row.macro_f1 for row in outcomes), default=0.0)
+    if choice.winner is None:
+        chosen = f"The best row's {top:.3f}"
+    else:
+        won = next(row for row in outcomes if row.combo == choice.winner).macro_f1
+        chosen = f"The winner's {won:.3f}"
+    base = next((row for row in rows if row.combo == BASELINE), None)
+    base_acc = f"dev accuracy {base.scifact.accuracy:.3f}" if base else "a lower dev accuracy"
+    counted = rows[0].averitec.counted if rows else 0
+    readable = rows[0].averitec.readable if rows else 0
+    floor = sum(on_grid_floor(row.thresholds.decide) for row in rows)
+    if floor:
+        floor_line = (
+            f"- **Grid floor.** {floor} of {len(rows)} combinations put `decide` on the grid "
+            f"floor ({bakeoff.GRID[0]:.2f}), marked `(grid floor)` above: SciFact train "
+            "accuracy was best at the lowest cut tried, so the best cut may lie below the grid."
+        )
+    else:
+        floor_line = (
+            f"- **Grid floor.** No combination puts `decide` on the grid floor "
+            f"({bakeoff.GRID[0]:.2f})."
+        )
+    return [
+        "## Limits",
+        "",
+        "- **Dev chooses and reports.** SciFact dev both picks the winner and reports it. "
+        f"{chosen} macro-F1 comes from the split that chose it, out of {len(rows)} "
+        f"combinations whose highest dev macro-F1 is {top:.3f}, so it is optimistic; a "
+        "held-out split would likely read lower.",
+        "- **int8 scores depend on the batch.** The candidate files are dynamically "
+        "quantised to int8, so a pair's probabilities depend on what else is in its batch. "
+        "Re-scoring 48 SciFact dev top-1 pairs one at a time moved probabilities by up to "
+        "0.197 (mean 0.008) and flipped 1 argmax (measured 2026-10-01 by the final review). "
+        f"That likely explains why `base` × k=1 × `max` reaches {base_acc} here "  # noqa: RUF001
+        "(0.597 even at decide 0.45) against the 0.609 published on 2026-09-12 with "
+        "dev-fitted cuts.",
+        "- **The AVeriTeC snapshot is new.** It is not the "
+        f"{PUBLISHED_AVERITEC} run's: coverage improved since (then {PUBLISHED_READABLE} of "
+        f"{PUBLISHED_ANSWERABLE} answerable claims were readable, now {readable} of "
+        f"{counted}), so AVeriTeC numbers here are not comparable with that run's.",
+        floor_line,
+    ]
+
+
+def render_report(
+    rows: Sequence[ReportRow],
+    choice: bakeoff.Choice,
+    *,
+    speed: Mapping[str, float],
+    sizes: Mapping[str, float | None],
+    counts: Mapping[str, int],
+    numeric: Mapping[int, NumericLayer],
+    today: str,
+    machine: str,
+) -> str:
+    scored = {row.combo.model for row in rows}
+    skipped = [m for m in CANDIDATES if m not in scored]
+    base_ave = rows[0].averitec if rows else None
+    counted = base_ave.counted if base_ave else 0
+    readable = base_ave.readable if base_ave else 0
+    baseline_ave = rows[0].averitec.majority_baseline if rows else 0.0
+    lines = [
+        f"# NLI bake-off — {today}",
+        "",
+        "- spec: `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`",
+        f"- calibrated on SciFact train ({counts.get('scifact-train', 0)} pairs); reported on "
+        f"SciFact dev ({counts.get('scifact-dev', 0)} pairs) and AVeriTeC dev "
+        f"({counts.get('averitec', 0)} claims, {counted} answerable (3-way gold label), "
+        f"{readable} of those with a readable source; frozen snapshot, browser not permitted)",
+        f"- embedder `{EMBEDDER}`, top {bakeoff.TOP_N} passages per item, numeric layer on",
+        f"- machine: {machine}",
+        "- coverage: "
+        + (f"not scored: {', '.join(skipped)}" if skipped else "all candidate models were scored"),
+        "",
+        "## Decision rule (fixed before any number was seen)",
+        "",
+        bakeoff.DECISION_RULE,
+        "",
+        "## Result",
+        "",
+    ]
+    lines.extend(_result_lines(rows, choice))
+    lines.extend(
+        ["", "## Models", "", "| model | repo | size (MB) | ms/pair |", "|---|---|---|---|"]
+    )
+    for model in dict.fromkeys(row.combo.model for row in rows):
+        size = sizes.get(model)
+        lines.append(
+            _md(
+                model,
+                f"`{CANDIDATES[model].repo}`",
+                "—" if size is None else f"{size:.0f}",
+                f"{speed.get(model, 0.0):.0f}",
+            )
+        )
+    lines.extend(
+        [
+            "",
+            "## Every combination",
+            "",
+            f"AVeriTeC majority baseline: {baseline_ave:.3f}. The {PUBLISHED_AVERITEC} run "
+            "reported 0.270 (all) and 0.361 (readable), but on a different snapshot with "
+            f"different cuts, over {PUBLISHED_ANSWERABLE} answerable / {PUBLISHED_READABLE} "
+            f"readable claims; this run counts {counted} answerable / {readable} readable. "
+            "A gap between the two runs is partly a coverage change, not only a model gain.",
+            "",
+            "| model | k | aggregation | decide | medium | high | dev acc | dev macro-F1 "
+            "| F1 S / R / NEI | rationale F1 | AVeriTeC all | AVeriTeC readable | w/o passage |",
+            "|---|---|---|---|---|---|---|---|---|---|---|---|---|",
+        ]
+    )
+    for row in rows:
+        f1 = row.scifact.f1
+        lines.append(
+            _md(
+                row.combo.model,
+                row.combo.k,
+                row.combo.aggregation,
+                _decide_cell(row.thresholds.decide),
+                f"{row.thresholds.medium:.6f}",
+                f"{row.thresholds.high:.6f}",
+                f"{row.scifact.accuracy:.3f}",
+                f"{row.scifact.macro_f1:.3f}",
+                f"{f1[Label.SUPPORTED]:.2f} / {f1[Label.REFUTED]:.2f} / {f1[Label.NEI]:.2f}",
+                f"{row.scifact.rationale_f1:.3f}",
+                f"{row.averitec.accuracy_all:.3f}",
+                f"{row.averitec.accuracy_readable:.3f}",
+                _without_passage_cell(
+                    row.scifact.asserted_without_passage + row.averitec.asserted_without_passage
+                ),
+            )
+        )
+    labels = sorted({label for row in rows for label in row.averitec.per_label})
+    lines.extend(
+        [
+            "",
+            "## AVeriTeC per label",
+            "",
+            "Cells are correct/n per gold label. `n (not scored)` marks a gold label with no "
+            "3-way verdict; it is not a model failure.",
+            "",
+            _md("model", "k", "aggregation", *labels),
+            _md(*(["---"] * (3 + len(labels)))),
+        ]
+    )
+    for row in rows:
+        cells = []
+        for label in labels:
+            seen, right = row.averitec.per_label.get(label, (0, 0))
+            cells.append(
+                f"{seen} (not scored)" if averitec.to_label(label) is None else f"{right}/{seen}"
+            )
+        lines.append(_md(row.combo.model, row.combo.k, row.combo.aggregation, *cells))
+    lines.extend(
+        [
+            "",
+            "## Tiers on SciFact dev",
+            "",
+            f"Cells are n · precision (95 % Wilson interval); n below {bakeoff.MIN_TIER_N} is "
+            "too few to judge.",
+            "",
+            "| model | k | aggregation | high | medium | low |",
+            "|---|---|---|---|---|---|",
+        ]
+    )
+    for row in rows:
+        tiers = {t.tier: t for t in row.scifact.tiers}
+        lines.append(
+            _md(
+                row.combo.model,
+                row.combo.k,
+                row.combo.aggregation,
+                *(_tier_cell(tiers[name]) for name in ("high", "medium", "low")),
+            )
+        )
+    lines.extend(["", *_numeric_lines(rows, numeric), "", *_limits_lines(rows, choice)])
+    return "\n".join(lines) + "\n"
+
+
+def cmd_report(args: argparse.Namespace) -> int:
+    if not items_path().exists() or not snapshot_path().exists():
+        print("error     run `snapshot` and `score --model base` first", file=sys.stderr)
+        return 2
+    if items_stale(items_path(), snapshot_path()):
+        print(
+            "error     the snapshot is newer than items.json, so the stored items are stale.\n"
+            "          The next `score` run rebuilds items.json automatically; after that, "
+            "re-score every model with `score --model <name>`.",
+            file=sys.stderr,
+        )
+        return 2
+    items = load_items(items_path())
+    snapshot = load_snapshot(snapshot_path())
+    rows: list[ReportRow] = []
+    speed: dict[str, float] = {}
+    numeric: dict[int, NumericLayer] = {}
+    for model, candidate in CANDIDATES.items():
+        path = bakeoff_dir() / f"{model}.npz"
+        if not path.exists():
+            print(f"skip      {model}: not scored", file=sys.stderr)
+            continue
+        try:
+            probs, speed[model] = load_probs(path, items)
+        except ValueError as error:
+            print(
+                f"error     {model}: stored scores do not match the current items ({error}); "
+                f"run `score --model {model}`",
+                file=sys.stderr,
+            )
+            return 2
+        scored = [bakeoff.Scored(i, p) for i, p in zip(items, probs, strict=True)]
+        train = [s for s in scored if s.item.dataset == "scifact-train"]
+        dev = [s for s in scored if s.item.dataset == "scifact-dev"]
+        ave = [s for s in scored if s.item.dataset == "averitec"]
+        for k in bakeoff.KS:
+            # The rule reads the items only, so every model gives the same firings.
+            if k not in numeric:
+                numeric[k] = NumericLayer(
+                    bakeoff.numeric_firings(train, k=k), bakeoff.numeric_firings(dev, k=k)
+                )
+            for aggregation in bakeoff.AGGREGATIONS:
+                thresholds = bakeoff.calibrate(
+                    bakeoff.calibration_rows(train, k=k, aggregation=aggregation)
+                )
+                rows.append(
+                    ReportRow(
+                        bakeoff.Combo(model, k, aggregation),
+                        candidate.large,
+                        thresholds,
+                        bakeoff.evaluate_scifact(dev, thresholds, k=k, aggregation=aggregation),
+                        bakeoff.evaluate_averitec(
+                            snapshot, ave, thresholds, k=k, aggregation=aggregation
+                        ),
+                    )
+                )
+    if "base" not in speed:
+        print(
+            "error     the baseline model is not scored; run `score --model base`", file=sys.stderr
+        )
+        return 2
+    choice = bakeoff.choose(outcomes_for(rows), baseline=BASELINE)
+    counts = {
+        "scifact-train": sum(i.dataset == "scifact-train" for i in items),
+        "scifact-dev": sum(i.dataset == "scifact-dev" for i in items),
+        "averitec": len(snapshot),
+    }
+    report = render_report(
+        rows,
+        choice,
+        speed=speed,
+        sizes={model: model_size_mb(model) for model in speed},
+        counts=counts,
+        numeric=numeric,
+        today=date.today().isoformat(),
+        machine=f"{platform.system()} {platform.machine()}",
+    )
+    out = (
+        Path(args.out)
+        if args.out
+        else Path("docs/eval") / f"{date.today().isoformat()}-nli-bakeoff.md"
+    )
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(report, encoding="utf-8")
+    print(report)
+    print(f"written   {out}")
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+    snap = sub.add_parser("snapshot", help="fetch AVeriTeC gold sources once (network)")
+    snap.add_argument("--limit", type=int, default=100, help="first N dev claims")
+    snap.add_argument("--sleep", type=float, default=1.0, help="seconds between fetches")
+    snap.add_argument("--resume", action="store_true", help="skip claims already stored")
+    score = sub.add_parser("score", help="run one NLI model over every stored pair")
+    score.add_argument("--model", required=True, choices=sorted(CANDIDATES))
+    score.add_argument("--rebuild-items", action="store_true", help="re-run retrieval")
+    report = sub.add_parser("report", help="calibrate, evaluate and choose (offline)")
+    report.add_argument("--out", default="")
+    args = parser.parse_args(argv)
+    commands = {"snapshot": cmd_snapshot, "score": cmd_score, "report": cmd_report}
+    return commands[args.command](args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/src/proofpath/eval/bakeoff.py b/src/proofpath/eval/bakeoff.py
new file mode 100644
index 0000000..575bb5e
--- /dev/null
+++ b/src/proofpath/eval/bakeoff.py
@@ -0,0 +1,525 @@
+"""NLI bake-off: aggregation variants, calibration, evaluation and the decision rule.
+
+Pure functions over stored NLI probabilities; nothing here touches the network or a
+model (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md). A variant moves into
+``pipeline.py`` only if the bake-off picks it.
+"""
+
+from __future__ import annotations
+
+import math
+from collections import Counter
+from collections.abc import Callable, Mapping, Sequence
+from dataclasses import dataclass
+
+import numpy as np
+
+from proofpath import numerics
+from proofpath.entailment import LABEL_ORDER
+from proofpath.eval import averitec, metrics
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+
+_S = LABEL_ORDER.index(Label.SUPPORTED)
+_R = LABEL_ORDER.index(Label.REFUTED)
+_N = LABEL_ORDER.index(Label.NEI)
+
+# Simplest first: this order is also the tie order of the decision rule.
+AGGREGATIONS: tuple[str, ...] = ("max", "max_nei", "margin")
+KS: tuple[int, ...] = (1, 2, 3)
+# Passages retrieved and scored per item; every k is a prefix of these.
+TOP_N = 5
+
+
+@dataclass(frozen=True)
+class Signal:
+    """The strongest assertion a combination proposes for one item, before ``decide``.
+
+    ``label`` is SUPPORTED or REFUTED, or NEI when the aggregation proposes nothing;
+    ``passage`` is what the assertion rests on.
+    """
+
+    label: Label
+    score: float
+    passage: Passage | None
+
+
+NOTHING = Signal(Label.NEI, 0.0, None)
+
+Aggregator = Callable[[Sequence[Passage], np.ndarray], Signal]
+
+
+def aggregate_max(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """Today's ``pipeline.aggregate``: the loudest SUPPORTED or REFUTED over the hits."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_max_nei(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """As ``max``, but a hit only counts when its assertion beats its own p(NEI)."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        neutral = float(row[_N])
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > neutral and value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_margin(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """max p(S) - max p(R): a supporting and a contradicting passage cancel out."""
+    if len(passages) != len(probs):
+        raise ValueError(f"{len(passages)} passages but {len(probs)} probability rows")
+    if len(passages) == 0:
+        return NOTHING
+    s_at = int(np.argmax(probs[:, _S]))
+    r_at = int(np.argmax(probs[:, _R]))
+    supported = float(probs[s_at, _S])
+    refuted = float(probs[r_at, _R])
+    if supported > refuted:
+        return Signal(Label.SUPPORTED, supported - refuted, passages[s_at])
+    if refuted > supported:
+        return Signal(Label.REFUTED, refuted - supported, passages[r_at])
+    return NOTHING
+
+
+AGGREGATORS: Mapping[str, Aggregator] = {
+    "max": aggregate_max,
+    "max_nei": aggregate_max_nei,
+    "margin": aggregate_margin,
+}
+
+
+def signal_for(
+    claim: str, passages: Sequence[Passage], probs: np.ndarray, *, k: int, aggregation: str
+) -> Signal:
+    """What one combination proposes for one item, numeric layer first (spec §10).
+
+    A numeric mismatch decides by rule at score 1.0, as ``pipeline.decide_indexed``
+    does, so no candidate model is credited with what the rule decided.
+    """
+    ruled = _rule_signal(claim, passages, k=k)
+    if ruled is not None:
+        return ruled
+    return AGGREGATORS[aggregation](passages[:k], probs[:k])
+
+
+def _rule_signal(claim: str, passages: Sequence[Passage], *, k: int) -> Signal | None:
+    """The numeric layer's REFUTED at score 1.0 over the first ``k`` hits, if it fires."""
+    if k < 1:
+        raise ValueError(f"k must be at least 1, got {k}")
+    numeric = numerics.check(claim, list(passages[:k]))
+    if numeric is not None and numeric.mismatch:
+        return Signal(Label.REFUTED, 1.0, numeric.passage)
+    return None
+
+
+def numeric_firings(scored: Sequence[Scored], *, k: int) -> tuple[int, int]:
+    """``(fired, correct)``: how often the numeric layer decides by rule at this k.
+
+    The same path as ``signal_for``, so these are exactly the verdicts scored 1.0 that
+    no model earned. A firing is correct when the gold label is REFUTED. It depends on
+    the items and k only, never on the model or the aggregation.
+    """
+    if k < 1:
+        raise ValueError(f"k must be at least 1, got {k}")
+    fired = correct = 0
+    for s in scored:
+        if _rule_signal(s.item.claim, s.item.passages, k=k) is not None:
+            fired += 1
+            correct += Label(s.item.gold) is Label.REFUTED
+    return fired, correct
+
+
+def decide(signal: Signal, thresholds: Thresholds) -> Label:
+    """The verdict a signal earns: below the cut, or with no passage, it is NEI."""
+    if signal.label is Label.NEI or signal.passage is None:
+        return Label.NEI
+    return signal.label if signal.score >= thresholds.decide else Label.NEI
+
+
+def wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]:
+    """95 % Wilson score interval for ``correct`` successes out of ``n``."""
+    if n <= 0:
+        raise ValueError("a Wilson interval needs at least one observation")
+    if not 0 <= correct <= n:
+        raise ValueError(f"correct must lie between 0 and n={n}, got {correct}")
+    p = correct / n
+    denominator = 1 + z * z / n
+    centre = (p + z * z / (2 * n)) / denominator
+    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator
+    return max(0.0, centre - half), min(1.0, centre + half)
+
+
+# Wider than eval_scifact's 0.30-0.95 grid: a ``margin`` score is a difference of two
+# probabilities and lives much nearer 0. Every combination is swept on the same grid.
+GRID: tuple[float, ...] = tuple(round(0.05 * i, 2) for i in range(1, 20))
+HIGH_TARGET = 0.85
+MEDIUM_TARGET = 0.70
+# Below this many dev verdicts a tier's precision is reported as "too few to judge".
+MIN_TIER_N = 20
+CUT_DECIMALS = 6
+
+
+@dataclass(frozen=True)
+class Scored:
+    item: Item
+    probs: np.ndarray  # (len(item.passages), 3) in LABEL_ORDER
+
+
+def _require(scored: Sequence[Scored], dataset: str) -> None:
+    wrong = sorted({s.item.dataset for s in scored if s.item.dataset != dataset})
+    if wrong:
+        raise ValueError(f"expected only {dataset} items, got {', '.join(wrong)}")
+
+
+def ship_cut(value: float) -> float:
+    """Round a cut up to ``CUT_DECIMALS`` — the rule of ``eval_scifact.ship_cut``.
+
+    Up, never down: a cut rounded down would hand a tier to scores the calibration
+    split never showed were that good.
+    """
+    scale: int = 10**CUT_DECIMALS
+    return min(1.0, math.ceil(value * scale) / scale)
+
+
+def calibration_rows(scored: Sequence[Scored], *, k: int, aggregation: str) -> list[metrics.Row]:
+    """``(gold, proposed label, score)`` rows for calibration, SciFact train only."""
+    _require(scored, "scifact-train")
+    rows: list[metrics.Row] = []
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        rows.append((Label(s.item.gold), signal.label, signal.score))
+    return rows
+
+
+def calibrate(rows: Sequence[metrics.Row]) -> Thresholds:
+    """Best-accuracy ``decide``, then the 0.85 / 0.70 precision cuts above it."""
+    best = metrics.sweep_decide(rows, grid=GRID)
+    raw_high, raw_medium = metrics.tier_cutpoints(
+        rows,
+        decide=best.threshold,
+        high_precision=HIGH_TARGET,
+        medium_precision=MEDIUM_TARGET,
+    )
+    high, medium = ship_cut(raw_high), ship_cut(raw_medium)
+    return Thresholds(decide=best.threshold, high=max(high, medium), medium=medium)
+
+
+@dataclass(frozen=True)
+class TierStat:
+    tier: str
+    n: int
+    correct: int
+
+    @property
+    def precision(self) -> float | None:
+        return self.correct / self.n if self.n >= MIN_TIER_N else None
+
+    @property
+    def interval(self) -> tuple[float, float] | None:
+        return wilson(self.correct, self.n) if self.n >= MIN_TIER_N else None
+
+
+@dataclass(frozen=True)
+class SciFactResult:
+    n: int
+    accuracy: float
+    macro_f1: float
+    f1: dict[Label, float]
+    rationale_f1: float
+    tiers: tuple[TierStat, ...]  # high, medium, low
+    asserted_without_passage: int
+
+
+def per_label_f1(gold: Sequence[Label], pred: Sequence[Label]) -> dict[Label, float]:
+    scores: dict[Label, float] = {}
+    for label in metrics.LABELS:
+        tp = sum(g is label and p is label for g, p in zip(gold, pred, strict=True))
+        fp = sum(g is not label and p is label for g, p in zip(gold, pred, strict=True))
+        fn = sum(g is label and p is not label for g, p in zip(gold, pred, strict=True))
+        scores[label] = 2 * tp / (2 * tp + fp + fn) if tp else 0.0
+    return scores
+
+
+def _unbacked(signal: Signal, thresholds: Thresholds) -> bool:
+    """An assertion over the cut with no passage under it (product rule 1)."""
+    return (
+        signal.label is not Label.NEI
+        and signal.score >= thresholds.decide
+        and signal.passage is None
+    )
+
+
+def evaluate_scifact(
+    scored: Sequence[Scored], thresholds: Thresholds, *, k: int, aggregation: str
+) -> SciFactResult:
+    _require(scored, "scifact-dev")
+    gold: list[Label] = []
+    pred: list[Label] = []
+    gold_rationale: list[frozenset[int]] = []
+    pred_rationale: list[frozenset[int]] = []
+    tiers = {"high": [0, 0], "medium": [0, 0], "low": [0, 0]}
+    unbacked = 0
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        unbacked += _unbacked(signal, thresholds)
+        label = decide(signal, thresholds)
+        truth = Label(s.item.gold)
+        gold.append(truth)
+        pred.append(label)
+        gold_rationale.append(s.item.rationale)
+        if label is not Label.NEI and signal.passage is not None:
+            pred_rationale.append(frozenset({signal.passage.index}))
+            counts = tiers[thresholds.tier(signal.score)]
+            counts[0] += 1
+            counts[1] += label is truth
+        else:
+            pred_rationale.append(frozenset())
+    return SciFactResult(
+        n=len(scored),
+        accuracy=metrics.accuracy(gold, pred),
+        macro_f1=metrics.macro_f1(gold, pred),
+        f1=per_label_f1(gold, pred),
+        rationale_f1=metrics.rationale_f1(gold_rationale, pred_rationale),
+        tiers=tuple(TierStat(name, n, c) for name, (n, c) in tiers.items()),
+        asserted_without_passage=unbacked,
+    )
+
+
+@dataclass(frozen=True)
+class AveritecResult:
+    n: int
+    counted: int  # claims with a 3-way gold label
+    accuracy_all: float
+    readable: int  # counted claims with a source whose snapshot text is not blank
+    accuracy_readable: float
+    majority_baseline: float
+    per_label: dict[str, tuple[int, int]]  # gold label -> (n, correct)
+    asserted_without_passage: int
+
+
+def evaluate_averitec(
+    claims: Sequence[SnapshotClaim],
+    scored: Sequence[Scored],
+    thresholds: Thresholds,
+    *,
+    k: int,
+    aggregation: str,
+) -> AveritecResult:
+    """Per claim, the rule of ``eval_averitec._decide_claim``.
+
+    The strongest decided non-NEI verdict across the claim's sources wins. A claim
+    with a readable source and no assertion is NEI. A claim with nothing readable is
+    unanswered, and is scored as NEI.
+
+    A claim is readable when any of its snapshot sources has non-blank text, as
+    ``_decide_claim``'s ``fetched_any`` has it, whether or not scored items exist.
+    Scored items whose group matches no snapshot claim are refused: a key-format drift
+    must not silently turn every claim into NEI (product rule 6).
+    """
+    _require(scored, "averitec")
+    by_group: dict[str, list[Scored]] = {}
+    for s in scored:
+        by_group.setdefault(s.item.group, []).append(s)
+    known = {f"averitec:{claim.claim_id}" for claim in claims}
+    orphans = sorted(set(by_group) - known)
+    if orphans:
+        raise ValueError(
+            f"{len(orphans)} scored groups match no snapshot claim: {', '.join(orphans[:5])}"
+        )
+    per_label: dict[str, tuple[int, int]] = {}
+    golds: list[Label] = []
+    correct_all = readable = correct_readable = unbacked = 0
+    for claim in claims:
+        group = by_group.get(f"averitec:{claim.claim_id}", [])
+        best: Signal | None = None
+        for s in group:
+            signal = signal_for(
+                s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation
+            )
+            unbacked += _unbacked(signal, thresholds)
+            if decide(signal, thresholds) is Label.NEI:
+                continue
+            if best is None or signal.score > best.score:
+                best = signal
+        predicted = best.label if best is not None else Label.NEI
+        gold = averitec.to_label(claim.gold)
+        correct = gold is not None and predicted is gold
+        seen, right = per_label.get(claim.gold, (0, 0))
+        per_label[claim.gold] = (seen + 1, right + int(correct))
+        if gold is None:
+            continue
+        golds.append(gold)
+        correct_all += correct
+        if any(source.text.strip() for source in claim.sources):
+            readable += 1
+            correct_readable += correct
+    majority = max(Counter(golds).values(), default=0)
+    return AveritecResult(
+        n=len(claims),
+        counted=len(golds),
+        accuracy_all=correct_all / len(golds) if golds else 0.0,
+        readable=readable,
+        accuracy_readable=correct_readable / readable if readable else 0.0,
+        majority_baseline=majority / len(golds) if golds else 0.0,
+        per_label=per_label,
+        asserted_without_passage=unbacked,
+    )
+
+
+TIE_MARGIN = 0.01
+LARGE_MARGIN = 0.03
+NO_CHANGE_MARGIN = 0.01
+# Absorbs float representation error only (0.627 - 0.597 is 0.02999999...), not display
+# rounding: a gap shown as 0.0295 must still fail a 0.03 margin.
+_EPS = 1e-9
+
+DECISION_RULE = """\
+The steps run in this order. Each one filters the rows the previous step left.
+
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Beats the baseline:** a row stays only if its SciFact dev macro-F1 beats
+   `base` × k=1 × `max` by **≥ 0.01**. If no row stays, the result is "no change". It is
+   written up as such, and the next package (B, coverage) starts.
+3. **Size gate:** a `large*` row stays only if it beats the highest macro-F1 of any
+   eligible `base` row by **≥ 0.03**. That is the strongest eligible `base` row, not the
+   one the tie rule would pick.
+4. **Winner:** the highest SciFact dev macro-F1 among the rows left.
+5. **Tie:** rows left within 0.01 of the winner go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`."""  # noqa: RUF001
+
+
+@dataclass(frozen=True)
+class Combo:
+    model: str
+    k: int
+    aggregation: str
+
+    def label(self) -> str:
+        return f"`{self.model}` × k={self.k} × `{self.aggregation}`"  # noqa: RUF001
+
+
+@dataclass(frozen=True)
+class Outcome:
+    combo: Combo
+    large: bool
+    macro_f1: float  # SciFact dev
+    averitec_readable: float
+    asserted_without_passage: int
+
+
+@dataclass(frozen=True)
+class Choice:
+    winner: Combo | None  # None means "no change"
+    reason: str
+
+
+def choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
+    """Apply ``DECISION_RULE`` step by step; each step filters the previous step's rows.
+
+    When the baseline row itself fails step 1, the reason says so: every margin is still
+    measured against its numbers, and a reader must not take it for a clean baseline.
+    """
+    choice = _choose(outcomes, baseline=baseline)
+    base = next(row for row in outcomes if row.combo == baseline)
+    if base.asserted_without_passage:
+        return Choice(
+            choice.winner,
+            "the baseline itself is not eligible: it asserts without a passage "
+            f"{base.asserted_without_passage} times; {choice.reason}",
+        )
+    return choice
+
+
+def _choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
+    by_combo = {row.combo: row for row in outcomes}
+    if len(by_combo) != len(outcomes):
+        duplicated = sorted(
+            combo.label() for combo, n in Counter(row.combo for row in outcomes).items() if n > 1
+        )
+        raise ValueError(f"duplicate combinations among the outcomes: {', '.join(duplicated)}")
+    if baseline not in by_combo:
+        raise ValueError(f"the baseline {baseline.label()} is not among the outcomes")
+    base = by_combo[baseline]
+
+    # Step 1: eligible.
+    eligible = [
+        row
+        for row in outcomes
+        if row.asserted_without_passage == 0
+        and row.averitec_readable >= base.averitec_readable - _EPS
+    ]
+    if not eligible:
+        return Choice(None, "no change: no row is eligible under step 1")
+
+    # Step 2: beats the baseline.
+    beating = [row for row in eligible if row.macro_f1 - base.macro_f1 >= NO_CHANGE_MARGIN - _EPS]
+    if not beating:
+        best = max(eligible, key=lambda row: row.macro_f1)
+        gain = best.macro_f1 - base.macro_f1
+        return Choice(
+            None,
+            f"no change: the best eligible row, {best.combo.label()}, is {gain:+.3f} "
+            f"macro-F1 over the baseline, below {NO_CHANGE_MARGIN}",
+        )
+
+    # Step 3: size gate, measured against the best eligible base row (step 1 rows).
+    best_base = max((row.macro_f1 for row in eligible if not row.large), default=base.macro_f1)
+    left = [
+        row for row in beating if not row.large or row.macro_f1 - best_base >= LARGE_MARGIN - _EPS
+    ]
+    if not left:
+        return Choice(
+            None,
+            "no change: the only rows that beat the baseline were large rows, and none "
+            f"beat the best eligible base row ({best_base:.3f}) by {LARGE_MARGIN}",
+        )
+
+    # Steps 4 and 5: highest macro-F1, then the tie rule.
+    top = max(left, key=lambda row: row.macro_f1)
+    contenders = [row for row in left if top.macro_f1 - row.macro_f1 <= TIE_MARGIN + _EPS]
+    winner = min(
+        contenders,
+        key=lambda row: (
+            row.combo.k,
+            AGGREGATIONS.index(row.combo.aggregation),
+            -row.macro_f1,
+        ),
+    )
+    gain = winner.macro_f1 - base.macro_f1
+    reason = f"{winner.combo.label()} is {gain:+.3f} macro-F1 over {baseline.label()}"
+    if winner is not top:
+        reason += (
+            f"; the tie rule chose it over {top.combo.label()}, "
+            f"the top macro-F1 row ({top.macro_f1:.3f})"
+        )
+    return Choice(winner.combo, reason)
+
+
+def best_eligible_base(outcomes: Sequence[Outcome], *, baseline: Combo) -> Outcome | None:
+    """The highest-macro-F1 non-large row that passes step 1, or None when none does.
+
+    Step 1 as ``choose`` applies it: no assertion without a passage, and AVeriTeC
+    readable accuracy not below the baseline's. The report uses it to state a winner's
+    gain over the strongest eligible ``base`` row, the bar of the size gate.
+    """
+    base = next((row for row in outcomes if row.combo == baseline), None)
+    if base is None:
+        raise ValueError(f"the baseline {baseline.label()} is not among the outcomes")
+    eligible = [
+        row
+        for row in outcomes
+        if not row.large
+        and row.asserted_without_passage == 0
+        and row.averitec_readable >= base.averitec_readable - _EPS
+    ]
+    return max(eligible, key=lambda row: row.macro_f1, default=None)
diff --git a/src/proofpath/eval/bakeoff_data.py b/src/proofpath/eval/bakeoff_data.py
new file mode 100644
index 0000000..c64ca21
--- /dev/null
+++ b/src/proofpath/eval/bakeoff_data.py
@@ -0,0 +1,232 @@
+"""Stored inputs of the NLI bake-off (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Three files under ``<cache>/datasets``, written once and read by every later step:
+the frozen AVeriTeC snapshot (page text per source URL), the retrieved items (top
+passages per claim, shared by every model), and one npz of raw NLI probabilities per
+model. Nothing here touches the network or a model.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from proofpath.eval import averitec
+from proofpath.fetch import Fetched
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+DATASETS: tuple[str, ...] = ("scifact-train", "scifact-dev", "averitec")
+
+
+@dataclass(frozen=True)
+class SnapshotSource:
+    """One source URL as the snapshot run found it. ``text`` is "" unless it was read."""
+
+    url: str
+    state: str
+    text: str
+
+
+@dataclass(frozen=True)
+class SnapshotClaim:
+    claim_id: int
+    claim: str
+    gold: str  # the AVeriTeC label, verbatim
+    sources: tuple[SnapshotSource, ...]
+    non_urls: int
+
+
+@dataclass(frozen=True)
+class Item:
+    """One (claim, source) unit with its top retrieved passages, best first.
+
+    ``group`` is the claim an item belongs to: a SciFact pair is its own group, an
+    AVeriTeC claim groups every readable source it has.
+    """
+
+    key: str
+    dataset: str
+    group: str
+    claim: str
+    gold: str
+    passages: tuple[Passage, ...]
+    rationale: frozenset[int] = frozenset()
+
+    def __post_init__(self) -> None:
+        if self.dataset not in DATASETS:
+            raise ValueError(f"unknown dataset {self.dataset!r}")
+
+
+def source_from_fetched(url: str, fetched: Fetched) -> SnapshotSource:
+    """The state and text one fetch earned, worded as ``eval_averitec.state_for`` does.
+
+    A page that was reached but yielded no text is not ``ok`` (product rule 6), and
+    only a read page keeps its text: a challenge page's words are not the source.
+    """
+    if fetched.ok and fetched.text.strip():
+        return SnapshotSource(url, fetched.outcome.value, fetched.text)
+    if fetched.ok:
+        return SnapshotSource(url, NO_TEXT, "")
+    return SnapshotSource(url, fetched.outcome.value, "")
+
+
+def pending(
+    claims: Sequence[averitec.Claim], done: Sequence[SnapshotClaim]
+) -> list[averitec.Claim]:
+    """The claims a resumed snapshot run still has to fetch, in dataset order."""
+    finished = {claim.claim_id for claim in done}
+    return [claim for claim in claims if claim.id not in finished]
+
+
+def _write_atomic(path: Path, payload: str) -> None:
+    """Write a whole file at once, so a Ctrl-C never leaves half of one."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    temporary.write_text(payload, encoding="utf-8")
+    temporary.replace(path)
+
+
+def save_snapshot(path: Path, claims: Sequence[SnapshotClaim]) -> None:
+    payload = [
+        {
+            "claim_id": claim.claim_id,
+            "claim": claim.claim,
+            "gold": claim.gold,
+            "non_urls": claim.non_urls,
+            "sources": [{"url": s.url, "state": s.state, "text": s.text} for s in claim.sources],
+        }
+        for claim in claims
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_snapshot(path: Path) -> list[SnapshotClaim]:
+    if not path.exists():
+        return []
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        SnapshotClaim(
+            claim_id=int(row["claim_id"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            sources=tuple(
+                SnapshotSource(str(s["url"]), str(s["state"]), str(s["text"]))
+                for s in row["sources"]
+            ),
+            non_urls=int(row["non_urls"]),
+        )
+        for row in raw
+    ]
+
+
+def save_items(path: Path, items: Sequence[Item]) -> None:
+    payload = [
+        {
+            "key": item.key,
+            "dataset": item.dataset,
+            "group": item.group,
+            "claim": item.claim,
+            "gold": item.gold,
+            "passages": [[p.text, p.source_id, p.index] for p in item.passages],
+            "rationale": sorted(item.rationale),
+        }
+        for item in items
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_items(path: Path) -> list[Item]:
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        Item(
+            key=str(row["key"]),
+            dataset=str(row["dataset"]),
+            group=str(row["group"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            passages=tuple(Passage(str(t), str(s), int(i)) for t, s, i in row["passages"]),
+            rationale=frozenset(int(i) for i in row["rationale"]),
+        )
+        for row in raw
+    ]
+
+
+def items_fingerprint(items: Sequence[Item]) -> str:
+    """A digest of what was scored: keys, claims and passage texts, in order.
+
+    Stored beside the probabilities so a rebuilt item file can never be paired with
+    probabilities computed on different passages.
+    """
+    digest = hashlib.sha256()
+    for item in items:
+        digest.update(item.key.encode("utf-8") + b"\0" + item.claim.encode("utf-8") + b"\0")
+        for passage in item.passages:
+            digest.update(passage.text.encode("utf-8") + b"\0")
+    return digest.hexdigest()
+
+
+def passage_offsets(items: Sequence[Item]) -> np.ndarray:
+    """Where each item's rows start in the flat ``(n_pairs, 3)`` array, plus the end."""
+    return np.cumsum([0, *(len(item.passages) for item in items)], dtype=np.int64)
+
+
+def split_at(flat: np.ndarray, offsets: np.ndarray) -> list[np.ndarray]:
+    """Cut ``flat`` into one block per pair of consecutive offsets. No validation."""
+    return [flat[offsets[n] : offsets[n + 1]] for n in range(len(offsets) - 1)]
+
+
+def save_probs(
+    path: Path, items: Sequence[Item], probs: Sequence[np.ndarray], *, ms_per_pair: float
+) -> None:
+    """Store one model's ``(n_passages, 3)`` probabilities per item, flattened."""
+    if len(items) != len(probs):
+        raise ValueError("one probability block per item is required")
+    for item, block in zip(items, probs, strict=True):
+        if np.asarray(block).shape != (len(item.passages), 3):
+            raise ValueError(f"probability block for {item.key} has the wrong shape")
+    offsets = passage_offsets(items)
+    flat = (
+        np.concatenate([np.asarray(b, dtype=np.float32) for b in probs])
+        if probs
+        else np.zeros((0, 3), dtype=np.float32)
+    )
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    with temporary.open("wb") as handle:
+        np.savez(
+            handle,
+            fingerprint=np.array(items_fingerprint(items)),
+            offsets=offsets,
+            probs=flat,
+            ms_per_pair=np.array(ms_per_pair, dtype=np.float64),
+        )
+    temporary.replace(path)
+
+
+def load_probs(path: Path, items: Sequence[Item]) -> tuple[list[np.ndarray], float]:
+    """The stored probabilities, one block per item, and the measured ms per pair."""
+    with np.load(path, allow_pickle=False) as data:
+        fingerprint = str(data["fingerprint"])
+        offsets = data["offsets"]
+        flat = data["probs"]
+        ms_per_pair = float(data["ms_per_pair"])
+    if fingerprint != items_fingerprint(items):
+        raise ValueError(f"{path.name} was scored on other items; run `score` again")
+    if len(offsets) != len(items) + 1:
+        raise ValueError(
+            f"{path.name} holds {len(offsets) - 1} blocks for {len(items)} items; run `score` again"
+        )
+    blocks = split_at(flat, offsets)
+    for item, block in zip(items, blocks, strict=True):
+        if len(block) != len(item.passages):
+            raise ValueError(
+                f"{path.name} holds {len(block)} rows for {item.key}, which has "
+                f"{len(item.passages)} passages; run `score` again"
+            )
+    return blocks, ms_per_pair
diff --git a/tests/test_bakeoff.py b/tests/test_bakeoff.py
new file mode 100644
index 0000000..17234db
--- /dev/null
+++ b/tests/test_bakeoff.py
@@ -0,0 +1,501 @@
+"""Offline tests for the bake-off's aggregation, calibration and decision rule."""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+from proofpath import pipeline
+from proofpath.eval import bakeoff
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim, SnapshotSource
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+from proofpath.retrieval import Hit
+
+# Rows are (SUPPORTED, REFUTED, NEI), the order of entailment.LABEL_ORDER.
+P = [Passage(f"passage {i}", "doc", i) for i in range(3)]
+
+
+def _probs(*rows: tuple[float, float, float]) -> np.ndarray:
+    return np.array(rows, dtype=np.float32)
+
+
+def test_max_takes_the_loudest_assertion_over_the_hits() -> None:
+    probs = _probs((0.6, 0.1, 0.3), (0.1, 0.8, 0.1))
+    signal = bakeoff.aggregate_max(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.REFUTED, P[1])
+    assert signal.score == pytest.approx(0.8)
+
+
+def test_max_nei_ignores_an_assertion_its_own_hit_calls_neutral() -> None:
+    probs = _probs((0.45, 0.05, 0.50), (0.40, 0.05, 0.30))
+    signal = bakeoff.aggregate_max_nei(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[1])
+    assert signal.score == pytest.approx(0.40)
+
+
+def test_max_nei_proposes_nothing_when_every_hit_is_neutral() -> None:
+    probs = _probs((0.2, 0.1, 0.7))
+    assert bakeoff.aggregate_max_nei(P[:1], probs) == bakeoff.NOTHING
+
+
+def test_margin_lets_a_contradicting_passage_cancel_a_supporting_one() -> None:
+    probs = _probs((0.90, 0.05, 0.05), (0.05, 0.85, 0.10))
+    signal = bakeoff.aggregate_margin(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[0])
+    assert signal.score == pytest.approx(0.05)
+
+
+def test_margin_with_an_exact_tie_proposes_nothing() -> None:
+    probs = _probs((0.5, 0.0, 0.5), (0.0, 0.5, 0.5))
+    assert bakeoff.aggregate_margin(P[:2], probs) == bakeoff.NOTHING
+
+
+@pytest.mark.parametrize("name", bakeoff.AGGREGATIONS)
+def test_no_hits_propose_nothing(name: str) -> None:
+    assert bakeoff.AGGREGATORS[name]([], np.zeros((0, 3), np.float32)) == bakeoff.NOTHING
+
+
+def test_signal_for_reads_only_the_first_k_hits() -> None:
+    probs = _probs((0.2, 0.1, 0.7), (0.95, 0.0, 0.05))
+    first = bakeoff.signal_for("a claim", P[:2], probs, k=1, aggregation="max")
+    assert first.passage == P[0]
+    both = bakeoff.signal_for("a claim", P[:2], probs, k=2, aggregation="max")
+    assert both.passage == P[1]
+
+
+def test_a_numeric_mismatch_decides_before_any_aggregation() -> None:
+    passages = [Passage("Mortality fell by 12% in the treated group.", "doc", 0)]
+    probs = _probs((0.99, 0.0, 0.01))
+    signal = bakeoff.signal_for(
+        "Mortality fell by 40% in the treated group.", passages, probs, k=1, aggregation="max"
+    )
+    assert signal == bakeoff.Signal(Label.REFUTED, 1.0, passages[0])
+
+
+def _numeric_item(key: str, dataset: str, gold: str, passage: str) -> bakeoff.Scored:
+    passages = (Passage(passage, key, 0), Passage("An unrelated sentence.", key, 1))
+    item = Item(key, dataset, key, "Mortality fell by 40% in the treated group.", gold, passages)
+    return bakeoff.Scored(item, _probs((0.99, 0.0, 0.01), (0.99, 0.0, 0.01)))
+
+
+def test_numeric_firings_count_rule_verdicts_and_the_refuted_ones_right() -> None:
+    scored = [
+        _numeric_item(
+            "a", "scifact-train", "REFUTED", "Mortality fell by 12% in the treated group."
+        ),
+        _numeric_item(
+            "b", "scifact-train", "SUPPORTED", "Mortality fell by 9% in the treated group."
+        ),
+        _numeric_item("c", "scifact-train", "SUPPORTED", "No figures here."),
+    ]
+    assert bakeoff.numeric_firings(scored, k=1) == (2, 1)
+
+
+def test_numeric_firings_agree_with_signal_for_at_every_k() -> None:
+    late = Item(
+        "d",
+        "scifact-dev",
+        "d",
+        "Mortality fell by 40% in the treated group.",
+        "REFUTED",
+        (Passage("No figures here.", "d", 0), Passage("Mortality fell by 12%.", "d", 1)),
+    )
+    scored = [bakeoff.Scored(late, _probs((0.2, 0.1, 0.7), (0.2, 0.1, 0.7)))]
+    for k in (1, 2):
+        ruled = sum(
+            bakeoff.signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation="max")
+            == bakeoff.Signal(Label.REFUTED, 1.0, s.item.passages[1])
+            for s in scored
+        )
+        assert bakeoff.numeric_firings(scored, k=k) == (ruled, ruled)
+    assert bakeoff.numeric_firings(scored, k=1) == (0, 0)
+    assert bakeoff.numeric_firings(scored, k=2) == (1, 1)
+
+
+@pytest.mark.parametrize("k", [0, -1])
+def test_numeric_firings_refuse_a_k_below_one(k: int) -> None:
+    with pytest.raises(ValueError, match="at least 1"):
+        bakeoff.numeric_firings([], k=k)
+
+
+def test_decide_needs_the_cut_and_a_passage() -> None:
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.6, P[0]), cuts) is Label.SUPPORTED
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.4, P[0]), cuts) is Label.NEI
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.9, None), cuts) is Label.NEI
+
+
+def test_wilson_interval_matches_known_values() -> None:
+    low, high = bakeoff.wilson(0, 10)
+    assert low == pytest.approx(0.0)
+    assert high == pytest.approx(0.2775, abs=1e-4)
+    low, high = bakeoff.wilson(10, 10)
+    assert low == pytest.approx(0.7225, abs=1e-4)
+    assert high == pytest.approx(1.0)
+
+
+def test_wilson_refuses_an_empty_sample() -> None:
+    with pytest.raises(ValueError):
+        bakeoff.wilson(0, 0)
+
+
+@pytest.mark.parametrize(("correct", "n"), [(-1, 10), (11, 10)])
+def test_wilson_refuses_a_count_outside_the_sample(correct: int, n: int) -> None:
+    with pytest.raises(ValueError, match="between 0 and n"):
+        bakeoff.wilson(correct, n)
+
+
+@pytest.mark.parametrize("k", [0, -1])
+def test_signal_for_refuses_a_k_below_one(k: int) -> None:
+    with pytest.raises(ValueError, match="k must be at least 1"):
+        bakeoff.signal_for("a claim", P[:1], _probs((0.9, 0.0, 0.1)), k=k, aggregation="max")
+
+
+def test_ship_cut_rounds_up_at_six_decimals_and_caps_at_one() -> None:
+    assert bakeoff.ship_cut(0.4579471) == pytest.approx(0.457948)
+    assert bakeoff.ship_cut(0.4579470) == pytest.approx(0.457947)
+    assert bakeoff.ship_cut(0.9999999) == 1.0
+    assert bakeoff.ship_cut(1.0) == 1.0
+
+
+@pytest.mark.parametrize("name", bakeoff.AGGREGATIONS)
+def test_aggregators_refuse_a_passage_and_row_count_mismatch(name: str) -> None:
+    probs = _probs((0.5, 0.2, 0.3), (0.1, 0.1, 0.8), (0.2, 0.7, 0.1))
+    with pytest.raises(ValueError):
+        bakeoff.AGGREGATORS[name](P[:2], probs)
+    with pytest.raises(ValueError):
+        bakeoff.AGGREGATORS[name](P[:3], probs[:2])
+
+
+def test_margin_reports_a_refuted_signal_when_the_contradiction_is_louder() -> None:
+    probs = _probs((0.10, 0.05, 0.85), (0.05, 0.80, 0.15))
+    signal = bakeoff.aggregate_margin(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.REFUTED, P[1])
+    assert signal.score == pytest.approx(0.70)
+
+
+@pytest.mark.parametrize(
+    "rows",
+    [
+        ((0.6, 0.1, 0.3), (0.1, 0.8, 0.1)),
+        ((0.7, 0.2, 0.1), (0.3, 0.7, 0.0)),  # tie between SUPPORTED and REFUTED
+        ((0.0, 0.0, 1.0), (0.0, 0.0, 1.0)),  # nothing asserted
+        ((0.4, 0.4, 0.2),),  # tie inside one row
+    ],
+)
+def test_max_agrees_with_the_pipeline_aggregate(
+    rows: tuple[tuple[float, float, float], ...],
+) -> None:
+    probs = _probs(*rows)
+    passages = P[: len(rows)]
+    hits = [Hit(passage, 0.5) for passage in passages]
+    cuts = Thresholds(decide=0.0, high=1.0, medium=1.0)
+    verdict = pipeline.aggregate(hits, probs, thresholds=cuts)
+    signal = bakeoff.aggregate_max(passages, probs)
+    assert (verdict.label, verdict.score, verdict.passage) == (
+        signal.label,
+        signal.score,
+        signal.passage,
+    )
+
+
+def test_max_matches_the_pipeline_when_nothing_is_asserted() -> None:
+    probs = _probs((0.0, 0.0, 1.0))
+    verdict = pipeline.aggregate(
+        [Hit(P[0], 0.5)], probs, thresholds=Thresholds(decide=0.0, high=1.0, medium=1.0)
+    )
+    assert verdict.label is Label.NEI and verdict.passage is None
+    assert bakeoff.aggregate_max(P[:1], probs) == bakeoff.NOTHING
+
+
+def test_decide_at_the_cut_asserts_and_keeps_a_refuted_signal() -> None:
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.5, P[0]), cuts) is Label.SUPPORTED
+    assert bakeoff.decide(bakeoff.Signal(Label.REFUTED, 0.5, P[0]), cuts) is Label.REFUTED
+    assert bakeoff.decide(bakeoff.Signal(Label.REFUTED, 0.49, P[0]), cuts) is Label.NEI
+    assert bakeoff.decide(bakeoff.NOTHING, cuts) is Label.NEI
+
+
+def _scored(
+    key: str,
+    dataset: str,
+    gold: str,
+    row: tuple[float, float, float],
+    *,
+    group: str | None = None,
+    rationale: frozenset[int] = frozenset({0}),
+) -> bakeoff.Scored:
+    item = Item(key, dataset, group or key, "a claim", gold, (Passage("p", key, 0),), rationale)
+    return bakeoff.Scored(item, _probs(row))
+
+
+def test_calibration_refuses_anything_but_scifact_train() -> None:
+    dev = [_scored("d", "scifact-dev", "SUPPORTED", (0.9, 0.0, 0.1))]
+    with pytest.raises(ValueError, match="scifact-train"):
+        bakeoff.calibration_rows(dev, k=1, aggregation="max")
+
+
+def test_evaluation_refuses_the_calibration_split() -> None:
+    train = [_scored("t", "scifact-train", "SUPPORTED", (0.9, 0.0, 0.1))]
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    with pytest.raises(ValueError, match="scifact-dev"):
+        bakeoff.evaluate_scifact(train, cuts, k=1, aggregation="max")
+
+
+def test_calibrate_picks_a_cut_that_separates_right_from_wrong() -> None:
+    rows = [(Label.SUPPORTED, Label.SUPPORTED, 0.9)] * 5 + [(Label.NEI, Label.SUPPORTED, 0.3)] * 5
+    cuts = bakeoff.calibrate(rows)
+    assert 0.3 < cuts.decide <= 0.9
+    assert cuts.decide <= cuts.medium <= cuts.high
+
+
+def test_tier_stats_hide_precision_below_the_minimum_n() -> None:
+    assert bakeoff.TierStat("high", 19, 19).precision is None
+    assert bakeoff.TierStat("high", 19, 19).interval is None
+    assert bakeoff.TierStat("high", 20, 15).precision == pytest.approx(0.75)
+
+
+def test_evaluate_scifact_counts_verdicts_tiers_and_rationale() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    scored = [
+        _scored("a", "scifact-dev", "SUPPORTED", (0.97, 0.01, 0.02)),  # high, right
+        _scored("b", "scifact-dev", "REFUTED", (0.80, 0.10, 0.10)),  # medium, wrong
+        _scored("c", "scifact-dev", "NEI", (0.30, 0.10, 0.60)),  # below decide
+    ]
+    result = bakeoff.evaluate_scifact(scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.accuracy == pytest.approx(2 / 3)
+    assert {t.tier: (t.n, t.correct) for t in result.tiers} == {
+        "high": (1, 1),
+        "medium": (1, 0),
+        "low": (0, 0),
+    }
+    assert result.asserted_without_passage == 0
+    # Gold {0} for all three; predicted {0}, {0}, {}: tp 2, fp 0, fn 1 -> P 1, R 2/3, F1 0.8.
+    assert result.rationale_f1 == pytest.approx(0.8)
+
+
+def test_evaluate_averitec_keeps_the_strongest_source_and_both_denominators() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [
+        SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u1", "ok", "t"),), 0),
+        SnapshotClaim(2, "c2", "Supported", (SnapshotSource("u2", "UNVERIFIED (x)", ""),), 0),
+        SnapshotClaim(3, "c3", "Conflicting Evidence/Cherrypicking", (), 0),
+    ]
+    scored = [
+        _scored("averitec:1:0", "averitec", "Refuted", (0.6, 0.1, 0.3), group="averitec:1"),
+        _scored("averitec:1:1", "averitec", "Refuted", (0.0, 0.9, 0.1), group="averitec:1"),
+    ]
+    result = bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.counted == 2  # the Conflicting claim has no 3-way label
+    assert result.accuracy_all == pytest.approx(0.5)  # claim 1 right, claim 2 unread → NEI
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(1.0)
+    assert result.majority_baseline == pytest.approx(0.5)
+    assert result.per_label == {
+        "Refuted": (1, 1),
+        "Supported": (1, 0),
+        "Conflicting Evidence/Cherrypicking": (1, 0),
+    }
+    assert result.asserted_without_passage == 0
+
+
+def test_a_readable_claim_whose_only_signal_is_below_the_cut_is_nei_and_readable() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [SnapshotClaim(1, "c1", "Not Enough Evidence", (SnapshotSource("u", "ok", "t"),), 0)]
+    scored = [_scored("averitec:1:0", "averitec", "x", (0.3, 0.1, 0.6), group="averitec:1")]
+    result = bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(1.0)
+    assert result.per_label == {"Not Enough Evidence": (1, 1)}
+
+
+def test_a_claim_with_readable_text_but_no_scored_items_is_still_readable() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u", "ok", "text"),), 0)]
+    result = bakeoff.evaluate_averitec(claims, [], cuts, k=1, aggregation="max")
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(0.0)
+
+
+def test_scored_items_matching_no_snapshot_claim_are_refused() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u", "ok", "t"),), 0)]
+    scored = [_scored("averitec:9:0", "averitec", "Refuted", (0.0, 0.9, 0.1), group="avt:9")]
+    with pytest.raises(ValueError, match="avt:9"):
+        bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
+
+
+BASE = bakeoff.Combo("base", 1, "max")
+
+
+def _o(
+    model: str, k: int, agg: str, f1: float, *, readable: float = 0.40, unbacked: int = 0
+) -> bakeoff.Outcome:
+    return bakeoff.Outcome(bakeoff.Combo(model, k, agg), model != "base", f1, readable, unbacked)
+
+
+def test_no_change_when_nothing_beats_the_baseline_by_a_hundredth() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.604)], baseline=BASE
+    )
+    assert choice.winner is None
+    assert "no change" in choice.reason
+
+
+def test_a_base_row_that_clears_the_margin_wins() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 3, "margin", 0.630)], baseline=BASE
+    )
+    assert choice.winner == bakeoff.Combo("base", 3, "margin")
+
+
+def test_a_large_row_needs_three_hundredths_over_the_best_base_row() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max_nei", 0.620),
+        _o("large", 1, "max", 0.645),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+    rows.append(_o("large-fever", 1, "max", 0.651))
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large-fever", 1, "max")
+
+
+def test_ties_go_to_the_smaller_k_then_the_simpler_aggregation() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.640),
+        _o("base", 2, "margin", 0.635),
+        _o("base", 2, "max_nei", 0.632),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+
+
+def test_a_row_that_asserts_without_a_passage_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.700, unbacked=1)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_a_row_worse_on_averitec_readable_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597, readable=0.40), _o("base", 2, "max", 0.700, readable=0.39)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_the_baseline_must_be_among_the_outcomes() -> None:
+    with pytest.raises(ValueError, match="baseline"):
+        bakeoff.choose([_o("base", 2, "max", 0.6)], baseline=BASE)
+
+
+def test_a_large_row_is_measured_against_the_best_base_row_not_the_tie_pick() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.640),
+        _o("base", 1, "max_nei", 0.631),
+        _o("large", 1, "max", 0.662),
+    ]
+    choice = bakeoff.choose(rows, baseline=BASE)
+    assert choice.winner == bakeoff.Combo("base", 1, "max_nei")
+    assert "k=3" in choice.reason  # names the top-F1 row it did not pick
+
+
+def test_a_large_row_that_clears_the_gate_over_the_best_base_row_wins() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max", 0.620),
+        _o("large", 1, "max", 0.645),
+        _o("large", 3, "max", 0.652),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large", 3, "max")
+
+
+def test_a_row_below_the_baseline_margin_is_dropped_before_the_tie_rule() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.612),
+        _o("base", 1, "max_nei", 0.605),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 3, "max")
+
+
+def test_a_large_row_exactly_three_hundredths_over_the_best_base_row_passes() -> None:
+    rows = [
+        _o("base", 1, "max", 0.500),
+        _o("base", 2, "max", 0.560),
+        _o("large", 1, "max", 0.590),
+    ]
+    assert 0.590 - 0.560 < 0.03  # float error: the nominal margin is not met exactly
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large", 1, "max")
+
+
+def test_a_row_exactly_one_hundredth_over_the_baseline_passes() -> None:
+    rows = [_o("base", 1, "max", 0.553), _o("base", 2, "max", 0.563)]
+    assert 0.563 - 0.553 < 0.01  # float error: the nominal margin is not met exactly
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max")
+
+
+def test_an_ineligible_baseline_does_not_stop_another_eligible_row_from_winning() -> None:
+    rows = [_o("base", 1, "max", 0.597, unbacked=1), _o("base", 2, "max", 0.620)]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max")
+
+
+def test_only_large_rows_beating_the_baseline_but_failing_the_size_gate_is_no_change() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max", 0.600),
+        _o("large", 1, "max", 0.620),
+    ]
+    choice = bakeoff.choose(rows, baseline=BASE)
+    assert choice.winner is None
+    assert choice.reason.startswith("no change:")
+    assert "large" in choice.reason
+
+
+def test_the_reason_says_so_when_the_baseline_itself_is_not_eligible() -> None:
+    won = bakeoff.choose(
+        [_o("base", 1, "max", 0.597, unbacked=2), _o("base", 2, "max", 0.620)], baseline=BASE
+    )
+    assert won.winner == bakeoff.Combo("base", 2, "max")
+    assert won.reason.startswith("the baseline itself is not eligible:")
+    unchanged = bakeoff.choose(
+        [_o("base", 1, "max", 0.597, unbacked=2), _o("base", 2, "max", 0.600)], baseline=BASE
+    )
+    assert unchanged.winner is None
+    assert unchanged.reason.startswith("the baseline itself is not eligible:")
+    assert "no change" in unchanged.reason
+
+
+def test_an_eligible_baseline_is_not_called_ineligible() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.620)], baseline=BASE
+    )
+    assert "not eligible" not in choice.reason
+
+
+def test_duplicate_combinations_among_the_outcomes_are_refused() -> None:
+    with pytest.raises(ValueError, match="duplicate"):
+        bakeoff.choose([_o("base", 1, "max", 0.597), _o("base", 1, "max", 0.600)], baseline=BASE)
+
+
+def test_best_eligible_base_skips_large_unbacked_and_averitec_worse_rows() -> None:
+    outcomes = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max", 0.610),
+        _o("base", 3, "max", 0.640, readable=0.30),
+        _o("base", 2, "margin", 0.645, unbacked=1),
+        _o("large", 1, "max", 0.700),
+    ]
+    best = bakeoff.best_eligible_base(outcomes, baseline=BASE)
+    assert best is not None
+    assert best.combo == bakeoff.Combo("base", 2, "max")
+
+
+def test_best_eligible_base_is_none_when_no_base_row_is_eligible() -> None:
+    outcomes = [_o("base", 1, "max", 0.597, unbacked=2), _o("large", 1, "max", 0.700)]
+    assert bakeoff.best_eligible_base(outcomes, baseline=BASE) is None
+
+
+def test_best_eligible_base_needs_the_baseline() -> None:
+    with pytest.raises(ValueError, match="not among the outcomes"):
+        bakeoff.best_eligible_base([_o("base", 2, "max", 0.6)], baseline=BASE)
diff --git a/tests/test_bakeoff_data.py b/tests/test_bakeoff_data.py
new file mode 100644
index 0000000..51f9ee1
--- /dev/null
+++ b/tests/test_bakeoff_data.py
@@ -0,0 +1,194 @@
+"""Offline tests for the bake-off's stored inputs (no network, no model)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from proofpath.eval import averitec
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    SnapshotSource,
+    items_fingerprint,
+    load_items,
+    load_probs,
+    load_snapshot,
+    passage_offsets,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+    split_at,
+)
+from proofpath.fetch import Fetched, Outcome
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+
+def _fetched(outcome: Outcome, text: str) -> Fetched:
+    return Fetched(
+        url="https://example.org/a",
+        final_url="https://example.org/a",
+        step=1,
+        outcome=outcome,
+        status=200 if outcome is Outcome.OK else 403,
+        content_type="text/html",
+        kind="html",
+        body=b"",
+        text=text,
+        notes=[],
+    )
+
+
+def _item(key: str, n: int = 2, dataset: str = "scifact-dev") -> Item:
+    passages = tuple(Passage(f"sentence {i} of {key}", "doc", i) for i in range(n))
+    return Item(key, dataset, key, "a claim", "SUPPORTED", passages, frozenset({0}))
+
+
+def test_a_read_page_keeps_its_text_and_ok_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "Some text."))
+    assert source == SnapshotSource("https://example.org/a", "ok", "Some text.")
+
+
+def test_a_reached_page_with_no_text_gets_the_no_text_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "   "))
+    assert source.state == NO_TEXT
+    assert source.text == ""
+
+
+def test_a_blocked_page_keeps_its_state_and_no_text() -> None:
+    blocked = _fetched(Outcome.BLOCKED_NO_BROWSER, "challenge page")
+    source = source_from_fetched("https://example.org/a", blocked)
+    assert source.state == Outcome.BLOCKED_NO_BROWSER.value
+    assert source.text == ""
+
+
+def test_snapshot_round_trips(tmp_path: Path) -> None:
+    claims = [
+        SnapshotClaim(
+            7, "c", "Refuted", (SnapshotSource("u", "ok", "t"), SnapshotSource("v", "x", "")), 1
+        )
+    ]
+    path = tmp_path / "snap.json"
+    save_snapshot(path, claims)
+    assert load_snapshot(path) == claims
+
+
+def test_a_missing_snapshot_loads_as_empty(tmp_path: Path) -> None:
+    assert load_snapshot(tmp_path / "absent.json") == []
+
+
+def test_pending_skips_claims_already_in_the_snapshot() -> None:
+    claims = [averitec.Claim(i, f"claim {i}", "Refuted", ("https://e.org",)) for i in range(3)]
+    done = [SnapshotClaim(1, "claim 1", "Refuted", (), 0)]
+    assert [c.id for c in pending(claims, done)] == [0, 2]
+
+
+def test_items_round_trip(tmp_path: Path) -> None:
+    items = [_item("a"), _item("b", 1, "averitec")]
+    path = tmp_path / "items.json"
+    save_items(path, items)
+    assert load_items(path) == items
+
+
+def test_an_unknown_dataset_is_refused() -> None:
+    with pytest.raises(ValueError, match="unknown dataset"):
+        _item("a", dataset="scifact-test")
+
+
+def test_probs_round_trip(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 3)]
+    probs = [np.full((2, 3), 0.25, np.float32), np.full((3, 3), 0.5, np.float32)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, probs, ms_per_pair=12.5)
+    loaded, ms = load_probs(path, items)
+    assert ms == pytest.approx(12.5)
+    assert [p.shape for p in loaded] == [(2, 3), (3, 3)]
+    assert np.allclose(loaded[1], 0.5)
+
+
+def test_probs_scored_on_other_items_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, [np.zeros((2, 3), np.float32)], ms_per_pair=1.0)
+    changed = [_item("z", 2)]
+    with pytest.raises(ValueError, match="other items"):
+        load_probs(path, changed)
+
+
+def test_probs_with_the_wrong_shape_are_refused(tmp_path: Path) -> None:
+    with pytest.raises(ValueError, match="shape"):
+        save_probs(tmp_path / "x.npz", [_item("a", 2)], [np.zeros((3, 3))], ms_per_pair=1.0)
+
+
+def test_the_fingerprint_changes_with_a_passage_text() -> None:
+    a = _item("a")
+    b = Item(a.key, a.dataset, a.group, a.claim, a.gold, (Passage("other", "doc", 0),))
+    assert items_fingerprint([a]) != items_fingerprint([b])
+
+
+def test_save_probs_refuses_a_block_count_that_differs_from_the_items(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 1)]
+    with pytest.raises(ValueError, match="one probability block per item"):
+        save_probs(tmp_path / "x.npz", items, [np.zeros((2, 3))], ms_per_pair=1.0)
+
+
+def test_an_empty_item_list_round_trips(tmp_path: Path) -> None:
+    path = tmp_path / "empty.npz"
+    save_probs(path, [], [], ms_per_pair=3.0)
+    loaded, ms = load_probs(path, [])
+    assert loaded == []
+    assert ms == pytest.approx(3.0)
+
+
+def test_the_fingerprint_changes_with_a_claim_or_a_key() -> None:
+    a = _item("a")
+    other_claim = Item(a.key, a.dataset, a.group, "another claim", a.gold, a.passages)
+    other_key = Item("z", a.dataset, a.group, a.claim, a.gold, a.passages)
+    assert items_fingerprint([a]) != items_fingerprint([other_claim])
+    assert items_fingerprint([a]) != items_fingerprint([other_key])
+
+
+def _save_with_offsets(path: Path, items: list[Item], offsets: list[int], rows: int) -> None:
+    np.savez(
+        path,
+        fingerprint=np.array(items_fingerprint(items)),
+        offsets=np.array(offsets, dtype=np.int64),
+        probs=np.zeros((rows, 3), dtype=np.float32),
+        ms_per_pair=np.array(1.0),
+    )
+
+
+def test_probs_with_the_wrong_number_of_offsets_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 1)]
+    path = tmp_path / "bad.npz"
+    _save_with_offsets(path, items, [0, 2], 2)
+    with pytest.raises(ValueError, match="blocks for 2 items"):
+        load_probs(path, items)
+
+
+def test_probs_whose_slice_does_not_match_the_passage_count_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 1)]
+    path = tmp_path / "bad.npz"
+    _save_with_offsets(path, items, [0, 1, 3], 3)
+    with pytest.raises(ValueError, match="rows for a"):
+        load_probs(path, items)
+
+
+def test_passage_offsets_start_at_zero_and_end_at_the_pair_count() -> None:
+    offsets = passage_offsets([_item("a", 2), _item("b", 0), _item("c", 3)])
+    assert offsets.tolist() == [0, 2, 2, 5]
+    assert offsets.dtype == np.int64
+    assert passage_offsets([]).tolist() == [0]
+
+
+def test_split_at_cuts_one_block_per_offset_pair() -> None:
+    flat = np.arange(15, dtype=np.float32).reshape(5, 3)
+    blocks = split_at(flat, np.array([0, 2, 2, 5]))
+    assert [b.shape for b in blocks] == [(2, 3), (0, 3), (3, 3)]
+    assert np.array_equal(blocks[2], flat[2:5])
+    assert split_at(np.zeros((0, 3)), np.array([0])) == []
diff --git a/tests/test_eval_nli_bakeoff.py b/tests/test_eval_nli_bakeoff.py
new file mode 100644
index 0000000..f063d6f
--- /dev/null
+++ b/tests/test_eval_nli_bakeoff.py
@@ -0,0 +1,493 @@
+"""Offline tests for scripts/eval_nli_bakeoff.py's importable parts."""
+
+from __future__ import annotations
+
+import importlib.util
+import os
+import sys
+from argparse import Namespace
+from collections.abc import Sequence
+from pathlib import Path
+from types import ModuleType
+
+import numpy as np
+import pytest
+
+from proofpath.eval import averitec, bakeoff
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim, SnapshotSource
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+
+SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "eval_nli_bakeoff.py"
+
+
+def _load_script() -> ModuleType:
+    spec = importlib.util.spec_from_file_location("eval_nli_bakeoff", SCRIPT_PATH)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+script = _load_script()
+
+
+def test_every_candidate_is_pinned_to_a_full_revision() -> None:
+    assert set(script.CANDIDATES) == {"base", "large", "large-fever"}
+    for candidate in script.CANDIDATES.values():
+        assert len(candidate.revision) == 40
+        int(candidate.revision, 16)
+
+
+def test_only_the_large_models_are_marked_large() -> None:
+    assert {name for name, c in script.CANDIDATES.items() if c.large} == {"large", "large-fever"}
+
+
+def test_each_candidate_names_an_onnx_file_per_machine() -> None:
+    for machine in ("arm64", "x86_64"):
+        for candidate in script.CANDIDATES.values():
+            assert candidate.onnx_file(machine).startswith("onnx/")
+            assert candidate.onnx_file(machine).endswith(".onnx")
+
+
+def test_the_baseline_is_base_k1_max() -> None:
+    assert (script.BASELINE.model, script.BASELINE.k, script.BASELINE.aggregation) == (
+        "base",
+        1,
+        "max",
+    )
+
+
+def test_coreml_is_never_handed_to_the_int8_models() -> None:
+    assert "CoreMLExecutionProvider" not in script.nli_providers()
+
+
+def _row(
+    model: str,
+    k: int,
+    agg: str,
+    f1: float,
+    tiers: tuple[int, int, int],
+    unbacked: int = 0,
+    *,
+    decide: float = 0.45,
+    high: float = 0.99933,
+    readable: float = 0.36,
+) -> object:
+    sci = bakeoff.SciFactResult(
+        n=340,
+        accuracy=f1 + 0.01,
+        macro_f1=f1,
+        f1={Label.SUPPORTED: 0.7, Label.REFUTED: 0.5, Label.NEI: 0.6},
+        rationale_f1=0.3,
+        tiers=(
+            bakeoff.TierStat("high", tiers[0], tiers[0]),
+            bakeoff.TierStat("medium", tiers[1], tiers[1] // 2),
+            bakeoff.TierStat("low", tiers[2], 0),
+        ),
+        asserted_without_passage=unbacked,
+    )
+    ave = bakeoff.AveritecResult(
+        n=100,
+        counted=89,
+        accuracy_all=0.27,
+        readable=61,
+        accuracy_readable=readable,
+        majority_baseline=0.708,
+        per_label={"Refuted": (63, 20), "Supported": (10, 7)},
+        asserted_without_passage=0,
+    )
+    return script.ReportRow(
+        bakeoff.Combo(model, k, agg),
+        model != "base",
+        Thresholds(decide=decide, high=high, medium=0.457948),
+        sci,
+        ave,
+    )
+
+
+NUMERIC = {
+    1: script.NumericLayer(train=(4, 0), dev=(1, 1)),
+    2: script.NumericLayer(train=(5, 0), dev=(1, 1)),
+}
+
+
+def _render(
+    choice: bakeoff.Choice,
+    unbacked: int = 0,
+    models: tuple[str, ...] = ("base", "large", "large-fever"),
+    rows: list[object] | None = None,
+) -> str:
+    if rows is None:
+        rows = [
+            _row("base", 1, "max", 0.597, (21, 113, 1)),
+            _row("large", 2, "margin", 0.650, (30, 40, 25), unbacked),
+            _row("large-fever", 1, "max", 0.600, (25, 50, 20)),
+        ]
+    rows = [r for r in rows if r.combo.model in models]
+    return script.render_report(
+        rows,
+        choice,
+        speed={"base": 185.0, "large": 540.0, "large-fever": 540.0},
+        sizes={"base": 244.4, "large": None},
+        counts={"scifact-train": 919, "scifact-dev": 340, "averitec": 100},
+        numeric=NUMERIC,
+        today="2026-10-02",
+        machine="Darwin arm64",
+    )
+
+
+def _section(text: str, heading: str) -> str:
+    """The body of one ``## heading`` section, up to the next one."""
+    body = text.split(f"## {heading}\n", 1)[1]
+    return body.split("\n## ", 1)[0]
+
+
+def test_the_winner_is_named_once_when_the_reason_starts_with_it() -> None:
+    winner = bakeoff.Combo("large", 2, "margin")
+    reason = f"{winner.label()} is +0.053 macro-F1 over {script.BASELINE.label()}"
+    text = _render(bakeoff.Choice(winner, reason))
+    result = _section(text, "Result")
+    assert result.count(winner.label()) == 1
+    assert f"**Winner:** {reason}." in result
+
+
+def test_a_reason_that_does_not_start_with_the_winner_keeps_the_label() -> None:
+    winner = bakeoff.Combo("large", 2, "margin")
+    text = _render(bakeoff.Choice(winner, "the baseline itself is not eligible: x"))
+    assert f"**Winner:** {winner.label()} — the baseline itself is not eligible: x." in text
+
+
+def test_the_result_states_the_gain_over_the_best_eligible_base_row() -> None:
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("base", 2, "max", 0.610, (1, 10, 10)),
+        # Higher macro-F1, but worse on AVeriTeC readable: not eligible.
+        _row("base", 3, "max", 0.640, (1, 10, 10), readable=0.30),
+        # Higher macro-F1, but asserts without a passage: not eligible.
+        _row("base", 2, "margin", 0.645, (1, 10, 10), unbacked=1),
+        _row("large", 2, "margin", 0.650, (30, 40, 25)),
+    ]
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"), rows=rows)
+    result = _section(text, "Result")
+    best = bakeoff.Combo("base", 2, "max").label()
+    assert f"Over the best eligible `base` row, {best} (0.610), the winner is +0.040" in result
+
+
+def test_a_base_winner_that_is_the_best_base_row_says_so() -> None:
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("base", 2, "max", 0.620, (1, 10, 10)),
+    ]
+    text = _render(bakeoff.Choice(bakeoff.Combo("base", 2, "max"), "why"), rows=rows)
+    assert "The winner is itself the best eligible `base` row." in text
+
+
+def test_no_change_has_no_gain_line() -> None:
+    result = _section(_render(bakeoff.Choice(None, "no change: x")), "Result")
+    assert "best eligible `base` row" not in result
+
+
+def test_a_decide_cut_on_the_grid_floor_is_marked() -> None:
+    floor = bakeoff.GRID[0]
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("large", 1, "max", 0.600, (1, 10, 10), decide=floor),
+    ]
+    text = _render(bakeoff.Choice(None, "x"), rows=rows)
+    assert f"| large | 1 | max | {floor:.2f} (grid floor) |" in text
+    assert "| base | 1 | max | 0.45 |" in text
+    assert f"1 of 2 combinations put `decide` on the grid floor ({floor:.2f})" in text
+
+
+def test_no_grid_floor_cut_is_said_in_words() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "(grid floor) |" not in text
+    assert "No combination puts `decide` on the grid floor" in text
+
+
+def test_the_numeric_layer_table_has_one_row_per_k() -> None:
+    section = _section(_render(bakeoff.Choice(None, "x")), "Numeric layer")
+    assert "| k | SciFact train fired | train correct | SciFact dev fired | dev correct |" in (
+        section
+    )
+    assert "| 1 | 4 | 0 | 1 | 1 |" in section
+    assert "| 2 | 5 | 0 | 1 | 1 |" in section
+
+
+def test_a_high_tier_made_only_of_rule_firings_is_said_to_come_from_the_rule() -> None:
+    rows = [
+        # high cut 1.0, one dev high verdict, one dev firing at k=1: the rule's.
+        _row("base", 1, "max", 0.597, (1, 113, 1), high=1.0),
+        # high cut 1.0 but two dev high verdicts against one firing: not only the rule.
+        _row("base", 2, "max", 0.600, (2, 10, 10), high=1.0),
+        # high cut below 1.0 with model verdicts in the tier.
+        _row("large", 1, "max", 0.600, (25, 50, 20)),
+    ]
+    section = _section(_render(bakeoff.Choice(None, "x"), rows=rows), "Numeric layer")
+    assert "The `high` cut is 1.000000 in 2 of 3 combinations." in section
+    assert "In 1 of 3, every SciFact dev `high` verdict is a rule firing" in section
+    assert "comes from the rule, not the model" in section
+
+
+def test_the_limits_state_dev_optimism_with_the_maximum_over_all_rows() -> None:
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"))
+    limits = _section(text, "Limits")
+    assert "The winner's 0.650 macro-F1 comes from the split that chose it" in limits
+    assert "out of 3 combinations whose highest dev macro-F1 is 0.650" in limits
+
+
+def test_the_limits_state_batch_dependence_snapshot_drift_and_grid_floor() -> None:
+    limits = _section(_render(bakeoff.Choice(None, "x")), "Limits")
+    assert "up to 0.197 (mean 0.008) and flipped 1 argmax" in limits
+    assert "measured 2026-10-01 by the final review" in limits
+    assert "reaches dev accuracy 0.607 here" in limits
+    assert "against the 0.609 published on 2026-09-12" in limits
+    assert "then 61 of 89 answerable claims were readable, now 61 of 89" in limits
+    assert "**Grid floor.**" in limits
+
+
+def test_the_old_averitec_numbers_carry_their_denominators_not_a_comparison() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "Published numbers to compare" not in text
+    assert "on a different snapshot with different cuts" in text
+    assert "over 89 answerable / 61 readable claims; this run counts 89 answerable" in text
+    assert "partly a coverage change" in text
+
+
+def test_the_report_prints_the_rule_verbatim_and_the_winner() -> None:
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"))
+    assert bakeoff.DECISION_RULE in text
+    assert "**Winner:** `large` × k=2 × `margin`" in text  # noqa: RUF001
+    assert "| base | 1 | max | 0.45 |" in text
+    assert "| large | 2 | margin | 0.45 |" in text
+
+
+def test_the_report_says_no_change_in_words() -> None:
+    text = _render(bakeoff.Choice(None, "no change: below 0.01"))
+    assert "**No change.**" in text
+    assert "no change: below 0.01" in text
+
+
+def test_a_tier_below_the_minimum_n_is_too_few_to_judge() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "| 1 · too few to judge |" in text
+
+
+def test_a_tier_at_or_above_the_minimum_n_shows_precision_and_wilson() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    low, high = bakeoff.wilson(21, 21)
+    assert f"21 · 1.000 (Wilson {low:.2f}–{high:.2f})" in text  # noqa: RUF001
+    assert "| base | 1 | max | 21 · 1.000" in text
+
+
+def test_unscored_models_are_listed_in_the_report() -> None:
+    text = _render(bakeoff.Choice(None, "x"), models=("base",))
+    assert "not scored: large, large-fever" in text
+
+
+def test_a_complete_run_says_all_candidates_were_scored() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert set(script.CANDIDATES) == {"base", "large", "large-fever"}
+    assert "all candidate models were scored" in text
+    assert "not scored:" not in text
+
+
+def test_a_missing_candidate_is_named_not_scored() -> None:
+    text = _render(bakeoff.Choice(None, "x"), models=("base", "large"))
+    assert "not scored: large-fever" in text
+    assert "all candidate models were scored" not in text
+
+
+def test_the_calibrated_cut_points_print_at_six_decimals() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "| 0.45 | 0.457948 | 0.999330 |" in text
+
+
+def test_a_gold_label_without_a_verdict_is_not_scored_rather_than_wrong() -> None:
+    rows = [_row("base", 1, "max", 0.597, (21, 113, 1))]
+    rows[0].averitec.per_label["Conflicting Evidence/Cherrypicking"] = (5, 0)
+    text = script.render_report(
+        rows,
+        bakeoff.Choice(None, "x"),
+        speed={"base": 185.0},
+        sizes={"base": None},
+        counts={"averitec": 100},
+        numeric=NUMERIC,
+        today="2026-10-02",
+        machine="Darwin arm64",
+    )
+    assert "5 (not scored)" in text
+    assert "0/5" not in text
+    assert "not a model failure" in text
+
+
+def test_the_averitec_header_states_every_denominator() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "100 claims, 89 answerable (3-way gold label), 61 of those with a readable" in text
+
+
+def test_averitec_per_label_counts_cover_every_combination() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "## AVeriTeC per label" in text
+    assert "| model | k | aggregation | Refuted | Supported |" in text
+    assert "| base | 1 | max | 20/63 | 7/10 |" in text
+    assert "| large | 2 | margin | 20/63 | 7/10 |" in text
+
+
+def test_a_nonzero_without_passage_count_is_flagged() -> None:
+    text = _render(bakeoff.Choice(None, "x"), unbacked=3)
+    assert "**3 — rule 1 violated**" in text
+    assert "| 0 |" in text
+
+
+def test_a_clean_run_has_no_rule_1_flag() -> None:
+    assert "rule 1 violated" not in _render(bakeoff.Choice(None, "x"))
+
+
+def test_an_unknown_model_size_is_a_dash_not_zero() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "| large | `cross-encoder/nli-deberta-v3-large` | — |" in text
+
+
+def _touch(path: Path, mtime: float) -> Path:
+    path.write_text("[]", encoding="utf-8")
+    os.utime(path, (mtime, mtime))
+    return path
+
+
+def test_items_are_stale_when_the_snapshot_is_newer(tmp_path: Path) -> None:
+    items = _touch(tmp_path / "items.json", 1_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 2_000)
+    assert script.items_stale(items, snapshot) is True
+
+
+def test_items_are_fresh_when_built_after_the_snapshot(tmp_path: Path) -> None:
+    items = _touch(tmp_path / "items.json", 2_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 1_000)
+    assert script.items_stale(items, snapshot) is False
+
+
+def test_report_with_a_stale_npz_names_the_model_and_the_fix(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    items = _touch(tmp_path / "items.json", 2_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 1_000)
+    (tmp_path / "base.npz").write_bytes(b"")
+    monkeypatch.setattr(script, "items_path", lambda: items)
+    monkeypatch.setattr(script, "snapshot_path", lambda: snapshot)
+    monkeypatch.setattr(script, "bakeoff_dir", lambda: tmp_path)
+    monkeypatch.setattr(script, "load_items", lambda _p: [])
+    monkeypatch.setattr(script, "load_snapshot", lambda _p: [])
+
+    def _stale(_path: Path, _items: object) -> object:
+        raise ValueError("base.npz was scored on other items; run `score` again")
+
+    monkeypatch.setattr(script, "load_probs", _stale)
+    assert script.cmd_report(Namespace(out="")) == 2
+    err = capsys.readouterr().err
+    assert "base" in err
+    assert "score --model base" in err
+    assert "Traceback" not in err
+
+
+def test_report_with_stale_items_does_not_advise_rebuild_items_flag(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    items = _touch(tmp_path / "items.json", 1_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 2_000)
+    monkeypatch.setattr(script, "items_path", lambda: items)
+    monkeypatch.setattr(script, "snapshot_path", lambda: snapshot)
+    assert script.cmd_report(Namespace(out="")) == 2
+    err = capsys.readouterr().err
+    assert "--rebuild-items" not in err
+    assert "score --model <name>" in err
+
+
+def test_split_blocks_cuts_the_flat_scores_per_item() -> None:
+    items = [_item("a", 2), _item("b", 0), _item("c", 3)]
+    flat = np.arange(15, dtype=np.float32).reshape(5, 3)
+    blocks = script.split_blocks(flat, items)
+    assert [b.shape for b in blocks] == [(2, 3), (0, 3), (3, 3)]
+    assert np.array_equal(blocks[2], flat[2:5])
+
+
+def test_split_blocks_refuses_a_row_count_that_differs_from_the_passages() -> None:
+    with pytest.raises(ValueError, match="score rows"):
+        script.split_blocks(np.zeros((4, 3), np.float32), [_item("a", 2)])
+
+
+def _item(key: str, n: int) -> Item:
+    passages = tuple(Passage(f"s{i}", "doc", i) for i in range(n))
+    return Item(key, "scifact-dev", key, "claim", "SUPPORTED", passages)
+
+
+class _FakeEmbedder:
+    """Deterministic unit vectors from the words of a text; no model."""
+
+    name = "fake"
+    dim = 16
+
+    def embed(self, texts: Sequence[str]) -> np.ndarray:
+        out = np.zeros((len(texts), self.dim), dtype=np.float32)
+        for row, text in enumerate(texts):
+            for word in text.lower().split():
+                out[row, sum(map(ord, word)) % self.dim] += 1.0
+            out[row, 0] += 0.01
+        return out / np.linalg.norm(out, axis=1, keepdims=True)
+
+
+def test_averitec_items_group_by_claim_skip_blank_sources_and_key_by_position() -> None:
+    snapshot = [
+        SnapshotClaim(
+            7,
+            "the claim",
+            "Refuted",
+            (
+                SnapshotSource("https://a", "ok", "First sentence. Second sentence."),
+                SnapshotSource("https://b", "blocked", ""),
+                SnapshotSource("https://c", "ok", "   "),
+                SnapshotSource("https://d", "ok", "Only one."),
+            ),
+            0,
+        )
+    ]
+    items = script.averitec_items(snapshot, _FakeEmbedder())
+    assert [i.key for i in items] == ["averitec:7:0", "averitec:7:3"]
+    assert {i.group for i in items} == {"averitec:7"}
+    assert {i.dataset for i in items} == {"averitec"}
+    assert items[0].claim == "the claim"
+    assert items[0].gold == "Refuted"
+    assert {p.source_id for p in items[0].passages} == {"https://a"}
+    assert [p.source_id for p in items[1].passages] == ["https://d"]
+    assert len(items[0].passages) <= bakeoff.TOP_N
+
+
+def test_snapshot_returns_130_after_an_interrupt(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    from proofpath import verify
+
+    closed: list[bool] = []
+
+    class _Fetcher:
+        def fetch(self, _url: str) -> object:
+            raise KeyboardInterrupt
+
+    class _Engine:
+        fetcher = _Fetcher()
+
+        def close(self) -> None:
+            closed.append(True)
+
+    claim = averitec.Claim(1, "c", "Refuted", ("https://e.org",))
+    monkeypatch.setattr(script, "snapshot_path", lambda: tmp_path / "snap.json")
+    monkeypatch.setattr(script.averitec, "ensure_downloaded", lambda _d: tmp_path)
+    monkeypatch.setattr(script.averitec, "load", lambda _p, limit=None: [claim])
+    monkeypatch.setattr(verify.Engine, "default", lambda *_a, **_k: _Engine())
+    monkeypatch.setattr("proofpath.config.load_config", lambda *_a, **_k: None)
+    code = script.cmd_snapshot(Namespace(resume=False, limit=0, sleep=0.0))
+    assert code == 130
+    assert closed == [True]