From 9870a029ab7b7e110765974cbe727b3b9c41be57 Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:18:29 +0300
Subject: [PATCH 01/10] docs: add the NLI bake-off design and the next-work
 backlog

k=3 was already measured and does not raise accuracy, so the next accuracy step
is a measured comparison of NLI models with a decision rule fixed in advance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 docs/superpowers/OPEN-ITEMS.md                |  30 ++++
 .../specs/2026-10-01-nli-bakeoff-design.md    | 154 ++++++++++++++++++
 2 files changed, 184 insertions(+)
 create mode 100644 docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md

diff --git a/docs/superpowers/OPEN-ITEMS.md b/docs/superpowers/OPEN-ITEMS.md
index 727b2a1..0da0e2c 100644
--- a/docs/superpowers/OPEN-ITEMS.md
+++ b/docs/superpowers/OPEN-ITEMS.md
@@ -578,3 +578,33 @@ third needed a design decision and is now decided and implemented (19.1).
 | 19.1 | ~~Underspecified claims pass too easily~~ **Decided and implemented 2026-09-30** | "openai won" was `SUPPORTED (found by proofpath)`, medium, on a single vanity-award page ("OpenAI's recognition with a 2026 Global Recognition Award"). The passage does entail the claim, so rules 1–3 hold; but a claim with no object ("won *what*?") is satisfied by any win. **Decided (the user): rule-based, note only.** `claims.is_underspecified` counts content words in the text the models check (`claim.hypothesis or claim.text`, so a translated claim is counted in English): hashtags, handles, punctuation and function words (the sentence openers plus auxiliaries, negation, prepositions, pronouns) are dropped; a number is one term ("12.5") and so is a run of capitalised words, a name ("Elon Musk"). **≤ 2 content terms is underspecified** (`UNDERSPECIFIED_MAX_WORDS`), **unless a linking verb stands between two content terms** — a complete predication ("The Earth is flat", "Revenue was 12.5%"). "openai won", "Elon Musk won", "OpenAI failed" and "It is true" are underspecified; "OpenAI won the award" is not. Every verdict finding on such a claim (all kinds, cited or found) carries `report.UNDERSPECIFIED` in its detail, after `ABSTRACT_BASIS` and before `FOUND_BY_PROOFPATH`; the SARIF message appends it the same way. No judge, no new dependency, and kind, level, tier and exit code are unchanged. A supported *cited* claim still produces no finding, so it gets no note. Known limits: a passive with a linking verb ("OpenAI was acquired") counts as complete, and a cited Turkish sentence is counted untranslated, so it reads slightly shorter. In title case or all caps (every content word capitalised) no names are joined, so a verb is never swallowed into one. Known small misses, documented and left: a sentence-initial adverb joins the name after it ("Yesterday Elon Musk won" is flagged), a dotted abbreviation splits ("U.S. won" is not flagged), a "BREAKING:" prefix counts as a content word, and a two-word name in title case ("Elon Musk Won") is three terms, so it is not flagged. The note is worded for every verdict kind (`underspecified claim: it names no object, time or scope, so many different events match it; check that the passage is about the one meant`). Source credibility (vanity awards, content farms) stays out of scope (§3). Evidence-search spec §2 records the rule |
 | 19.2 | A bot wall inside a site's full template can read as `ok` | challenge *phrases* now count only on pages under `fetch.CHALLENGE_MAX_WORDS` (300) extracted words, so a long real article that says "CAPTCHA" is not blocked. A wall served inside the site's own layout, with menus not in `<nav>`, can exceed 300 words and be read as content (review, 2026-09-30: a 364-word probe). A title/h1 signal was considered and rejected: it would block real pages titled "CAPTCHA". `cf-chl` stays unconditional. Cloudflare's underscore traces (`_cf_chl_opt`, `__cf_chl_f_tk`) are not matched by `cf-chl` — a candidate structural marker |
 | 19.3 | The markdown `## Checked` table does not mark abstract-grade support | a cited claim SUPPORTED by an abstract produces no finding (by design), so its row in `report.py`'s Checked table is the only place it appears, and it carries no abstract mark. The source's own `LOW CONFIDENCE (abstract only)` finding and the coverage line still state it; a per-row mark would make the table honest on its own (review, 2026-09-30) |
+
+## 20. What to improve next — 2026-10-01
+
+Two read-only sweeps (OPEN-ITEMS plus `docs/eval`, and the pipeline code) were turned into
+five packages. The user's decisions:
+
+- **A starts first.** Its first plan was "k=1 → k=3" plus retrieval tweaks. That plan was
+  dropped once the 2026-09-12 sweep showed k=3 does not raise accuracy (0.603 against
+  0.609). A became a measured NLI bake-off with a decision rule fixed in advance
+  (`docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`). The candidates are the
+  current base model and the two large int8 exports, cross-encoder and MoritzLaurer
+  FEVER/ANLI, downloaded for the eval only.
+- **Calibration** is fitted on SciFact **train** and reported on dev.
+- **Wayback** is tried after 429/5xx only. A URL that robots.txt disallows is never read
+  from the archive.
+
+### Backlog (packages B–E)
+
+| # | Item | Note |
+|---|---|---|
+| 20.1 | Unpaywall never runs by default | `oa.py` skips Unpaywall without a contact email, and the default is `""` (`config.py`). Add a `/config` "Contact email" row, in the same pattern as the Tavily row |
+| 20.2 | Optional Semantic Scholar API key | lifts the 1.1 s/request throttle; same as 7.9 and 8.21 |
+| 20.3 | Web pages use the paper full-text bar | `FULLTEXT_MIN_WORDS=1500` labels a normal news article "abstract only". Web needs its own lower bar (see 8.16) |
+| 20.4 | Wayback after 429/5xx | `fetch.py` skips the archive on rate limits and server errors. Pick the snapshot nearest the citation year, not the newest |
+| 20.5 | Crossref `/works/{doi}` is fetched three times per source | in resolve, the retraction check and the OA chain; add an in-memory memo |
+| 20.6 | Concurrent cold resolve/OA | thread pool over the thread-safe `polite.py` throttle. Needs per-thread cache connections first (10.11). Same as 11.10 |
+| 20.7 | Skip model loads on a fully cached re-run | same as 10.2 |
+| 20.8 | Bibliography parsing that was measured but never shipped | two-column rejoin (9.10/12.1), multi-block headless fallback (11.11), more headings (16.5), `[1]-[3]` ranges (9.4) |
+| 20.9 | The judge's quote is never checked | `judge._collect` does not verify that the quoted span occurs in the passage. Checking it is a cheap way to enforce product rule 1 |
+| 20.10 | Eval honesty | add a `--judge` flag to `scripts/eval_averitec.py`; do one run with the browser allowed; show the 0.404 search score in the README "Measured" table |
diff --git a/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md b/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md
new file mode 100644
index 0000000..ade0255
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md
@@ -0,0 +1,154 @@
+# NLI bake-off — design (2026-10-01)
+
+## Why
+
+The entailment step is where proofpath loses most of its accuracy, and the obvious fix
+has already been measured and does not work:
+
+- SciFact dev retrieval recall goes from 0.560 at k=1 to 0.852 at k=3, but accuracy does
+  not follow: k=1 0.609, k=2 0.609, k=3 0.603 (`docs/eval/2026-09-12-scifact-dev.md`).
+  Taking the strongest of more passages adds as many wrong assertions as right ones.
+- The model is overconfident: 116 verdicts score in [0.90, 1.00] and only 71.6 % of them
+  are right. The `medium` cut (0.457948) sits on `decide` (0.45), so the report really
+  has two tiers, and every cut was fitted and measured on the same split.
+- On AVeriTeC every one of the 19 `Supported` claims was missed, and readable-source
+  accuracy is 0.361 against a 0.708 baseline (`docs/eval/2026-09-16-averitec.md`).
+
+So the next step is not a code change to the product. It is a measured comparison of
+NLI models and aggregation rules, with the winner chosen by a rule fixed in advance.
+Product integration is a separate spec written after the result.
+
+## Candidates
+
+| id | repo | export | size | trained on |
+|---|---|---|---|---|
+| `base` (current) | `cross-encoder/nli-deberta-v3-base` @ `6c749ce` | int8 per CPU | 244 MB | SNLI + MNLI |
+| `large` | `cross-encoder/nli-deberta-v3-large` @ `bab4bc7` | int8 per CPU | 643 MB | SNLI + MNLI |
+| `large-fever` | `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` @ `b3546ea` | `onnx/model_quantized.onnx` | 643 MB | MNLI, FEVER-NLI, ANLI, LingNLI, WANLI |
+
+All three load through the existing `entailment.OnnxNli(repo_id=, revision=, onnx_file=)`;
+`label_permutation` already maps each model's `id2label`. Revisions are pinned to full
+SHAs in the script. The downloads land on the developer's machine for this eval only;
+nothing in the product downloads them (product rule 5).
+
+## Data
+
+- **SciFact train** — calibration only (`eval.scifact.load(split="train")`).
+- **SciFact dev** — the reported, held-out split. It is the same 340 pairs as before, so
+  the numbers compare directly with 2026-09-12.
+- **AVeriTeC dev, first 100 claims** — a frozen snapshot. The 2026-09-16 run's fetched
+  text has expired from the cache (7-day TTL), and re-fetching for each candidate would
+  compare models on different pages. One network pass with the current ladder, browser
+  not permitted (the default install, as in the published number), writes
+  `<cache>/datasets/averitec_snapshot.json`: per claim, the gold label, and per source
+  URL the fetch state and the extracted text. The snapshot stays outside the repo
+  (third-party page text). Every candidate is scored from that one file.
+
+## Method
+
+`scripts/eval_nli_bakeoff.py`, dev-only, three subcommands:
+
+1. `snapshot` — builds the AVeriTeC snapshot (network, `--sleep`, `--resume`). It reuses
+   `eval_averitec.py`'s engine construction, `state_for` and per-URL fetch loop.
+2. `score --model <id>` — offline. Splits each source into sentences with
+   `retrieval.split_sentences`, as the product does, and retrieves the top 5 with
+   `bge-small-en-v1.5` once; the retrieval is shared by every model and stored. Runs the
+   model's NLI once over every (passage, claim) pair in that top 5 and stores the raw
+   probabilities in `<cache>/datasets/bakeoff/<id>.npz`, with ms/pair. The numeric layer
+   runs unchanged on the same top-k (`numerics.check`), so a candidate is never credited
+   with what the rule decided.
+3. `report` — offline. From the stored probabilities it computes every combination of
+   model × k ∈ {1, 2, 3} × aggregation, calibrates each on SciFact train, scores it on
+   SciFact dev and on AVeriTeC, and writes `docs/eval/<date>-nli-bakeoff.md`.
+
+### Aggregation variants
+
+Pure functions over `(hits, probs)` that return a `Verdict`. They live in the script; a
+variant moves into `pipeline.py` only if it wins.
+
+- `max` — today's `pipeline.aggregate`: the strongest p(SUPPORTED) or p(REFUTED) over
+  the hits decides.
+- `max_nei` — as `max`, but a hit only counts when its asserted label beats its own
+  p(NEI).
+- `margin` — score = max p(S) − max p(R) over the hits. The sign gives the label and the
+  absolute value is the score, so a supporting passage and a contradicting passage on
+  the same source cancel out instead of the louder one winning.
+
+### Calibration
+
+For each combination, on SciFact **train**:
+
+- `decide` is the best-accuracy cut from `eval.metrics.sweep_decide`.
+- `high` and `medium` come from `eval.metrics.tier_cutpoints` at the existing 0.85 and
+  0.70 precision targets.
+
+Those cuts are then applied unchanged to dev and AVeriTeC. Each tier's dev precision is
+reported with its n and a 95 % Wilson interval. A tier that holds fewer than 20 dev
+verdicts is reported as "too few to judge" rather than given a precision.
+
+### AVeriTeC scoring
+
+Per claim, the same rule as `eval_averitec._decide_claim`:
+
+- the strongest non-NEI verdict that carries a passage, across the claim's sources, wins;
+- a claim with a readable source but no assertion is `NEI`;
+- a claim with no readable source is unanswered.
+
+Two numbers are reported: 3-way accuracy over all answerable claims (comparable to
+0.270), and over claims with at least one readable source (comparable to 0.361). The
+0.708 majority baseline is printed beside both.
+
+## Metrics in the report
+
+For every combination:
+
+- SciFact dev: accuracy, macro-F1, per-label F1, rationale F1, and the tier table;
+- AVeriTeC: both 3-way accuracies and the per-label counts;
+- for each model: ms/pair (Apple Silicon CPU, int8) and download size;
+- "asserted without a passage", which must be 0 in every row (product rule 1).
+
+## Decision rule (fixed before any number is seen)
+
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Winner:** the highest SciFact dev macro-F1 among eligible rows.
+3. **Size gate:** a `large*` row wins only if it beats the best eligible `base` row by
+   **≥ 0.03** macro-F1. Otherwise the best `base` row wins.
+4. **Tie:** two rows within 0.01 of each other go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`.
+5. If nothing beats `base` × k=1 × `max` by ≥ 0.01 macro-F1, the result is "no change".
+   It is written up as such, and the next package (B, coverage) starts.
+
+The rule is copied into the report verbatim, next to the numbers it chose.
+
+## What happens after
+
+A follow-up spec, written after the report:
+
+- **`base` wins with a new k or aggregation:** change `pipeline.aggregate` and
+  `DEFAULT_THRESHOLDS`, and update `cache.model_id` keying. Old verdicts invalidate
+  through the threshold key that already exists.
+- **A `large*` row wins:** add an opt-in "accurate" NLI model. It is downloaded only with
+  consent (product rule 5, spec §7.1) and is never the default, because the default
+  install must stay small. It carries its own thresholds.
+
+Either way, the README "Measured" table, spec §14 and the tier note are updated with the
+new numbers.
+
+## Tests (no network)
+
+- The aggregation variants are tested on hand-built probability rows: the `margin`
+  cancellation, the NEI veto in `max_nei`, and no passage → no assertion.
+- The train/dev split guard: calibration must never see a dev pair id.
+- Wilson interval values, and the "too few to judge" cut-off.
+- The snapshot round trip: write, load, and `--resume` skipping finished ids, against a
+  fixture `Fetched`.
+- The decision rule as a function over synthetic result rows, covering the size gate,
+  the tie order and "no change".
+
+## Out of scope
+
+- Changing the embedder or chunking (OPEN-ITEMS 3.6 stays open).
+- Fetch coverage, including running with the browser allowed; package B covers that.
+- Any change to product code or defaults; that is the follow-up spec.
+- A FEVER-trained `base` model: it has no ONNX export, and exporting one needs torch.

From 354f274551f4b3d0ea44668a1bc1aed95e8f62ab Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:26:29 +0300
Subject: [PATCH 02/10] docs: add the NLI bake-off implementation plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 .../plans/2026-10-01-nli-bakeoff.md           | 2006 +++++++++++++++++
 1 file changed, 2006 insertions(+)
 create mode 100644 docs/superpowers/plans/2026-10-01-nli-bakeoff.md

diff --git a/docs/superpowers/plans/2026-10-01-nli-bakeoff.md b/docs/superpowers/plans/2026-10-01-nli-bakeoff.md
new file mode 100644
index 0000000..0874a57
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-01-nli-bakeoff.md
@@ -0,0 +1,2006 @@
+# NLI Bake-off Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Measure three NLI models × k ∈ {1,2,3} × three aggregation rules on SciFact (calibrated on train, reported on dev) and a frozen AVeriTeC snapshot, and pick a winner by a rule fixed in advance.
+
+**Architecture:** The pure logic lives in two new modules under `src/proofpath/eval/`:
+- `bakeoff_data.py`: stored inputs and their I/O;
+- `bakeoff.py`: aggregation, calibration, evaluation and the decision rule.
+
+Both are covered by `mypy --strict` and unit tests. The dev-only script `scripts/eval_nli_bakeoff.py` does the network and model work (`snapshot`, `score`) and writes the markdown (`report`). Nothing in the product pipeline changes.
+
+**Tech Stack:** Python 3.10+, numpy, onnxruntime through `proofpath.entailment.OnnxNli`, fastembed through `proofpath.retrieval.FastEmbedder`, pytest.
+
+**Spec:** `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`
+
+## Global Constraints
+
+- Python 3.10+. Every function signature carries type annotations. `ruff check`, `ruff format --check` and `mypy` (strict, `src/`) are clean.
+- `pathlib` everywhere. Every text read and write passes `encoding="utf-8"`.
+- Unit tests make no network calls.
+- Code, identifiers and comments are in English.
+- Do not change product code (`pipeline.py`, `verify.py`, `entailment.py`, defaults). The bake-off only reads it.
+- Model revisions are pinned to full 40-character SHAs:
+  - base `6c749ce3425cd33b46d187e45b92bbf96ee12ec7`
+  - large `bab4bc7178836f731dcfd18c06ca9def0a137712`
+  - large-fever `b3546ea6b0346eb6f8d5d68b13c7dc6d0376b3d7`
+- The AVeriTeC snapshot is fetched with the browser **not permitted** and `interactive=False`.
+- Third-party page text (the snapshot) stays under `<cache>/datasets/` and never goes into the repo.
+- The decision rule is the spec's, verbatim. Numbers: TIE 0.01, LARGE 0.03, NO-CHANGE 0.01, MIN_TIER_N 20, precision targets 0.85 / 0.70.
+- Subagents never commit. The controller stages and commits on `feat/nli-bakeoff` after a clean review.
+- The code blocks in this plan are not pre-formatted. Run `uv run ruff format <new or changed files>` before every `--check`.
+- When a step says "append" to a test file, put its `import` lines in the file's top import block. A mid-file import fails ruff's E402.
+
+**Drift from the spec (binding):** the spec put the aggregation variants "in the script". They live in `src/proofpath/eval/bakeoff.py` instead, beside `metrics.py` and `scifact.py`. That way they get `mypy --strict` and clean imports in tests. That package is eval tooling, not the product pipeline, so "no product change" still holds.
+
+## File Structure
+
+| File | Responsibility |
+|---|---|
+| Create `src/proofpath/eval/bakeoff_data.py` | `SnapshotSource`, `SnapshotClaim`, `Item`; snapshot/items JSON and probability npz I/O; `pending`, `source_from_fetched`, `items_fingerprint` |
+| Create `src/proofpath/eval/bakeoff.py` | `Signal`, three aggregators, `signal_for`, `decide`, `wilson`, `calibrate`, `calibration_rows`, `evaluate_scifact`, `evaluate_averitec`, `Combo`/`Outcome`/`Choice`, `choose`, `DECISION_RULE` |
+| Create `scripts/eval_nli_bakeoff.py` | `snapshot`, `score` and `report` subcommands; `CANDIDATES`; `render_report` |
+| Create `tests/test_bakeoff_data.py` | I/O round trips, resume, stale-probability guards |
+| Create `tests/test_bakeoff.py` | aggregation, Wilson, calibration split guard, evaluation, decision rule |
+| Create `tests/test_eval_nli_bakeoff.py` | the script imports without side effects; candidate pins; `render_report` |
+| Create (Task 7) `docs/eval/<date>-nli-bakeoff.md` | the result |
+| Modify (Task 7) `docs/superpowers/OPEN-ITEMS.md` | §20: the outcome line |
+
+---
+
+### Task 1: Stored inputs — `bakeoff_data.py`
+
+**Files:**
+- Create: `src/proofpath/eval/bakeoff_data.py`
+- Test: `tests/test_bakeoff_data.py`
+
+**Interfaces:**
+- Consumes:
+  - `proofpath.models.Passage(text: str, source_id: str, index: int)`
+  - `proofpath.fetch.Fetched` (`.ok`, `.text`, `.outcome.value`)
+  - `proofpath.fetch.Outcome`
+  - `proofpath.providers.NO_TEXT: str`
+  - `proofpath.eval.averitec.Claim(id, text, label, source_urls, non_urls)`
+- Produces:
+  - `DATASETS: tuple[str, ...] = ("scifact-train", "scifact-dev", "averitec")`
+  - `SnapshotSource(url: str, state: str, text: str)`
+  - `SnapshotClaim(claim_id: int, claim: str, gold: str, sources: tuple[SnapshotSource, ...], non_urls: int)`
+  - `Item(key: str, dataset: str, group: str, claim: str, gold: str, passages: tuple[Passage, ...], rationale: frozenset[int] = frozenset())`
+  - `source_from_fetched(url: str, fetched: Fetched) -> SnapshotSource`
+  - `pending(claims: Sequence[averitec.Claim], done: Sequence[SnapshotClaim]) -> list[averitec.Claim]`
+  - `save_snapshot(path: Path, claims: Sequence[SnapshotClaim]) -> None` / `load_snapshot(path: Path) -> list[SnapshotClaim]`
+  - `save_items(path: Path, items: Sequence[Item]) -> None` / `load_items(path: Path) -> list[Item]`
+  - `items_fingerprint(items: Sequence[Item]) -> str`
+  - `save_probs(path: Path, items: Sequence[Item], probs: Sequence[np.ndarray], *, ms_per_pair: float) -> None`
+  - `load_probs(path: Path, items: Sequence[Item]) -> tuple[list[np.ndarray], float]`
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+"""Offline tests for the bake-off's stored inputs (no network, no model)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from proofpath.eval import averitec
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    SnapshotSource,
+    items_fingerprint,
+    load_items,
+    load_probs,
+    load_snapshot,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+)
+from proofpath.fetch import Fetched, Outcome
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+
+def _fetched(outcome: Outcome, text: str) -> Fetched:
+    return Fetched(
+        url="https://example.org/a",
+        final_url="https://example.org/a",
+        step=1,
+        outcome=outcome,
+        status=200 if outcome is Outcome.OK else 403,
+        content_type="text/html",
+        kind="html",
+        body=b"",
+        text=text,
+        notes=[],
+    )
+
+
+def _item(key: str, n: int = 2, dataset: str = "scifact-dev") -> Item:
+    passages = tuple(Passage(f"sentence {i} of {key}", "doc", i) for i in range(n))
+    return Item(key, dataset, key, "a claim", "SUPPORTED", passages, frozenset({0}))
+
+
+def test_a_read_page_keeps_its_text_and_ok_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "Some text."))
+    assert source == SnapshotSource("https://example.org/a", "ok", "Some text.")
+
+
+def test_a_reached_page_with_no_text_gets_the_no_text_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "   "))
+    assert source.state == NO_TEXT
+    assert source.text == ""
+
+
+def test_a_blocked_page_keeps_its_state_and_no_text() -> None:
+    blocked = _fetched(Outcome.BLOCKED_NO_BROWSER, "challenge page")
+    source = source_from_fetched("https://example.org/a", blocked)
+    assert source.state == Outcome.BLOCKED_NO_BROWSER.value
+    assert source.text == ""
+
+
+def test_snapshot_round_trips(tmp_path: Path) -> None:
+    claims = [
+        SnapshotClaim(
+            7, "c", "Refuted", (SnapshotSource("u", "ok", "t"), SnapshotSource("v", "x", "")), 1
+        )
+    ]
+    path = tmp_path / "snap.json"
+    save_snapshot(path, claims)
+    assert load_snapshot(path) == claims
+
+
+def test_a_missing_snapshot_loads_as_empty(tmp_path: Path) -> None:
+    assert load_snapshot(tmp_path / "absent.json") == []
+
+
+def test_pending_skips_claims_already_in_the_snapshot() -> None:
+    claims = [averitec.Claim(i, f"claim {i}", "Refuted", ("https://e.org",)) for i in range(3)]
+    done = [SnapshotClaim(1, "claim 1", "Refuted", (), 0)]
+    assert [c.id for c in pending(claims, done)] == [0, 2]
+
+
+def test_items_round_trip(tmp_path: Path) -> None:
+    items = [_item("a"), _item("b", 1, "averitec")]
+    path = tmp_path / "items.json"
+    save_items(path, items)
+    assert load_items(path) == items
+
+
+def test_an_unknown_dataset_is_refused() -> None:
+    with pytest.raises(ValueError, match="unknown dataset"):
+        _item("a", dataset="scifact-test")
+
+
+def test_probs_round_trip(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 3)]
+    probs = [np.full((2, 3), 0.25, np.float32), np.full((3, 3), 0.5, np.float32)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, probs, ms_per_pair=12.5)
+    loaded, ms = load_probs(path, items)
+    assert ms == pytest.approx(12.5)
+    assert [p.shape for p in loaded] == [(2, 3), (3, 3)]
+    assert np.allclose(loaded[1], 0.5)
+
+
+def test_probs_scored_on_other_items_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, [np.zeros((2, 3), np.float32)], ms_per_pair=1.0)
+    changed = [_item("z", 2)]
+    with pytest.raises(ValueError, match="other items"):
+        load_probs(path, changed)
+
+
+def test_probs_with_the_wrong_shape_are_refused(tmp_path: Path) -> None:
+    with pytest.raises(ValueError, match="shape"):
+        save_probs(tmp_path / "x.npz", [_item("a", 2)], [np.zeros((3, 3))], ms_per_pair=1.0)
+
+
+def test_the_fingerprint_changes_with_a_passage_text() -> None:
+    a = _item("a")
+    b = Item(a.key, a.dataset, a.group, a.claim, a.gold, (Passage("other", "doc", 0),))
+    assert items_fingerprint([a]) != items_fingerprint([b])
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff_data.py -q`
+Expected: FAIL with `ModuleNotFoundError: No module named 'proofpath.eval.bakeoff_data'`
+
+- [ ] **Step 3: Write the implementation**
+
+```python
+"""Stored inputs of the NLI bake-off (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Three files under ``<cache>/datasets``, written once and read by every later step:
+the frozen AVeriTeC snapshot (page text per source URL), the retrieved items (top
+passages per claim, shared by every model), and one npz of raw NLI probabilities per
+model. Nothing here touches the network or a model.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from proofpath.eval import averitec
+from proofpath.fetch import Fetched
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+DATASETS: tuple[str, ...] = ("scifact-train", "scifact-dev", "averitec")
+
+
+@dataclass(frozen=True)
+class SnapshotSource:
+    """One source URL as the snapshot run found it. ``text`` is "" unless it was read."""
+
+    url: str
+    state: str
+    text: str
+
+
+@dataclass(frozen=True)
+class SnapshotClaim:
+    claim_id: int
+    claim: str
+    gold: str  # the AVeriTeC label, verbatim
+    sources: tuple[SnapshotSource, ...]
+    non_urls: int
+
+
+@dataclass(frozen=True)
+class Item:
+    """One (claim, source) unit with its top retrieved passages, best first.
+
+    ``group`` is the claim an item belongs to: a SciFact pair is its own group, an
+    AVeriTeC claim groups every readable source it has.
+    """
+
+    key: str
+    dataset: str
+    group: str
+    claim: str
+    gold: str
+    passages: tuple[Passage, ...]
+    rationale: frozenset[int] = frozenset()
+
+    def __post_init__(self) -> None:
+        if self.dataset not in DATASETS:
+            raise ValueError(f"unknown dataset {self.dataset!r}")
+
+
+def source_from_fetched(url: str, fetched: Fetched) -> SnapshotSource:
+    """The state and text one fetch earned, worded as ``eval_averitec.state_for`` does.
+
+    A page that was reached but yielded no text is not ``ok`` (product rule 6), and
+    only a read page keeps its text: a challenge page's words are not the source.
+    """
+    if fetched.ok and fetched.text.strip():
+        return SnapshotSource(url, fetched.outcome.value, fetched.text)
+    if fetched.ok:
+        return SnapshotSource(url, NO_TEXT, "")
+    return SnapshotSource(url, fetched.outcome.value, "")
+
+
+def pending(
+    claims: Sequence[averitec.Claim], done: Sequence[SnapshotClaim]
+) -> list[averitec.Claim]:
+    """The claims a resumed snapshot run still has to fetch, in dataset order."""
+    finished = {claim.claim_id for claim in done}
+    return [claim for claim in claims if claim.id not in finished]
+
+
+def _write_atomic(path: Path, payload: str) -> None:
+    """Write a whole file at once, so a Ctrl-C never leaves half of one."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    temporary.write_text(payload, encoding="utf-8")
+    temporary.replace(path)
+
+
+def save_snapshot(path: Path, claims: Sequence[SnapshotClaim]) -> None:
+    payload = [
+        {
+            "claim_id": claim.claim_id,
+            "claim": claim.claim,
+            "gold": claim.gold,
+            "non_urls": claim.non_urls,
+            "sources": [{"url": s.url, "state": s.state, "text": s.text} for s in claim.sources],
+        }
+        for claim in claims
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_snapshot(path: Path) -> list[SnapshotClaim]:
+    if not path.exists():
+        return []
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        SnapshotClaim(
+            claim_id=int(row["claim_id"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            sources=tuple(
+                SnapshotSource(str(s["url"]), str(s["state"]), str(s["text"]))
+                for s in row["sources"]
+            ),
+            non_urls=int(row["non_urls"]),
+        )
+        for row in raw
+    ]
+
+
+def save_items(path: Path, items: Sequence[Item]) -> None:
+    payload = [
+        {
+            "key": item.key,
+            "dataset": item.dataset,
+            "group": item.group,
+            "claim": item.claim,
+            "gold": item.gold,
+            "passages": [[p.text, p.source_id, p.index] for p in item.passages],
+            "rationale": sorted(item.rationale),
+        }
+        for item in items
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_items(path: Path) -> list[Item]:
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        Item(
+            key=str(row["key"]),
+            dataset=str(row["dataset"]),
+            group=str(row["group"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            passages=tuple(Passage(str(t), str(s), int(i)) for t, s, i in row["passages"]),
+            rationale=frozenset(int(i) for i in row["rationale"]),
+        )
+        for row in raw
+    ]
+
+
+def items_fingerprint(items: Sequence[Item]) -> str:
+    """A digest of what was scored: keys, claims and passage texts, in order.
+
+    Stored beside the probabilities so a rebuilt item file can never be paired with
+    probabilities computed on different passages.
+    """
+    digest = hashlib.sha256()
+    for item in items:
+        digest.update(item.key.encode("utf-8") + b"\0" + item.claim.encode("utf-8") + b"\0")
+        for passage in item.passages:
+            digest.update(passage.text.encode("utf-8") + b"\0")
+    return digest.hexdigest()
+
+
+def save_probs(
+    path: Path, items: Sequence[Item], probs: Sequence[np.ndarray], *, ms_per_pair: float
+) -> None:
+    """Store one model's ``(n_passages, 3)`` probabilities per item, flattened."""
+    if len(items) != len(probs):
+        raise ValueError("one probability block per item is required")
+    for item, block in zip(items, probs, strict=True):
+        if np.asarray(block).shape != (len(item.passages), 3):
+            raise ValueError(f"probability block for {item.key} has the wrong shape")
+    offsets = np.cumsum([0, *(len(item.passages) for item in items)], dtype=np.int64)
+    flat = (
+        np.concatenate([np.asarray(b, dtype=np.float32) for b in probs])
+        if probs
+        else np.zeros((0, 3), dtype=np.float32)
+    )
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    with temporary.open("wb") as handle:
+        np.savez(
+            handle,
+            fingerprint=np.array(items_fingerprint(items)),
+            offsets=offsets,
+            probs=flat,
+            ms_per_pair=np.array(ms_per_pair, dtype=np.float64),
+        )
+    temporary.replace(path)
+
+
+def load_probs(path: Path, items: Sequence[Item]) -> tuple[list[np.ndarray], float]:
+    """The stored probabilities, one block per item, and the measured ms per pair."""
+    with np.load(path, allow_pickle=False) as data:
+        fingerprint = str(data["fingerprint"])
+        offsets = data["offsets"]
+        flat = data["probs"]
+        ms_per_pair = float(data["ms_per_pair"])
+    if fingerprint != items_fingerprint(items):
+        raise ValueError(f"{path.name} was scored on other items; run `score` again")
+    return [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))], ms_per_pair
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff_data.py -q`
+Expected: 12 passed
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src/proofpath/eval/bakeoff_data.py tests/test_bakeoff_data.py && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 2: Signals and aggregation — `bakeoff.py` part 1
+
+**Files:**
+- Create: `src/proofpath/eval/bakeoff.py`
+- Test: `tests/test_bakeoff.py`
+
+**Interfaces:**
+- Consumes:
+  - `proofpath.numerics.check(claim: str, passages: Sequence[Passage]) -> NumericResult | None` (`.mismatch`, `.passage`)
+  - `proofpath.entailment.LABEL_ORDER = (SUPPORTED, REFUTED, NEI)`
+  - `proofpath.pipeline.Thresholds(decide, high, medium)` with `.tier(score)`
+- Produces:
+  - `AGGREGATIONS = ("max", "max_nei", "margin")`
+  - `KS = (1, 2, 3)`
+  - `TOP_N = 5`
+  - `Signal(label: Label, score: float, passage: Passage | None)`
+  - `NOTHING: Signal`
+  - `aggregate_max` / `aggregate_max_nei` / `aggregate_margin(passages: Sequence[Passage], probs: np.ndarray) -> Signal`
+  - `AGGREGATORS: Mapping[str, Aggregator]`
+  - `signal_for(claim: str, passages: Sequence[Passage], probs: np.ndarray, *, k: int, aggregation: str) -> Signal`
+  - `decide(signal: Signal, thresholds: Thresholds) -> Label`
+  - `wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]`
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+"""Offline tests for the bake-off's aggregation, calibration and decision rule."""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+from proofpath.eval import bakeoff
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+
+# Rows are (SUPPORTED, REFUTED, NEI), the order of entailment.LABEL_ORDER.
+P = [Passage(f"passage {i}", "doc", i) for i in range(3)]
+
+
+def _probs(*rows: tuple[float, float, float]) -> np.ndarray:
+    return np.array(rows, dtype=np.float32)
+
+
+def test_max_takes_the_loudest_assertion_over_the_hits() -> None:
+    probs = _probs((0.6, 0.1, 0.3), (0.1, 0.8, 0.1))
+    signal = bakeoff.aggregate_max(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.REFUTED, P[1])
+    assert signal.score == pytest.approx(0.8)
+
+
+def test_max_nei_ignores_an_assertion_its_own_hit_calls_neutral() -> None:
+    probs = _probs((0.45, 0.05, 0.50), (0.40, 0.05, 0.30))
+    signal = bakeoff.aggregate_max_nei(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[1])
+    assert signal.score == pytest.approx(0.40)
+
+
+def test_max_nei_proposes_nothing_when_every_hit_is_neutral() -> None:
+    probs = _probs((0.2, 0.1, 0.7))
+    assert bakeoff.aggregate_max_nei(P[:1], probs) == bakeoff.NOTHING
+
+
+def test_margin_lets_a_contradicting_passage_cancel_a_supporting_one() -> None:
+    probs = _probs((0.90, 0.05, 0.05), (0.05, 0.85, 0.10))
+    signal = bakeoff.aggregate_margin(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[0])
+    assert signal.score == pytest.approx(0.05)
+
+
+def test_margin_with_an_exact_tie_proposes_nothing() -> None:
+    probs = _probs((0.5, 0.0, 0.5), (0.0, 0.5, 0.5))
+    assert bakeoff.aggregate_margin(P[:2], probs) == bakeoff.NOTHING
+
+
+@pytest.mark.parametrize("name", bakeoff.AGGREGATIONS)
+def test_no_hits_propose_nothing(name: str) -> None:
+    assert bakeoff.AGGREGATORS[name]([], np.zeros((0, 3), np.float32)) == bakeoff.NOTHING
+
+
+def test_signal_for_reads_only_the_first_k_hits() -> None:
+    probs = _probs((0.2, 0.1, 0.7), (0.95, 0.0, 0.05))
+    first = bakeoff.signal_for("a claim", P[:2], probs, k=1, aggregation="max")
+    assert first.passage == P[0]
+    both = bakeoff.signal_for("a claim", P[:2], probs, k=2, aggregation="max")
+    assert both.passage == P[1]
+
+
+def test_a_numeric_mismatch_decides_before_any_aggregation() -> None:
+    passages = [Passage("Mortality fell by 12% in the treated group.", "doc", 0)]
+    probs = _probs((0.99, 0.0, 0.01))
+    signal = bakeoff.signal_for(
+        "Mortality fell by 40% in the treated group.", passages, probs, k=1, aggregation="max"
+    )
+    assert signal == bakeoff.Signal(Label.REFUTED, 1.0, passages[0])
+
+
+def test_decide_needs_the_cut_and_a_passage() -> None:
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.6, P[0]), cuts) is Label.SUPPORTED
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.4, P[0]), cuts) is Label.NEI
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.9, None), cuts) is Label.NEI
+
+
+def test_wilson_interval_matches_known_values() -> None:
+    low, high = bakeoff.wilson(0, 10)
+    assert low == pytest.approx(0.0)
+    assert high == pytest.approx(0.2775, abs=1e-4)
+    low, high = bakeoff.wilson(10, 10)
+    assert low == pytest.approx(0.7225, abs=1e-4)
+    assert high == pytest.approx(1.0)
+
+
+def test_wilson_refuses_an_empty_sample() -> None:
+    with pytest.raises(ValueError):
+        bakeoff.wilson(0, 0)
+```
+
+The numeric test relies on `numerics.check` refuting 40 % against a single 12 %. If the layer stays silent on this pair, change the sentences to a pair that `tests/test_numerics.py` already shows mismatching, and copy that pair verbatim.
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: FAIL with `ImportError: cannot import name 'bakeoff'`
+
+- [ ] **Step 3: Write the implementation**
+
+```python
+"""NLI bake-off: aggregation variants, calibration, evaluation and the decision rule.
+
+Pure functions over stored NLI probabilities; nothing here touches the network or a
+model (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md). A variant moves into
+``pipeline.py`` only if the bake-off picks it.
+"""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Callable, Mapping, Sequence
+from dataclasses import dataclass
+
+import numpy as np
+
+from proofpath import numerics
+from proofpath.entailment import LABEL_ORDER
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+
+_S = LABEL_ORDER.index(Label.SUPPORTED)
+_R = LABEL_ORDER.index(Label.REFUTED)
+_N = LABEL_ORDER.index(Label.NEI)
+
+# Simplest first: this order is also the tie order of the decision rule.
+AGGREGATIONS: tuple[str, ...] = ("max", "max_nei", "margin")
+KS: tuple[int, ...] = (1, 2, 3)
+# Passages retrieved and scored per item; every k is a prefix of these.
+TOP_N = 5
+
+
+@dataclass(frozen=True)
+class Signal:
+    """The strongest assertion a combination proposes for one item, before ``decide``.
+
+    ``label`` is SUPPORTED or REFUTED, or NEI when the aggregation proposes nothing;
+    ``passage`` is what the assertion rests on.
+    """
+
+    label: Label
+    score: float
+    passage: Passage | None
+
+
+NOTHING = Signal(Label.NEI, 0.0, None)
+
+Aggregator = Callable[[Sequence[Passage], np.ndarray], Signal]
+
+
+def aggregate_max(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """Today's ``pipeline.aggregate``: the loudest SUPPORTED or REFUTED over the hits."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_max_nei(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """As ``max``, but a hit only counts when its assertion beats its own p(NEI)."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        neutral = float(row[_N])
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > neutral and value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_margin(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """max p(S) − max p(R): a supporting and a contradicting passage cancel out."""
+    if len(passages) == 0:
+        return NOTHING
+    s_at = int(np.argmax(probs[:, _S]))
+    r_at = int(np.argmax(probs[:, _R]))
+    supported = float(probs[s_at, _S])
+    refuted = float(probs[r_at, _R])
+    if supported > refuted:
+        return Signal(Label.SUPPORTED, supported - refuted, passages[s_at])
+    if refuted > supported:
+        return Signal(Label.REFUTED, refuted - supported, passages[r_at])
+    return NOTHING
+
+
+AGGREGATORS: Mapping[str, Aggregator] = {
+    "max": aggregate_max,
+    "max_nei": aggregate_max_nei,
+    "margin": aggregate_margin,
+}
+
+
+def signal_for(
+    claim: str, passages: Sequence[Passage], probs: np.ndarray, *, k: int, aggregation: str
+) -> Signal:
+    """What one combination proposes for one item, numeric layer first (spec §10).
+
+    A numeric mismatch decides by rule at score 1.0, as ``pipeline.decide_indexed``
+    does, so no candidate model is credited with what the rule decided.
+    """
+    top = list(passages[:k])
+    numeric = numerics.check(claim, top)
+    if numeric is not None and numeric.mismatch:
+        return Signal(Label.REFUTED, 1.0, numeric.passage)
+    return AGGREGATORS[aggregation](top, probs[:k])
+
+
+def decide(signal: Signal, thresholds: Thresholds) -> Label:
+    """The verdict a signal earns: below the cut, or with no passage, it is NEI."""
+    if signal.label is Label.NEI or signal.passage is None:
+        return Label.NEI
+    return signal.label if signal.score >= thresholds.decide else Label.NEI
+
+
+def wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]:
+    """95 % Wilson score interval for ``correct`` successes out of ``n``."""
+    if n <= 0:
+        raise ValueError("a Wilson interval needs at least one observation")
+    p = correct / n
+    denominator = 1 + z * z / n
+    centre = (p + z * z / (2 * n)) / denominator
+    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator
+    return max(0.0, centre - half), min(1.0, centre + half)
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: 13 passed (11 test functions; the parametrised one runs three times)
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src tests && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 3: Calibration and evaluation — `bakeoff.py` part 2
+
+**Files:**
+- Modify: `src/proofpath/eval/bakeoff.py` (append)
+- Test: `tests/test_bakeoff.py` (append)
+
+**Interfaces:**
+- Consumes:
+  - Task 1: `Item`, `SnapshotClaim`
+  - Task 2: `signal_for`, `decide`, `wilson`
+  - `proofpath.eval.metrics`: `Row`, `LABELS`, `accuracy`, `macro_f1`, `rationale_f1`, `sweep_decide`, `tier_cutpoints`
+  - `proofpath.eval.averitec.to_label(str) -> Label | None`
+- Produces:
+  - `GRID: tuple[float, ...]` (0.05 … 0.95)
+  - `HIGH_TARGET = 0.85`, `MEDIUM_TARGET = 0.70`, `MIN_TIER_N = 20`
+  - `Scored(item: Item, probs: np.ndarray)`
+  - `ship_cut(value: float) -> float`
+  - `calibration_rows(scored: Sequence[Scored], *, k: int, aggregation: str) -> list[metrics.Row]`, which raises `ValueError` on any item that is not `scifact-train`
+  - `calibrate(rows: Sequence[metrics.Row]) -> Thresholds`
+  - `TierStat(tier: str, n: int, correct: int)` with `.precision -> float | None` (None below `MIN_TIER_N`) and `.interval -> tuple[float, float] | None`
+  - `SciFactResult(n, accuracy, macro_f1, f1: dict[Label, float], rationale_f1, tiers: tuple[TierStat, ...], asserted_without_passage)`
+  - `evaluate_scifact(scored, thresholds, *, k, aggregation) -> SciFactResult`, which accepts only `scifact-dev`
+  - `AveritecResult(n, counted, accuracy_all, readable, accuracy_readable, majority_baseline, per_label: dict[str, tuple[int, int]], asserted_without_passage)`
+  - `evaluate_averitec(claims: Sequence[SnapshotClaim], scored, thresholds, *, k, aggregation) -> AveritecResult`, which accepts only `averitec`
+
+- [ ] **Step 1: Write the failing tests (append to `tests/test_bakeoff.py`)**
+
+```python
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim, SnapshotSource
+
+
+def _scored(
+    key: str,
+    dataset: str,
+    gold: str,
+    row: tuple[float, float, float],
+    *,
+    group: str | None = None,
+    rationale: frozenset[int] = frozenset({0}),
+) -> bakeoff.Scored:
+    item = Item(key, dataset, group or key, "a claim", gold, (Passage("p", key, 0),), rationale)
+    return bakeoff.Scored(item, _probs(row))
+
+
+def test_calibration_refuses_anything_but_scifact_train() -> None:
+    dev = [_scored("d", "scifact-dev", "SUPPORTED", (0.9, 0.0, 0.1))]
+    with pytest.raises(ValueError, match="scifact-train"):
+        bakeoff.calibration_rows(dev, k=1, aggregation="max")
+
+
+def test_evaluation_refuses_the_calibration_split() -> None:
+    train = [_scored("t", "scifact-train", "SUPPORTED", (0.9, 0.0, 0.1))]
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    with pytest.raises(ValueError, match="scifact-dev"):
+        bakeoff.evaluate_scifact(train, cuts, k=1, aggregation="max")
+
+
+def test_calibrate_picks_a_cut_that_separates_right_from_wrong() -> None:
+    rows = [(Label.SUPPORTED, Label.SUPPORTED, 0.9)] * 5 + [(Label.NEI, Label.SUPPORTED, 0.3)] * 5
+    cuts = bakeoff.calibrate(rows)
+    assert 0.3 < cuts.decide <= 0.9
+    assert cuts.decide <= cuts.medium <= cuts.high
+
+
+def test_tier_stats_hide_precision_below_the_minimum_n() -> None:
+    assert bakeoff.TierStat("high", 19, 19).precision is None
+    assert bakeoff.TierStat("high", 19, 19).interval is None
+    assert bakeoff.TierStat("high", 20, 15).precision == pytest.approx(0.75)
+
+
+def test_evaluate_scifact_counts_verdicts_tiers_and_rationale() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    scored = [
+        _scored("a", "scifact-dev", "SUPPORTED", (0.97, 0.01, 0.02)),  # high, right
+        _scored("b", "scifact-dev", "REFUTED", (0.80, 0.10, 0.10)),  # medium, wrong
+        _scored("c", "scifact-dev", "NEI", (0.30, 0.10, 0.60)),  # below decide
+    ]
+    result = bakeoff.evaluate_scifact(scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.accuracy == pytest.approx(2 / 3)
+    assert {t.tier: (t.n, t.correct) for t in result.tiers} == {
+        "high": (1, 1),
+        "medium": (1, 0),
+        "low": (0, 0),
+    }
+    assert result.asserted_without_passage == 0
+    assert 0.0 < result.rationale_f1 <= 1.0
+
+
+def test_evaluate_averitec_keeps_the_strongest_source_and_both_denominators() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [
+        SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u1", "ok", "t"),), 0),
+        SnapshotClaim(2, "c2", "Supported", (SnapshotSource("u2", "UNVERIFIED (x)", ""),), 0),
+        SnapshotClaim(3, "c3", "Conflicting Evidence/Cherrypicking", (), 0),
+    ]
+    scored = [
+        _scored("averitec:1:0", "averitec", "Refuted", (0.6, 0.1, 0.3), group="averitec:1"),
+        _scored("averitec:1:1", "averitec", "Refuted", (0.0, 0.9, 0.1), group="averitec:1"),
+    ]
+    result = bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.counted == 2  # the Conflicting claim has no 3-way label
+    assert result.accuracy_all == pytest.approx(0.5)  # claim 1 right, claim 2 unread → NEI
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(1.0)
+    assert result.majority_baseline == pytest.approx(0.5)
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: FAIL with `AttributeError: module 'proofpath.eval.bakeoff' has no attribute 'Scored'`
+
+- [ ] **Step 3: Write the implementation (append to `bakeoff.py`; merge the imports into the top block)**
+
+```python
+from collections import Counter
+
+from proofpath.eval import averitec, metrics
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim
+
+# Wider than eval_scifact's 0.30–0.95 grid: a ``margin`` score is a difference of two
+# probabilities and lives much nearer 0. Every combination is swept on the same grid.
+GRID: tuple[float, ...] = tuple(round(0.05 * i, 2) for i in range(1, 20))
+HIGH_TARGET = 0.85
+MEDIUM_TARGET = 0.70
+# Below this many dev verdicts a tier's precision is reported as "too few to judge".
+MIN_TIER_N = 20
+CUT_DECIMALS = 6
+
+
+@dataclass(frozen=True)
+class Scored:
+    item: Item
+    probs: np.ndarray  # (len(item.passages), 3) in LABEL_ORDER
+
+
+def _require(scored: Sequence[Scored], dataset: str) -> None:
+    wrong = sorted({s.item.dataset for s in scored if s.item.dataset != dataset})
+    if wrong:
+        raise ValueError(f"expected only {dataset} items, got {', '.join(wrong)}")
+
+
+def ship_cut(value: float) -> float:
+    """Round a cut up to ``CUT_DECIMALS`` — the rule of ``eval_scifact.ship_cut``.
+
+    Up, never down: a cut rounded down would hand a tier to scores the calibration
+    split never showed were that good.
+    """
+    scale = 10**CUT_DECIMALS
+    return min(1.0, math.ceil(value * scale) / scale)
+
+
+def calibration_rows(scored: Sequence[Scored], *, k: int, aggregation: str) -> list[metrics.Row]:
+    """``(gold, proposed label, score)`` rows for calibration, SciFact train only."""
+    _require(scored, "scifact-train")
+    rows: list[metrics.Row] = []
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        rows.append((Label(s.item.gold), signal.label, signal.score))
+    return rows
+
+
+def calibrate(rows: Sequence[metrics.Row]) -> Thresholds:
+    """Best-accuracy ``decide``, then the 0.85 / 0.70 precision cuts above it."""
+    best = metrics.sweep_decide(rows, grid=GRID)
+    raw_high, raw_medium = metrics.tier_cutpoints(
+        rows,
+        decide=best.threshold,
+        high_precision=HIGH_TARGET,
+        medium_precision=MEDIUM_TARGET,
+    )
+    high, medium = ship_cut(raw_high), ship_cut(raw_medium)
+    return Thresholds(decide=best.threshold, high=max(high, medium), medium=medium)
+
+
+@dataclass(frozen=True)
+class TierStat:
+    tier: str
+    n: int
+    correct: int
+
+    @property
+    def precision(self) -> float | None:
+        return self.correct / self.n if self.n >= MIN_TIER_N else None
+
+    @property
+    def interval(self) -> tuple[float, float] | None:
+        return wilson(self.correct, self.n) if self.n >= MIN_TIER_N else None
+
+
+@dataclass(frozen=True)
+class SciFactResult:
+    n: int
+    accuracy: float
+    macro_f1: float
+    f1: dict[Label, float]
+    rationale_f1: float
+    tiers: tuple[TierStat, ...]  # high, medium, low
+    asserted_without_passage: int
+
+
+def per_label_f1(gold: Sequence[Label], pred: Sequence[Label]) -> dict[Label, float]:
+    scores: dict[Label, float] = {}
+    for label in metrics.LABELS:
+        tp = sum(g is label and p is label for g, p in zip(gold, pred, strict=True))
+        fp = sum(g is not label and p is label for g, p in zip(gold, pred, strict=True))
+        fn = sum(g is label and p is not label for g, p in zip(gold, pred, strict=True))
+        scores[label] = 2 * tp / (2 * tp + fp + fn) if tp else 0.0
+    return scores
+
+
+def _unbacked(signal: Signal, thresholds: Thresholds) -> bool:
+    """An assertion over the cut with no passage under it (product rule 1)."""
+    return (
+        signal.label is not Label.NEI
+        and signal.score >= thresholds.decide
+        and signal.passage is None
+    )
+
+
+def evaluate_scifact(
+    scored: Sequence[Scored], thresholds: Thresholds, *, k: int, aggregation: str
+) -> SciFactResult:
+    _require(scored, "scifact-dev")
+    gold: list[Label] = []
+    pred: list[Label] = []
+    gold_rationale: list[frozenset[int]] = []
+    pred_rationale: list[frozenset[int]] = []
+    tiers = {"high": [0, 0], "medium": [0, 0], "low": [0, 0]}
+    unbacked = 0
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        unbacked += _unbacked(signal, thresholds)
+        label = decide(signal, thresholds)
+        truth = Label(s.item.gold)
+        gold.append(truth)
+        pred.append(label)
+        gold_rationale.append(s.item.rationale)
+        if label is not Label.NEI and signal.passage is not None:
+            pred_rationale.append(frozenset({signal.passage.index}))
+            counts = tiers[thresholds.tier(signal.score)]
+            counts[0] += 1
+            counts[1] += label is truth
+        else:
+            pred_rationale.append(frozenset())
+    return SciFactResult(
+        n=len(scored),
+        accuracy=metrics.accuracy(gold, pred),
+        macro_f1=metrics.macro_f1(gold, pred),
+        f1=per_label_f1(gold, pred),
+        rationale_f1=metrics.rationale_f1(gold_rationale, pred_rationale),
+        tiers=tuple(TierStat(name, n, c) for name, (n, c) in tiers.items()),
+        asserted_without_passage=unbacked,
+    )
+
+
+@dataclass(frozen=True)
+class AveritecResult:
+    n: int
+    counted: int  # claims with a 3-way gold label
+    accuracy_all: float
+    readable: int  # counted claims with at least one readable source
+    accuracy_readable: float
+    majority_baseline: float
+    per_label: dict[str, tuple[int, int]]  # gold label -> (n, correct)
+    asserted_without_passage: int
+
+
+def evaluate_averitec(
+    claims: Sequence[SnapshotClaim],
+    scored: Sequence[Scored],
+    thresholds: Thresholds,
+    *,
+    k: int,
+    aggregation: str,
+) -> AveritecResult:
+    """Per claim, the rule of ``eval_averitec._decide_claim``.
+
+    The strongest decided non-NEI verdict across the claim's sources wins. A claim
+    with a readable source and no assertion is NEI. A claim with nothing readable is
+    unanswered, and is scored as NEI.
+    """
+    _require(scored, "averitec")
+    by_group: dict[str, list[Scored]] = {}
+    for s in scored:
+        by_group.setdefault(s.item.group, []).append(s)
+    per_label: dict[str, tuple[int, int]] = {}
+    golds: list[Label] = []
+    correct_all = readable = correct_readable = unbacked = 0
+    for claim in claims:
+        group = by_group.get(f"averitec:{claim.claim_id}", [])
+        best: Signal | None = None
+        for s in group:
+            signal = signal_for(
+                s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation
+            )
+            unbacked += _unbacked(signal, thresholds)
+            if decide(signal, thresholds) is Label.NEI:
+                continue
+            if best is None or signal.score > best.score:
+                best = signal
+        predicted = best.label if best is not None else Label.NEI
+        gold = averitec.to_label(claim.gold)
+        correct = gold is not None and predicted is gold
+        seen, right = per_label.get(claim.gold, (0, 0))
+        per_label[claim.gold] = (seen + 1, right + int(correct))
+        if gold is None:
+            continue
+        golds.append(gold)
+        correct_all += correct
+        if group:
+            readable += 1
+            correct_readable += correct
+    majority = max(Counter(golds).values(), default=0)
+    return AveritecResult(
+        n=len(claims),
+        counted=len(golds),
+        accuracy_all=correct_all / len(golds) if golds else 0.0,
+        readable=readable,
+        accuracy_readable=correct_readable / readable if readable else 0.0,
+        majority_baseline=majority / len(golds) if golds else 0.0,
+        per_label=per_label,
+        asserted_without_passage=unbacked,
+    )
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: all passed
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src tests && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 4: The decision rule — `bakeoff.py` part 3
+
+**Files:**
+- Modify: `src/proofpath/eval/bakeoff.py` (append)
+- Test: `tests/test_bakeoff.py` (append)
+
+**Interfaces:**
+- Consumes: `AGGREGATIONS` from Task 2.
+- Produces:
+  - `TIE_MARGIN = 0.01`, `LARGE_MARGIN = 0.03`, `NO_CHANGE_MARGIN = 0.01`
+  - `DECISION_RULE: str`
+  - `Combo(model: str, k: int, aggregation: str)` with `.label() -> str`
+  - `Outcome(combo: Combo, large: bool, macro_f1: float, averitec_readable: float, asserted_without_passage: int)`
+  - `Choice(winner: Combo | None, reason: str)`
+  - `choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice`
+
+- [ ] **Step 1: Write the failing tests (append)**
+
+```python
+BASE = bakeoff.Combo("base", 1, "max")
+
+
+def _o(
+    model: str, k: int, agg: str, f1: float, *, readable: float = 0.40, unbacked: int = 0
+) -> bakeoff.Outcome:
+    return bakeoff.Outcome(bakeoff.Combo(model, k, agg), model != "base", f1, readable, unbacked)
+
+
+def test_no_change_when_nothing_beats_the_baseline_by_a_hundredth() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.604)], baseline=BASE
+    )
+    assert choice.winner is None
+    assert "no change" in choice.reason
+
+
+def test_a_base_row_that_clears_the_margin_wins() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 3, "margin", 0.630)], baseline=BASE
+    )
+    assert choice.winner == bakeoff.Combo("base", 3, "margin")
+
+
+def test_a_large_row_needs_three_hundredths_over_the_best_base_row() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max_nei", 0.620),
+        _o("large", 1, "max", 0.645),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+    rows.append(_o("large-fever", 1, "max", 0.651))
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large-fever", 1, "max")
+
+
+def test_ties_go_to_the_smaller_k_then_the_simpler_aggregation() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.640),
+        _o("base", 2, "margin", 0.635),
+        _o("base", 2, "max_nei", 0.632),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+
+
+def test_a_row_that_asserts_without_a_passage_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.700, unbacked=1)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_a_row_worse_on_averitec_readable_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597, readable=0.40), _o("base", 2, "max", 0.700, readable=0.39)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_the_baseline_must_be_among_the_outcomes() -> None:
+    with pytest.raises(ValueError, match="baseline"):
+        bakeoff.choose([_o("base", 2, "max", 0.6)], baseline=BASE)
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: FAIL with `AttributeError: module 'proofpath.eval.bakeoff' has no attribute 'Combo'`
+
+- [ ] **Step 3: Write the implementation (append)**
+
+```python
+TIE_MARGIN = 0.01
+LARGE_MARGIN = 0.03
+NO_CHANGE_MARGIN = 0.01
+# Float slack, so a difference printed as 0.030 counts as 0.03.
+_EPS = 1e-9
+
+DECISION_RULE = """\
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Winner:** the highest SciFact dev macro-F1 among eligible rows.
+3. **Size gate:** a `large*` row wins only if it beats the best eligible `base` row by
+   **≥ 0.03** macro-F1. Otherwise the best `base` row wins.
+4. **Tie:** two rows within 0.01 of each other go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`.
+5. If nothing beats `base` × k=1 × `max` by ≥ 0.01 macro-F1, the result is "no change".
+   It is written up as such, and the next package (B, coverage) starts."""
+
+
+@dataclass(frozen=True)
+class Combo:
+    model: str
+    k: int
+    aggregation: str
+
+    def label(self) -> str:
+        return f"`{self.model}` × k={self.k} × `{self.aggregation}`"
+
+
+@dataclass(frozen=True)
+class Outcome:
+    combo: Combo
+    large: bool
+    macro_f1: float  # SciFact dev
+    averitec_readable: float
+    asserted_without_passage: int
+
+
+@dataclass(frozen=True)
+class Choice:
+    winner: Combo | None  # None means "no change"
+    reason: str
+
+
+def _pick(rows: Sequence[Outcome]) -> Outcome:
+    top = max(row.macro_f1 for row in rows)
+    contenders = [row for row in rows if top - row.macro_f1 <= TIE_MARGIN + _EPS]
+    return min(
+        contenders,
+        key=lambda row: (
+            row.combo.k,
+            AGGREGATIONS.index(row.combo.aggregation),
+            -row.macro_f1,
+        ),
+    )
+
+
+def choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
+    """Apply ``DECISION_RULE`` to the measured rows."""
+    by_combo = {row.combo: row for row in outcomes}
+    if baseline not in by_combo:
+        raise ValueError(f"the baseline {baseline.label()} is not among the outcomes")
+    base = by_combo[baseline]
+    eligible = [
+        row
+        for row in outcomes
+        if row.asserted_without_passage == 0
+        and row.averitec_readable >= base.averitec_readable - _EPS
+    ]
+    small = [row for row in eligible if not row.large]
+    large = [row for row in eligible if row.large]
+    if not small:
+        return Choice(None, "no change: no eligible base row (the baseline itself failed rule 1)")
+    candidate = _pick(small)
+    if large:
+        best_large = _pick(large)
+        if best_large.macro_f1 - candidate.macro_f1 >= LARGE_MARGIN - _EPS:
+            candidate = best_large
+    gain = candidate.macro_f1 - base.macro_f1
+    if gain < NO_CHANGE_MARGIN - _EPS:
+        return Choice(
+            None,
+            f"no change: the best eligible row, {candidate.combo.label()}, is "
+            f"{gain:+.3f} macro-F1 over the baseline, below {NO_CHANGE_MARGIN}",
+        )
+    return Choice(
+        candidate.combo,
+        f"{candidate.combo.label()} is {gain:+.3f} macro-F1 over {baseline.label()}",
+    )
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_bakeoff.py -q`
+Expected: all passed
+
+- [ ] **Step 5: Lint and type-check**
+
+Run: `uv run ruff check src tests && uv run ruff format --check src tests && uv run mypy`
+Expected: no errors
+
+---
+
+### Task 5: The script's `snapshot` and `score` subcommands
+
+**Files:**
+- Create: `scripts/eval_nli_bakeoff.py`
+- Test: `tests/test_eval_nli_bakeoff.py`
+
+**Interfaces:**
+- Consumes:
+  - Tasks 1–4
+  - `proofpath.retrieval.rank`, `split_sentences`, `FastEmbedder`
+  - `proofpath.entailment.OnnxNli(cache_dir=, repo_id=, revision=, onnx_file=, providers=)` and `pick_onnx_file`
+  - `proofpath.verify.Engine.default(config, interactive=False, browser=False)`, then `.fetcher.fetch(url)` and `.close()`
+  - `proofpath.eval.scifact.ensure_downloaded/load` and `proofpath.eval.averitec.ensure_downloaded/load`
+- Produces:
+  - `Candidate(repo, revision, onnx_file: Callable[[str], str], large: bool)`
+  - `CANDIDATES: dict[str, Candidate]`
+  - `BASELINE = bakeoff.Combo("base", 1, "max")`
+  - `bakeoff_dir() -> Path`, `snapshot_path() -> Path`, `items_path() -> Path`
+  - `nli_providers() -> list[str]`
+  - `build_items(embedder: retrieval.Embedder) -> list[Item]`
+  - `main(argv: list[str] | None = None) -> int`, with subcommands `snapshot`, `score` and `report`. `report` is a stub that returns 2 until Task 6.
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+"""Offline tests for scripts/eval_nli_bakeoff.py's importable parts."""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+from types import ModuleType
+
+SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "eval_nli_bakeoff.py"
+
+
+def _load_script() -> ModuleType:
+    spec = importlib.util.spec_from_file_location("eval_nli_bakeoff", SCRIPT_PATH)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+script = _load_script()
+
+
+def test_every_candidate_is_pinned_to_a_full_revision() -> None:
+    assert set(script.CANDIDATES) == {"base", "large", "large-fever"}
+    for candidate in script.CANDIDATES.values():
+        assert len(candidate.revision) == 40
+        int(candidate.revision, 16)
+
+
+def test_only_the_large_models_are_marked_large() -> None:
+    assert {name for name, c in script.CANDIDATES.items() if c.large} == {"large", "large-fever"}
+
+
+def test_each_candidate_names_an_onnx_file_per_machine() -> None:
+    for machine in ("arm64", "x86_64"):
+        for candidate in script.CANDIDATES.values():
+            assert candidate.onnx_file(machine).startswith("onnx/")
+            assert candidate.onnx_file(machine).endswith(".onnx")
+
+
+def test_the_baseline_is_base_k1_max() -> None:
+    assert (script.BASELINE.model, script.BASELINE.k, script.BASELINE.aggregation) == (
+        "base",
+        1,
+        "max",
+    )
+
+
+def test_coreml_is_never_handed_to_the_int8_models() -> None:
+    assert "CoreMLExecutionProvider" not in script.nli_providers()
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: FAIL with `FileNotFoundError` (the script does not exist yet)
+
+- [ ] **Step 3: Write the script**
+
+```python
+"""NLI bake-off harness (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Usage:
+    uv run python scripts/eval_nli_bakeoff.py snapshot [--limit 100] [--sleep 1.0] [--resume]
+    uv run python scripts/eval_nli_bakeoff.py score --model base|large|large-fever
+        [--rebuild-items]
+    uv run python scripts/eval_nli_bakeoff.py report [--out docs/eval/<date>-nli-bakeoff.md]
+
+``snapshot`` is the only network step, apart from the one-time dataset and model
+downloads. ``score`` runs one NLI model over every stored (passage, claim) pair.
+``report`` is offline. Importing this module touches neither the network nor a model.
+"""
+
+from __future__ import annotations
+
+import argparse
+import platform
+import sys
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from proofpath import retrieval
+from proofpath.entailment import pick_onnx_file
+from proofpath.eval import averitec, bakeoff, scifact
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    load_items,
+    load_snapshot,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+)
+from proofpath.models import Passage
+from proofpath.paths import cache_dir, models_dir
+
+EMBEDDER = "BAAI/bge-small-en-v1.5"
+
+
+@dataclass(frozen=True)
+class Candidate:
+    repo: str
+    revision: str
+    onnx_file: Callable[[str], str]  # platform.machine() -> file inside the repo
+    large: bool
+
+
+def _fever_file(_machine: str) -> str:
+    # The repo ships one dynamic-int8 export for every CPU.
+    return "onnx/model_quantized.onnx"
+
+
+CANDIDATES: dict[str, Candidate] = {
+    "base": Candidate(
+        "cross-encoder/nli-deberta-v3-base",
+        "6c749ce3425cd33b46d187e45b92bbf96ee12ec7",
+        pick_onnx_file,
+        large=False,
+    ),
+    "large": Candidate(
+        "cross-encoder/nli-deberta-v3-large",
+        "bab4bc7178836f731dcfd18c06ca9def0a137712",
+        pick_onnx_file,
+        large=True,
+    ),
+    "large-fever": Candidate(
+        "MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
+        "b3546ea6b0346eb6f8d5d68b13c7dc6d0376b3d7",
+        _fever_file,
+        large=True,
+    ),
+}
+BASELINE = bakeoff.Combo("base", 1, "max")
+# Pairs handed to the scorer per call: several of its 16-pair batches, so progress
+# can be printed without starving the batches.
+SCORE_CHUNK = 400
+
+
+def bakeoff_dir() -> Path:
+    return cache_dir() / "datasets" / "bakeoff"
+
+
+def snapshot_path() -> Path:
+    return cache_dir() / "datasets" / "averitec_snapshot.json"
+
+
+def items_path() -> Path:
+    return bakeoff_dir() / "items.json"
+
+
+def nli_providers() -> list[str]:
+    """CUDA first where present, then CPU.
+
+    CoreML is left out for every candidate. All three files are int8, and the
+    product leaves CoreML out for int8 too (``entailment.providers_for``).
+    """
+    from proofpath.device import onnx_providers
+
+    return [p for p in onnx_providers() if p != "CoreMLExecutionProvider"]
+
+
+def _ranked(
+    claim: str, passages: list[Passage], embedder: retrieval.Embedder
+) -> tuple[Passage, ...]:
+    hits = retrieval.rank(claim, passages, embedder, k=bakeoff.TOP_N)
+    return tuple(hit.passage for hit in hits)
+
+
+def build_items(embedder: retrieval.Embedder) -> list[Item]:
+    """SciFact train and dev pairs, then every readable AVeriTeC source, top 5 each.
+
+    Passages are split exactly as the product splits them: SciFact ships its
+    sentences, and a page goes through ``retrieval.split_sentences`` as in
+    ``eval_averitec._decide_claim``.
+    """
+    items: list[Item] = []
+    tarball = scifact.ensure_downloaded(cache_dir())
+    for split in ("train", "dev"):
+        data = scifact.load(tarball, split=split)
+        for pair in data.pairs():
+            doc = data.corpus[pair.doc_id]
+            passages = [Passage(t, str(doc.doc_id), i) for i, t in enumerate(doc.sentences)]
+            key = f"scifact-{split}:{pair.claim_id}:{pair.doc_id}"
+            items.append(
+                Item(
+                    key,
+                    f"scifact-{split}",
+                    key,
+                    pair.claim,
+                    pair.label.value,
+                    _ranked(pair.claim, passages, embedder),
+                    pair.rationale,
+                )
+            )
+    for claim in load_snapshot(snapshot_path()):
+        for position, source in enumerate(claim.sources):
+            sentences = retrieval.split_sentences(source.text) if source.text.strip() else []
+            if not sentences:
+                continue
+            passages = [Passage(s, source.url, i) for i, s in enumerate(sentences)]
+            items.append(
+                Item(
+                    f"averitec:{claim.claim_id}:{position}",
+                    "averitec",
+                    f"averitec:{claim.claim_id}",
+                    claim.claim,
+                    claim.gold,
+                    _ranked(claim.claim, passages, embedder),
+                )
+            )
+    return items
+
+
+def cmd_snapshot(args: argparse.Namespace) -> int:
+    from proofpath.config import load_config
+    from proofpath.verify import Engine
+
+    path = snapshot_path()
+    if path.exists() and not args.resume:
+        print(
+            f"error     {path} already holds a snapshot.\n"
+            "          Pass --resume to continue it, or delete the file to start over.",
+            file=sys.stderr,
+        )
+        return 2
+    claims = averitec.load(averitec.ensure_downloaded(cache_dir()), limit=args.limit or None)
+    stored = load_snapshot(path) if args.resume else []
+    todo = pending(claims, stored)
+    print(f"snapshot  {len(claims)} claims, {len(stored)} done, {len(todo)} to fetch")
+    # Browser not permitted and no prompts: the default install, as in the published
+    # AVeriTeC number (spec: Data).
+    engine = Engine.default(load_config(), interactive=False, browser=False)
+    try:
+        for position, claim in enumerate(todo, start=1):
+            sources = []
+            for url in claim.source_urls:
+                if args.sleep:
+                    time.sleep(args.sleep)
+                sources.append(source_from_fetched(url, engine.fetcher.fetch(url)))
+            stored.append(
+                SnapshotClaim(claim.id, claim.text, claim.label, tuple(sources), claim.non_urls)
+            )
+            save_snapshot(path, stored)
+            read = sum(bool(s.text) for s in sources)
+            print(
+                f"  {position}/{len(todo)}  claim {claim.id}  read {read}/{len(sources)}",
+                file=sys.stderr,
+            )
+    except KeyboardInterrupt:
+        print(f"\ninterrupted; {path} is usable with --resume", file=sys.stderr)
+    finally:
+        engine.close()
+    return 0
+
+
+def _items(rebuild: bool) -> list[Item]:
+    if items_path().exists() and not rebuild:
+        return load_items(items_path())
+    embedder = retrieval.FastEmbedder(EMBEDDER, cache_dir=models_dir())
+    try:
+        items = build_items(embedder)
+    finally:
+        embedder.close()
+    save_items(items_path(), items)
+    print(f"items     {len(items)} written to {items_path()}")
+    return items
+
+
+def cmd_score(args: argparse.Namespace) -> int:
+    from proofpath.entailment import OnnxNli
+
+    if not snapshot_path().exists():
+        print("error     no AVeriTeC snapshot; run `snapshot` first", file=sys.stderr)
+        return 2
+    items = _items(args.rebuild_items)
+    candidate = CANDIDATES[args.model]
+    scorer = OnnxNli(
+        cache_dir=models_dir(),
+        repo_id=candidate.repo,
+        revision=candidate.revision,
+        onnx_file=candidate.onnx_file(platform.machine()),
+        providers=nli_providers(),
+    )
+    print(f"nli       {scorer.name}  providers={scorer.providers}")
+    pairs = [(p.text, item.claim) for item in items for p in item.passages]
+    try:
+        started = time.perf_counter()
+        blocks: list[np.ndarray] = []
+        for start in range(0, len(pairs), SCORE_CHUNK):
+            blocks.append(scorer.score(pairs[start : start + SCORE_CHUNK]))
+            done = min(start + SCORE_CHUNK, len(pairs))
+            print(
+                f"  {done}/{len(pairs)} pairs  {time.perf_counter() - started:.0f}s",
+                file=sys.stderr,
+            )
+        elapsed = time.perf_counter() - started
+    finally:
+        scorer.close()
+    flat = np.concatenate(blocks) if blocks else np.zeros((0, 3), dtype=np.float32)
+    offsets = np.cumsum([0, *(len(item.passages) for item in items)])
+    probs = [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))]
+    ms_per_pair = elapsed / max(len(pairs), 1) * 1000
+    out = bakeoff_dir() / f"{args.model}.npz"
+    save_probs(out, items, probs, ms_per_pair=ms_per_pair)
+    print(f"scored    {len(pairs)} pairs, {ms_per_pair:.0f} ms/pair -> {out}")
+    return 0
+
+
+def cmd_report(args: argparse.Namespace) -> int:
+    print("error     report is implemented in Task 6", file=sys.stderr)
+    return 2
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+    snap = sub.add_parser("snapshot", help="fetch AVeriTeC gold sources once (network)")
+    snap.add_argument("--limit", type=int, default=100, help="first N dev claims")
+    snap.add_argument("--sleep", type=float, default=1.0, help="seconds between fetches")
+    snap.add_argument("--resume", action="store_true", help="skip claims already stored")
+    score = sub.add_parser("score", help="run one NLI model over every stored pair")
+    score.add_argument("--model", required=True, choices=sorted(CANDIDATES))
+    score.add_argument("--rebuild-items", action="store_true", help="re-run retrieval")
+    report = sub.add_parser("report", help="calibrate, evaluate and choose (offline)")
+    report.add_argument("--out", default="")
+    args = parser.parse_args(argv)
+    commands = {"snapshot": cmd_snapshot, "score": cmd_score, "report": cmd_report}
+    return commands[args.command](args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: 5 passed
+
+- [ ] **Step 5: Lint**
+
+Run: `uv run ruff check scripts/eval_nli_bakeoff.py tests && uv run ruff format --check scripts tests`
+Expected: no errors. Wrap any line over 100 characters as ruff formats it.
+
+- [ ] **Step 6: Smoke-test that each model loads (controller-run, network, ~1.3 GB once)**
+
+Run:
+```bash
+uv run python - <<'EOF'
+import platform, importlib.util, sys
+spec = importlib.util.spec_from_file_location("b", "scripts/eval_nli_bakeoff.py")
+m = importlib.util.module_from_spec(spec); sys.modules["b"] = m; spec.loader.exec_module(m)
+from proofpath.entailment import OnnxNli
+from proofpath.paths import models_dir
+pairs = [("The cat sat on the mat.", "A cat is sitting."), ("The cat sat on the mat.", "No animal is present.")]
+for name, c in m.CANDIDATES.items():
+    s = OnnxNli(cache_dir=models_dir(), repo_id=c.repo, revision=c.revision,
+                onnx_file=c.onnx_file(platform.machine()), providers=m.nli_providers())
+    print(name, s.score(pairs).round(3).tolist()); s.close()
+EOF
+```
+Expected: for every model, row 0 has its largest value in column 0 (SUPPORTED) and row 1 in column 1 (REFUTED). If `large-fever` fails to load on arm64 (no int8 kernel), stop and report back; do not substitute a different file without asking.
+
+---
+
+### Task 6: The `report` subcommand and `render_report`
+
+**Files:**
+- Modify: `scripts/eval_nli_bakeoff.py`. Replace `cmd_report` and add `ReportRow`, `model_size_mb` and `render_report`.
+- Test: `tests/test_eval_nli_bakeoff.py` (append)
+
+**Interfaces:**
+- Consumes:
+  - Tasks 1–5
+  - `huggingface_hub.try_to_load_from_cache(repo_id, filename, cache_dir=, revision=)`
+- Produces:
+  - `ReportRow(combo: bakeoff.Combo, large: bool, thresholds: Thresholds, scifact: bakeoff.SciFactResult, averitec: bakeoff.AveritecResult)`
+  - `render_report(rows, choice, *, speed: Mapping[str, float], sizes: Mapping[str, float | None], counts: Mapping[str, int], today: str, machine: str) -> str`
+
+- [ ] **Step 1: Write the failing tests (append)**
+
+```python
+from proofpath.eval import bakeoff
+from proofpath.models import Label
+from proofpath.pipeline import Thresholds
+
+
+def _row(model: str, k: int, agg: str, f1: float, tiers: tuple[int, int, int]) -> object:
+    sci = bakeoff.SciFactResult(
+        n=340,
+        accuracy=f1 + 0.01,
+        macro_f1=f1,
+        f1={Label.SUPPORTED: 0.7, Label.REFUTED: 0.5, Label.NEI: 0.6},
+        rationale_f1=0.3,
+        tiers=(
+            bakeoff.TierStat("high", tiers[0], tiers[0]),
+            bakeoff.TierStat("medium", tiers[1], tiers[1] // 2),
+            bakeoff.TierStat("low", tiers[2], 0),
+        ),
+        asserted_without_passage=0,
+    )
+    ave = bakeoff.AveritecResult(
+        n=100,
+        counted=89,
+        accuracy_all=0.27,
+        readable=61,
+        accuracy_readable=0.36,
+        majority_baseline=0.708,
+        per_label={"Refuted": (63, 20)},
+        asserted_without_passage=0,
+    )
+    return script.ReportRow(
+        bakeoff.Combo(model, k, agg),
+        model != "base",
+        Thresholds(decide=0.45, high=0.99933, medium=0.457948),
+        sci,
+        ave,
+    )
+
+
+def _render(choice: bakeoff.Choice) -> str:
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("large", 2, "margin", 0.650, (30, 40, 25)),
+    ]
+    return script.render_report(
+        rows,
+        choice,
+        speed={"base": 185.0, "large": 540.0},
+        sizes={"base": 244.4, "large": None},
+        counts={"scifact-train": 919, "scifact-dev": 340, "averitec": 100},
+        today="2026-10-02",
+        machine="Darwin arm64",
+    )
+
+
+def test_the_report_prints_the_rule_verbatim_and_the_winner() -> None:
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"))
+    assert bakeoff.DECISION_RULE in text
+    assert "**Winner:** `large` × k=2 × `margin`" in text
+    assert "| base | 1 | max | 0.45 |" in text
+    assert "| large | 2 | margin | 0.45 |" in text
+
+
+def test_the_report_says_no_change_in_words() -> None:
+    text = _render(bakeoff.Choice(None, "no change: below 0.01"))
+    assert "**No change.**" in text
+    assert "no change: below 0.01" in text
+
+
+def test_a_tier_below_the_minimum_n_is_too_few_to_judge() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "too few to judge" in text
+
+
+def test_an_unknown_model_size_is_a_dash_not_zero() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "| large | `cross-encoder/nli-deberta-v3-large` | — |" in text
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: FAIL with `AttributeError: module 'eval_nli_bakeoff' has no attribute 'ReportRow'`
+
+- [ ] **Step 3: Write the implementation**
+
+Add `from collections.abc import Callable, Mapping, Sequence`, `from datetime import date`, `from proofpath.eval.bakeoff_data import load_probs` and `from proofpath.pipeline import Thresholds` to the imports. Then replace `cmd_report`:
+
+```python
+@dataclass(frozen=True)
+class ReportRow:
+    combo: bakeoff.Combo
+    large: bool
+    thresholds: Thresholds
+    scifact: bakeoff.SciFactResult
+    averitec: bakeoff.AveritecResult
+
+
+def model_size_mb(model: str) -> float | None:
+    """Size of the ONNX file the run used, or None when it is not in the cache."""
+    from huggingface_hub import try_to_load_from_cache
+
+    candidate = CANDIDATES[model]
+    found = try_to_load_from_cache(
+        candidate.repo,
+        candidate.onnx_file(platform.machine()),
+        cache_dir=str(models_dir()),
+        revision=candidate.revision,
+    )
+    return Path(found).stat().st_size / 1e6 if isinstance(found, str) else None
+
+
+def _md(*cells: object) -> str:
+    return "| " + " | ".join(str(c) for c in cells) + " |"
+
+
+def _tier_lines(row: ReportRow) -> list[str]:
+    lines = [
+        f"### {row.combo.label()}  (decide {row.thresholds.decide:.2f}, "
+        f"medium {row.thresholds.medium:.6f}, high {row.thresholds.high:.6f})",
+        "",
+        "| tier | n | precision | 95 % Wilson |",
+        "|---|---|---|---|",
+    ]
+    for tier in row.scifact.tiers:
+        if tier.precision is None or tier.interval is None:
+            lines.append(_md(tier.tier, tier.n, "too few to judge", "—"))
+        else:
+            low, high = tier.interval
+            lines.append(_md(tier.tier, tier.n, f"{tier.precision:.3f}", f"{low:.2f}–{high:.2f}"))
+    lines.append("")
+    return lines
+
+
+def render_report(
+    rows: Sequence[ReportRow],
+    choice: bakeoff.Choice,
+    *,
+    speed: Mapping[str, float],
+    sizes: Mapping[str, float | None],
+    counts: Mapping[str, int],
+    today: str,
+    machine: str,
+) -> str:
+    readable = rows[0].averitec.readable if rows else 0
+    baseline_ave = rows[0].averitec.majority_baseline if rows else 0.0
+    lines = [
+        f"# NLI bake-off — {today}",
+        "",
+        "- spec: `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`",
+        f"- calibrated on SciFact train ({counts.get('scifact-train', 0)} pairs); reported on "
+        f"SciFact dev ({counts.get('scifact-dev', 0)} pairs) and AVeriTeC dev "
+        f"({counts.get('averitec', 0)} claims, {readable} answerable with a readable source; "
+        "frozen snapshot, browser not permitted)",
+        f"- embedder `{EMBEDDER}`, top {bakeoff.TOP_N} passages per item, numeric layer on",
+        f"- machine: {machine}",
+        "",
+        "## Decision rule (fixed before any number was seen)",
+        "",
+        bakeoff.DECISION_RULE,
+        "",
+        "## Result",
+        "",
+    ]
+    if choice.winner is None:
+        lines.append(f"**No change.** {choice.reason}")
+    else:
+        lines.append(f"**Winner:** {choice.winner.label()} — {choice.reason}")
+    lines.extend(
+        ["", "## Models", "", "| model | repo | size (MB) | ms/pair |", "|---|---|---|---|"]
+    )
+    for model in dict.fromkeys(row.combo.model for row in rows):
+        size = sizes.get(model)
+        lines.append(
+            _md(
+                model,
+                f"`{CANDIDATES[model].repo}`",
+                "—" if size is None else f"{size:.0f}",
+                f"{speed.get(model, 0.0):.0f}",
+            )
+        )
+    lines.extend(
+        [
+            "",
+            "## Every combination",
+            "",
+            f"AVeriTeC majority baseline: {baseline_ave:.3f}. Published numbers to compare: "
+            "0.270 (all) and 0.361 (readable), 2026-09-16.",
+            "",
+            "| model | k | aggregation | decide | dev acc | dev macro-F1 | F1 S / R / NEI "
+            "| rationale F1 | AVeriTeC all | AVeriTeC readable | w/o passage |",
+            "|---|---|---|---|---|---|---|---|---|---|---|",
+        ]
+    )
+    for row in rows:
+        f1 = row.scifact.f1
+        lines.append(
+            _md(
+                row.combo.model,
+                row.combo.k,
+                row.combo.aggregation,
+                f"{row.thresholds.decide:.2f}",
+                f"{row.scifact.accuracy:.3f}",
+                f"{row.scifact.macro_f1:.3f}",
+                f"{f1[Label.SUPPORTED]:.2f} / {f1[Label.REFUTED]:.2f} / {f1[Label.NEI]:.2f}",
+                f"{row.scifact.rationale_f1:.3f}",
+                f"{row.averitec.accuracy_all:.3f}",
+                f"{row.averitec.accuracy_readable:.3f}",
+                row.scifact.asserted_without_passage + row.averitec.asserted_without_passage,
+            )
+        )
+    lines.extend(["", "## Tiers on SciFact dev", ""])
+    shown = [BASELINE] + ([choice.winner] if choice.winner and choice.winner != BASELINE else [])
+    for combo in shown:
+        lines.extend(_tier_lines(next(r for r in rows if r.combo == combo)))
+    return "\n".join(lines) + "\n"
+
+
+def cmd_report(args: argparse.Namespace) -> int:
+    if not items_path().exists() or not snapshot_path().exists():
+        print("error     run `snapshot` and `score --model base` first", file=sys.stderr)
+        return 2
+    items = load_items(items_path())
+    snapshot = load_snapshot(snapshot_path())
+    rows: list[ReportRow] = []
+    speed: dict[str, float] = {}
+    for model, candidate in CANDIDATES.items():
+        path = bakeoff_dir() / f"{model}.npz"
+        if not path.exists():
+            print(f"skip      {model}: not scored", file=sys.stderr)
+            continue
+        probs, speed[model] = load_probs(path, items)
+        scored = [bakeoff.Scored(i, p) for i, p in zip(items, probs, strict=True)]
+        train = [s for s in scored if s.item.dataset == "scifact-train"]
+        dev = [s for s in scored if s.item.dataset == "scifact-dev"]
+        ave = [s for s in scored if s.item.dataset == "averitec"]
+        for k in bakeoff.KS:
+            for aggregation in bakeoff.AGGREGATIONS:
+                thresholds = bakeoff.calibrate(
+                    bakeoff.calibration_rows(train, k=k, aggregation=aggregation)
+                )
+                rows.append(
+                    ReportRow(
+                        bakeoff.Combo(model, k, aggregation),
+                        candidate.large,
+                        thresholds,
+                        bakeoff.evaluate_scifact(dev, thresholds, k=k, aggregation=aggregation),
+                        bakeoff.evaluate_averitec(
+                            snapshot, ave, thresholds, k=k, aggregation=aggregation
+                        ),
+                    )
+                )
+    if "base" not in speed:
+        print(
+            "error     the baseline model is not scored; run `score --model base`", file=sys.stderr
+        )
+        return 2
+    outcomes = [
+        bakeoff.Outcome(
+            r.combo,
+            r.large,
+            r.scifact.macro_f1,
+            r.averitec.accuracy_readable,
+            r.scifact.asserted_without_passage + r.averitec.asserted_without_passage,
+        )
+        for r in rows
+    ]
+    choice = bakeoff.choose(outcomes, baseline=BASELINE)
+    counts = {
+        "scifact-train": sum(i.dataset == "scifact-train" for i in items),
+        "scifact-dev": sum(i.dataset == "scifact-dev" for i in items),
+        "averitec": len(snapshot),
+    }
+    report = render_report(
+        rows,
+        choice,
+        speed=speed,
+        sizes={model: model_size_mb(model) for model in speed},
+        counts=counts,
+        today=date.today().isoformat(),
+        machine=f"{platform.system()} {platform.machine()}",
+    )
+    out = (
+        Path(args.out)
+        if args.out
+        else Path("docs/eval") / f"{date.today().isoformat()}-nli-bakeoff.md"
+    )
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(report, encoding="utf-8")
+    print(report)
+    print(f"written   {out}")
+    return 0
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `uv run pytest tests/test_eval_nli_bakeoff.py -q`
+Expected: 9 passed
+
+- [ ] **Step 5: Full gate**
+
+Run: `uv run ruff check src tests scripts && uv run ruff format --check . && uv run mypy && uv run pytest -q`
+Expected: no lint or type errors; the whole suite passes.
+
+---
+
+### Task 7: Run the bake-off and write it up (controller)
+
+This task is long-running measurement, not code. The controller runs it, and a reviewer checks the write-up against the numbers.
+
+- [ ] **Step 1: Snapshot (network, about 30–60 min at `--sleep 1.0`)**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py snapshot --limit 100 --sleep 1.0 --resume`
+Expected: `averitec_snapshot.json` holds 100 claims. If it is interrupted, re-run the same command.
+
+- [ ] **Step 2: Score the baseline first**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py score --model base`
+Expected: `bakeoff/base.npz` is written. The ms/pair for base should be the same order as the 185 ms/pair measured on 2026-09-12; it may be lower, because batches are now filled.
+
+- [ ] **Step 3: Sanity-check before spending hours on the large models**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py report --out /tmp/bakeoff-base.md`
+Expected: the `base` × k=1 × `max` row has a dev accuracy near 0.609. It will not match exactly: the calibration now comes from train, not dev. If it is off by more than 0.05, stop and investigate with superpowers:systematic-debugging before going on.
+
+- [ ] **Step 4: Score both large models (about 1 hour each on Apple Silicon CPU)**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py score --model large && uv run python scripts/eval_nli_bakeoff.py score --model large-fever`
+
+- [ ] **Step 5: Write the report**
+
+Run: `uv run python scripts/eval_nli_bakeoff.py report`
+Expected: `docs/eval/<date>-nli-bakeoff.md`, with the winner or "No change" stated by the rule.
+
+- [ ] **Step 6: Record the outcome**
+
+Append one row to the backlog table in OPEN-ITEMS §20:
+
+`| 20.11 | NLI bake-off result | <winner or "no change">, <one line of numbers>; docs/eval/<date>-nli-bakeoff.md. Next: <follow-up spec, or package B> |`
+
+Change no README, spec §14 or product numbers here. That belongs to the follow-up spec.
+
+- [ ] **Step 7: Final gate and commit**
+
+Run: `uv run ruff check . && uv run ruff format --check . && uv run mypy && uv run pytest -q`
+Then the controller commits:
+- the script, the two modules, the tests, the report and OPEN-ITEMS;
+- on `feat/nli-bakeoff`, with the message `feat(eval): add the NLI bake-off harness and its first result`.
+
+Push and merge only with the user's go.

From 15059c4c0cd2ea86818aa820b06f04bc683c5090 Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:28:47 +0300
Subject: [PATCH 03/10] feat(eval): add the bake-off's stored inputs (snapshot,
 items, probabilities)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 src/proofpath/eval/bakeoff_data.py | 211 +++++++++++++++++++++++++++++
 tests/test_bakeoff_data.py         | 129 ++++++++++++++++++
 2 files changed, 340 insertions(+)
 create mode 100644 src/proofpath/eval/bakeoff_data.py
 create mode 100644 tests/test_bakeoff_data.py

diff --git a/src/proofpath/eval/bakeoff_data.py b/src/proofpath/eval/bakeoff_data.py
new file mode 100644
index 0000000..5b3d8c5
--- /dev/null
+++ b/src/proofpath/eval/bakeoff_data.py
@@ -0,0 +1,211 @@
+"""Stored inputs of the NLI bake-off (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Three files under ``<cache>/datasets``, written once and read by every later step:
+the frozen AVeriTeC snapshot (page text per source URL), the retrieved items (top
+passages per claim, shared by every model), and one npz of raw NLI probabilities per
+model. Nothing here touches the network or a model.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from proofpath.eval import averitec
+from proofpath.fetch import Fetched
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+DATASETS: tuple[str, ...] = ("scifact-train", "scifact-dev", "averitec")
+
+
+@dataclass(frozen=True)
+class SnapshotSource:
+    """One source URL as the snapshot run found it. ``text`` is "" unless it was read."""
+
+    url: str
+    state: str
+    text: str
+
+
+@dataclass(frozen=True)
+class SnapshotClaim:
+    claim_id: int
+    claim: str
+    gold: str  # the AVeriTeC label, verbatim
+    sources: tuple[SnapshotSource, ...]
+    non_urls: int
+
+
+@dataclass(frozen=True)
+class Item:
+    """One (claim, source) unit with its top retrieved passages, best first.
+
+    ``group`` is the claim an item belongs to: a SciFact pair is its own group, an
+    AVeriTeC claim groups every readable source it has.
+    """
+
+    key: str
+    dataset: str
+    group: str
+    claim: str
+    gold: str
+    passages: tuple[Passage, ...]
+    rationale: frozenset[int] = frozenset()
+
+    def __post_init__(self) -> None:
+        if self.dataset not in DATASETS:
+            raise ValueError(f"unknown dataset {self.dataset!r}")
+
+
+def source_from_fetched(url: str, fetched: Fetched) -> SnapshotSource:
+    """The state and text one fetch earned, worded as ``eval_averitec.state_for`` does.
+
+    A page that was reached but yielded no text is not ``ok`` (product rule 6), and
+    only a read page keeps its text: a challenge page's words are not the source.
+    """
+    if fetched.ok and fetched.text.strip():
+        return SnapshotSource(url, fetched.outcome.value, fetched.text)
+    if fetched.ok:
+        return SnapshotSource(url, NO_TEXT, "")
+    return SnapshotSource(url, fetched.outcome.value, "")
+
+
+def pending(
+    claims: Sequence[averitec.Claim], done: Sequence[SnapshotClaim]
+) -> list[averitec.Claim]:
+    """The claims a resumed snapshot run still has to fetch, in dataset order."""
+    finished = {claim.claim_id for claim in done}
+    return [claim for claim in claims if claim.id not in finished]
+
+
+def _write_atomic(path: Path, payload: str) -> None:
+    """Write a whole file at once, so a Ctrl-C never leaves half of one."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    temporary.write_text(payload, encoding="utf-8")
+    temporary.replace(path)
+
+
+def save_snapshot(path: Path, claims: Sequence[SnapshotClaim]) -> None:
+    payload = [
+        {
+            "claim_id": claim.claim_id,
+            "claim": claim.claim,
+            "gold": claim.gold,
+            "non_urls": claim.non_urls,
+            "sources": [{"url": s.url, "state": s.state, "text": s.text} for s in claim.sources],
+        }
+        for claim in claims
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_snapshot(path: Path) -> list[SnapshotClaim]:
+    if not path.exists():
+        return []
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        SnapshotClaim(
+            claim_id=int(row["claim_id"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            sources=tuple(
+                SnapshotSource(str(s["url"]), str(s["state"]), str(s["text"]))
+                for s in row["sources"]
+            ),
+            non_urls=int(row["non_urls"]),
+        )
+        for row in raw
+    ]
+
+
+def save_items(path: Path, items: Sequence[Item]) -> None:
+    payload = [
+        {
+            "key": item.key,
+            "dataset": item.dataset,
+            "group": item.group,
+            "claim": item.claim,
+            "gold": item.gold,
+            "passages": [[p.text, p.source_id, p.index] for p in item.passages],
+            "rationale": sorted(item.rationale),
+        }
+        for item in items
+    ]
+    _write_atomic(path, json.dumps(payload, indent=1))
+
+
+def load_items(path: Path) -> list[Item]:
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    return [
+        Item(
+            key=str(row["key"]),
+            dataset=str(row["dataset"]),
+            group=str(row["group"]),
+            claim=str(row["claim"]),
+            gold=str(row["gold"]),
+            passages=tuple(Passage(str(t), str(s), int(i)) for t, s, i in row["passages"]),
+            rationale=frozenset(int(i) for i in row["rationale"]),
+        )
+        for row in raw
+    ]
+
+
+def items_fingerprint(items: Sequence[Item]) -> str:
+    """A digest of what was scored: keys, claims and passage texts, in order.
+
+    Stored beside the probabilities so a rebuilt item file can never be paired with
+    probabilities computed on different passages.
+    """
+    digest = hashlib.sha256()
+    for item in items:
+        digest.update(item.key.encode("utf-8") + b"\0" + item.claim.encode("utf-8") + b"\0")
+        for passage in item.passages:
+            digest.update(passage.text.encode("utf-8") + b"\0")
+    return digest.hexdigest()
+
+
+def save_probs(
+    path: Path, items: Sequence[Item], probs: Sequence[np.ndarray], *, ms_per_pair: float
+) -> None:
+    """Store one model's ``(n_passages, 3)`` probabilities per item, flattened."""
+    if len(items) != len(probs):
+        raise ValueError("one probability block per item is required")
+    for item, block in zip(items, probs, strict=True):
+        if np.asarray(block).shape != (len(item.passages), 3):
+            raise ValueError(f"probability block for {item.key} has the wrong shape")
+    offsets = np.cumsum([0, *(len(item.passages) for item in items)], dtype=np.int64)
+    flat = (
+        np.concatenate([np.asarray(b, dtype=np.float32) for b in probs])
+        if probs
+        else np.zeros((0, 3), dtype=np.float32)
+    )
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + ".tmp")
+    with temporary.open("wb") as handle:
+        np.savez(
+            handle,
+            fingerprint=np.array(items_fingerprint(items)),
+            offsets=offsets,
+            probs=flat,
+            ms_per_pair=np.array(ms_per_pair, dtype=np.float64),
+        )
+    temporary.replace(path)
+
+
+def load_probs(path: Path, items: Sequence[Item]) -> tuple[list[np.ndarray], float]:
+    """The stored probabilities, one block per item, and the measured ms per pair."""
+    with np.load(path, allow_pickle=False) as data:
+        fingerprint = str(data["fingerprint"])
+        offsets = data["offsets"]
+        flat = data["probs"]
+        ms_per_pair = float(data["ms_per_pair"])
+    if fingerprint != items_fingerprint(items):
+        raise ValueError(f"{path.name} was scored on other items; run `score` again")
+    return [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))], ms_per_pair
diff --git a/tests/test_bakeoff_data.py b/tests/test_bakeoff_data.py
new file mode 100644
index 0000000..ec5dbb6
--- /dev/null
+++ b/tests/test_bakeoff_data.py
@@ -0,0 +1,129 @@
+"""Offline tests for the bake-off's stored inputs (no network, no model)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from proofpath.eval import averitec
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    SnapshotSource,
+    items_fingerprint,
+    load_items,
+    load_probs,
+    load_snapshot,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+)
+from proofpath.fetch import Fetched, Outcome
+from proofpath.models import Passage
+from proofpath.providers import NO_TEXT
+
+
+def _fetched(outcome: Outcome, text: str) -> Fetched:
+    return Fetched(
+        url="https://example.org/a",
+        final_url="https://example.org/a",
+        step=1,
+        outcome=outcome,
+        status=200 if outcome is Outcome.OK else 403,
+        content_type="text/html",
+        kind="html",
+        body=b"",
+        text=text,
+        notes=[],
+    )
+
+
+def _item(key: str, n: int = 2, dataset: str = "scifact-dev") -> Item:
+    passages = tuple(Passage(f"sentence {i} of {key}", "doc", i) for i in range(n))
+    return Item(key, dataset, key, "a claim", "SUPPORTED", passages, frozenset({0}))
+
+
+def test_a_read_page_keeps_its_text_and_ok_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "Some text."))
+    assert source == SnapshotSource("https://example.org/a", "ok", "Some text.")
+
+
+def test_a_reached_page_with_no_text_gets_the_no_text_state() -> None:
+    source = source_from_fetched("https://example.org/a", _fetched(Outcome.OK, "   "))
+    assert source.state == NO_TEXT
+    assert source.text == ""
+
+
+def test_a_blocked_page_keeps_its_state_and_no_text() -> None:
+    blocked = _fetched(Outcome.BLOCKED_NO_BROWSER, "challenge page")
+    source = source_from_fetched("https://example.org/a", blocked)
+    assert source.state == Outcome.BLOCKED_NO_BROWSER.value
+    assert source.text == ""
+
+
+def test_snapshot_round_trips(tmp_path: Path) -> None:
+    claims = [
+        SnapshotClaim(
+            7, "c", "Refuted", (SnapshotSource("u", "ok", "t"), SnapshotSource("v", "x", "")), 1
+        )
+    ]
+    path = tmp_path / "snap.json"
+    save_snapshot(path, claims)
+    assert load_snapshot(path) == claims
+
+
+def test_a_missing_snapshot_loads_as_empty(tmp_path: Path) -> None:
+    assert load_snapshot(tmp_path / "absent.json") == []
+
+
+def test_pending_skips_claims_already_in_the_snapshot() -> None:
+    claims = [averitec.Claim(i, f"claim {i}", "Refuted", ("https://e.org",)) for i in range(3)]
+    done = [SnapshotClaim(1, "claim 1", "Refuted", (), 0)]
+    assert [c.id for c in pending(claims, done)] == [0, 2]
+
+
+def test_items_round_trip(tmp_path: Path) -> None:
+    items = [_item("a"), _item("b", 1, "averitec")]
+    path = tmp_path / "items.json"
+    save_items(path, items)
+    assert load_items(path) == items
+
+
+def test_an_unknown_dataset_is_refused() -> None:
+    with pytest.raises(ValueError, match="unknown dataset"):
+        _item("a", dataset="scifact-test")
+
+
+def test_probs_round_trip(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 3)]
+    probs = [np.full((2, 3), 0.25, np.float32), np.full((3, 3), 0.5, np.float32)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, probs, ms_per_pair=12.5)
+    loaded, ms = load_probs(path, items)
+    assert ms == pytest.approx(12.5)
+    assert [p.shape for p in loaded] == [(2, 3), (3, 3)]
+    assert np.allclose(loaded[1], 0.5)
+
+
+def test_probs_scored_on_other_items_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2)]
+    path = tmp_path / "base.npz"
+    save_probs(path, items, [np.zeros((2, 3), np.float32)], ms_per_pair=1.0)
+    changed = [_item("z", 2)]
+    with pytest.raises(ValueError, match="other items"):
+        load_probs(path, changed)
+
+
+def test_probs_with_the_wrong_shape_are_refused(tmp_path: Path) -> None:
+    with pytest.raises(ValueError, match="shape"):
+        save_probs(tmp_path / "x.npz", [_item("a", 2)], [np.zeros((3, 3))], ms_per_pair=1.0)
+
+
+def test_the_fingerprint_changes_with_a_passage_text() -> None:
+    a = _item("a")
+    b = Item(a.key, a.dataset, a.group, a.claim, a.gold, (Passage("other", "doc", 0),))
+    assert items_fingerprint([a]) != items_fingerprint([b])

From 52f7db835bf306eedaf8400debf81deb95c289c4 Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:30:24 +0300
Subject: [PATCH 04/10] feat(eval): add the bake-off's aggregation variants and
 Wilson interval

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 src/proofpath/eval/bakeoff.py | 127 ++++++++++++++++++++++++++++
 tests/test_bakeoff.py         | 151 ++++++++++++++++++++++++++++++++++
 2 files changed, 278 insertions(+)
 create mode 100644 src/proofpath/eval/bakeoff.py
 create mode 100644 tests/test_bakeoff.py

diff --git a/src/proofpath/eval/bakeoff.py b/src/proofpath/eval/bakeoff.py
new file mode 100644
index 0000000..a5ed29a
--- /dev/null
+++ b/src/proofpath/eval/bakeoff.py
@@ -0,0 +1,127 @@
+"""NLI bake-off: aggregation variants, calibration, evaluation and the decision rule.
+
+Pure functions over stored NLI probabilities; nothing here touches the network or a
+model (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md). A variant moves into
+``pipeline.py`` only if the bake-off picks it.
+"""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Callable, Mapping, Sequence
+from dataclasses import dataclass
+
+import numpy as np
+
+from proofpath import numerics
+from proofpath.entailment import LABEL_ORDER
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+
+_S = LABEL_ORDER.index(Label.SUPPORTED)
+_R = LABEL_ORDER.index(Label.REFUTED)
+_N = LABEL_ORDER.index(Label.NEI)
+
+# Simplest first: this order is also the tie order of the decision rule.
+AGGREGATIONS: tuple[str, ...] = ("max", "max_nei", "margin")
+KS: tuple[int, ...] = (1, 2, 3)
+# Passages retrieved and scored per item; every k is a prefix of these.
+TOP_N = 5
+
+
+@dataclass(frozen=True)
+class Signal:
+    """The strongest assertion a combination proposes for one item, before ``decide``.
+
+    ``label`` is SUPPORTED or REFUTED, or NEI when the aggregation proposes nothing;
+    ``passage`` is what the assertion rests on.
+    """
+
+    label: Label
+    score: float
+    passage: Passage | None
+
+
+NOTHING = Signal(Label.NEI, 0.0, None)
+
+Aggregator = Callable[[Sequence[Passage], np.ndarray], Signal]
+
+
+def aggregate_max(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """Today's ``pipeline.aggregate``: the loudest SUPPORTED or REFUTED over the hits."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_max_nei(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """As ``max``, but a hit only counts when its assertion beats its own p(NEI)."""
+    best = NOTHING
+    for passage, row in zip(passages, probs, strict=True):
+        neutral = float(row[_N])
+        for label, column in ((Label.SUPPORTED, _S), (Label.REFUTED, _R)):
+            value = float(row[column])
+            if value > neutral and value > best.score:
+                best = Signal(label, value, passage)
+    return best
+
+
+def aggregate_margin(passages: Sequence[Passage], probs: np.ndarray) -> Signal:
+    """max p(S) - max p(R): a supporting and a contradicting passage cancel out."""
+    if len(passages) != len(probs):
+        raise ValueError(f"{len(passages)} passages but {len(probs)} probability rows")
+    if len(passages) == 0:
+        return NOTHING
+    s_at = int(np.argmax(probs[:, _S]))
+    r_at = int(np.argmax(probs[:, _R]))
+    supported = float(probs[s_at, _S])
+    refuted = float(probs[r_at, _R])
+    if supported > refuted:
+        return Signal(Label.SUPPORTED, supported - refuted, passages[s_at])
+    if refuted > supported:
+        return Signal(Label.REFUTED, refuted - supported, passages[r_at])
+    return NOTHING
+
+
+AGGREGATORS: Mapping[str, Aggregator] = {
+    "max": aggregate_max,
+    "max_nei": aggregate_max_nei,
+    "margin": aggregate_margin,
+}
+
+
+def signal_for(
+    claim: str, passages: Sequence[Passage], probs: np.ndarray, *, k: int, aggregation: str
+) -> Signal:
+    """What one combination proposes for one item, numeric layer first (spec §10).
+
+    A numeric mismatch decides by rule at score 1.0, as ``pipeline.decide_indexed``
+    does, so no candidate model is credited with what the rule decided.
+    """
+    top = list(passages[:k])
+    numeric = numerics.check(claim, top)
+    if numeric is not None and numeric.mismatch:
+        return Signal(Label.REFUTED, 1.0, numeric.passage)
+    return AGGREGATORS[aggregation](top, probs[:k])
+
+
+def decide(signal: Signal, thresholds: Thresholds) -> Label:
+    """The verdict a signal earns: below the cut, or with no passage, it is NEI."""
+    if signal.label is Label.NEI or signal.passage is None:
+        return Label.NEI
+    return signal.label if signal.score >= thresholds.decide else Label.NEI
+
+
+def wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]:
+    """95 % Wilson score interval for ``correct`` successes out of ``n``."""
+    if n <= 0:
+        raise ValueError("a Wilson interval needs at least one observation")
+    p = correct / n
+    denominator = 1 + z * z / n
+    centre = (p + z * z / (2 * n)) / denominator
+    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator
+    return max(0.0, centre - half), min(1.0, centre + half)
diff --git a/tests/test_bakeoff.py b/tests/test_bakeoff.py
new file mode 100644
index 0000000..3a80402
--- /dev/null
+++ b/tests/test_bakeoff.py
@@ -0,0 +1,151 @@
+"""Offline tests for the bake-off's aggregation, calibration and decision rule."""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+from proofpath import pipeline
+from proofpath.eval import bakeoff
+from proofpath.models import Label, Passage
+from proofpath.pipeline import Thresholds
+from proofpath.retrieval import Hit
+
+# Rows are (SUPPORTED, REFUTED, NEI), the order of entailment.LABEL_ORDER.
+P = [Passage(f"passage {i}", "doc", i) for i in range(3)]
+
+
+def _probs(*rows: tuple[float, float, float]) -> np.ndarray:
+    return np.array(rows, dtype=np.float32)
+
+
+def test_max_takes_the_loudest_assertion_over_the_hits() -> None:
+    probs = _probs((0.6, 0.1, 0.3), (0.1, 0.8, 0.1))
+    signal = bakeoff.aggregate_max(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.REFUTED, P[1])
+    assert signal.score == pytest.approx(0.8)
+
+
+def test_max_nei_ignores_an_assertion_its_own_hit_calls_neutral() -> None:
+    probs = _probs((0.45, 0.05, 0.50), (0.40, 0.05, 0.30))
+    signal = bakeoff.aggregate_max_nei(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[1])
+    assert signal.score == pytest.approx(0.40)
+
+
+def test_max_nei_proposes_nothing_when_every_hit_is_neutral() -> None:
+    probs = _probs((0.2, 0.1, 0.7))
+    assert bakeoff.aggregate_max_nei(P[:1], probs) == bakeoff.NOTHING
+
+
+def test_margin_lets_a_contradicting_passage_cancel_a_supporting_one() -> None:
+    probs = _probs((0.90, 0.05, 0.05), (0.05, 0.85, 0.10))
+    signal = bakeoff.aggregate_margin(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.SUPPORTED, P[0])
+    assert signal.score == pytest.approx(0.05)
+
+
+def test_margin_with_an_exact_tie_proposes_nothing() -> None:
+    probs = _probs((0.5, 0.0, 0.5), (0.0, 0.5, 0.5))
+    assert bakeoff.aggregate_margin(P[:2], probs) == bakeoff.NOTHING
+
+
+@pytest.mark.parametrize("name", bakeoff.AGGREGATIONS)
+def test_no_hits_propose_nothing(name: str) -> None:
+    assert bakeoff.AGGREGATORS[name]([], np.zeros((0, 3), np.float32)) == bakeoff.NOTHING
+
+
+def test_signal_for_reads_only_the_first_k_hits() -> None:
+    probs = _probs((0.2, 0.1, 0.7), (0.95, 0.0, 0.05))
+    first = bakeoff.signal_for("a claim", P[:2], probs, k=1, aggregation="max")
+    assert first.passage == P[0]
+    both = bakeoff.signal_for("a claim", P[:2], probs, k=2, aggregation="max")
+    assert both.passage == P[1]
+
+
+def test_a_numeric_mismatch_decides_before_any_aggregation() -> None:
+    passages = [Passage("Mortality fell by 12% in the treated group.", "doc", 0)]
+    probs = _probs((0.99, 0.0, 0.01))
+    signal = bakeoff.signal_for(
+        "Mortality fell by 40% in the treated group.", passages, probs, k=1, aggregation="max"
+    )
+    assert signal == bakeoff.Signal(Label.REFUTED, 1.0, passages[0])
+
+
+def test_decide_needs_the_cut_and_a_passage() -> None:
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.6, P[0]), cuts) is Label.SUPPORTED
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.4, P[0]), cuts) is Label.NEI
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.9, None), cuts) is Label.NEI
+
+
+def test_wilson_interval_matches_known_values() -> None:
+    low, high = bakeoff.wilson(0, 10)
+    assert low == pytest.approx(0.0)
+    assert high == pytest.approx(0.2775, abs=1e-4)
+    low, high = bakeoff.wilson(10, 10)
+    assert low == pytest.approx(0.7225, abs=1e-4)
+    assert high == pytest.approx(1.0)
+
+
+def test_wilson_refuses_an_empty_sample() -> None:
+    with pytest.raises(ValueError):
+        bakeoff.wilson(0, 0)
+
+
+@pytest.mark.parametrize("name", bakeoff.AGGREGATIONS)
+def test_aggregators_refuse_a_passage_and_row_count_mismatch(name: str) -> None:
+    probs = _probs((0.5, 0.2, 0.3), (0.1, 0.1, 0.8), (0.2, 0.7, 0.1))
+    with pytest.raises(ValueError):
+        bakeoff.AGGREGATORS[name](P[:2], probs)
+    with pytest.raises(ValueError):
+        bakeoff.AGGREGATORS[name](P[:3], probs[:2])
+
+
+def test_margin_reports_a_refuted_signal_when_the_contradiction_is_louder() -> None:
+    probs = _probs((0.10, 0.05, 0.85), (0.05, 0.80, 0.15))
+    signal = bakeoff.aggregate_margin(P[:2], probs)
+    assert (signal.label, signal.passage) == (Label.REFUTED, P[1])
+    assert signal.score == pytest.approx(0.70)
+
+
+@pytest.mark.parametrize(
+    "rows",
+    [
+        ((0.6, 0.1, 0.3), (0.1, 0.8, 0.1)),
+        ((0.7, 0.2, 0.1), (0.3, 0.7, 0.0)),  # tie between SUPPORTED and REFUTED
+        ((0.0, 0.0, 1.0), (0.0, 0.0, 1.0)),  # nothing asserted
+        ((0.4, 0.4, 0.2),),  # tie inside one row
+    ],
+)
+def test_max_agrees_with_the_pipeline_aggregate(
+    rows: tuple[tuple[float, float, float], ...],
+) -> None:
+    probs = _probs(*rows)
+    passages = P[: len(rows)]
+    hits = [Hit(passage, 0.5) for passage in passages]
+    cuts = Thresholds(decide=0.0, high=1.0, medium=1.0)
+    verdict = pipeline.aggregate(hits, probs, thresholds=cuts)
+    signal = bakeoff.aggregate_max(passages, probs)
+    assert (verdict.label, verdict.score, verdict.passage) == (
+        signal.label,
+        signal.score,
+        signal.passage,
+    )
+
+
+def test_max_matches_the_pipeline_when_nothing_is_asserted() -> None:
+    probs = _probs((0.0, 0.0, 1.0))
+    verdict = pipeline.aggregate(
+        [Hit(P[0], 0.5)], probs, thresholds=Thresholds(decide=0.0, high=1.0, medium=1.0)
+    )
+    assert verdict.label is Label.NEI and verdict.passage is None
+    assert bakeoff.aggregate_max(P[:1], probs) == bakeoff.NOTHING
+
+
+def test_decide_at_the_cut_asserts_and_keeps_a_refuted_signal() -> None:
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.5, P[0]), cuts) is Label.SUPPORTED
+    assert bakeoff.decide(bakeoff.Signal(Label.REFUTED, 0.5, P[0]), cuts) is Label.REFUTED
+    assert bakeoff.decide(bakeoff.Signal(Label.REFUTED, 0.49, P[0]), cuts) is Label.NEI
+    assert bakeoff.decide(bakeoff.NOTHING, cuts) is Label.NEI

From 438ad10ab88093aed4db38ab6d121c82e361dc6b Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:33:02 +0300
Subject: [PATCH 05/10] feat(eval): calibrate on SciFact train and score dev
 and AVeriTeC

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 src/proofpath/eval/bakeoff.py | 209 ++++++++++++++++++++++++++++++++++
 tests/test_bakeoff.py         |  79 +++++++++++++
 2 files changed, 288 insertions(+)

diff --git a/src/proofpath/eval/bakeoff.py b/src/proofpath/eval/bakeoff.py
index a5ed29a..3d01ac9 100644
--- a/src/proofpath/eval/bakeoff.py
+++ b/src/proofpath/eval/bakeoff.py
@@ -8,6 +8,7 @@
 from __future__ import annotations
 
 import math
+from collections import Counter
 from collections.abc import Callable, Mapping, Sequence
 from dataclasses import dataclass
 
@@ -15,6 +16,8 @@
 
 from proofpath import numerics
 from proofpath.entailment import LABEL_ORDER
+from proofpath.eval import averitec, metrics
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim
 from proofpath.models import Label, Passage
 from proofpath.pipeline import Thresholds
 
@@ -125,3 +128,209 @@ def wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]:
     centre = (p + z * z / (2 * n)) / denominator
     half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator
     return max(0.0, centre - half), min(1.0, centre + half)
+
+
+# Wider than eval_scifact's 0.30-0.95 grid: a ``margin`` score is a difference of two
+# probabilities and lives much nearer 0. Every combination is swept on the same grid.
+GRID: tuple[float, ...] = tuple(round(0.05 * i, 2) for i in range(1, 20))
+HIGH_TARGET = 0.85
+MEDIUM_TARGET = 0.70
+# Below this many dev verdicts a tier's precision is reported as "too few to judge".
+MIN_TIER_N = 20
+CUT_DECIMALS = 6
+
+
+@dataclass(frozen=True)
+class Scored:
+    item: Item
+    probs: np.ndarray  # (len(item.passages), 3) in LABEL_ORDER
+
+
+def _require(scored: Sequence[Scored], dataset: str) -> None:
+    wrong = sorted({s.item.dataset for s in scored if s.item.dataset != dataset})
+    if wrong:
+        raise ValueError(f"expected only {dataset} items, got {', '.join(wrong)}")
+
+
+def ship_cut(value: float) -> float:
+    """Round a cut up to ``CUT_DECIMALS`` — the rule of ``eval_scifact.ship_cut``.
+
+    Up, never down: a cut rounded down would hand a tier to scores the calibration
+    split never showed were that good.
+    """
+    scale: int = 10**CUT_DECIMALS
+    return min(1.0, math.ceil(value * scale) / scale)
+
+
+def calibration_rows(scored: Sequence[Scored], *, k: int, aggregation: str) -> list[metrics.Row]:
+    """``(gold, proposed label, score)`` rows for calibration, SciFact train only."""
+    _require(scored, "scifact-train")
+    rows: list[metrics.Row] = []
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        rows.append((Label(s.item.gold), signal.label, signal.score))
+    return rows
+
+
+def calibrate(rows: Sequence[metrics.Row]) -> Thresholds:
+    """Best-accuracy ``decide``, then the 0.85 / 0.70 precision cuts above it."""
+    best = metrics.sweep_decide(rows, grid=GRID)
+    raw_high, raw_medium = metrics.tier_cutpoints(
+        rows,
+        decide=best.threshold,
+        high_precision=HIGH_TARGET,
+        medium_precision=MEDIUM_TARGET,
+    )
+    high, medium = ship_cut(raw_high), ship_cut(raw_medium)
+    return Thresholds(decide=best.threshold, high=max(high, medium), medium=medium)
+
+
+@dataclass(frozen=True)
+class TierStat:
+    tier: str
+    n: int
+    correct: int
+
+    @property
+    def precision(self) -> float | None:
+        return self.correct / self.n if self.n >= MIN_TIER_N else None
+
+    @property
+    def interval(self) -> tuple[float, float] | None:
+        return wilson(self.correct, self.n) if self.n >= MIN_TIER_N else None
+
+
+@dataclass(frozen=True)
+class SciFactResult:
+    n: int
+    accuracy: float
+    macro_f1: float
+    f1: dict[Label, float]
+    rationale_f1: float
+    tiers: tuple[TierStat, ...]  # high, medium, low
+    asserted_without_passage: int
+
+
+def per_label_f1(gold: Sequence[Label], pred: Sequence[Label]) -> dict[Label, float]:
+    scores: dict[Label, float] = {}
+    for label in metrics.LABELS:
+        tp = sum(g is label and p is label for g, p in zip(gold, pred, strict=True))
+        fp = sum(g is not label and p is label for g, p in zip(gold, pred, strict=True))
+        fn = sum(g is label and p is not label for g, p in zip(gold, pred, strict=True))
+        scores[label] = 2 * tp / (2 * tp + fp + fn) if tp else 0.0
+    return scores
+
+
+def _unbacked(signal: Signal, thresholds: Thresholds) -> bool:
+    """An assertion over the cut with no passage under it (product rule 1)."""
+    return (
+        signal.label is not Label.NEI
+        and signal.score >= thresholds.decide
+        and signal.passage is None
+    )
+
+
+def evaluate_scifact(
+    scored: Sequence[Scored], thresholds: Thresholds, *, k: int, aggregation: str
+) -> SciFactResult:
+    _require(scored, "scifact-dev")
+    gold: list[Label] = []
+    pred: list[Label] = []
+    gold_rationale: list[frozenset[int]] = []
+    pred_rationale: list[frozenset[int]] = []
+    tiers = {"high": [0, 0], "medium": [0, 0], "low": [0, 0]}
+    unbacked = 0
+    for s in scored:
+        signal = signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation)
+        unbacked += _unbacked(signal, thresholds)
+        label = decide(signal, thresholds)
+        truth = Label(s.item.gold)
+        gold.append(truth)
+        pred.append(label)
+        gold_rationale.append(s.item.rationale)
+        if label is not Label.NEI and signal.passage is not None:
+            pred_rationale.append(frozenset({signal.passage.index}))
+            counts = tiers[thresholds.tier(signal.score)]
+            counts[0] += 1
+            counts[1] += label is truth
+        else:
+            pred_rationale.append(frozenset())
+    return SciFactResult(
+        n=len(scored),
+        accuracy=metrics.accuracy(gold, pred),
+        macro_f1=metrics.macro_f1(gold, pred),
+        f1=per_label_f1(gold, pred),
+        rationale_f1=metrics.rationale_f1(gold_rationale, pred_rationale),
+        tiers=tuple(TierStat(name, n, c) for name, (n, c) in tiers.items()),
+        asserted_without_passage=unbacked,
+    )
+
+
+@dataclass(frozen=True)
+class AveritecResult:
+    n: int
+    counted: int  # claims with a 3-way gold label
+    accuracy_all: float
+    readable: int  # counted claims with at least one readable source
+    accuracy_readable: float
+    majority_baseline: float
+    per_label: dict[str, tuple[int, int]]  # gold label -> (n, correct)
+    asserted_without_passage: int
+
+
+def evaluate_averitec(
+    claims: Sequence[SnapshotClaim],
+    scored: Sequence[Scored],
+    thresholds: Thresholds,
+    *,
+    k: int,
+    aggregation: str,
+) -> AveritecResult:
+    """Per claim, the rule of ``eval_averitec._decide_claim``.
+
+    The strongest decided non-NEI verdict across the claim's sources wins. A claim
+    with a readable source and no assertion is NEI. A claim with nothing readable is
+    unanswered, and is scored as NEI.
+    """
+    _require(scored, "averitec")
+    by_group: dict[str, list[Scored]] = {}
+    for s in scored:
+        by_group.setdefault(s.item.group, []).append(s)
+    per_label: dict[str, tuple[int, int]] = {}
+    golds: list[Label] = []
+    correct_all = readable = correct_readable = unbacked = 0
+    for claim in claims:
+        group = by_group.get(f"averitec:{claim.claim_id}", [])
+        best: Signal | None = None
+        for s in group:
+            signal = signal_for(
+                s.item.claim, s.item.passages, s.probs, k=k, aggregation=aggregation
+            )
+            unbacked += _unbacked(signal, thresholds)
+            if decide(signal, thresholds) is Label.NEI:
+                continue
+            if best is None or signal.score > best.score:
+                best = signal
+        predicted = best.label if best is not None else Label.NEI
+        gold = averitec.to_label(claim.gold)
+        correct = gold is not None and predicted is gold
+        seen, right = per_label.get(claim.gold, (0, 0))
+        per_label[claim.gold] = (seen + 1, right + int(correct))
+        if gold is None:
+            continue
+        golds.append(gold)
+        correct_all += correct
+        if group:
+            readable += 1
+            correct_readable += correct
+    majority = max(Counter(golds).values(), default=0)
+    return AveritecResult(
+        n=len(claims),
+        counted=len(golds),
+        accuracy_all=correct_all / len(golds) if golds else 0.0,
+        readable=readable,
+        accuracy_readable=correct_readable / readable if readable else 0.0,
+        majority_baseline=majority / len(golds) if golds else 0.0,
+        per_label=per_label,
+        asserted_without_passage=unbacked,
+    )
diff --git a/tests/test_bakeoff.py b/tests/test_bakeoff.py
index 3a80402..2e384ca 100644
--- a/tests/test_bakeoff.py
+++ b/tests/test_bakeoff.py
@@ -7,6 +7,7 @@
 
 from proofpath import pipeline
 from proofpath.eval import bakeoff
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim, SnapshotSource
 from proofpath.models import Label, Passage
 from proofpath.pipeline import Thresholds
 from proofpath.retrieval import Hit
@@ -149,3 +150,81 @@ def test_decide_at_the_cut_asserts_and_keeps_a_refuted_signal() -> None:
     assert bakeoff.decide(bakeoff.Signal(Label.REFUTED, 0.5, P[0]), cuts) is Label.REFUTED
     assert bakeoff.decide(bakeoff.Signal(Label.REFUTED, 0.49, P[0]), cuts) is Label.NEI
     assert bakeoff.decide(bakeoff.NOTHING, cuts) is Label.NEI
+
+
+def _scored(
+    key: str,
+    dataset: str,
+    gold: str,
+    row: tuple[float, float, float],
+    *,
+    group: str | None = None,
+    rationale: frozenset[int] = frozenset({0}),
+) -> bakeoff.Scored:
+    item = Item(key, dataset, group or key, "a claim", gold, (Passage("p", key, 0),), rationale)
+    return bakeoff.Scored(item, _probs(row))
+
+
+def test_calibration_refuses_anything_but_scifact_train() -> None:
+    dev = [_scored("d", "scifact-dev", "SUPPORTED", (0.9, 0.0, 0.1))]
+    with pytest.raises(ValueError, match="scifact-train"):
+        bakeoff.calibration_rows(dev, k=1, aggregation="max")
+
+
+def test_evaluation_refuses_the_calibration_split() -> None:
+    train = [_scored("t", "scifact-train", "SUPPORTED", (0.9, 0.0, 0.1))]
+    cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
+    with pytest.raises(ValueError, match="scifact-dev"):
+        bakeoff.evaluate_scifact(train, cuts, k=1, aggregation="max")
+
+
+def test_calibrate_picks_a_cut_that_separates_right_from_wrong() -> None:
+    rows = [(Label.SUPPORTED, Label.SUPPORTED, 0.9)] * 5 + [(Label.NEI, Label.SUPPORTED, 0.3)] * 5
+    cuts = bakeoff.calibrate(rows)
+    assert 0.3 < cuts.decide <= 0.9
+    assert cuts.decide <= cuts.medium <= cuts.high
+
+
+def test_tier_stats_hide_precision_below_the_minimum_n() -> None:
+    assert bakeoff.TierStat("high", 19, 19).precision is None
+    assert bakeoff.TierStat("high", 19, 19).interval is None
+    assert bakeoff.TierStat("high", 20, 15).precision == pytest.approx(0.75)
+
+
+def test_evaluate_scifact_counts_verdicts_tiers_and_rationale() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    scored = [
+        _scored("a", "scifact-dev", "SUPPORTED", (0.97, 0.01, 0.02)),  # high, right
+        _scored("b", "scifact-dev", "REFUTED", (0.80, 0.10, 0.10)),  # medium, wrong
+        _scored("c", "scifact-dev", "NEI", (0.30, 0.10, 0.60)),  # below decide
+    ]
+    result = bakeoff.evaluate_scifact(scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.accuracy == pytest.approx(2 / 3)
+    assert {t.tier: (t.n, t.correct) for t in result.tiers} == {
+        "high": (1, 1),
+        "medium": (1, 0),
+        "low": (0, 0),
+    }
+    assert result.asserted_without_passage == 0
+    assert 0.0 < result.rationale_f1 <= 1.0
+
+
+def test_evaluate_averitec_keeps_the_strongest_source_and_both_denominators() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [
+        SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u1", "ok", "t"),), 0),
+        SnapshotClaim(2, "c2", "Supported", (SnapshotSource("u2", "UNVERIFIED (x)", ""),), 0),
+        SnapshotClaim(3, "c3", "Conflicting Evidence/Cherrypicking", (), 0),
+    ]
+    scored = [
+        _scored("averitec:1:0", "averitec", "Refuted", (0.6, 0.1, 0.3), group="averitec:1"),
+        _scored("averitec:1:1", "averitec", "Refuted", (0.0, 0.9, 0.1), group="averitec:1"),
+    ]
+    result = bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
+    assert result.n == 3
+    assert result.counted == 2  # the Conflicting claim has no 3-way label
+    assert result.accuracy_all == pytest.approx(0.5)  # claim 1 right, claim 2 unread → NEI
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(1.0)
+    assert result.majority_baseline == pytest.approx(0.5)

From fbbc071014f0b06e0b2a525cfdec6c9605d67277 Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:38:57 +0300
Subject: [PATCH 06/10] feat(eval): add the bake-off decision rule, steps in a
 fixed order

The first wording left the step order open; read as "tie rule first" it let
a large model through on a 0.022 lead. Clarified before any number was seen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 .../specs/2026-10-01-nli-bakeoff-design.md    |  20 ++-
 src/proofpath/eval/bakeoff.py                 | 109 +++++++++++++++
 tests/test_bakeoff.py                         | 124 ++++++++++++++++++
 3 files changed, 247 insertions(+), 6 deletions(-)

diff --git a/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md b/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md
index ade0255..277b8f6 100644
--- a/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md
+++ b/docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md
@@ -109,15 +109,23 @@ For every combination:
 
 ## Decision rule (fixed before any number is seen)
 
+The steps run in this order. Each one filters the rows the previous step left.
+
 1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
    not below the `base` × k=1 × `max` row.
-2. **Winner:** the highest SciFact dev macro-F1 among eligible rows.
-3. **Size gate:** a `large*` row wins only if it beats the best eligible `base` row by
-   **≥ 0.03** macro-F1. Otherwise the best `base` row wins.
-4. **Tie:** two rows within 0.01 of each other go to the smaller k, then to the simpler
+2. **Beats the baseline:** a row stays only if its SciFact dev macro-F1 beats
+   `base` × k=1 × `max` by **≥ 0.01**. If no row stays, the result is "no change". It is
+   written up as such, and the next package (B, coverage) starts.
+3. **Size gate:** a `large*` row stays only if it beats the highest macro-F1 of any
+   eligible `base` row by **≥ 0.03**. That is the strongest eligible `base` row, not the
+   one the tie rule would pick.
+4. **Winner:** the highest SciFact dev macro-F1 among the rows left.
+5. **Tie:** rows left within 0.01 of the winner go to the smaller k, then to the simpler
    aggregation, in the order `max`, `max_nei`, `margin`.
-5. If nothing beats `base` × k=1 × `max` by ≥ 0.01 macro-F1, the result is "no change".
-   It is written up as such, and the next package (B, coverage) starts.
+
+Clarified on 2026-10-01, before any number was seen. The first wording did not fix the
+order of the steps. Read as "tie rule first", it let a `large` row through on a 0.022
+lead, because it was measured against a simpler but weaker `base` row.
 
 The rule is copied into the report verbatim, next to the numbers it chose.
 
diff --git a/src/proofpath/eval/bakeoff.py b/src/proofpath/eval/bakeoff.py
index 3d01ac9..b358256 100644
--- a/src/proofpath/eval/bakeoff.py
+++ b/src/proofpath/eval/bakeoff.py
@@ -334,3 +334,112 @@ def evaluate_averitec(
         per_label=per_label,
         asserted_without_passage=unbacked,
     )
+
+
+TIE_MARGIN = 0.01
+LARGE_MARGIN = 0.03
+NO_CHANGE_MARGIN = 0.01
+# Absorbs float representation error only (0.627 - 0.597 is 0.02999999...), not display
+# rounding: a gap shown as 0.0295 must still fail a 0.03 margin.
+_EPS = 1e-9
+
+DECISION_RULE = """\
+The steps run in this order. Each one filters the rows the previous step left.
+
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Beats the baseline:** a row stays only if its SciFact dev macro-F1 beats
+   `base` × k=1 × `max` by **≥ 0.01**. If no row stays, the result is "no change". It is
+   written up as such, and the next package (B, coverage) starts.
+3. **Size gate:** a `large*` row stays only if it beats the highest macro-F1 of any
+   eligible `base` row by **≥ 0.03**. That is the strongest eligible `base` row, not the
+   one the tie rule would pick.
+4. **Winner:** the highest SciFact dev macro-F1 among the rows left.
+5. **Tie:** rows left within 0.01 of the winner go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`."""  # noqa: RUF001
+
+
+@dataclass(frozen=True)
+class Combo:
+    model: str
+    k: int
+    aggregation: str
+
+    def label(self) -> str:
+        return f"`{self.model}` × k={self.k} × `{self.aggregation}`"  # noqa: RUF001
+
+
+@dataclass(frozen=True)
+class Outcome:
+    combo: Combo
+    large: bool
+    macro_f1: float  # SciFact dev
+    averitec_readable: float
+    asserted_without_passage: int
+
+
+@dataclass(frozen=True)
+class Choice:
+    winner: Combo | None  # None means "no change"
+    reason: str
+
+
+def choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
+    """Apply ``DECISION_RULE`` step by step; each step filters the previous step's rows."""
+    by_combo = {row.combo: row for row in outcomes}
+    if baseline not in by_combo:
+        raise ValueError(f"the baseline {baseline.label()} is not among the outcomes")
+    base = by_combo[baseline]
+
+    # Step 1: eligible.
+    eligible = [
+        row
+        for row in outcomes
+        if row.asserted_without_passage == 0
+        and row.averitec_readable >= base.averitec_readable - _EPS
+    ]
+    if not eligible:
+        return Choice(None, "no change: no row is eligible under step 1")
+
+    # Step 2: beats the baseline.
+    beating = [row for row in eligible if row.macro_f1 - base.macro_f1 >= NO_CHANGE_MARGIN - _EPS]
+    if not beating:
+        best = max(eligible, key=lambda row: row.macro_f1)
+        gain = best.macro_f1 - base.macro_f1
+        return Choice(
+            None,
+            f"no change: the best eligible row, {best.combo.label()}, is {gain:+.3f} "
+            f"macro-F1 over the baseline, below {NO_CHANGE_MARGIN}",
+        )
+
+    # Step 3: size gate, measured against the best eligible base row (step 1 rows).
+    best_base = max((row.macro_f1 for row in eligible if not row.large), default=base.macro_f1)
+    left = [
+        row for row in beating if not row.large or row.macro_f1 - best_base >= LARGE_MARGIN - _EPS
+    ]
+    if not left:
+        return Choice(
+            None,
+            "no change: the only rows that beat the baseline were large rows, and none "
+            f"beat the best eligible base row ({best_base:.3f}) by {LARGE_MARGIN}",
+        )
+
+    # Steps 4 and 5: highest macro-F1, then the tie rule.
+    top = max(left, key=lambda row: row.macro_f1)
+    contenders = [row for row in left if top.macro_f1 - row.macro_f1 <= TIE_MARGIN + _EPS]
+    winner = min(
+        contenders,
+        key=lambda row: (
+            row.combo.k,
+            AGGREGATIONS.index(row.combo.aggregation),
+            -row.macro_f1,
+        ),
+    )
+    gain = winner.macro_f1 - base.macro_f1
+    reason = f"{winner.combo.label()} is {gain:+.3f} macro-F1 over {baseline.label()}"
+    if winner is not top:
+        reason += (
+            f"; the tie rule chose it over {top.combo.label()}, "
+            f"the top macro-F1 row ({top.macro_f1:.3f})"
+        )
+    return Choice(winner.combo, reason)
diff --git a/tests/test_bakeoff.py b/tests/test_bakeoff.py
index 2e384ca..e1d4c62 100644
--- a/tests/test_bakeoff.py
+++ b/tests/test_bakeoff.py
@@ -228,3 +228,127 @@ def test_evaluate_averitec_keeps_the_strongest_source_and_both_denominators() ->
     assert result.readable == 1
     assert result.accuracy_readable == pytest.approx(1.0)
     assert result.majority_baseline == pytest.approx(0.5)
+
+
+BASE = bakeoff.Combo("base", 1, "max")
+
+
+def _o(
+    model: str, k: int, agg: str, f1: float, *, readable: float = 0.40, unbacked: int = 0
+) -> bakeoff.Outcome:
+    return bakeoff.Outcome(bakeoff.Combo(model, k, agg), model != "base", f1, readable, unbacked)
+
+
+def test_no_change_when_nothing_beats_the_baseline_by_a_hundredth() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.604)], baseline=BASE
+    )
+    assert choice.winner is None
+    assert "no change" in choice.reason
+
+
+def test_a_base_row_that_clears_the_margin_wins() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 3, "margin", 0.630)], baseline=BASE
+    )
+    assert choice.winner == bakeoff.Combo("base", 3, "margin")
+
+
+def test_a_large_row_needs_three_hundredths_over_the_best_base_row() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max_nei", 0.620),
+        _o("large", 1, "max", 0.645),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+    rows.append(_o("large-fever", 1, "max", 0.651))
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large-fever", 1, "max")
+
+
+def test_ties_go_to_the_smaller_k_then_the_simpler_aggregation() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.640),
+        _o("base", 2, "margin", 0.635),
+        _o("base", 2, "max_nei", 0.632),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max_nei")
+
+
+def test_a_row_that_asserts_without_a_passage_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.700, unbacked=1)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_a_row_worse_on_averitec_readable_is_never_eligible() -> None:
+    rows = [_o("base", 1, "max", 0.597, readable=0.40), _o("base", 2, "max", 0.700, readable=0.39)]
+    assert bakeoff.choose(rows, baseline=BASE).winner is None
+
+
+def test_the_baseline_must_be_among_the_outcomes() -> None:
+    with pytest.raises(ValueError, match="baseline"):
+        bakeoff.choose([_o("base", 2, "max", 0.6)], baseline=BASE)
+
+
+def test_a_large_row_is_measured_against_the_best_base_row_not_the_tie_pick() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.640),
+        _o("base", 1, "max_nei", 0.631),
+        _o("large", 1, "max", 0.662),
+    ]
+    choice = bakeoff.choose(rows, baseline=BASE)
+    assert choice.winner == bakeoff.Combo("base", 1, "max_nei")
+    assert "k=3" in choice.reason  # names the top-F1 row it did not pick
+
+
+def test_a_large_row_that_clears_the_gate_over_the_best_base_row_wins() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max", 0.620),
+        _o("large", 1, "max", 0.645),
+        _o("large", 3, "max", 0.652),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large", 3, "max")
+
+
+def test_a_row_below_the_baseline_margin_is_dropped_before_the_tie_rule() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 3, "max", 0.612),
+        _o("base", 1, "max_nei", 0.605),
+    ]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 3, "max")
+
+
+def test_a_large_row_exactly_three_hundredths_over_the_best_base_row_passes() -> None:
+    rows = [
+        _o("base", 1, "max", 0.500),
+        _o("base", 2, "max", 0.560),
+        _o("large", 1, "max", 0.590),
+    ]
+    assert 0.590 - 0.560 < 0.03  # float error: the nominal margin is not met exactly
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("large", 1, "max")
+
+
+def test_a_row_exactly_one_hundredth_over_the_baseline_passes() -> None:
+    rows = [_o("base", 1, "max", 0.553), _o("base", 2, "max", 0.563)]
+    assert 0.563 - 0.553 < 0.01  # float error: the nominal margin is not met exactly
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max")
+
+
+def test_an_ineligible_baseline_does_not_stop_another_eligible_row_from_winning() -> None:
+    rows = [_o("base", 1, "max", 0.597, unbacked=1), _o("base", 2, "max", 0.620)]
+    assert bakeoff.choose(rows, baseline=BASE).winner == bakeoff.Combo("base", 2, "max")
+
+
+def test_only_large_rows_beating_the_baseline_but_failing_the_size_gate_is_no_change() -> None:
+    rows = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max", 0.600),
+        _o("large", 1, "max", 0.620),
+    ]
+    choice = bakeoff.choose(rows, baseline=BASE)
+    assert choice.winner is None
+    assert choice.reason.startswith("no change:")
+    assert "large" in choice.reason

From c61855aa49b7b50578840e0ea8bbb66da0eea96a Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:39:25 +0300
Subject: [PATCH 07/10] feat(eval): add the bake-off script's snapshot and
 score steps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 scripts/eval_nli_bakeoff.py    | 278 +++++++++++++++++++++++++++++++++
 tests/test_eval_nli_bakeoff.py |  53 +++++++
 2 files changed, 331 insertions(+)
 create mode 100644 scripts/eval_nli_bakeoff.py
 create mode 100644 tests/test_eval_nli_bakeoff.py

diff --git a/scripts/eval_nli_bakeoff.py b/scripts/eval_nli_bakeoff.py
new file mode 100644
index 0000000..b6a5baa
--- /dev/null
+++ b/scripts/eval_nli_bakeoff.py
@@ -0,0 +1,278 @@
+"""NLI bake-off harness (docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md).
+
+Usage:
+    uv run python scripts/eval_nli_bakeoff.py snapshot [--limit 100] [--sleep 1.0] [--resume]
+    uv run python scripts/eval_nli_bakeoff.py score --model base|large|large-fever
+        [--rebuild-items]
+    uv run python scripts/eval_nli_bakeoff.py report [--out docs/eval/<date>-nli-bakeoff.md]
+
+``snapshot`` is the only network step, apart from the one-time dataset and model
+downloads. ``score`` runs one NLI model over every stored (passage, claim) pair.
+``report`` is offline. Importing this module touches neither the network nor a model.
+"""
+
+from __future__ import annotations
+
+import argparse
+import platform
+import sys
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from proofpath import retrieval
+from proofpath.entailment import pick_onnx_file
+from proofpath.eval import averitec, bakeoff, scifact
+from proofpath.eval.bakeoff_data import (
+    Item,
+    SnapshotClaim,
+    load_items,
+    load_snapshot,
+    pending,
+    save_items,
+    save_probs,
+    save_snapshot,
+    source_from_fetched,
+)
+from proofpath.models import Passage
+from proofpath.paths import cache_dir, models_dir
+
+EMBEDDER = "BAAI/bge-small-en-v1.5"
+
+
+@dataclass(frozen=True)
+class Candidate:
+    repo: str
+    revision: str
+    onnx_file: Callable[[str], str]  # platform.machine() -> file inside the repo
+    large: bool
+
+
+def _fever_file(_machine: str) -> str:
+    # The repo ships one dynamic-int8 export for every CPU.
+    return "onnx/model_quantized.onnx"
+
+
+CANDIDATES: dict[str, Candidate] = {
+    "base": Candidate(
+        "cross-encoder/nli-deberta-v3-base",
+        "6c749ce3425cd33b46d187e45b92bbf96ee12ec7",
+        pick_onnx_file,
+        large=False,
+    ),
+    "large": Candidate(
+        "cross-encoder/nli-deberta-v3-large",
+        "bab4bc7178836f731dcfd18c06ca9def0a137712",
+        pick_onnx_file,
+        large=True,
+    ),
+    "large-fever": Candidate(
+        "MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
+        "b3546ea6b0346eb6f8d5d68b13c7dc6d0376b3d7",
+        _fever_file,
+        large=True,
+    ),
+}
+BASELINE = bakeoff.Combo("base", 1, "max")
+# Pairs handed to the scorer per call: several of its 16-pair batches, so progress
+# can be printed without starving the batches.
+SCORE_CHUNK = 400
+
+
+def bakeoff_dir() -> Path:
+    return cache_dir() / "datasets" / "bakeoff"
+
+
+def snapshot_path() -> Path:
+    return cache_dir() / "datasets" / "averitec_snapshot.json"
+
+
+def items_path() -> Path:
+    return bakeoff_dir() / "items.json"
+
+
+def nli_providers() -> list[str]:
+    """CUDA first where present, then CPU.
+
+    CoreML is left out for every candidate. All three files are int8, and the
+    product leaves CoreML out for int8 too (``entailment.providers_for``).
+    """
+    from proofpath.device import onnx_providers
+
+    return [p for p in onnx_providers() if p != "CoreMLExecutionProvider"]
+
+
+def _ranked(
+    claim: str, passages: list[Passage], embedder: retrieval.Embedder
+) -> tuple[Passage, ...]:
+    hits = retrieval.rank(claim, passages, embedder, k=bakeoff.TOP_N)
+    return tuple(hit.passage for hit in hits)
+
+
+def build_items(embedder: retrieval.Embedder) -> list[Item]:
+    """SciFact train and dev pairs, then every readable AVeriTeC source, top 5 each.
+
+    Passages are split exactly as the product splits them: SciFact ships its
+    sentences, and a page goes through ``retrieval.split_sentences`` as in
+    ``eval_averitec._decide_claim``.
+    """
+    items: list[Item] = []
+    tarball = scifact.ensure_downloaded(cache_dir())
+    for split in ("train", "dev"):
+        data = scifact.load(tarball, split=split)
+        for pair in data.pairs():
+            doc = data.corpus[pair.doc_id]
+            passages = [Passage(t, str(doc.doc_id), i) for i, t in enumerate(doc.sentences)]
+            key = f"scifact-{split}:{pair.claim_id}:{pair.doc_id}"
+            items.append(
+                Item(
+                    key,
+                    f"scifact-{split}",
+                    key,
+                    pair.claim,
+                    pair.label.value,
+                    _ranked(pair.claim, passages, embedder),
+                    pair.rationale,
+                )
+            )
+    for claim in load_snapshot(snapshot_path()):
+        for position, source in enumerate(claim.sources):
+            sentences = retrieval.split_sentences(source.text) if source.text.strip() else []
+            if not sentences:
+                continue
+            passages = [Passage(s, source.url, i) for i, s in enumerate(sentences)]
+            items.append(
+                Item(
+                    f"averitec:{claim.claim_id}:{position}",
+                    "averitec",
+                    f"averitec:{claim.claim_id}",
+                    claim.claim,
+                    claim.gold,
+                    _ranked(claim.claim, passages, embedder),
+                )
+            )
+    return items
+
+
+def cmd_snapshot(args: argparse.Namespace) -> int:
+    from proofpath.config import load_config
+    from proofpath.verify import Engine
+
+    path = snapshot_path()
+    if path.exists() and not args.resume:
+        print(
+            f"error     {path} already holds a snapshot.\n"
+            "          Pass --resume to continue it, or delete the file to start over.",
+            file=sys.stderr,
+        )
+        return 2
+    claims = averitec.load(averitec.ensure_downloaded(cache_dir()), limit=args.limit or None)
+    stored = load_snapshot(path) if args.resume else []
+    todo = pending(claims, stored)
+    print(f"snapshot  {len(claims)} claims, {len(stored)} done, {len(todo)} to fetch")
+    # Browser not permitted and no prompts: the default install, as in the published
+    # AVeriTeC number (spec: Data).
+    engine = Engine.default(load_config(), interactive=False, browser=False)
+    try:
+        for position, claim in enumerate(todo, start=1):
+            sources = []
+            for url in claim.source_urls:
+                if args.sleep:
+                    time.sleep(args.sleep)
+                sources.append(source_from_fetched(url, engine.fetcher.fetch(url)))
+            stored.append(
+                SnapshotClaim(claim.id, claim.text, claim.label, tuple(sources), claim.non_urls)
+            )
+            save_snapshot(path, stored)
+            read = sum(bool(s.text) for s in sources)
+            print(
+                f"  {position}/{len(todo)}  claim {claim.id}  read {read}/{len(sources)}",
+                file=sys.stderr,
+            )
+    except KeyboardInterrupt:
+        print(f"\ninterrupted; {path} is usable with --resume", file=sys.stderr)
+    finally:
+        engine.close()
+    return 0
+
+
+def _items(rebuild: bool) -> list[Item]:
+    if items_path().exists() and not rebuild:
+        return load_items(items_path())
+    embedder = retrieval.FastEmbedder(EMBEDDER, cache_dir=models_dir())
+    try:
+        items = build_items(embedder)
+    finally:
+        embedder.close()
+    save_items(items_path(), items)
+    print(f"items     {len(items)} written to {items_path()}")
+    return items
+
+
+def cmd_score(args: argparse.Namespace) -> int:
+    from proofpath.entailment import OnnxNli
+
+    if not snapshot_path().exists():
+        print("error     no AVeriTeC snapshot; run `snapshot` first", file=sys.stderr)
+        return 2
+    items = _items(args.rebuild_items)
+    candidate = CANDIDATES[args.model]
+    scorer = OnnxNli(
+        cache_dir=models_dir(),
+        repo_id=candidate.repo,
+        revision=candidate.revision,
+        onnx_file=candidate.onnx_file(platform.machine()),
+        providers=nli_providers(),
+    )
+    print(f"nli       {scorer.name}  providers={scorer.providers}")
+    pairs = [(p.text, item.claim) for item in items for p in item.passages]
+    try:
+        started = time.perf_counter()
+        blocks: list[np.ndarray] = []
+        for start in range(0, len(pairs), SCORE_CHUNK):
+            blocks.append(scorer.score(pairs[start : start + SCORE_CHUNK]))
+            done = min(start + SCORE_CHUNK, len(pairs))
+            print(
+                f"  {done}/{len(pairs)} pairs  {time.perf_counter() - started:.0f}s",
+                file=sys.stderr,
+            )
+        elapsed = time.perf_counter() - started
+    finally:
+        scorer.close()
+    flat = np.concatenate(blocks) if blocks else np.zeros((0, 3), dtype=np.float32)
+    offsets = np.cumsum([0, *(len(item.passages) for item in items)])
+    probs = [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))]
+    ms_per_pair = elapsed / max(len(pairs), 1) * 1000
+    out = bakeoff_dir() / f"{args.model}.npz"
+    save_probs(out, items, probs, ms_per_pair=ms_per_pair)
+    print(f"scored    {len(pairs)} pairs, {ms_per_pair:.0f} ms/pair -> {out}")
+    return 0
+
+
+def cmd_report(args: argparse.Namespace) -> int:
+    print("error     report is implemented in Task 6", file=sys.stderr)
+    return 2
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+    snap = sub.add_parser("snapshot", help="fetch AVeriTeC gold sources once (network)")
+    snap.add_argument("--limit", type=int, default=100, help="first N dev claims")
+    snap.add_argument("--sleep", type=float, default=1.0, help="seconds between fetches")
+    snap.add_argument("--resume", action="store_true", help="skip claims already stored")
+    score = sub.add_parser("score", help="run one NLI model over every stored pair")
+    score.add_argument("--model", required=True, choices=sorted(CANDIDATES))
+    score.add_argument("--rebuild-items", action="store_true", help="re-run retrieval")
+    report = sub.add_parser("report", help="calibrate, evaluate and choose (offline)")
+    report.add_argument("--out", default="")
+    args = parser.parse_args(argv)
+    commands = {"snapshot": cmd_snapshot, "score": cmd_score, "report": cmd_report}
+    return commands[args.command](args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/tests/test_eval_nli_bakeoff.py b/tests/test_eval_nli_bakeoff.py
new file mode 100644
index 0000000..d3f1ec6
--- /dev/null
+++ b/tests/test_eval_nli_bakeoff.py
@@ -0,0 +1,53 @@
+"""Offline tests for scripts/eval_nli_bakeoff.py's importable parts."""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+from types import ModuleType
+
+SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "eval_nli_bakeoff.py"
+
+
+def _load_script() -> ModuleType:
+    spec = importlib.util.spec_from_file_location("eval_nli_bakeoff", SCRIPT_PATH)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+script = _load_script()
+
+
+def test_every_candidate_is_pinned_to_a_full_revision() -> None:
+    assert set(script.CANDIDATES) == {"base", "large", "large-fever"}
+    for candidate in script.CANDIDATES.values():
+        assert len(candidate.revision) == 40
+        int(candidate.revision, 16)
+
+
+def test_only_the_large_models_are_marked_large() -> None:
+    assert {name for name, c in script.CANDIDATES.items() if c.large} == {"large", "large-fever"}
+
+
+def test_each_candidate_names_an_onnx_file_per_machine() -> None:
+    for machine in ("arm64", "x86_64"):
+        for candidate in script.CANDIDATES.values():
+            assert candidate.onnx_file(machine).startswith("onnx/")
+            assert candidate.onnx_file(machine).endswith(".onnx")
+
+
+def test_the_baseline_is_base_k1_max() -> None:
+    assert (script.BASELINE.model, script.BASELINE.k, script.BASELINE.aggregation) == (
+        "base",
+        1,
+        "max",
+    )
+
+
+def test_coreml_is_never_handed_to_the_int8_models() -> None:
+    assert "CoreMLExecutionProvider" not in script.nli_providers()

From 6fb54fc61683610c4b898326f6381cfedff5bb44 Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 11:57:54 +0300
Subject: [PATCH 08/10] feat(eval): add the bake-off report, with coverage
 stated in the file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 scripts/eval_nli_bakeoff.py    | 278 ++++++++++++++++++++++++++++++++-
 tests/test_eval_nli_bakeoff.py | 218 ++++++++++++++++++++++++++
 2 files changed, 491 insertions(+), 5 deletions(-)

diff --git a/scripts/eval_nli_bakeoff.py b/scripts/eval_nli_bakeoff.py
index b6a5baa..faa5b13 100644
--- a/scripts/eval_nli_bakeoff.py
+++ b/scripts/eval_nli_bakeoff.py
@@ -17,8 +17,9 @@
 import platform
 import sys
 import time
-from collections.abc import Callable
+from collections.abc import Callable, Mapping, Sequence
 from dataclasses import dataclass
+from datetime import date
 from pathlib import Path
 
 import numpy as np
@@ -30,6 +31,7 @@
     Item,
     SnapshotClaim,
     load_items,
+    load_probs,
     load_snapshot,
     pending,
     save_items,
@@ -37,8 +39,9 @@
     save_snapshot,
     source_from_fetched,
 )
-from proofpath.models import Passage
+from proofpath.models import Label, Passage
 from proofpath.paths import cache_dir, models_dir
+from proofpath.pipeline import Thresholds
 
 EMBEDDER = "BAAI/bge-small-en-v1.5"
 
@@ -199,9 +202,16 @@ def cmd_snapshot(args: argparse.Namespace) -> int:
     return 0
 
 
+def items_stale(items: Path, snapshot: Path) -> bool:
+    """True when the snapshot was written after ``items`` was built."""
+    return snapshot.stat().st_mtime > items.stat().st_mtime
+
+
 def _items(rebuild: bool) -> list[Item]:
     if items_path().exists() and not rebuild:
-        return load_items(items_path())
+        if not items_stale(items_path(), snapshot_path()):
+            return load_items(items_path())
+        print("items     the snapshot is newer than items.json; rebuilding")
     embedder = retrieval.FastEmbedder(EMBEDDER, cache_dir=models_dir())
     try:
         items = build_items(embedder)
@@ -252,9 +262,267 @@ def cmd_score(args: argparse.Namespace) -> int:
     return 0
 
 
+@dataclass(frozen=True)
+class ReportRow:
+    combo: bakeoff.Combo
+    large: bool
+    thresholds: Thresholds
+    scifact: bakeoff.SciFactResult
+    averitec: bakeoff.AveritecResult
+
+
+def model_size_mb(model: str) -> float | None:
+    """Size of the ONNX file the run used, or None when it is not in the cache."""
+    from huggingface_hub import try_to_load_from_cache
+
+    candidate = CANDIDATES[model]
+    found = try_to_load_from_cache(
+        candidate.repo,
+        candidate.onnx_file(platform.machine()),
+        cache_dir=str(models_dir()),
+        revision=candidate.revision,
+    )
+    return Path(found).stat().st_size / 1e6 if isinstance(found, str) else None
+
+
+def _md(*cells: object) -> str:
+    return "| " + " | ".join(str(c) for c in cells) + " |"
+
+
+def _tier_cell(tier: bakeoff.TierStat) -> str:
+    if tier.precision is None or tier.interval is None:
+        return f"{tier.n} · too few to judge"
+    low, high = tier.interval
+    return f"{tier.n} · {tier.precision:.3f} (Wilson {low:.2f}–{high:.2f})"  # noqa: RUF001
+
+
+def _without_passage_cell(count: int) -> str:
+    return f"**{count} — rule 1 violated**" if count else "0"
+
+
+def render_report(
+    rows: Sequence[ReportRow],
+    choice: bakeoff.Choice,
+    *,
+    speed: Mapping[str, float],
+    sizes: Mapping[str, float | None],
+    counts: Mapping[str, int],
+    today: str,
+    machine: str,
+) -> str:
+    scored = {row.combo.model for row in rows}
+    skipped = [m for m in CANDIDATES if m not in scored]
+    base_ave = rows[0].averitec if rows else None
+    counted = base_ave.counted if base_ave else 0
+    readable = base_ave.readable if base_ave else 0
+    baseline_ave = rows[0].averitec.majority_baseline if rows else 0.0
+    lines = [
+        f"# NLI bake-off — {today}",
+        "",
+        "- spec: `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`",
+        f"- calibrated on SciFact train ({counts.get('scifact-train', 0)} pairs); reported on "
+        f"SciFact dev ({counts.get('scifact-dev', 0)} pairs) and AVeriTeC dev "
+        f"({counts.get('averitec', 0)} claims, {counted} answerable (3-way gold label), "
+        f"{readable} of those with a readable source; frozen snapshot, browser not permitted)",
+        f"- embedder `{EMBEDDER}`, top {bakeoff.TOP_N} passages per item, numeric layer on",
+        f"- machine: {machine}",
+        "- coverage: "
+        + (f"not scored: {', '.join(skipped)}" if skipped else "all candidate models were scored"),
+        "",
+        "## Decision rule (fixed before any number was seen)",
+        "",
+        bakeoff.DECISION_RULE,
+        "",
+        "## Result",
+        "",
+    ]
+    if choice.winner is None:
+        lines.append(f"**No change.** {choice.reason}")
+    else:
+        lines.append(f"**Winner:** {choice.winner.label()} — {choice.reason}")
+    lines.extend(
+        ["", "## Models", "", "| model | repo | size (MB) | ms/pair |", "|---|---|---|---|"]
+    )
+    for model in dict.fromkeys(row.combo.model for row in rows):
+        size = sizes.get(model)
+        lines.append(
+            _md(
+                model,
+                f"`{CANDIDATES[model].repo}`",
+                "—" if size is None else f"{size:.0f}",
+                f"{speed.get(model, 0.0):.0f}",
+            )
+        )
+    lines.extend(
+        [
+            "",
+            "## Every combination",
+            "",
+            f"AVeriTeC majority baseline: {baseline_ave:.3f}. Published numbers to compare: "
+            "0.270 (all) and 0.361 (readable), 2026-09-16.",
+            "",
+            "| model | k | aggregation | decide | medium | high | dev acc | dev macro-F1 "
+            "| F1 S / R / NEI | rationale F1 | AVeriTeC all | AVeriTeC readable | w/o passage |",
+            "|---|---|---|---|---|---|---|---|---|---|---|---|---|",
+        ]
+    )
+    for row in rows:
+        f1 = row.scifact.f1
+        lines.append(
+            _md(
+                row.combo.model,
+                row.combo.k,
+                row.combo.aggregation,
+                f"{row.thresholds.decide:.2f}",
+                f"{row.thresholds.medium:.6f}",
+                f"{row.thresholds.high:.6f}",
+                f"{row.scifact.accuracy:.3f}",
+                f"{row.scifact.macro_f1:.3f}",
+                f"{f1[Label.SUPPORTED]:.2f} / {f1[Label.REFUTED]:.2f} / {f1[Label.NEI]:.2f}",
+                f"{row.scifact.rationale_f1:.3f}",
+                f"{row.averitec.accuracy_all:.3f}",
+                f"{row.averitec.accuracy_readable:.3f}",
+                _without_passage_cell(
+                    row.scifact.asserted_without_passage + row.averitec.asserted_without_passage
+                ),
+            )
+        )
+    labels = sorted({label for row in rows for label in row.averitec.per_label})
+    lines.extend(
+        [
+            "",
+            "## AVeriTeC per label",
+            "",
+            "Cells are correct/n per gold label. `n (not scored)` marks a gold label with no "
+            "3-way verdict; it is not a model failure.",
+            "",
+            _md("model", "k", "aggregation", *labels),
+            _md(*(["---"] * (3 + len(labels)))),
+        ]
+    )
+    for row in rows:
+        cells = []
+        for label in labels:
+            seen, right = row.averitec.per_label.get(label, (0, 0))
+            cells.append(
+                f"{seen} (not scored)" if averitec.to_label(label) is None else f"{right}/{seen}"
+            )
+        lines.append(_md(row.combo.model, row.combo.k, row.combo.aggregation, *cells))
+    lines.extend(
+        [
+            "",
+            "## Tiers on SciFact dev",
+            "",
+            f"Cells are n · precision (95 % Wilson interval); n below {bakeoff.MIN_TIER_N} is "
+            "too few to judge.",
+            "",
+            "| model | k | aggregation | high | medium | low |",
+            "|---|---|---|---|---|---|",
+        ]
+    )
+    for row in rows:
+        tiers = {t.tier: t for t in row.scifact.tiers}
+        lines.append(
+            _md(
+                row.combo.model,
+                row.combo.k,
+                row.combo.aggregation,
+                *(_tier_cell(tiers[name]) for name in ("high", "medium", "low")),
+            )
+        )
+    return "\n".join(lines) + "\n"
+
+
 def cmd_report(args: argparse.Namespace) -> int:
-    print("error     report is implemented in Task 6", file=sys.stderr)
-    return 2
+    if not items_path().exists() or not snapshot_path().exists():
+        print("error     run `snapshot` and `score --model base` first", file=sys.stderr)
+        return 2
+    if items_stale(items_path(), snapshot_path()):
+        print(
+            "error     the snapshot is newer than items.json, so the stored items are stale.\n"
+            "          The next `score` run rebuilds items.json automatically; after that, "
+            "re-score every model with `score --model <name>`.",
+            file=sys.stderr,
+        )
+        return 2
+    items = load_items(items_path())
+    snapshot = load_snapshot(snapshot_path())
+    rows: list[ReportRow] = []
+    speed: dict[str, float] = {}
+    for model, candidate in CANDIDATES.items():
+        path = bakeoff_dir() / f"{model}.npz"
+        if not path.exists():
+            print(f"skip      {model}: not scored", file=sys.stderr)
+            continue
+        try:
+            probs, speed[model] = load_probs(path, items)
+        except ValueError as error:
+            print(
+                f"error     {model}: stored scores do not match the current items ({error}); "
+                f"run `score --model {model}`",
+                file=sys.stderr,
+            )
+            return 2
+        scored = [bakeoff.Scored(i, p) for i, p in zip(items, probs, strict=True)]
+        train = [s for s in scored if s.item.dataset == "scifact-train"]
+        dev = [s for s in scored if s.item.dataset == "scifact-dev"]
+        ave = [s for s in scored if s.item.dataset == "averitec"]
+        for k in bakeoff.KS:
+            for aggregation in bakeoff.AGGREGATIONS:
+                thresholds = bakeoff.calibrate(
+                    bakeoff.calibration_rows(train, k=k, aggregation=aggregation)
+                )
+                rows.append(
+                    ReportRow(
+                        bakeoff.Combo(model, k, aggregation),
+                        candidate.large,
+                        thresholds,
+                        bakeoff.evaluate_scifact(dev, thresholds, k=k, aggregation=aggregation),
+                        bakeoff.evaluate_averitec(
+                            snapshot, ave, thresholds, k=k, aggregation=aggregation
+                        ),
+                    )
+                )
+    if "base" not in speed:
+        print(
+            "error     the baseline model is not scored; run `score --model base`", file=sys.stderr
+        )
+        return 2
+    outcomes = [
+        bakeoff.Outcome(
+            r.combo,
+            r.large,
+            r.scifact.macro_f1,
+            r.averitec.accuracy_readable,
+            r.scifact.asserted_without_passage + r.averitec.asserted_without_passage,
+        )
+        for r in rows
+    ]
+    choice = bakeoff.choose(outcomes, baseline=BASELINE)
+    counts = {
+        "scifact-train": sum(i.dataset == "scifact-train" for i in items),
+        "scifact-dev": sum(i.dataset == "scifact-dev" for i in items),
+        "averitec": len(snapshot),
+    }
+    report = render_report(
+        rows,
+        choice,
+        speed=speed,
+        sizes={model: model_size_mb(model) for model in speed},
+        counts=counts,
+        today=date.today().isoformat(),
+        machine=f"{platform.system()} {platform.machine()}",
+    )
+    out = (
+        Path(args.out)
+        if args.out
+        else Path("docs/eval") / f"{date.today().isoformat()}-nli-bakeoff.md"
+    )
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(report, encoding="utf-8")
+    print(report)
+    print(f"written   {out}")
+    return 0
 
 
 def main(argv: list[str] | None = None) -> int:
diff --git a/tests/test_eval_nli_bakeoff.py b/tests/test_eval_nli_bakeoff.py
index d3f1ec6..0a7ec06 100644
--- a/tests/test_eval_nli_bakeoff.py
+++ b/tests/test_eval_nli_bakeoff.py
@@ -3,10 +3,18 @@
 from __future__ import annotations
 
 import importlib.util
+import os
 import sys
+from argparse import Namespace
 from pathlib import Path
 from types import ModuleType
 
+import pytest
+
+from proofpath.eval import bakeoff
+from proofpath.models import Label
+from proofpath.pipeline import Thresholds
+
 SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "eval_nli_bakeoff.py"
 
 
@@ -51,3 +59,213 @@ def test_the_baseline_is_base_k1_max() -> None:
 
 def test_coreml_is_never_handed_to_the_int8_models() -> None:
     assert "CoreMLExecutionProvider" not in script.nli_providers()
+
+
+def _row(
+    model: str,
+    k: int,
+    agg: str,
+    f1: float,
+    tiers: tuple[int, int, int],
+    unbacked: int = 0,
+) -> object:
+    sci = bakeoff.SciFactResult(
+        n=340,
+        accuracy=f1 + 0.01,
+        macro_f1=f1,
+        f1={Label.SUPPORTED: 0.7, Label.REFUTED: 0.5, Label.NEI: 0.6},
+        rationale_f1=0.3,
+        tiers=(
+            bakeoff.TierStat("high", tiers[0], tiers[0]),
+            bakeoff.TierStat("medium", tiers[1], tiers[1] // 2),
+            bakeoff.TierStat("low", tiers[2], 0),
+        ),
+        asserted_without_passage=unbacked,
+    )
+    ave = bakeoff.AveritecResult(
+        n=100,
+        counted=89,
+        accuracy_all=0.27,
+        readable=61,
+        accuracy_readable=0.36,
+        majority_baseline=0.708,
+        per_label={"Refuted": (63, 20), "Supported": (10, 7)},
+        asserted_without_passage=0,
+    )
+    return script.ReportRow(
+        bakeoff.Combo(model, k, agg),
+        model != "base",
+        Thresholds(decide=0.45, high=0.99933, medium=0.457948),
+        sci,
+        ave,
+    )
+
+
+def _render(
+    choice: bakeoff.Choice,
+    unbacked: int = 0,
+    models: tuple[str, ...] = ("base", "large", "large-fever"),
+) -> str:
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("large", 2, "margin", 0.650, (30, 40, 25), unbacked),
+        _row("large-fever", 1, "max", 0.600, (25, 50, 20)),
+    ]
+    rows = [r for r in rows if r.combo.model in models]
+    return script.render_report(
+        rows,
+        choice,
+        speed={"base": 185.0, "large": 540.0, "large-fever": 540.0},
+        sizes={"base": 244.4, "large": None},
+        counts={"scifact-train": 919, "scifact-dev": 340, "averitec": 100},
+        today="2026-10-02",
+        machine="Darwin arm64",
+    )
+
+
+def test_the_report_prints_the_rule_verbatim_and_the_winner() -> None:
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"))
+    assert bakeoff.DECISION_RULE in text
+    assert "**Winner:** `large` × k=2 × `margin`" in text  # noqa: RUF001
+    assert "| base | 1 | max | 0.45 |" in text
+    assert "| large | 2 | margin | 0.45 |" in text
+
+
+def test_the_report_says_no_change_in_words() -> None:
+    text = _render(bakeoff.Choice(None, "no change: below 0.01"))
+    assert "**No change.**" in text
+    assert "no change: below 0.01" in text
+
+
+def test_a_tier_below_the_minimum_n_is_too_few_to_judge() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "| 1 · too few to judge |" in text
+
+
+def test_a_tier_at_or_above_the_minimum_n_shows_precision_and_wilson() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    low, high = bakeoff.wilson(21, 21)
+    assert f"21 · 1.000 (Wilson {low:.2f}–{high:.2f})" in text  # noqa: RUF001
+    assert "| base | 1 | max | 21 · 1.000" in text
+
+
+def test_unscored_models_are_listed_in_the_report() -> None:
+    text = _render(bakeoff.Choice(None, "x"), models=("base",))
+    assert "not scored: large, large-fever" in text
+
+
+def test_a_complete_run_says_all_candidates_were_scored() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert set(script.CANDIDATES) == {"base", "large", "large-fever"}
+    assert "all candidate models were scored" in text
+    assert "not scored:" not in text
+
+
+def test_a_missing_candidate_is_named_not_scored() -> None:
+    text = _render(bakeoff.Choice(None, "x"), models=("base", "large"))
+    assert "not scored: large-fever" in text
+    assert "all candidate models were scored" not in text
+
+
+def test_the_calibrated_cut_points_print_at_six_decimals() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "| 0.45 | 0.457948 | 0.999330 |" in text
+
+
+def test_a_gold_label_without_a_verdict_is_not_scored_rather_than_wrong() -> None:
+    rows = [_row("base", 1, "max", 0.597, (21, 113, 1))]
+    rows[0].averitec.per_label["Conflicting Evidence/Cherrypicking"] = (5, 0)
+    text = script.render_report(
+        rows,
+        bakeoff.Choice(None, "x"),
+        speed={"base": 185.0},
+        sizes={"base": None},
+        counts={"averitec": 100},
+        today="2026-10-02",
+        machine="Darwin arm64",
+    )
+    assert "5 (not scored)" in text
+    assert "0/5" not in text
+    assert "not a model failure" in text
+
+
+def test_the_averitec_header_states_every_denominator() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "100 claims, 89 answerable (3-way gold label), 61 of those with a readable" in text
+
+
+def test_averitec_per_label_counts_cover_every_combination() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "## AVeriTeC per label" in text
+    assert "| model | k | aggregation | Refuted | Supported |" in text
+    assert "| base | 1 | max | 20/63 | 7/10 |" in text
+    assert "| large | 2 | margin | 20/63 | 7/10 |" in text
+
+
+def test_a_nonzero_without_passage_count_is_flagged() -> None:
+    text = _render(bakeoff.Choice(None, "x"), unbacked=3)
+    assert "**3 — rule 1 violated**" in text
+    assert "| 0 |" in text
+
+
+def test_a_clean_run_has_no_rule_1_flag() -> None:
+    assert "rule 1 violated" not in _render(bakeoff.Choice(None, "x"))
+
+
+def test_an_unknown_model_size_is_a_dash_not_zero() -> None:
+    text = _render(bakeoff.Choice(None, "no change"))
+    assert "| large | `cross-encoder/nli-deberta-v3-large` | — |" in text
+
+
+def _touch(path: Path, mtime: float) -> Path:
+    path.write_text("[]", encoding="utf-8")
+    os.utime(path, (mtime, mtime))
+    return path
+
+
+def test_items_are_stale_when_the_snapshot_is_newer(tmp_path: Path) -> None:
+    items = _touch(tmp_path / "items.json", 1_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 2_000)
+    assert script.items_stale(items, snapshot) is True
+
+
+def test_items_are_fresh_when_built_after_the_snapshot(tmp_path: Path) -> None:
+    items = _touch(tmp_path / "items.json", 2_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 1_000)
+    assert script.items_stale(items, snapshot) is False
+
+
+def test_report_with_a_stale_npz_names_the_model_and_the_fix(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    items = _touch(tmp_path / "items.json", 2_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 1_000)
+    (tmp_path / "base.npz").write_bytes(b"")
+    monkeypatch.setattr(script, "items_path", lambda: items)
+    monkeypatch.setattr(script, "snapshot_path", lambda: snapshot)
+    monkeypatch.setattr(script, "bakeoff_dir", lambda: tmp_path)
+    monkeypatch.setattr(script, "load_items", lambda _p: [])
+    monkeypatch.setattr(script, "load_snapshot", lambda _p: [])
+
+    def _stale(_path: Path, _items: object) -> object:
+        raise ValueError("base.npz was scored on other items; run `score` again")
+
+    monkeypatch.setattr(script, "load_probs", _stale)
+    assert script.cmd_report(Namespace(out="")) == 2
+    err = capsys.readouterr().err
+    assert "base" in err
+    assert "score --model base" in err
+    assert "Traceback" not in err
+
+
+def test_report_with_stale_items_does_not_advise_rebuild_items_flag(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    items = _touch(tmp_path / "items.json", 1_000)
+    snapshot = _touch(tmp_path / "snapshot.json", 2_000)
+    monkeypatch.setattr(script, "items_path", lambda: items)
+    monkeypatch.setattr(script, "snapshot_path", lambda: snapshot)
+    assert script.cmd_report(Namespace(out="")) == 2
+    err = capsys.readouterr().err
+    assert "--rebuild-items" not in err
+    assert "score --model <name>" in err

From 1a7474903c96ab9cc45ee4a1dda975d8170ab65d Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 12:31:00 +0300
Subject: [PATCH 09/10] feat(eval): harden the bake-off and record its first
 result

The final review found the report under-stated its own limits: rule-decided
rows set the high cut, int8 scores move with the batch, the AVeriTeC snapshot
is new, and dev both chooses and reports. The report now says so in the file.

Result: MoritzLaurer large mnli-fever-anli-ling-wanli at k=2 wins on SciFact
dev macro-F1 (0.697 vs 0.580 baseline); AVeriTeC stays below its majority
baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 docs/eval/2026-10-01-nli-bakeoff.md | 158 ++++++++++++++++++
 docs/superpowers/OPEN-ITEMS.md      |   1 +
 scripts/eval_nli_bakeoff.py         | 207 +++++++++++++++++++++---
 src/proofpath/eval/bakeoff.py       |  92 ++++++++++-
 src/proofpath/eval/bakeoff_data.py  |  25 ++-
 tests/test_bakeoff.py               | 149 ++++++++++++++++-
 tests/test_bakeoff_data.py          |  65 ++++++++
 tests/test_eval_nli_bakeoff.py      | 240 ++++++++++++++++++++++++++--
 8 files changed, 898 insertions(+), 39 deletions(-)
 create mode 100644 docs/eval/2026-10-01-nli-bakeoff.md

diff --git a/docs/eval/2026-10-01-nli-bakeoff.md b/docs/eval/2026-10-01-nli-bakeoff.md
new file mode 100644
index 0000000..e7fcd77
--- /dev/null
+++ b/docs/eval/2026-10-01-nli-bakeoff.md
@@ -0,0 +1,158 @@
+# NLI bake-off — 2026-10-01
+
+- spec: `docs/superpowers/specs/2026-10-01-nli-bakeoff-design.md`
+- calibrated on SciFact train (919 pairs); reported on SciFact dev (340 pairs) and AVeriTeC dev (100 claims, 89 answerable (3-way gold label), 80 of those with a readable source; frozen snapshot, browser not permitted)
+- embedder `BAAI/bge-small-en-v1.5`, top 5 passages per item, numeric layer on
+- machine: Darwin arm64
+- coverage: all candidate models were scored
+
+## Decision rule (fixed before any number was seen)
+
+The steps run in this order. Each one filters the rows the previous step left.
+
+1. **Eligible:** zero assertions without a passage, and AVeriTeC readable-subset accuracy
+   not below the `base` × k=1 × `max` row.
+2. **Beats the baseline:** a row stays only if its SciFact dev macro-F1 beats
+   `base` × k=1 × `max` by **≥ 0.01**. If no row stays, the result is "no change". It is
+   written up as such, and the next package (B, coverage) starts.
+3. **Size gate:** a `large*` row stays only if it beats the highest macro-F1 of any
+   eligible `base` row by **≥ 0.03**. That is the strongest eligible `base` row, not the
+   one the tie rule would pick.
+4. **Winner:** the highest SciFact dev macro-F1 among the rows left.
+5. **Tie:** rows left within 0.01 of the winner go to the smaller k, then to the simpler
+   aggregation, in the order `max`, `max_nei`, `margin`.
+
+## Result
+
+**Winner:** `large-fever` × k=2 × `max` is +0.117 macro-F1 over `base` × k=1 × `max`.
+
+Over the best eligible `base` row, `base` × k=2 × `margin` (0.608), the winner is +0.089 macro-F1.
+
+## Models
+
+| model | repo | size (MB) | ms/pair |
+|---|---|---|---|
+| base | `cross-encoder/nli-deberta-v3-base` | 244 | 19 |
+| large | `cross-encoder/nli-deberta-v3-large` | 643 | 55 |
+| large-fever | `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` | 643 | 54 |
+
+## Every combination
+
+AVeriTeC majority baseline: 0.708. The 2026-09-16 run reported 0.270 (all) and 0.361 (readable), but on a different snapshot with different cuts, over 89 answerable / 61 readable claims; this run counts 89 answerable / 80 readable. A gap between the two runs is partly a coverage change, not only a model gain.
+
+| model | k | aggregation | decide | medium | high | dev acc | dev macro-F1 | F1 S / R / NEI | rationale F1 | AVeriTeC all | AVeriTeC readable | w/o passage |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| base | 1 | max | 0.20 | 0.915356 | 1.000000 | 0.591 | 0.580 | 0.54 / 0.57 / 0.63 | 0.299 | 0.404 | 0.438 | 0 |
+| base | 1 | max_nei | 0.40 | 0.915356 | 1.000000 | 0.594 | 0.578 | 0.52 / 0.57 / 0.65 | 0.291 | 0.360 | 0.388 | 0 |
+| base | 1 | margin | 0.20 | 0.878350 | 1.000000 | 0.591 | 0.579 | 0.54 / 0.57 / 0.63 | 0.296 | 0.393 | 0.425 | 0 |
+| base | 2 | max | 0.20 | 0.999791 | 1.000000 | 0.609 | 0.606 | 0.61 / 0.59 / 0.62 | 0.324 | 0.494 | 0.537 | 0 |
+| base | 2 | max_nei | 0.40 | 0.999791 | 1.000000 | 0.594 | 0.589 | 0.57 / 0.58 / 0.62 | 0.316 | 0.472 | 0.512 | 0 |
+| base | 2 | margin | 0.15 | 0.992588 | 1.000000 | 0.609 | 0.608 | 0.60 / 0.61 / 0.62 | 0.326 | 0.483 | 0.525 | 0 |
+| base | 3 | max | 0.80 | 1.000000 | 1.000000 | 0.594 | 0.588 | 0.59 / 0.55 / 0.63 | 0.339 | 0.494 | 0.537 | 0 |
+| base | 3 | max_nei | 0.80 | 1.000000 | 1.000000 | 0.594 | 0.588 | 0.59 / 0.55 / 0.63 | 0.339 | 0.494 | 0.537 | 0 |
+| base | 3 | margin | 0.05 (grid floor) | 1.000000 | 1.000000 | 0.562 | 0.563 | 0.61 / 0.54 / 0.54 | 0.330 | 0.551 | 0.600 | 0 |
+| large | 1 | max | 0.70 | 0.924491 | 1.000000 | 0.582 | 0.572 | 0.49 / 0.61 / 0.62 | 0.292 | 0.393 | 0.425 | 0 |
+| large | 1 | max_nei | 0.70 | 0.924491 | 1.000000 | 0.582 | 0.572 | 0.49 / 0.61 / 0.62 | 0.292 | 0.393 | 0.425 | 0 |
+| large | 1 | margin | 0.70 | 0.920455 | 1.000000 | 0.582 | 0.573 | 0.49 / 0.61 / 0.62 | 0.288 | 0.393 | 0.425 | 0 |
+| large | 2 | max | 0.80 | 0.999586 | 1.000000 | 0.609 | 0.607 | 0.54 / 0.65 / 0.63 | 0.338 | 0.427 | 0.463 | 0 |
+| large | 2 | max_nei | 0.80 | 0.999586 | 1.000000 | 0.609 | 0.607 | 0.54 / 0.65 / 0.63 | 0.338 | 0.427 | 0.463 | 0 |
+| large | 2 | margin | 0.70 | 0.986530 | 1.000000 | 0.597 | 0.597 | 0.52 / 0.66 / 0.61 | 0.331 | 0.416 | 0.450 | 0 |
+| large | 3 | max | 0.90 | 0.999698 | 1.000000 | 0.597 | 0.595 | 0.57 / 0.60 / 0.61 | 0.343 | 0.494 | 0.537 | 0 |
+| large | 3 | max_nei | 0.90 | 0.999698 | 1.000000 | 0.597 | 0.595 | 0.57 / 0.60 / 0.61 | 0.343 | 0.494 | 0.537 | 0 |
+| large | 3 | margin | 0.90 | 0.999396 | 1.000000 | 0.588 | 0.585 | 0.52 / 0.63 / 0.61 | 0.334 | 0.449 | 0.487 | 0 |
+| large-fever | 1 | max | 0.05 (grid floor) | 0.074872 | 0.997081 | 0.647 | 0.642 | 0.61 / 0.64 / 0.67 | 0.339 | 0.472 | 0.512 | 0 |
+| large-fever | 1 | max_nei | 0.40 | 0.440642 | 0.997081 | 0.632 | 0.624 | 0.53 / 0.67 / 0.67 | 0.313 | 0.360 | 0.388 | 0 |
+| large-fever | 1 | margin | 0.05 (grid floor) | 0.064411 | 1.000000 | 0.638 | 0.633 | 0.59 / 0.64 / 0.66 | 0.338 | 0.472 | 0.512 | 0 |
+| large-fever | 2 | max | 0.25 | 0.252721 | 1.000000 | 0.697 | 0.697 | 0.66 / 0.72 / 0.71 | 0.383 | 0.438 | 0.475 | 0 |
+| large-fever | 2 | max_nei | 0.45 | 0.465651 | 1.000000 | 0.679 | 0.680 | 0.64 / 0.71 / 0.69 | 0.372 | 0.416 | 0.450 | 0 |
+| large-fever | 2 | margin | 0.20 | 0.232304 | 1.000000 | 0.688 | 0.690 | 0.65 / 0.73 / 0.69 | 0.373 | 0.438 | 0.475 | 0 |
+| large-fever | 3 | max | 0.45 | 0.552317 | 1.000000 | 0.674 | 0.672 | 0.65 / 0.68 / 0.69 | 0.356 | 0.472 | 0.512 | 0 |
+| large-fever | 3 | max_nei | 0.45 | 0.552317 | 1.000000 | 0.671 | 0.669 | 0.64 / 0.68 / 0.68 | 0.356 | 0.472 | 0.512 | 0 |
+| large-fever | 3 | margin | 0.20 | 0.438075 | 1.000000 | 0.674 | 0.675 | 0.65 / 0.70 / 0.67 | 0.361 | 0.483 | 0.525 | 0 |
+
+## AVeriTeC per label
+
+Cells are correct/n per gold label. `n (not scored)` marks a gold label with no 3-way verdict; it is not a model failure.
+
+| model | k | aggregation | Conflicting Evidence/Cherrypicking | Not Enough Evidence | Refuted | Supported |
+| --- | --- | --- | --- | --- | --- | --- |
+| base | 1 | max | 11 (not scored) | 5/7 | 30/63 | 1/19 |
+| base | 1 | max_nei | 11 (not scored) | 5/7 | 27/63 | 0/19 |
+| base | 1 | margin | 11 (not scored) | 5/7 | 29/63 | 1/19 |
+| base | 2 | max | 11 (not scored) | 2/7 | 41/63 | 1/19 |
+| base | 2 | max_nei | 11 (not scored) | 2/7 | 39/63 | 1/19 |
+| base | 2 | margin | 11 (not scored) | 2/7 | 40/63 | 1/19 |
+| base | 3 | max | 11 (not scored) | 3/7 | 40/63 | 1/19 |
+| base | 3 | max_nei | 11 (not scored) | 3/7 | 40/63 | 1/19 |
+| base | 3 | margin | 11 (not scored) | 2/7 | 46/63 | 1/19 |
+| large | 1 | max | 11 (not scored) | 5/7 | 27/63 | 3/19 |
+| large | 1 | max_nei | 11 (not scored) | 5/7 | 27/63 | 3/19 |
+| large | 1 | margin | 11 (not scored) | 5/7 | 27/63 | 3/19 |
+| large | 2 | max | 11 (not scored) | 3/7 | 31/63 | 4/19 |
+| large | 2 | max_nei | 11 (not scored) | 3/7 | 31/63 | 4/19 |
+| large | 2 | margin | 11 (not scored) | 3/7 | 31/63 | 3/19 |
+| large | 3 | max | 11 (not scored) | 2/7 | 38/63 | 4/19 |
+| large | 3 | max_nei | 11 (not scored) | 2/7 | 38/63 | 4/19 |
+| large | 3 | margin | 11 (not scored) | 2/7 | 35/63 | 3/19 |
+| large-fever | 1 | max | 11 (not scored) | 5/7 | 34/63 | 3/19 |
+| large-fever | 1 | max_nei | 11 (not scored) | 6/7 | 24/63 | 2/19 |
+| large-fever | 1 | margin | 11 (not scored) | 5/7 | 34/63 | 3/19 |
+| large-fever | 2 | max | 11 (not scored) | 4/7 | 33/63 | 2/19 |
+| large-fever | 2 | max_nei | 11 (not scored) | 4/7 | 31/63 | 2/19 |
+| large-fever | 2 | margin | 11 (not scored) | 4/7 | 33/63 | 2/19 |
+| large-fever | 3 | max | 11 (not scored) | 4/7 | 35/63 | 3/19 |
+| large-fever | 3 | max_nei | 11 (not scored) | 4/7 | 35/63 | 3/19 |
+| large-fever | 3 | margin | 11 (not scored) | 4/7 | 36/63 | 3/19 |
+
+## Tiers on SciFact dev
+
+Cells are n · precision (95 % Wilson interval); n below 20 is too few to judge.
+
+| model | k | aggregation | high | medium | low |
+|---|---|---|---|---|---|
+| base | 1 | max | 1 · too few to judge | 109 · 0.725 (Wilson 0.63–0.80) | 33 · 0.515 (Wilson 0.35–0.67) |
+| base | 1 | max_nei | 1 · too few to judge | 109 · 0.725 (Wilson 0.63–0.80) | 18 · too few to judge |
+| base | 1 | margin | 1 · too few to judge | 112 · 0.732 (Wilson 0.64–0.81) | 27 · 0.481 (Wilson 0.31–0.66) |
+| base | 2 | max | 1 · too few to judge | 13 · too few to judge | 175 · 0.611 (Wilson 0.54–0.68) |
+| base | 2 | max_nei | 1 · too few to judge | 13 · too few to judge | 158 · 0.614 (Wilson 0.54–0.69) |
+| base | 2 | margin | 1 · too few to judge | 75 · 0.800 (Wilson 0.70–0.87) | 110 · 0.527 (Wilson 0.43–0.62) |
+| base | 3 | max | 1 · too few to judge | 0 · too few to judge | 181 · 0.608 (Wilson 0.54–0.68) |
+| base | 3 | max_nei | 1 · too few to judge | 0 · too few to judge | 181 · 0.608 (Wilson 0.54–0.68) |
+| base | 3 | margin | 1 · too few to judge | 0 · too few to judge | 220 · 0.559 (Wilson 0.49–0.62) |
+| large | 1 | max | 1 · too few to judge | 112 · 0.759 (Wilson 0.67–0.83) | 8 · too few to judge |
+| large | 1 | max_nei | 1 · too few to judge | 112 · 0.759 (Wilson 0.67–0.83) | 8 · too few to judge |
+| large | 1 | margin | 1 · too few to judge | 112 · 0.750 (Wilson 0.66–0.82) | 7 · too few to judge |
+| large | 2 | max | 1 · too few to judge | 31 · 0.935 (Wilson 0.79–0.98) | 117 · 0.650 (Wilson 0.56–0.73) |
+| large | 2 | max_nei | 1 · too few to judge | 31 · 0.935 (Wilson 0.79–0.98) | 117 · 0.650 (Wilson 0.56–0.73) |
+| large | 2 | margin | 1 · too few to judge | 93 · 0.828 (Wilson 0.74–0.89) | 48 · 0.500 (Wilson 0.36–0.64) |
+| large | 3 | max | 1 · too few to judge | 34 · 0.882 (Wilson 0.73–0.95) | 129 · 0.605 (Wilson 0.52–0.68) |
+| large | 3 | max_nei | 1 · too few to judge | 34 · 0.882 (Wilson 0.73–0.95) | 129 · 0.605 (Wilson 0.52–0.68) |
+| large | 3 | margin | 1 · too few to judge | 28 · 0.929 (Wilson 0.77–0.98) | 114 · 0.640 (Wilson 0.55–0.72) |
+| large-fever | 1 | max | 27 · 0.963 (Wilson 0.82–0.99) | 121 · 0.678 (Wilson 0.59–0.75) | 5 · too few to judge |
+| large-fever | 1 | max_nei | 27 · 0.963 (Wilson 0.82–0.99) | 93 · 0.763 (Wilson 0.67–0.84) | 0 · too few to judge |
+| large-fever | 1 | margin | 1 · too few to judge | 147 · 0.735 (Wilson 0.66–0.80) | 1 · too few to judge |
+| large-fever | 2 | max | 1 · too few to judge | 165 · 0.776 (Wilson 0.71–0.83) | 0 · too few to judge |
+| large-fever | 2 | max_nei | 1 · too few to judge | 155 · 0.781 (Wilson 0.71–0.84) | 0 · too few to judge |
+| large-fever | 2 | margin | 1 · too few to judge | 161 · 0.783 (Wilson 0.71–0.84) | 3 · too few to judge |
+| large-fever | 3 | max | 1 · too few to judge | 167 · 0.743 (Wilson 0.67–0.80) | 6 · too few to judge |
+| large-fever | 3 | max_nei | 1 · too few to judge | 167 · 0.743 (Wilson 0.67–0.80) | 5 · too few to judge |
+| large-fever | 3 | margin | 1 · too few to judge | 161 · 0.758 (Wilson 0.69–0.82) | 21 · 0.429 (Wilson 0.24–0.63) |
+
+## Numeric layer
+
+The numeric layer (spec §10) runs on the same top-k passages before any model. A mismatch decides REFUTED by rule at score 1.0, and no model is credited with it. It depends on k only, so one row per k covers every model and aggregation. A firing is correct when the gold label is REFUTED.
+
+| k | SciFact train fired | train correct | SciFact dev fired | dev correct |
+|---|---|---|---|---|
+| 1 | 4 | 0 | 1 | 1 |
+| 2 | 5 | 0 | 1 | 1 |
+| 3 | 6 | 0 | 1 | 1 |
+
+Rule-decided rows score exactly 1.0, the top of the scale, so they are among the first rows the `high` cut's walk down SciFact train meets: a wrong firing lowers the precision every lower cut is measured with, and a `high` cut of 1.000000 means no score below 1.0 held 0.85 precision, so only a score of 1.0 reaches the tier. The `high` cut is 1.000000 in 25 of 27 combinations. In 25 of 27, every SciFact dev `high` verdict is a rule firing (the high-tier n equals the dev firings at that k): there the high tier on dev comes from the rule, not the model.
+
+## Limits
+
+- **Dev chooses and reports.** SciFact dev both picks the winner and reports it. The winner's 0.697 macro-F1 comes from the split that chose it, out of 27 combinations whose highest dev macro-F1 is 0.697, so it is optimistic; a held-out split would likely read lower.
+- **int8 scores depend on the batch.** The candidate files are dynamically quantised to int8, so a pair's probabilities depend on what else is in its batch. Re-scoring 48 SciFact dev top-1 pairs one at a time moved probabilities by up to 0.197 (mean 0.008) and flipped 1 argmax (measured 2026-10-01 by the final review). That likely explains why `base` × k=1 × `max` reaches dev accuracy 0.591 here (0.597 even at decide 0.45) against the 0.609 published on 2026-09-12 with dev-fitted cuts.
+- **The AVeriTeC snapshot is new.** It is not the 2026-09-16 run's: coverage improved since (then 61 of 89 answerable claims were readable, now 80 of 89), so AVeriTeC numbers here are not comparable with that run's.
+- **Grid floor.** 3 of 27 combinations put `decide` on the grid floor (0.05), marked `(grid floor)` above: SciFact train accuracy was best at the lowest cut tried, so the best cut may lie below the grid.
diff --git a/docs/superpowers/OPEN-ITEMS.md b/docs/superpowers/OPEN-ITEMS.md
index 0da0e2c..91955b5 100644
--- a/docs/superpowers/OPEN-ITEMS.md
+++ b/docs/superpowers/OPEN-ITEMS.md
@@ -608,3 +608,4 @@ five packages. The user's decisions:
 | 20.8 | Bibliography parsing that was measured but never shipped | two-column rejoin (9.10/12.1), multi-block headless fallback (11.11), more headings (16.5), `[1]-[3]` ranges (9.4) |
 | 20.9 | The judge's quote is never checked | `judge._collect` does not verify that the quoted span occurs in the passage. Checking it is a cheap way to enforce product rule 1 |
 | 20.10 | Eval honesty | add a `--judge` flag to `scripts/eval_averitec.py`; do one run with the browser allowed; show the 0.404 search score in the README "Measured" table |
+| 20.11 | NLI bake-off result | **Winner: `large-fever` (MoritzLaurer DeBERTa-v3-large mnli-fever-anli-ling-wanli) × k=2 × `max`.** SciFact dev macro-F1 0.697, against 0.580 for the `base` × k=1 × `max` baseline and 0.608 for the best `base` row. Accuracy 0.697. AVeriTeC 3-way 0.438, or 0.475 on the readable subset, against a 0.708 majority baseline. Cost: 643 MB and 54 ms/pair, against 244 MB and 19 ms/pair. Full report: `docs/eval/2026-10-01-nli-bakeoff.md`. Next: a follow-up spec for an opt-in "accurate" NLI model (consent download, spec §7.1). It must settle three open points. (a) Exclude rule-decided rows from the tier walk: the numeric layer fires 4–6 times on train with 0 correct, so it pins `high` at 1.000000. (b) The winner's `medium` cut sits on `decide`, so the report shows only two tiers. (c) On AVeriTeC the winner is below `base` × k=2 (0.475 against 0.537), and `Supported` is still 2/19 |
diff --git a/scripts/eval_nli_bakeoff.py b/scripts/eval_nli_bakeoff.py
index faa5b13..07b7e2f 100644
--- a/scripts/eval_nli_bakeoff.py
+++ b/scripts/eval_nli_bakeoff.py
@@ -33,11 +33,13 @@
     load_items,
     load_probs,
     load_snapshot,
+    passage_offsets,
     pending,
     save_items,
     save_probs,
     save_snapshot,
     source_from_fetched,
+    split_at,
 )
 from proofpath.models import Label, Passage
 from proofpath.paths import cache_dir, models_dir
@@ -129,6 +131,7 @@ def build_items(embedder: retrieval.Embedder) -> list[Item]:
         for pair in data.pairs():
             doc = data.corpus[pair.doc_id]
             passages = [Passage(t, str(doc.doc_id), i) for i, t in enumerate(doc.sentences)]
+            # Keys are not unique (dev repeats the pair 1245:7662395); consumers are positional.
             key = f"scifact-{split}:{pair.claim_id}:{pair.doc_id}"
             items.append(
                 Item(
@@ -141,7 +144,14 @@ def build_items(embedder: retrieval.Embedder) -> list[Item]:
                     pair.rationale,
                 )
             )
-    for claim in load_snapshot(snapshot_path()):
+    items.extend(averitec_items(load_snapshot(snapshot_path()), embedder))
+    return items
+
+
+def averitec_items(snapshot: Sequence[SnapshotClaim], embedder: retrieval.Embedder) -> list[Item]:
+    """One item per readable source; the key carries the source's position in its claim."""
+    items: list[Item] = []
+    for claim in snapshot:
         for position, source in enumerate(claim.sources):
             sentences = retrieval.split_sentences(source.text) if source.text.strip() else []
             if not sentences:
@@ -160,6 +170,14 @@ def build_items(embedder: retrieval.Embedder) -> list[Item]:
     return items
 
 
+def split_blocks(flat: np.ndarray, items: Sequence[Item]) -> list[np.ndarray]:
+    """Cut the flat ``(n_pairs, 3)`` scores into one block per item, in item order."""
+    offsets = passage_offsets(items)
+    if offsets[-1] != len(flat):
+        raise ValueError(f"{len(flat)} score rows for {offsets[-1]} passages")
+    return split_at(flat, offsets)
+
+
 def cmd_snapshot(args: argparse.Namespace) -> int:
     from proofpath.config import load_config
     from proofpath.verify import Engine
@@ -197,6 +215,7 @@ def cmd_snapshot(args: argparse.Namespace) -> int:
             )
     except KeyboardInterrupt:
         print(f"\ninterrupted; {path} is usable with --resume", file=sys.stderr)
+        return 130
     finally:
         engine.close()
     return 0
@@ -253,8 +272,7 @@ def cmd_score(args: argparse.Namespace) -> int:
     finally:
         scorer.close()
     flat = np.concatenate(blocks) if blocks else np.zeros((0, 3), dtype=np.float32)
-    offsets = np.cumsum([0, *(len(item.passages) for item in items)])
-    probs = [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))]
+    probs = split_blocks(flat, items)
     ms_per_pair = elapsed / max(len(pairs), 1) * 1000
     out = bakeoff_dir() / f"{args.model}.npz"
     save_probs(out, items, probs, ms_per_pair=ms_per_pair)
@@ -271,6 +289,28 @@ class ReportRow:
     averitec: bakeoff.AveritecResult
 
 
+@dataclass(frozen=True)
+class NumericLayer:
+    """Numeric-layer firings at one k, as ``(fired, correct)``; model-independent."""
+
+    train: tuple[int, int]
+    dev: tuple[int, int]
+
+
+def outcomes_for(rows: Sequence[ReportRow]) -> list[bakeoff.Outcome]:
+    """The numbers the decision rule reads, one per report row."""
+    return [
+        bakeoff.Outcome(
+            r.combo,
+            r.large,
+            r.scifact.macro_f1,
+            r.averitec.accuracy_readable,
+            r.scifact.asserted_without_passage + r.averitec.asserted_without_passage,
+        )
+        for r in rows
+    ]
+
+
 def model_size_mb(model: str) -> float | None:
     """Size of the ONNX file the run used, or None when it is not in the cache."""
     from huggingface_hub import try_to_load_from_cache
@@ -300,6 +340,132 @@ def _without_passage_cell(count: int) -> str:
     return f"**{count} — rule 1 violated**" if count else "0"
 
 
+def on_grid_floor(decide: float) -> bool:
+    """True when ``decide`` is the lowest cut the sweep tried: the best may lie below."""
+    return decide == bakeoff.GRID[0]
+
+
+def _decide_cell(decide: float) -> str:
+    return f"{decide:.2f} (grid floor)" if on_grid_floor(decide) else f"{decide:.2f}"
+
+
+# The last AVeriTeC run before this one (docs/eval/2026-09-16-averitec.md): its snapshot,
+# its cuts and its denominators.
+PUBLISHED_AVERITEC = "2026-09-16"
+PUBLISHED_ANSWERABLE = 89
+PUBLISHED_READABLE = 61
+
+
+def _result_lines(rows: Sequence[ReportRow], choice: bakeoff.Choice) -> list[str]:
+    """The verdict of the rule, the winner named once, and its gain over the best base row."""
+    if choice.winner is None:
+        return [f"**No change.** {choice.reason}"]
+    label = choice.winner.label()
+    if choice.reason.startswith(label):
+        lines = [f"**Winner:** {choice.reason}."]
+    else:
+        lines = [f"**Winner:** {label} — {choice.reason}."]
+    outcomes = outcomes_for(rows)
+    winner = next(row for row in outcomes if row.combo == choice.winner)
+    best = bakeoff.best_eligible_base(outcomes, baseline=BASELINE)
+    if best is None:
+        lines.append("No `base` row is eligible under step 1, so the winner has no base bar.")
+    elif best.combo == winner.combo:
+        lines.append("The winner is itself the best eligible `base` row.")
+    else:
+        lines.append(
+            f"Over the best eligible `base` row, {best.combo.label()} "
+            f"({best.macro_f1:.3f}), the winner is {winner.macro_f1 - best.macro_f1:+.3f} "
+            "macro-F1."
+        )
+    return [lines[0], "", *lines[1:]]
+
+
+def _numeric_lines(rows: Sequence[ReportRow], numeric: Mapping[int, NumericLayer]) -> list[str]:
+    """Per-k firings, and how much of the `high` tier the rule rather than a model holds."""
+    lines = [
+        "## Numeric layer",
+        "",
+        "The numeric layer (spec §10) runs on the same top-k passages before any model. A "
+        "mismatch decides REFUTED by rule at score 1.0, and no model is credited with it. "
+        "It depends on k only, so one row per k covers every model and aggregation. A "
+        "firing is correct when the gold label is REFUTED.",
+        "",
+        "| k | SciFact train fired | train correct | SciFact dev fired | dev correct |",
+        "|---|---|---|---|---|",
+    ]
+    for k in sorted(numeric):
+        layer = numeric[k]
+        lines.append(_md(k, layer.train[0], layer.train[1], layer.dev[0], layer.dev[1]))
+    at_one = sum(row.thresholds.high >= 1.0 for row in rows)
+    rule_only = 0
+    for row in rows:
+        high = next(t for t in row.scifact.tiers if t.tier == "high")
+        fired = numeric[row.combo.k].dev[0] if row.combo.k in numeric else -1
+        rule_only += high.n > 0 and high.n == fired
+    lines.extend(
+        [
+            "",
+            "Rule-decided rows score exactly 1.0, the top of the scale, so they are among "
+            "the first rows the `high` cut's walk down SciFact train meets: a wrong firing "
+            "lowers the precision every lower cut is measured with, and a `high` cut of "
+            "1.000000 means no score below 1.0 held "
+            f"{bakeoff.HIGH_TARGET:.2f} precision, so only a score of 1.0 reaches the tier. "
+            f"The `high` cut is 1.000000 in {at_one} of {len(rows)} combinations. In "
+            f"{rule_only} of {len(rows)}, every SciFact dev `high` verdict is a rule firing "
+            "(the high-tier n equals the dev firings at that k): there the high tier on "
+            "dev comes from the rule, not the model.",
+        ]
+    )
+    return lines
+
+
+def _limits_lines(rows: Sequence[ReportRow], choice: bakeoff.Choice) -> list[str]:
+    outcomes = outcomes_for(rows)
+    top = max((row.macro_f1 for row in outcomes), default=0.0)
+    if choice.winner is None:
+        chosen = f"The best row's {top:.3f}"
+    else:
+        won = next(row for row in outcomes if row.combo == choice.winner).macro_f1
+        chosen = f"The winner's {won:.3f}"
+    base = next((row for row in rows if row.combo == BASELINE), None)
+    base_acc = f"dev accuracy {base.scifact.accuracy:.3f}" if base else "a lower dev accuracy"
+    counted = rows[0].averitec.counted if rows else 0
+    readable = rows[0].averitec.readable if rows else 0
+    floor = sum(on_grid_floor(row.thresholds.decide) for row in rows)
+    if floor:
+        floor_line = (
+            f"- **Grid floor.** {floor} of {len(rows)} combinations put `decide` on the grid "
+            f"floor ({bakeoff.GRID[0]:.2f}), marked `(grid floor)` above: SciFact train "
+            "accuracy was best at the lowest cut tried, so the best cut may lie below the grid."
+        )
+    else:
+        floor_line = (
+            f"- **Grid floor.** No combination puts `decide` on the grid floor "
+            f"({bakeoff.GRID[0]:.2f})."
+        )
+    return [
+        "## Limits",
+        "",
+        "- **Dev chooses and reports.** SciFact dev both picks the winner and reports it. "
+        f"{chosen} macro-F1 comes from the split that chose it, out of {len(rows)} "
+        f"combinations whose highest dev macro-F1 is {top:.3f}, so it is optimistic; a "
+        "held-out split would likely read lower.",
+        "- **int8 scores depend on the batch.** The candidate files are dynamically "
+        "quantised to int8, so a pair's probabilities depend on what else is in its batch. "
+        "Re-scoring 48 SciFact dev top-1 pairs one at a time moved probabilities by up to "
+        "0.197 (mean 0.008) and flipped 1 argmax (measured 2026-10-01 by the final review). "
+        f"That likely explains why `base` × k=1 × `max` reaches {base_acc} here "  # noqa: RUF001
+        "(0.597 even at decide 0.45) against the 0.609 published on 2026-09-12 with "
+        "dev-fitted cuts.",
+        "- **The AVeriTeC snapshot is new.** It is not the "
+        f"{PUBLISHED_AVERITEC} run's: coverage improved since (then {PUBLISHED_READABLE} of "
+        f"{PUBLISHED_ANSWERABLE} answerable claims were readable, now {readable} of "
+        f"{counted}), so AVeriTeC numbers here are not comparable with that run's.",
+        floor_line,
+    ]
+
+
 def render_report(
     rows: Sequence[ReportRow],
     choice: bakeoff.Choice,
@@ -307,6 +473,7 @@ def render_report(
     speed: Mapping[str, float],
     sizes: Mapping[str, float | None],
     counts: Mapping[str, int],
+    numeric: Mapping[int, NumericLayer],
     today: str,
     machine: str,
 ) -> str:
@@ -336,10 +503,7 @@ def render_report(
         "## Result",
         "",
     ]
-    if choice.winner is None:
-        lines.append(f"**No change.** {choice.reason}")
-    else:
-        lines.append(f"**Winner:** {choice.winner.label()} — {choice.reason}")
+    lines.extend(_result_lines(rows, choice))
     lines.extend(
         ["", "## Models", "", "| model | repo | size (MB) | ms/pair |", "|---|---|---|---|"]
     )
@@ -358,8 +522,11 @@ def render_report(
             "",
             "## Every combination",
             "",
-            f"AVeriTeC majority baseline: {baseline_ave:.3f}. Published numbers to compare: "
-            "0.270 (all) and 0.361 (readable), 2026-09-16.",
+            f"AVeriTeC majority baseline: {baseline_ave:.3f}. The {PUBLISHED_AVERITEC} run "
+            "reported 0.270 (all) and 0.361 (readable), but on a different snapshot with "
+            f"different cuts, over {PUBLISHED_ANSWERABLE} answerable / {PUBLISHED_READABLE} "
+            f"readable claims; this run counts {counted} answerable / {readable} readable. "
+            "A gap between the two runs is partly a coverage change, not only a model gain.",
             "",
             "| model | k | aggregation | decide | medium | high | dev acc | dev macro-F1 "
             "| F1 S / R / NEI | rationale F1 | AVeriTeC all | AVeriTeC readable | w/o passage |",
@@ -373,7 +540,7 @@ def render_report(
                 row.combo.model,
                 row.combo.k,
                 row.combo.aggregation,
-                f"{row.thresholds.decide:.2f}",
+                _decide_cell(row.thresholds.decide),
                 f"{row.thresholds.medium:.6f}",
                 f"{row.thresholds.high:.6f}",
                 f"{row.scifact.accuracy:.3f}",
@@ -430,6 +597,7 @@ def render_report(
                 *(_tier_cell(tiers[name]) for name in ("high", "medium", "low")),
             )
         )
+    lines.extend(["", *_numeric_lines(rows, numeric), "", *_limits_lines(rows, choice)])
     return "\n".join(lines) + "\n"
 
 
@@ -449,6 +617,7 @@ def cmd_report(args: argparse.Namespace) -> int:
     snapshot = load_snapshot(snapshot_path())
     rows: list[ReportRow] = []
     speed: dict[str, float] = {}
+    numeric: dict[int, NumericLayer] = {}
     for model, candidate in CANDIDATES.items():
         path = bakeoff_dir() / f"{model}.npz"
         if not path.exists():
@@ -468,6 +637,11 @@ def cmd_report(args: argparse.Namespace) -> int:
         dev = [s for s in scored if s.item.dataset == "scifact-dev"]
         ave = [s for s in scored if s.item.dataset == "averitec"]
         for k in bakeoff.KS:
+            # The rule reads the items only, so every model gives the same firings.
+            if k not in numeric:
+                numeric[k] = NumericLayer(
+                    bakeoff.numeric_firings(train, k=k), bakeoff.numeric_firings(dev, k=k)
+                )
             for aggregation in bakeoff.AGGREGATIONS:
                 thresholds = bakeoff.calibrate(
                     bakeoff.calibration_rows(train, k=k, aggregation=aggregation)
@@ -488,17 +662,7 @@ def cmd_report(args: argparse.Namespace) -> int:
             "error     the baseline model is not scored; run `score --model base`", file=sys.stderr
         )
         return 2
-    outcomes = [
-        bakeoff.Outcome(
-            r.combo,
-            r.large,
-            r.scifact.macro_f1,
-            r.averitec.accuracy_readable,
-            r.scifact.asserted_without_passage + r.averitec.asserted_without_passage,
-        )
-        for r in rows
-    ]
-    choice = bakeoff.choose(outcomes, baseline=BASELINE)
+    choice = bakeoff.choose(outcomes_for(rows), baseline=BASELINE)
     counts = {
         "scifact-train": sum(i.dataset == "scifact-train" for i in items),
         "scifact-dev": sum(i.dataset == "scifact-dev" for i in items),
@@ -510,6 +674,7 @@ def cmd_report(args: argparse.Namespace) -> int:
         speed=speed,
         sizes={model: model_size_mb(model) for model in speed},
         counts=counts,
+        numeric=numeric,
         today=date.today().isoformat(),
         machine=f"{platform.system()} {platform.machine()}",
     )
diff --git a/src/proofpath/eval/bakeoff.py b/src/proofpath/eval/bakeoff.py
index b358256..575bb5e 100644
--- a/src/proofpath/eval/bakeoff.py
+++ b/src/proofpath/eval/bakeoff.py
@@ -105,11 +105,37 @@ def signal_for(
     A numeric mismatch decides by rule at score 1.0, as ``pipeline.decide_indexed``
     does, so no candidate model is credited with what the rule decided.
     """
-    top = list(passages[:k])
-    numeric = numerics.check(claim, top)
+    ruled = _rule_signal(claim, passages, k=k)
+    if ruled is not None:
+        return ruled
+    return AGGREGATORS[aggregation](passages[:k], probs[:k])
+
+
+def _rule_signal(claim: str, passages: Sequence[Passage], *, k: int) -> Signal | None:
+    """The numeric layer's REFUTED at score 1.0 over the first ``k`` hits, if it fires."""
+    if k < 1:
+        raise ValueError(f"k must be at least 1, got {k}")
+    numeric = numerics.check(claim, list(passages[:k]))
     if numeric is not None and numeric.mismatch:
         return Signal(Label.REFUTED, 1.0, numeric.passage)
-    return AGGREGATORS[aggregation](top, probs[:k])
+    return None
+
+
+def numeric_firings(scored: Sequence[Scored], *, k: int) -> tuple[int, int]:
+    """``(fired, correct)``: how often the numeric layer decides by rule at this k.
+
+    The same path as ``signal_for``, so these are exactly the verdicts scored 1.0 that
+    no model earned. A firing is correct when the gold label is REFUTED. It depends on
+    the items and k only, never on the model or the aggregation.
+    """
+    if k < 1:
+        raise ValueError(f"k must be at least 1, got {k}")
+    fired = correct = 0
+    for s in scored:
+        if _rule_signal(s.item.claim, s.item.passages, k=k) is not None:
+            fired += 1
+            correct += Label(s.item.gold) is Label.REFUTED
+    return fired, correct
 
 
 def decide(signal: Signal, thresholds: Thresholds) -> Label:
@@ -123,6 +149,8 @@ def wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]:
     """95 % Wilson score interval for ``correct`` successes out of ``n``."""
     if n <= 0:
         raise ValueError("a Wilson interval needs at least one observation")
+    if not 0 <= correct <= n:
+        raise ValueError(f"correct must lie between 0 and n={n}, got {correct}")
     p = correct / n
     denominator = 1 + z * z / n
     centre = (p + z * z / (2 * n)) / denominator
@@ -271,7 +299,7 @@ class AveritecResult:
     n: int
     counted: int  # claims with a 3-way gold label
     accuracy_all: float
-    readable: int  # counted claims with at least one readable source
+    readable: int  # counted claims with a source whose snapshot text is not blank
     accuracy_readable: float
     majority_baseline: float
     per_label: dict[str, tuple[int, int]]  # gold label -> (n, correct)
@@ -291,11 +319,22 @@ def evaluate_averitec(
     The strongest decided non-NEI verdict across the claim's sources wins. A claim
     with a readable source and no assertion is NEI. A claim with nothing readable is
     unanswered, and is scored as NEI.
+
+    A claim is readable when any of its snapshot sources has non-blank text, as
+    ``_decide_claim``'s ``fetched_any`` has it, whether or not scored items exist.
+    Scored items whose group matches no snapshot claim are refused: a key-format drift
+    must not silently turn every claim into NEI (product rule 6).
     """
     _require(scored, "averitec")
     by_group: dict[str, list[Scored]] = {}
     for s in scored:
         by_group.setdefault(s.item.group, []).append(s)
+    known = {f"averitec:{claim.claim_id}" for claim in claims}
+    orphans = sorted(set(by_group) - known)
+    if orphans:
+        raise ValueError(
+            f"{len(orphans)} scored groups match no snapshot claim: {', '.join(orphans[:5])}"
+        )
     per_label: dict[str, tuple[int, int]] = {}
     golds: list[Label] = []
     correct_all = readable = correct_readable = unbacked = 0
@@ -320,7 +359,7 @@ def evaluate_averitec(
             continue
         golds.append(gold)
         correct_all += correct
-        if group:
+        if any(source.text.strip() for source in claim.sources):
             readable += 1
             correct_readable += correct
     majority = max(Counter(golds).values(), default=0)
@@ -385,8 +424,29 @@ class Choice:
 
 
 def choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
-    """Apply ``DECISION_RULE`` step by step; each step filters the previous step's rows."""
+    """Apply ``DECISION_RULE`` step by step; each step filters the previous step's rows.
+
+    When the baseline row itself fails step 1, the reason says so: every margin is still
+    measured against its numbers, and a reader must not take it for a clean baseline.
+    """
+    choice = _choose(outcomes, baseline=baseline)
+    base = next(row for row in outcomes if row.combo == baseline)
+    if base.asserted_without_passage:
+        return Choice(
+            choice.winner,
+            "the baseline itself is not eligible: it asserts without a passage "
+            f"{base.asserted_without_passage} times; {choice.reason}",
+        )
+    return choice
+
+
+def _choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
     by_combo = {row.combo: row for row in outcomes}
+    if len(by_combo) != len(outcomes):
+        duplicated = sorted(
+            combo.label() for combo, n in Counter(row.combo for row in outcomes).items() if n > 1
+        )
+        raise ValueError(f"duplicate combinations among the outcomes: {', '.join(duplicated)}")
     if baseline not in by_combo:
         raise ValueError(f"the baseline {baseline.label()} is not among the outcomes")
     base = by_combo[baseline]
@@ -443,3 +503,23 @@ def choose(outcomes: Sequence[Outcome], *, baseline: Combo) -> Choice:
             f"the top macro-F1 row ({top.macro_f1:.3f})"
         )
     return Choice(winner.combo, reason)
+
+
+def best_eligible_base(outcomes: Sequence[Outcome], *, baseline: Combo) -> Outcome | None:
+    """The highest-macro-F1 non-large row that passes step 1, or None when none does.
+
+    Step 1 as ``choose`` applies it: no assertion without a passage, and AVeriTeC
+    readable accuracy not below the baseline's. The report uses it to state a winner's
+    gain over the strongest eligible ``base`` row, the bar of the size gate.
+    """
+    base = next((row for row in outcomes if row.combo == baseline), None)
+    if base is None:
+        raise ValueError(f"the baseline {baseline.label()} is not among the outcomes")
+    eligible = [
+        row
+        for row in outcomes
+        if not row.large
+        and row.asserted_without_passage == 0
+        and row.averitec_readable >= base.averitec_readable - _EPS
+    ]
+    return max(eligible, key=lambda row: row.macro_f1, default=None)
diff --git a/src/proofpath/eval/bakeoff_data.py b/src/proofpath/eval/bakeoff_data.py
index 5b3d8c5..c64ca21 100644
--- a/src/proofpath/eval/bakeoff_data.py
+++ b/src/proofpath/eval/bakeoff_data.py
@@ -171,6 +171,16 @@ def items_fingerprint(items: Sequence[Item]) -> str:
     return digest.hexdigest()
 
 
+def passage_offsets(items: Sequence[Item]) -> np.ndarray:
+    """Where each item's rows start in the flat ``(n_pairs, 3)`` array, plus the end."""
+    return np.cumsum([0, *(len(item.passages) for item in items)], dtype=np.int64)
+
+
+def split_at(flat: np.ndarray, offsets: np.ndarray) -> list[np.ndarray]:
+    """Cut ``flat`` into one block per pair of consecutive offsets. No validation."""
+    return [flat[offsets[n] : offsets[n + 1]] for n in range(len(offsets) - 1)]
+
+
 def save_probs(
     path: Path, items: Sequence[Item], probs: Sequence[np.ndarray], *, ms_per_pair: float
 ) -> None:
@@ -180,7 +190,7 @@ def save_probs(
     for item, block in zip(items, probs, strict=True):
         if np.asarray(block).shape != (len(item.passages), 3):
             raise ValueError(f"probability block for {item.key} has the wrong shape")
-    offsets = np.cumsum([0, *(len(item.passages) for item in items)], dtype=np.int64)
+    offsets = passage_offsets(items)
     flat = (
         np.concatenate([np.asarray(b, dtype=np.float32) for b in probs])
         if probs
@@ -208,4 +218,15 @@ def load_probs(path: Path, items: Sequence[Item]) -> tuple[list[np.ndarray], flo
         ms_per_pair = float(data["ms_per_pair"])
     if fingerprint != items_fingerprint(items):
         raise ValueError(f"{path.name} was scored on other items; run `score` again")
-    return [flat[offsets[n] : offsets[n + 1]] for n in range(len(items))], ms_per_pair
+    if len(offsets) != len(items) + 1:
+        raise ValueError(
+            f"{path.name} holds {len(offsets) - 1} blocks for {len(items)} items; run `score` again"
+        )
+    blocks = split_at(flat, offsets)
+    for item, block in zip(items, blocks, strict=True):
+        if len(block) != len(item.passages):
+            raise ValueError(
+                f"{path.name} holds {len(block)} rows for {item.key}, which has "
+                f"{len(item.passages)} passages; run `score` again"
+            )
+    return blocks, ms_per_pair
diff --git a/tests/test_bakeoff.py b/tests/test_bakeoff.py
index e1d4c62..17234db 100644
--- a/tests/test_bakeoff.py
+++ b/tests/test_bakeoff.py
@@ -73,6 +73,52 @@ def test_a_numeric_mismatch_decides_before_any_aggregation() -> None:
     assert signal == bakeoff.Signal(Label.REFUTED, 1.0, passages[0])
 
 
+def _numeric_item(key: str, dataset: str, gold: str, passage: str) -> bakeoff.Scored:
+    passages = (Passage(passage, key, 0), Passage("An unrelated sentence.", key, 1))
+    item = Item(key, dataset, key, "Mortality fell by 40% in the treated group.", gold, passages)
+    return bakeoff.Scored(item, _probs((0.99, 0.0, 0.01), (0.99, 0.0, 0.01)))
+
+
+def test_numeric_firings_count_rule_verdicts_and_the_refuted_ones_right() -> None:
+    scored = [
+        _numeric_item(
+            "a", "scifact-train", "REFUTED", "Mortality fell by 12% in the treated group."
+        ),
+        _numeric_item(
+            "b", "scifact-train", "SUPPORTED", "Mortality fell by 9% in the treated group."
+        ),
+        _numeric_item("c", "scifact-train", "SUPPORTED", "No figures here."),
+    ]
+    assert bakeoff.numeric_firings(scored, k=1) == (2, 1)
+
+
+def test_numeric_firings_agree_with_signal_for_at_every_k() -> None:
+    late = Item(
+        "d",
+        "scifact-dev",
+        "d",
+        "Mortality fell by 40% in the treated group.",
+        "REFUTED",
+        (Passage("No figures here.", "d", 0), Passage("Mortality fell by 12%.", "d", 1)),
+    )
+    scored = [bakeoff.Scored(late, _probs((0.2, 0.1, 0.7), (0.2, 0.1, 0.7)))]
+    for k in (1, 2):
+        ruled = sum(
+            bakeoff.signal_for(s.item.claim, s.item.passages, s.probs, k=k, aggregation="max")
+            == bakeoff.Signal(Label.REFUTED, 1.0, s.item.passages[1])
+            for s in scored
+        )
+        assert bakeoff.numeric_firings(scored, k=k) == (ruled, ruled)
+    assert bakeoff.numeric_firings(scored, k=1) == (0, 0)
+    assert bakeoff.numeric_firings(scored, k=2) == (1, 1)
+
+
+@pytest.mark.parametrize("k", [0, -1])
+def test_numeric_firings_refuse_a_k_below_one(k: int) -> None:
+    with pytest.raises(ValueError, match="at least 1"):
+        bakeoff.numeric_firings([], k=k)
+
+
 def test_decide_needs_the_cut_and_a_passage() -> None:
     cuts = Thresholds(decide=0.5, high=0.9, medium=0.7)
     assert bakeoff.decide(bakeoff.Signal(Label.SUPPORTED, 0.6, P[0]), cuts) is Label.SUPPORTED
@@ -94,6 +140,25 @@ def test_wilson_refuses_an_empty_sample() -> None:
         bakeoff.wilson(0, 0)
 
 
+@pytest.mark.parametrize(("correct", "n"), [(-1, 10), (11, 10)])
+def test_wilson_refuses_a_count_outside_the_sample(correct: int, n: int) -> None:
+    with pytest.raises(ValueError, match="between 0 and n"):
+        bakeoff.wilson(correct, n)
+
+
+@pytest.mark.parametrize("k", [0, -1])
+def test_signal_for_refuses_a_k_below_one(k: int) -> None:
+    with pytest.raises(ValueError, match="k must be at least 1"):
+        bakeoff.signal_for("a claim", P[:1], _probs((0.9, 0.0, 0.1)), k=k, aggregation="max")
+
+
+def test_ship_cut_rounds_up_at_six_decimals_and_caps_at_one() -> None:
+    assert bakeoff.ship_cut(0.4579471) == pytest.approx(0.457948)
+    assert bakeoff.ship_cut(0.4579470) == pytest.approx(0.457947)
+    assert bakeoff.ship_cut(0.9999999) == 1.0
+    assert bakeoff.ship_cut(1.0) == 1.0
+
+
 @pytest.mark.parametrize("name", bakeoff.AGGREGATIONS)
 def test_aggregators_refuse_a_passage_and_row_count_mismatch(name: str) -> None:
     probs = _probs((0.5, 0.2, 0.3), (0.1, 0.1, 0.8), (0.2, 0.7, 0.1))
@@ -207,7 +272,8 @@ def test_evaluate_scifact_counts_verdicts_tiers_and_rationale() -> None:
         "low": (0, 0),
     }
     assert result.asserted_without_passage == 0
-    assert 0.0 < result.rationale_f1 <= 1.0
+    # Gold {0} for all three; predicted {0}, {0}, {}: tp 2, fp 0, fn 1 -> P 1, R 2/3, F1 0.8.
+    assert result.rationale_f1 == pytest.approx(0.8)
 
 
 def test_evaluate_averitec_keeps_the_strongest_source_and_both_denominators() -> None:
@@ -228,6 +294,38 @@ def test_evaluate_averitec_keeps_the_strongest_source_and_both_denominators() ->
     assert result.readable == 1
     assert result.accuracy_readable == pytest.approx(1.0)
     assert result.majority_baseline == pytest.approx(0.5)
+    assert result.per_label == {
+        "Refuted": (1, 1),
+        "Supported": (1, 0),
+        "Conflicting Evidence/Cherrypicking": (1, 0),
+    }
+    assert result.asserted_without_passage == 0
+
+
+def test_a_readable_claim_whose_only_signal_is_below_the_cut_is_nei_and_readable() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [SnapshotClaim(1, "c1", "Not Enough Evidence", (SnapshotSource("u", "ok", "t"),), 0)]
+    scored = [_scored("averitec:1:0", "averitec", "x", (0.3, 0.1, 0.6), group="averitec:1")]
+    result = bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(1.0)
+    assert result.per_label == {"Not Enough Evidence": (1, 1)}
+
+
+def test_a_claim_with_readable_text_but_no_scored_items_is_still_readable() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u", "ok", "text"),), 0)]
+    result = bakeoff.evaluate_averitec(claims, [], cuts, k=1, aggregation="max")
+    assert result.readable == 1
+    assert result.accuracy_readable == pytest.approx(0.0)
+
+
+def test_scored_items_matching_no_snapshot_claim_are_refused() -> None:
+    cuts = Thresholds(decide=0.5, high=0.95, medium=0.7)
+    claims = [SnapshotClaim(1, "c1", "Refuted", (SnapshotSource("u", "ok", "t"),), 0)]
+    scored = [_scored("averitec:9:0", "averitec", "Refuted", (0.0, 0.9, 0.1), group="avt:9")]
+    with pytest.raises(ValueError, match="avt:9"):
+        bakeoff.evaluate_averitec(claims, scored, cuts, k=1, aggregation="max")
 
 
 BASE = bakeoff.Combo("base", 1, "max")
@@ -352,3 +450,52 @@ def test_only_large_rows_beating_the_baseline_but_failing_the_size_gate_is_no_ch
     assert choice.winner is None
     assert choice.reason.startswith("no change:")
     assert "large" in choice.reason
+
+
+def test_the_reason_says_so_when_the_baseline_itself_is_not_eligible() -> None:
+    won = bakeoff.choose(
+        [_o("base", 1, "max", 0.597, unbacked=2), _o("base", 2, "max", 0.620)], baseline=BASE
+    )
+    assert won.winner == bakeoff.Combo("base", 2, "max")
+    assert won.reason.startswith("the baseline itself is not eligible:")
+    unchanged = bakeoff.choose(
+        [_o("base", 1, "max", 0.597, unbacked=2), _o("base", 2, "max", 0.600)], baseline=BASE
+    )
+    assert unchanged.winner is None
+    assert unchanged.reason.startswith("the baseline itself is not eligible:")
+    assert "no change" in unchanged.reason
+
+
+def test_an_eligible_baseline_is_not_called_ineligible() -> None:
+    choice = bakeoff.choose(
+        [_o("base", 1, "max", 0.597), _o("base", 2, "max", 0.620)], baseline=BASE
+    )
+    assert "not eligible" not in choice.reason
+
+
+def test_duplicate_combinations_among_the_outcomes_are_refused() -> None:
+    with pytest.raises(ValueError, match="duplicate"):
+        bakeoff.choose([_o("base", 1, "max", 0.597), _o("base", 1, "max", 0.600)], baseline=BASE)
+
+
+def test_best_eligible_base_skips_large_unbacked_and_averitec_worse_rows() -> None:
+    outcomes = [
+        _o("base", 1, "max", 0.597),
+        _o("base", 2, "max", 0.610),
+        _o("base", 3, "max", 0.640, readable=0.30),
+        _o("base", 2, "margin", 0.645, unbacked=1),
+        _o("large", 1, "max", 0.700),
+    ]
+    best = bakeoff.best_eligible_base(outcomes, baseline=BASE)
+    assert best is not None
+    assert best.combo == bakeoff.Combo("base", 2, "max")
+
+
+def test_best_eligible_base_is_none_when_no_base_row_is_eligible() -> None:
+    outcomes = [_o("base", 1, "max", 0.597, unbacked=2), _o("large", 1, "max", 0.700)]
+    assert bakeoff.best_eligible_base(outcomes, baseline=BASE) is None
+
+
+def test_best_eligible_base_needs_the_baseline() -> None:
+    with pytest.raises(ValueError, match="not among the outcomes"):
+        bakeoff.best_eligible_base([_o("base", 2, "max", 0.6)], baseline=BASE)
diff --git a/tests/test_bakeoff_data.py b/tests/test_bakeoff_data.py
index ec5dbb6..51f9ee1 100644
--- a/tests/test_bakeoff_data.py
+++ b/tests/test_bakeoff_data.py
@@ -16,11 +16,13 @@
     load_items,
     load_probs,
     load_snapshot,
+    passage_offsets,
     pending,
     save_items,
     save_probs,
     save_snapshot,
     source_from_fetched,
+    split_at,
 )
 from proofpath.fetch import Fetched, Outcome
 from proofpath.models import Passage
@@ -127,3 +129,66 @@ def test_the_fingerprint_changes_with_a_passage_text() -> None:
     a = _item("a")
     b = Item(a.key, a.dataset, a.group, a.claim, a.gold, (Passage("other", "doc", 0),))
     assert items_fingerprint([a]) != items_fingerprint([b])
+
+
+def test_save_probs_refuses_a_block_count_that_differs_from_the_items(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 1)]
+    with pytest.raises(ValueError, match="one probability block per item"):
+        save_probs(tmp_path / "x.npz", items, [np.zeros((2, 3))], ms_per_pair=1.0)
+
+
+def test_an_empty_item_list_round_trips(tmp_path: Path) -> None:
+    path = tmp_path / "empty.npz"
+    save_probs(path, [], [], ms_per_pair=3.0)
+    loaded, ms = load_probs(path, [])
+    assert loaded == []
+    assert ms == pytest.approx(3.0)
+
+
+def test_the_fingerprint_changes_with_a_claim_or_a_key() -> None:
+    a = _item("a")
+    other_claim = Item(a.key, a.dataset, a.group, "another claim", a.gold, a.passages)
+    other_key = Item("z", a.dataset, a.group, a.claim, a.gold, a.passages)
+    assert items_fingerprint([a]) != items_fingerprint([other_claim])
+    assert items_fingerprint([a]) != items_fingerprint([other_key])
+
+
+def _save_with_offsets(path: Path, items: list[Item], offsets: list[int], rows: int) -> None:
+    np.savez(
+        path,
+        fingerprint=np.array(items_fingerprint(items)),
+        offsets=np.array(offsets, dtype=np.int64),
+        probs=np.zeros((rows, 3), dtype=np.float32),
+        ms_per_pair=np.array(1.0),
+    )
+
+
+def test_probs_with_the_wrong_number_of_offsets_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 1)]
+    path = tmp_path / "bad.npz"
+    _save_with_offsets(path, items, [0, 2], 2)
+    with pytest.raises(ValueError, match="blocks for 2 items"):
+        load_probs(path, items)
+
+
+def test_probs_whose_slice_does_not_match_the_passage_count_are_refused(tmp_path: Path) -> None:
+    items = [_item("a", 2), _item("b", 1)]
+    path = tmp_path / "bad.npz"
+    _save_with_offsets(path, items, [0, 1, 3], 3)
+    with pytest.raises(ValueError, match="rows for a"):
+        load_probs(path, items)
+
+
+def test_passage_offsets_start_at_zero_and_end_at_the_pair_count() -> None:
+    offsets = passage_offsets([_item("a", 2), _item("b", 0), _item("c", 3)])
+    assert offsets.tolist() == [0, 2, 2, 5]
+    assert offsets.dtype == np.int64
+    assert passage_offsets([]).tolist() == [0]
+
+
+def test_split_at_cuts_one_block_per_offset_pair() -> None:
+    flat = np.arange(15, dtype=np.float32).reshape(5, 3)
+    blocks = split_at(flat, np.array([0, 2, 2, 5]))
+    assert [b.shape for b in blocks] == [(2, 3), (0, 3), (3, 3)]
+    assert np.array_equal(blocks[2], flat[2:5])
+    assert split_at(np.zeros((0, 3)), np.array([0])) == []
diff --git a/tests/test_eval_nli_bakeoff.py b/tests/test_eval_nli_bakeoff.py
index 0a7ec06..f063d6f 100644
--- a/tests/test_eval_nli_bakeoff.py
+++ b/tests/test_eval_nli_bakeoff.py
@@ -6,13 +6,16 @@
 import os
 import sys
 from argparse import Namespace
+from collections.abc import Sequence
 from pathlib import Path
 from types import ModuleType
 
+import numpy as np
 import pytest
 
-from proofpath.eval import bakeoff
-from proofpath.models import Label
+from proofpath.eval import averitec, bakeoff
+from proofpath.eval.bakeoff_data import Item, SnapshotClaim, SnapshotSource
+from proofpath.models import Label, Passage
 from proofpath.pipeline import Thresholds
 
 SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "eval_nli_bakeoff.py"
@@ -68,6 +71,10 @@ def _row(
     f1: float,
     tiers: tuple[int, int, int],
     unbacked: int = 0,
+    *,
+    decide: float = 0.45,
+    high: float = 0.99933,
+    readable: float = 0.36,
 ) -> object:
     sci = bakeoff.SciFactResult(
         n=340,
@@ -87,7 +94,7 @@ def _row(
         counted=89,
         accuracy_all=0.27,
         readable=61,
-        accuracy_readable=0.36,
+        accuracy_readable=readable,
         majority_baseline=0.708,
         per_label={"Refuted": (63, 20), "Supported": (10, 7)},
         asserted_without_passage=0,
@@ -95,22 +102,30 @@ def _row(
     return script.ReportRow(
         bakeoff.Combo(model, k, agg),
         model != "base",
-        Thresholds(decide=0.45, high=0.99933, medium=0.457948),
+        Thresholds(decide=decide, high=high, medium=0.457948),
         sci,
         ave,
     )
 
 
+NUMERIC = {
+    1: script.NumericLayer(train=(4, 0), dev=(1, 1)),
+    2: script.NumericLayer(train=(5, 0), dev=(1, 1)),
+}
+
+
 def _render(
     choice: bakeoff.Choice,
     unbacked: int = 0,
     models: tuple[str, ...] = ("base", "large", "large-fever"),
+    rows: list[object] | None = None,
 ) -> str:
-    rows = [
-        _row("base", 1, "max", 0.597, (21, 113, 1)),
-        _row("large", 2, "margin", 0.650, (30, 40, 25), unbacked),
-        _row("large-fever", 1, "max", 0.600, (25, 50, 20)),
-    ]
+    if rows is None:
+        rows = [
+            _row("base", 1, "max", 0.597, (21, 113, 1)),
+            _row("large", 2, "margin", 0.650, (30, 40, 25), unbacked),
+            _row("large-fever", 1, "max", 0.600, (25, 50, 20)),
+        ]
     rows = [r for r in rows if r.combo.model in models]
     return script.render_report(
         rows,
@@ -118,11 +133,130 @@ def _render(
         speed={"base": 185.0, "large": 540.0, "large-fever": 540.0},
         sizes={"base": 244.4, "large": None},
         counts={"scifact-train": 919, "scifact-dev": 340, "averitec": 100},
+        numeric=NUMERIC,
         today="2026-10-02",
         machine="Darwin arm64",
     )
 
 
+def _section(text: str, heading: str) -> str:
+    """The body of one ``## heading`` section, up to the next one."""
+    body = text.split(f"## {heading}\n", 1)[1]
+    return body.split("\n## ", 1)[0]
+
+
+def test_the_winner_is_named_once_when_the_reason_starts_with_it() -> None:
+    winner = bakeoff.Combo("large", 2, "margin")
+    reason = f"{winner.label()} is +0.053 macro-F1 over {script.BASELINE.label()}"
+    text = _render(bakeoff.Choice(winner, reason))
+    result = _section(text, "Result")
+    assert result.count(winner.label()) == 1
+    assert f"**Winner:** {reason}." in result
+
+
+def test_a_reason_that_does_not_start_with_the_winner_keeps_the_label() -> None:
+    winner = bakeoff.Combo("large", 2, "margin")
+    text = _render(bakeoff.Choice(winner, "the baseline itself is not eligible: x"))
+    assert f"**Winner:** {winner.label()} — the baseline itself is not eligible: x." in text
+
+
+def test_the_result_states_the_gain_over_the_best_eligible_base_row() -> None:
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("base", 2, "max", 0.610, (1, 10, 10)),
+        # Higher macro-F1, but worse on AVeriTeC readable: not eligible.
+        _row("base", 3, "max", 0.640, (1, 10, 10), readable=0.30),
+        # Higher macro-F1, but asserts without a passage: not eligible.
+        _row("base", 2, "margin", 0.645, (1, 10, 10), unbacked=1),
+        _row("large", 2, "margin", 0.650, (30, 40, 25)),
+    ]
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"), rows=rows)
+    result = _section(text, "Result")
+    best = bakeoff.Combo("base", 2, "max").label()
+    assert f"Over the best eligible `base` row, {best} (0.610), the winner is +0.040" in result
+
+
+def test_a_base_winner_that_is_the_best_base_row_says_so() -> None:
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("base", 2, "max", 0.620, (1, 10, 10)),
+    ]
+    text = _render(bakeoff.Choice(bakeoff.Combo("base", 2, "max"), "why"), rows=rows)
+    assert "The winner is itself the best eligible `base` row." in text
+
+
+def test_no_change_has_no_gain_line() -> None:
+    result = _section(_render(bakeoff.Choice(None, "no change: x")), "Result")
+    assert "best eligible `base` row" not in result
+
+
+def test_a_decide_cut_on_the_grid_floor_is_marked() -> None:
+    floor = bakeoff.GRID[0]
+    rows = [
+        _row("base", 1, "max", 0.597, (21, 113, 1)),
+        _row("large", 1, "max", 0.600, (1, 10, 10), decide=floor),
+    ]
+    text = _render(bakeoff.Choice(None, "x"), rows=rows)
+    assert f"| large | 1 | max | {floor:.2f} (grid floor) |" in text
+    assert "| base | 1 | max | 0.45 |" in text
+    assert f"1 of 2 combinations put `decide` on the grid floor ({floor:.2f})" in text
+
+
+def test_no_grid_floor_cut_is_said_in_words() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "(grid floor) |" not in text
+    assert "No combination puts `decide` on the grid floor" in text
+
+
+def test_the_numeric_layer_table_has_one_row_per_k() -> None:
+    section = _section(_render(bakeoff.Choice(None, "x")), "Numeric layer")
+    assert "| k | SciFact train fired | train correct | SciFact dev fired | dev correct |" in (
+        section
+    )
+    assert "| 1 | 4 | 0 | 1 | 1 |" in section
+    assert "| 2 | 5 | 0 | 1 | 1 |" in section
+
+
+def test_a_high_tier_made_only_of_rule_firings_is_said_to_come_from_the_rule() -> None:
+    rows = [
+        # high cut 1.0, one dev high verdict, one dev firing at k=1: the rule's.
+        _row("base", 1, "max", 0.597, (1, 113, 1), high=1.0),
+        # high cut 1.0 but two dev high verdicts against one firing: not only the rule.
+        _row("base", 2, "max", 0.600, (2, 10, 10), high=1.0),
+        # high cut below 1.0 with model verdicts in the tier.
+        _row("large", 1, "max", 0.600, (25, 50, 20)),
+    ]
+    section = _section(_render(bakeoff.Choice(None, "x"), rows=rows), "Numeric layer")
+    assert "The `high` cut is 1.000000 in 2 of 3 combinations." in section
+    assert "In 1 of 3, every SciFact dev `high` verdict is a rule firing" in section
+    assert "comes from the rule, not the model" in section
+
+
+def test_the_limits_state_dev_optimism_with_the_maximum_over_all_rows() -> None:
+    text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"))
+    limits = _section(text, "Limits")
+    assert "The winner's 0.650 macro-F1 comes from the split that chose it" in limits
+    assert "out of 3 combinations whose highest dev macro-F1 is 0.650" in limits
+
+
+def test_the_limits_state_batch_dependence_snapshot_drift_and_grid_floor() -> None:
+    limits = _section(_render(bakeoff.Choice(None, "x")), "Limits")
+    assert "up to 0.197 (mean 0.008) and flipped 1 argmax" in limits
+    assert "measured 2026-10-01 by the final review" in limits
+    assert "reaches dev accuracy 0.607 here" in limits
+    assert "against the 0.609 published on 2026-09-12" in limits
+    assert "then 61 of 89 answerable claims were readable, now 61 of 89" in limits
+    assert "**Grid floor.**" in limits
+
+
+def test_the_old_averitec_numbers_carry_their_denominators_not_a_comparison() -> None:
+    text = _render(bakeoff.Choice(None, "x"))
+    assert "Published numbers to compare" not in text
+    assert "on a different snapshot with different cuts" in text
+    assert "over 89 answerable / 61 readable claims; this run counts 89 answerable" in text
+    assert "partly a coverage change" in text
+
+
 def test_the_report_prints_the_rule_verbatim_and_the_winner() -> None:
     text = _render(bakeoff.Choice(bakeoff.Combo("large", 2, "margin"), "why"))
     assert bakeoff.DECISION_RULE in text
@@ -181,6 +315,7 @@ def test_a_gold_label_without_a_verdict_is_not_scored_rather_than_wrong() -> Non
         speed={"base": 185.0},
         sizes={"base": None},
         counts={"averitec": 100},
+        numeric=NUMERIC,
         today="2026-10-02",
         machine="Darwin arm64",
     )
@@ -269,3 +404,90 @@ def test_report_with_stale_items_does_not_advise_rebuild_items_flag(
     err = capsys.readouterr().err
     assert "--rebuild-items" not in err
     assert "score --model <name>" in err
+
+
+def test_split_blocks_cuts_the_flat_scores_per_item() -> None:
+    items = [_item("a", 2), _item("b", 0), _item("c", 3)]
+    flat = np.arange(15, dtype=np.float32).reshape(5, 3)
+    blocks = script.split_blocks(flat, items)
+    assert [b.shape for b in blocks] == [(2, 3), (0, 3), (3, 3)]
+    assert np.array_equal(blocks[2], flat[2:5])
+
+
+def test_split_blocks_refuses_a_row_count_that_differs_from_the_passages() -> None:
+    with pytest.raises(ValueError, match="score rows"):
+        script.split_blocks(np.zeros((4, 3), np.float32), [_item("a", 2)])
+
+
+def _item(key: str, n: int) -> Item:
+    passages = tuple(Passage(f"s{i}", "doc", i) for i in range(n))
+    return Item(key, "scifact-dev", key, "claim", "SUPPORTED", passages)
+
+
+class _FakeEmbedder:
+    """Deterministic unit vectors from the words of a text; no model."""
+
+    name = "fake"
+    dim = 16
+
+    def embed(self, texts: Sequence[str]) -> np.ndarray:
+        out = np.zeros((len(texts), self.dim), dtype=np.float32)
+        for row, text in enumerate(texts):
+            for word in text.lower().split():
+                out[row, sum(map(ord, word)) % self.dim] += 1.0
+            out[row, 0] += 0.01
+        return out / np.linalg.norm(out, axis=1, keepdims=True)
+
+
+def test_averitec_items_group_by_claim_skip_blank_sources_and_key_by_position() -> None:
+    snapshot = [
+        SnapshotClaim(
+            7,
+            "the claim",
+            "Refuted",
+            (
+                SnapshotSource("https://a", "ok", "First sentence. Second sentence."),
+                SnapshotSource("https://b", "blocked", ""),
+                SnapshotSource("https://c", "ok", "   "),
+                SnapshotSource("https://d", "ok", "Only one."),
+            ),
+            0,
+        )
+    ]
+    items = script.averitec_items(snapshot, _FakeEmbedder())
+    assert [i.key for i in items] == ["averitec:7:0", "averitec:7:3"]
+    assert {i.group for i in items} == {"averitec:7"}
+    assert {i.dataset for i in items} == {"averitec"}
+    assert items[0].claim == "the claim"
+    assert items[0].gold == "Refuted"
+    assert {p.source_id for p in items[0].passages} == {"https://a"}
+    assert [p.source_id for p in items[1].passages] == ["https://d"]
+    assert len(items[0].passages) <= bakeoff.TOP_N
+
+
+def test_snapshot_returns_130_after_an_interrupt(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    from proofpath import verify
+
+    closed: list[bool] = []
+
+    class _Fetcher:
+        def fetch(self, _url: str) -> object:
+            raise KeyboardInterrupt
+
+    class _Engine:
+        fetcher = _Fetcher()
+
+        def close(self) -> None:
+            closed.append(True)
+
+    claim = averitec.Claim(1, "c", "Refuted", ("https://e.org",))
+    monkeypatch.setattr(script, "snapshot_path", lambda: tmp_path / "snap.json")
+    monkeypatch.setattr(script.averitec, "ensure_downloaded", lambda _d: tmp_path)
+    monkeypatch.setattr(script.averitec, "load", lambda _p, limit=None: [claim])
+    monkeypatch.setattr(verify.Engine, "default", lambda *_a, **_k: _Engine())
+    monkeypatch.setattr("proofpath.config.load_config", lambda *_a, **_k: None)
+    code = script.cmd_snapshot(Namespace(resume=False, limit=0, sleep=0.0))
+    assert code == 130
+    assert closed == [True]

From c5c88f46d776cf01e48429ba9c81ae62673e6d76 Mon Sep 17 00:00:00 2001
From: Yigtwxx <yigiterdogan023@gmail.com>
Date: Thu, 1 Oct 2026 12:34:56 +0300
Subject: [PATCH 10/10] docs(changelog): note the NLI bake-off under Unreleased

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRNnJrkUuFNdBcR5vXRgTp
---
 CHANGELOG.md | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 156e60f..1bef957 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,21 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+### Measured
+- **An NLI bake-off** (`scripts/eval_nli_bakeoff.py`, report in
+  `docs/eval/2026-10-01-nli-bakeoff.md`). It compares three NLI models, three
+  values of k and three aggregation rules, 27 combinations in all. Each combination
+  is calibrated on SciFact train and scored on SciFact dev and a frozen AVeriTeC
+  snapshot. The winner is picked by a rule written down before any number was
+  seen.
+  - **Winner:** `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` with
+    k=2. SciFact dev macro-F1 is 0.697, against 0.580 for today's default and 0.608
+    for the best configuration of the default model.
+  - **AVeriTeC** stays below its majority baseline: 0.475 on claims with a readable
+    source, against 0.708.
+  - The report lists its own limits. The main one is that dev both chose the winner
+    and reports its score.
+
 ## [0.4.9] - 2026-09-30
 
 Wikipedia can be read again, and a verdict now says more about how much it rests