From 41b0fc28d6f51646064448a90d0420213b8a4122 Mon Sep 17 00:00:00 2001 From: Yigtwxx Date: Mon, 5 Oct 2026 13:26:00 +0300 Subject: [PATCH 1/2] feat(eval): measure AVeriTeC with the browser and the judge (20.10) Every published AVeriTeC number so far was measured without the browser step and without the judge, so neither one's effect on the open-web gap is known. The harness now keeps one record per source: the ladder step that read it, the models' verdict, and the judge's opinion when the verdict was in the escalation band. From that record the report adds two numbers beside the product's: the accuracy with the browser's pages dropped, and a hypothetical column in which the judge decides the low band. The product never lets the judge change a verdict (spec 11.1), and the report says so. verify.escalates is public now, so the eval measures the exact band the product sends to the judge instead of a copy of the rule. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K46EEsxMvuAuM8wjvPNWLM --- docs/superpowers/OPEN-ITEMS.md | 24 ++ scripts/eval_averitec.py | 534 +++++++++++++++++++++++++++--- src/proofpath/verify.py | 12 +- tests/test_eval_averitec.py | 570 +++++++++++++++++++++++++++++++++ tests/test_verify.py | 18 +- 5 files changed, 1096 insertions(+), 62 deletions(-) diff --git a/docs/superpowers/OPEN-ITEMS.md b/docs/superpowers/OPEN-ITEMS.md index 4cb3975..64fa906 100644 --- a/docs/superpowers/OPEN-ITEMS.md +++ b/docs/superpowers/OPEN-ITEMS.md @@ -610,3 +610,27 @@ five packages. The user's decisions: | 20.10 | Eval honesty | add a `--judge` flag to `scripts/eval_averitec.py`; do one run with the browser allowed; show the 0.404 search score in the README "Measured" table | | 20.11 | NLI bake-off result | **Winner: `large-fever` (MoritzLaurer DeBERTa-v3-large mnli-fever-anli-ling-wanli) × k=2 × `max`.** SciFact dev macro-F1 0.697, against 0.580 for the `base` × k=1 × `max` baseline and 0.608 for the best `base` row. Accuracy 0.697. AVeriTeC 3-way 0.438, or 0.475 on the readable subset, against a 0.708 majority baseline. Cost: 643 MB and 54 ms/pair, against 244 MB and 19 ms/pair. Full report: `docs/eval/2026-10-01-nli-bakeoff.md`. Next: (a) and (b) are settled by the accurate-NLI design (`docs/superpowers/specs/2026-10-01-accurate-nli-design.md`), which adds the winner as an opt-in "accurate" profile behind a consent download (spec §7.1); (c) stays open. (a) **settled** — rule-decided rows are left out of the tier walk (the numeric layer fired 4–6 times on train with 0 correct and pinned `high` at 1.000000); the accurate cuts are `high` 0.999142. (b) **settled** — the `medium` cut still sits on `decide`, and the report now says so itself (`NO_LOW_TIER`, default runs included). (c) **still open** — on AVeriTeC the winner is below `base` × k=2 (0.475 against 0.537), and `Supported` is still 2/19; the README recommends the profile for scientific sources, not for news | | 20.12 | Accurate-NLI follow-ups | (1) A shared `browser.read_terminal_answer` helper: `model_gate.ask_model_terminal` repeats `browser.ask_terminal`'s prompt loop and imports the private `_CHOICE_ANSWERS` to do it. (2) The `-q` `nli` line carries the fallback reason but not the advice. (3) Ctrl-C and `/stop` do not interrupt a running 643 MB download. (4) An "installed" but corrupt accurate model fails the run instead of falling back. (5) A multilingual NLI model is still deferred (out of scope in the accurate-NLI design); non-English claims still need the judge to translate | + +### 20.10 decisions — 2026-10-05 + +The user picked the recommended option on all four questions. + +- **The judge is scored as a hypothetical column.** The product never lets the judge change + a verdict (spec §11.1), so with gold URLs `--judge` cannot move the product's number. The + report keeps the product's accuracy and adds a second one, labelled as something the + product does not do. In it, a confident model verdict (medium or high tier) still wins. + Otherwise the judge's opinions on the escalated sources decide (`verify.escalates`). + A majority of the asserting votes picks the label: an NEI opinion abstains, and a + tie is NEI. An escalation the judge did not answer votes with the models' label. +- **Three runs and an ablation.** (1) gold URLs, browser, judge. (2) search, browser, + sentence queries. (3) search, browser, judge queries. Every source records the ladder + step that read it. The "without the browser" number comes from the same run, with the + step-3 sources dropped. Wayback was never tried for those pages, so the ablation is a + lower bound on the pages such a run reads. It is not a bound on its accuracy, which can + move either way. Runs fetch live (`--fresh`), because a cache hit has no step. Each + setup gets its own results file and report. +- **Rate limits.** When Groq rate-limits, the judge falls back to the local + `qwen3.5:9b`, as the product does. The report counts opinions per model and splits the + hypothetical accuracy by the model whose vote won. +- **Git.** One feature branch and one PR: the script, its tests, the `docs/eval` + reports and the README "Measured" row. diff --git a/scripts/eval_averitec.py b/scripts/eval_averitec.py index 48f0d5f..a1ecd4e 100644 --- a/scripts/eval_averitec.py +++ b/scripts/eval_averitec.py @@ -1,8 +1,9 @@ """Phase 10 harness: measure proofpath end to end on AVeriTeC dev (spec section 14). Usage: - uv run python scripts/eval_averitec.py [--limit 100] [--no-browser] [--sleep 1.0] - [--resume] [--search] [--out docs/eval/-averitec.md] + uv run python scripts/eval_averitec.py [--limit 100] [--browser | --no-browser] + [--judge] [--fresh] [--sleep 1.0] [--resume] [--search] + [--out docs/eval/-averitec.md] SciFact measures retrieval and entailment over abstracts we are handed. AVeriTeC measures the whole product: a real-world claim, its real source pages, fetched over @@ -17,6 +18,13 @@ answer. Source coverage is reported per state, never collapsed (product rule 6): a run that reached nothing must not read like a run that reached everything. +Every source keeps the ladder step that read it and, under ``--judge``, the judge's +opinion when its verdict was in the escalation band. Two more numbers come from +those rows: the accuracy with the browser's pages dropped (what a run without it +would have read, less whatever Wayback would have rescued), and a hypothetical one +in which the judge's opinions decide the low band. The product never does the +second (spec section 11.1); the report says so. + The live half below ``main`` is run by hand. Importing this module touches neither the network nor a model. """ @@ -28,8 +36,8 @@ import sys import time from collections import Counter -from collections.abc import Sequence -from dataclasses import dataclass, replace +from collections.abc import Callable, Sequence +from dataclasses import asdict, dataclass, replace from datetime import date from pathlib import Path from typing import TYPE_CHECKING @@ -37,16 +45,18 @@ from proofpath import claims as claims_mod from proofpath import ingest, pipeline, retrieval +from proofpath import judge as judge_mod from proofpath.config import NliProfileName, SearchConfig, load_config from proofpath.eval import averitec from proofpath.fetch import Fetched -from proofpath.models import Label, Passage +from proofpath.models import Label, Passage, Verdict from proofpath.paths import cache_dir from proofpath.polite import ProviderError +from proofpath.providers import NO_TEXT from proofpath.report import LANGUAGE_UNSUPPORTED, SEARCH_UNAVAILABLE from proofpath.search import Searcher, SearchKeyError, canonical, search_claim from proofpath.search.queries import plan_queries -from proofpath.verify import NO_TEXT, Engine +from proofpath.verify import Engine, escalates if TYPE_CHECKING: from proofpath.judge import Judge @@ -69,6 +79,36 @@ SENTENCE_PATH = "sentence queries, no judge" RESULTS_NAME = "averitec_results.json" SEARCH_RESULTS_NAME = "averitec_search_results.json" +# The ladder step the ablation drops (``fetch.STEP_NAMES``). +BROWSER_STEP = 3 +# Who decided a claim's hypothetical judged label, when it was not a judge model. +DECIDED_BY_MODELS = "models (medium or high tier)" +NO_OPINION = "models (the judge gave no opinion)" +NOTHING_ESCALATED = "nothing escalated" +# A row from a results file written before sources were kept: only its label is known. +UNRECORDED = "not recorded (row has no per-source record)" + + +@dataclass(frozen=True) +class Source: + """One fetchable source of one claim, as the run read it. + + ``step`` is the ladder step that read the page, or the last one tried when it + failed; ``None`` for a cache hit, which carries no step. ``label`` is the models' + verdict, ``None`` when nothing was read. ``escalated`` says the verdict sat in the + band the product hands the judge (``verify.escalates``); ``judge_label`` and + ``judge_model`` are the judge's opinion of it, when one came back. + """ + + url: str + state: str + step: int | None + label: str | None + score: float = 0.0 + tier: str | None = None + escalated: bool = False + judge_label: str | None = None + judge_model: str | None = None @dataclass(frozen=True) @@ -86,6 +126,78 @@ class Row: predicted: str | None states: tuple[str, ...] # one honesty/fetch state per source value urls: tuple[str, ...] # the fetchable URLs the states came from, in fetch order + # One per fetchable URL. Empty in results files written before 2026-10-05. + sources: tuple[Source, ...] = () + + +def _best(sources: Sequence[Source]) -> Source: + """The strongest source, the first one on a tie, as ``_decide_claim`` always kept.""" + best = sources[0] + for source in sources[1:]: + if source.score > best.score: + best = source + return best + + +def _asserts(label: str | None) -> bool: + return label is not None and label != Label.NEI.value + + +def product_label(sources: Sequence[Source]) -> str | None: + """What the product says of the claim: its strongest assertion over every source, + NEI when something was read and nothing asserted, ``None`` when nothing was read.""" + asserted = [source for source in sources if _asserts(source.label)] + if asserted: + return _best(asserted).label + return Label.NEI.value if any(source.label is not None for source in sources) else None + + +def judged_label(sources: Sequence[Source]) -> str | None: + """The claim's label if the judge's opinions decided the low band. + + The product never does this (spec section 11.1): this is the column that says + whether it should. A confident model verdict (medium or high, not escalated) + still wins, as the escalation rule implies. Otherwise every escalated source + votes with the judge's label, or with the models' label when the judge gave no + opinion. An NEI vote abstains; the majority of the asserting votes wins, and a + tie is NEI. + """ + confident = [s for s in sources if _asserts(s.label) and not s.escalated] + if confident: + return _best(confident).label + votes = [s.judge_label or s.label for s in sources if s.escalated] + tally = Counter(vote for vote in votes if _asserts(vote)).most_common() + if tally: + if len(tally) > 1 and tally[0][1] == tally[1][1]: + return Label.NEI.value + return tally[0][0] + return Label.NEI.value if any(source.label is not None for source in sources) else None + + +def judge_decider(sources: Sequence[Source]) -> str: + """Who decided ``judged_label``: the models, or the judge model(s) whose vote won. + + When the vote ends in NEI (every vote abstained, or a tie) no vote won, and every + escalated source's answerer is named: together they are what left it at NEI.""" + if any(_asserts(s.label) and not s.escalated for s in sources): + return DECIDED_BY_MODELS + escalated = [s for s in sources if s.escalated] + if not escalated: + return NOTHING_ESCALATED + winner = judged_label(sources) + voters = [s for s in escalated if (s.judge_label or s.label) == winner and _asserts(winner)] + return " + ".join(sorted({s.judge_model or NO_OPINION for s in voters or escalated})) + + +def without_browser(sources: Sequence[Source]) -> tuple[Source, ...]: + """The sources as a run without the browser would have them: the pages the browser + read are unread. A cache hit has no step, so it is kept (the report counts those).""" + return tuple( + replace(source, label=None, escalated=False, judge_label=None, judge_model=None) + if source.step == BROWSER_STEP and source.label is not None + else source + for source in sources + ) def _is_archive(url: str) -> bool: @@ -190,6 +302,33 @@ def _site(url: str) -> str: return (urlsplit(url).hostname or "").removeprefix("www.") +@dataclass(frozen=True) +class Ablation: + """The run with the browser's pages dropped. A page the browser read was never + offered to the Wayback step, which a run without the browser tries next, so this + is a lower bound on the pages such a run reads -- not on its accuracy, which can + move either way (a wrong verdict dropped can turn a claim into a right NEI).""" + + read_sources: int + browser_sources: int # of ``read_sources``, read at ``BROWSER_STEP`` + unknown_step: int # of ``read_sources``, cache hits: kept, since no step is known + unrecorded: int # rows with URLs and no per-source record: scored as the product did + accuracy_3way: float + + +@dataclass(frozen=True) +class JudgedScore: + """The hypothetical column: what ``judged_label`` would have scored.""" + + accuracy_3way: float + per_label: dict[str, tuple[int, int]] # gold label -> (n, correct) + escalated: int # sources in the judge's band + unanswered: int # of those, how many got no opinion back + opinions: Counter[str] # judge model -> opinions it gave + deciders: dict[str, tuple[int, int]] # ``judge_decider`` -> (3-way n, correct) + unrecorded: int # rows with URLs and no per-source record: scored as the product did + + @dataclass(frozen=True) class AveritecResult: n: int @@ -203,9 +342,82 @@ class AveritecResult: coverage: Counter[str] # source state -> how many values ended there archive_urls: int # of ``total_urls``, how many were web.archive.org snapshots total_urls: int + ablation: Ablation | None = None # ``None`` when no row carries its sources + judged: JudgedScore | None = None # ``None`` unless the run had a judge -def score(rows: Sequence[Row]) -> AveritecResult: +def _three_way( + rows: Sequence[Row], label_of: Callable[[Row], str | None] +) -> tuple[float, dict[str, tuple[int, int]]]: + """3-way accuracy and per-gold-label (n, correct) for another way of labelling the + same rows, scored by the same rule as ``score``: no label is NEI.""" + per_label: dict[str, tuple[int, int]] = {} + counted = correct = 0 + for row in rows: + gold = averitec.to_label(row.gold) + if gold is None: + continue + predicted = label_of(row) + right = (Label(predicted) if predicted is not None else Label.NEI) is gold + seen, hits = per_label.get(row.gold, (0, 0)) + per_label[row.gold] = (seen + 1, hits + int(right)) + counted += 1 + correct += int(right) + return (correct / counted if counted else 0.0), per_label + + +def _unrecorded(row: Row) -> bool: + """Written before sources were kept (a ``--resume`` of an older file): the row has + URLs but no per-source record, so the product's own label is all that is known.""" + return not row.sources and bool(row.urls) + + +def _ablation(rows: Sequence[Row]) -> Ablation | None: + if not any(row.sources for row in rows): + return None + read = [s for row in rows for s in row.sources if s.label is not None] + accuracy, _ = _three_way( + rows, + lambda row: product_label(without_browser(row.sources)) if row.sources else row.predicted, + ) + return Ablation( + read_sources=len(read), + browser_sources=sum(1 for s in read if s.step == BROWSER_STEP), + unknown_step=sum(1 for s in read if s.step is None), + unrecorded=sum(1 for row in rows if _unrecorded(row)), + accuracy_3way=accuracy, + ) + + +def _judged_label_of(row: Row) -> str | None: + return judged_label(row.sources) if row.sources else row.predicted + + +def _judged(rows: Sequence[Row]) -> JudgedScore: + accuracy, per_label = _three_way(rows, _judged_label_of) + escalated = [s for row in rows for s in row.sources if s.escalated] + deciders: dict[str, tuple[int, int]] = {} + for row in rows: + gold = averitec.to_label(row.gold) + if gold is None: + continue + predicted = _judged_label_of(row) + right = (Label(predicted) if predicted is not None else Label.NEI) is gold + decider = UNRECORDED if _unrecorded(row) else judge_decider(row.sources) + seen, hits = deciders.get(decider, (0, 0)) + deciders[decider] = (seen + 1, hits + int(right)) + return JudgedScore( + accuracy_3way=accuracy, + per_label=per_label, + escalated=len(escalated), + unanswered=sum(1 for s in escalated if s.judge_label is None), + opinions=Counter(s.judge_model for s in escalated if s.judge_model is not None), + deciders=deciders, + unrecorded=sum(1 for row in rows if _unrecorded(row)), + ) + + +def score(rows: Sequence[Row], *, judge: bool = False) -> AveritecResult: """Turn decided rows into the numbers the report prints. A row with no prediction counts as NEI: having fetched nothing, NEI is the only @@ -245,6 +457,8 @@ def score(rows: Sequence[Row]) -> AveritecResult: coverage=coverage, archive_urls=archive_urls, total_urls=total_urls, + ablation=_ablation(rows), + judged=_judged(rows) if judge else None, ) @@ -252,8 +466,21 @@ def _md_row(*cells: object) -> str: return "| " + " | ".join(str(c) for c in cells) + " |" -def render_report(result: AveritecResult, *, date: str, limit: int, path: str = GOLD_PATH) -> str: - """The markdown skeleton. ``## Notes`` is left for the controller to fill in.""" +def render_report( + result: AveritecResult, + *, + date: str, + limit: int, + path: str = GOLD_PATH, + browser: bool | None = None, + fresh: bool = False, + judge: str = "", +) -> str: + """The markdown skeleton. ``## Notes`` is left for the controller to fill in. + + ``browser`` is ``None`` when the run left the browser to the config, so the report + claims nothing about it; ``judge`` is the judge's name, empty when there was none. + """ # ``result.counted`` is the denominator the 3-way accuracy and its baseline are # read over. Printed beside `n`, because "0.80 over 100 claims" would otherwise # claim a coverage the number does not have when some claims were never scorable. @@ -262,6 +489,18 @@ def render_report(result: AveritecResult, *, date: str, limit: int, path: str = "", f"- claims: {result.n} (limit={limit or 'none'})", f"- measured path: {path}", + ] + if browser is not None: + lines.append(f"- browser step: {'allowed' if browser else 'not used'}") + lines.append(f"- fetch: {'live, cache bypassed' if fresh else 'cache allowed'}") + if judge: + role = ( + "its opinions are the hypothetical column below; the verdicts are the models'" + if path == GOLD_PATH + else "writes the search queries; its opinions are the hypothetical column below" + ) + lines.append(f"- judge: {judge} ({role})") + lines += [ f"- dataset: `{averitec.URL}` (sha256 {averitec.SHA256[:12]}…)", "- one run of the whole product: real claims, real source pages, real fetch ladder.", "", @@ -298,22 +537,115 @@ def render_report(result: AveritecResult, *, date: str, limit: int, path: str = # report says how much of its own coverage came from one. lines.append("") if result.total_urls: - share = 100.0 * result.archive_urls / result.total_urls + percent = 100.0 * result.archive_urls / result.total_urls lines.append( f"{result.archive_urls} of {result.total_urls} source URLs are " - f"{ARCHIVE_HOST} snapshots ({share:.1f} %)." + f"{ARCHIVE_HOST} snapshots ({percent:.1f} %)." ) else: lines.append(f"No source URLs were measured, so no {ARCHIVE_HOST} share applies.") + if result.ablation is not None: + lines.extend(_ablation_lines(result.ablation)) + if result.judged is not None: + lines.extend(_judged_lines(result.judged, product=result.accuracy_3way)) lines.extend(["", "## Notes", "", "", ""]) return "\n".join(lines) + "\n" +def _ablation_lines(ablation: Ablation) -> list[str]: + lines = [ + "", + "## Without the browser", + "", + f"{ablation.browser_sources} of {ablation.read_sources} read sources were read by " + f"the browser step. Treating them as unread gives a 3-way accuracy of " + f"**{ablation.accuracy_3way:.3f}**.", + "", + "Those pages never reached the Wayback step, which a run without the browser " + "tries next, so this is a lower bound on the pages such a run reads. It is not a " + "bound on its accuracy: dropping a page can lose a right verdict or a wrong one.", + ] + if ablation.unknown_step: + lines.extend( + [ + "", + f"{ablation.unknown_step} read sources came from the cache, so the step " + "that read them is unknown; they are kept as read.", + ] + ) + if ablation.unrecorded: + lines.extend(["", _unrecorded_line(ablation.unrecorded)]) + return lines + + +def _unrecorded_line(count: int) -> str: + return ( + f"{count} rows have no per-source record (written before sources were kept); " + "they are scored with the product's own label." + ) + + +def _judged_lines(judged: JudgedScore, *, product: float) -> list[str]: + lines = [ + "", + "## Judge (hypothetical)", + "", + "The product never lets the judge change a verdict (spec section 11.1). This " + "column is what the accuracy would be if it did: a medium or high model verdict " + "still wins; otherwise the judge's opinions on the escalated sources decide, by " + "majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An " + "escalation the judge did not answer votes with the models' label.", + "", + f"3-way accuracy **{judged.accuracy_3way:.3f}**, against {product:.3f} for the " + f"product. {judged.escalated} sources were escalated; {judged.unanswered} got " + "no opinion back." + + (f" {_unrecorded_line(judged.unrecorded)}" if judged.unrecorded else ""), + "", + "| label | n | correct | accuracy |", + "|---|---|---|---|", + ] + for label in [label for label in averitec.LABELS if label in judged.per_label]: + seen, right = judged.per_label[label] + lines.append(_md_row(label, seen, right, f"{right / seen:.3f}" if seen else "—")) + lines.extend(["", "| judge model | opinions |", "|---|---|"]) + for model, count in judged.opinions.most_common(): + lines.append(_md_row(model, count)) + if not judged.opinions: + lines.append(_md_row("none", 0)) + lines.extend(["", "| decided by | n | correct | accuracy |", "|---|---|---|---|"]) + for decider, (seen, right) in sorted(judged.deciders.items()): + lines.append(_md_row(decider, seen, right, f"{right / seen:.3f}" if seen else "—")) + return lines + + # --- live half: run by hand, not covered by tests ------------------------ -def _results_path(*, search: bool) -> Path: - return cache_dir() / "datasets" / (SEARCH_RESULTS_NAME if search else RESULTS_NAME) +def _setup_suffix(*, browser: bool, judge: bool, fresh: bool) -> str: + """What sets this run apart from the legacy setup, for its file names.""" + return ( + ("-browser" if browser else "") + ("-judge" if judge else "") + ("-fresh" if fresh else "") + ) + + +def _results_path( + *, search: bool, browser: bool = False, judge: bool = False, fresh: bool = False +) -> Path: + """One file per run setup, so a resume does not mix rows from two setups. The + legacy names stay the runs made without ``--browser``, ``--judge`` or ``--fresh``; + they are shared by ``--no-browser`` and by a run that leaves the browser to the + config, as they always were, so that older files can still be resumed.""" + name = Path(SEARCH_RESULTS_NAME if search else RESULTS_NAME) + suffix = _setup_suffix(browser=browser, judge=judge, fresh=fresh) + return cache_dir() / "datasets" / f"{name.stem}{suffix}{name.suffix}" + + +def _report_path(*, day: str, search: bool, browser: bool, judge: bool, fresh: bool) -> Path: + """The default ``--out``: one report per run setup, so two runs on one day keep both.""" + suffix = ("-search" if search else "") + _setup_suffix( + browser=browser, judge=judge, fresh=fresh + ) + return Path("docs/eval") / f"{day}-averitec{suffix}.md" def _load_rows(path: Path) -> list[Row]: @@ -327,6 +659,7 @@ def _load_rows(path: Path) -> list[Row]: predicted=None if item["predicted"] is None else str(item["predicted"]), states=tuple(str(state) for state in item["states"]), urls=tuple(str(url) for url in item["urls"]), + sources=tuple(Source(**source) for source in item.get("sources", ())), ) for item in raw ] @@ -343,6 +676,7 @@ def _save_rows(path: Path, rows: Sequence[Row]) -> None: "predicted": row.predicted, "states": list(row.states), "urls": list(row.urls), + "sources": [asdict(source) for source in row.sources], } for row in rows ], @@ -354,26 +688,38 @@ def _save_rows(path: Path, rows: Sequence[Row]) -> None: def _decide_claim( - engine: Engine, claim: averitec.Claim, *, sleep: float, anonymous: bool = False + engine: Engine, + claim: averitec.Claim, + *, + sleep: float, + anonymous: bool = False, + fresh: bool = False, + judge: Judge | None = None, ) -> Row: """Fetch every source of one claim and keep the strongest non-NEI verdict. ``anonymous`` is for pages the search found: fetched without the contact address, - as the product fetches them (round 2).""" + as the product fetches them (round 2). ``fresh`` climbs past the cache, so every + source keeps the step that read it. ``judge`` is asked about the sources whose + verdict sits in the product's escalation band; its opinions are recorded beside + the verdicts and never change ``predicted`` (spec section 11.1).""" states: list[str] = [] - best: pipeline.Verdict | None = None - fetched_any = False + sources: list[Source] = [] + # What the judge is shown for each escalated source, by its index in ``sources``. + escalated: dict[int, Verdict] = {} for url in claim.source_urls: # Before every fetch, not between them: the ladder has no crawl delay of its # own, so this is the whole of the run's politeness and it has to hold across # claims too, not just within one. if sleep: time.sleep(sleep) - fetched = engine.fetcher.fetch(url, anonymous=anonymous) - states.append(state_for(fetched)) + fetched = engine.fetcher.fetch(url, anonymous=anonymous, use_cache=not fresh) + state = state_for(fetched) + states.append(state) + step = None if fetched.from_cache else fetched.step if not fetched.ok or not fetched.text.strip(): + sources.append(Source(url=url, state=state, step=step, label=None)) continue - fetched_any = True passages = [ Passage(sentence, url, ordinal) for ordinal, sentence in enumerate(retrieval.split_sentences(fetched.text)) @@ -386,41 +732,91 @@ def _decide_claim( k=engine.k, thresholds=engine.thresholds, ) - # No passage, no verdict (product rule 1): an assertion without the sentence - # it rests on is dropped rather than reported. - if verdict.label is Label.NEI or verdict.passage is None: - continue - if best is None or verdict.score > best.score: - best = verdict + if escalates(verdict): + escalated[len(sources)] = verdict + # A source whose verdict is NEI was still read: it stays in the row, and + # ``product_label`` keeps only assertions, each with the passage it rests on + # (product rule 1, enforced by ``Verdict`` itself). + sources.append( + Source( + url=url, + state=state, + step=step, + label=verdict.label.value, + score=verdict.score, + tier=verdict.tier, + escalated=escalates(verdict), + ) + ) + if judge is not None and escalated: + sources = _ask_judge(judge, claim.text, sources, escalated) # One state per value the dataset gave, fetchable or not, so the coverage table # accounts for every annotation rather than only the ones we could try. states.extend([NOT_A_URL] * claim.non_urls) if not claim.source_urls and not claim.non_urls: states.append(NO_SOURCE) - if best is not None: - predicted: str | None = best.label.value - elif fetched_any: - predicted = Label.NEI.value - else: - predicted = None return Row( claim_id=claim.id, gold=claim.label, - predicted=predicted, + predicted=product_label(sources), states=tuple(states), urls=claim.source_urls, + sources=tuple(sources), ) +def _ask_judge( + judge: Judge, claim: str, sources: list[Source], escalated: dict[int, Verdict] +) -> list[Source]: + """The judge's opinion on each escalated source, shown exactly what ``verify`` + shows it: the claim, the passage, the models' verdict and its tier.""" + items = [ + judge_mod.JudgeItem( + id=f"s{index}", + claim=claim, + passage=verdict.passage.text, + verdict=verdict.label, + tier=verdict.tier, + ) + for index, verdict in escalated.items() + if verdict.passage is not None # ``escalates`` already refused the others + ] + opinions = judge.review(items) + answered = list(sources) + for index in escalated: + opinion = opinions.get(f"s{index}") + if opinion is not None: + answered[index] = replace( + sources[index], judge_label=opinion.label.value, judge_model=opinion.model + ) + return answered + + NLI_PROFILE: NliProfileName = "default" def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--limit", type=int, default=100, help="first N dev claims") - parser.add_argument( + browser_flags = parser.add_mutually_exclusive_group() + browser_flags.add_argument( "--no-browser", action="store_true", help="never use the browser step of the ladder" ) + browser_flags.add_argument( + "--browser", + action="store_true", + help="allow the browser step whatever the config says (installs it if missing)", + ) + parser.add_argument( + "--judge", + action="store_true", + help="ask the configured judge about the escalated verdicts (a hypothetical column)", + ) + parser.add_argument( + "--fresh", + action="store_true", + help="fetch every page live, past the cache, so each source keeps its ladder step", + ) parser.add_argument("--sleep", type=float, default=1.0, help="seconds between fetches") parser.add_argument("--resume", action="store_true", help="skip claim ids already scored") parser.add_argument( @@ -431,7 +827,9 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument("--out", default="") args = parser.parse_args(argv) - results = _results_path(search=args.search) + results = _results_path( + search=args.search, browser=args.browser, judge=args.judge, fresh=args.fresh + ) if results.exists() and not args.resume: # The file is the only record of a run that costs hours of network; a fresh # run must not quietly replace it. Checked before anything is downloaded, so @@ -443,6 +841,21 @@ def main(argv: list[str] | None = None) -> int: ) return 2 + config = load_config() + key: judge_mod.ApiKey | None = None + if args.judge: + # Checked before anything is downloaded, as the CLI's ``--judge`` checks it: a + # run told to measure the judge must not quietly measure the models alone. + key = judge_mod.resolve_api_key(config.judge.api_key_env) + if config.judge.api_key_env and key is None: + print( + f"error {config.judge.api_key_env} is not set (put it in .env)", + file=sys.stderr, + ) + return 2 + if config.judge.provider == "gemini": + print(f"note {judge_mod.GEMINI_DATA_USE}", file=sys.stderr) + dataset = averitec.ensure_downloaded(cache_dir()) claims = averitec.load(dataset, limit=args.limit or None) print(f"dataset averitec/dev {len(claims)} claims (sha256 {averitec.SHA256[:12]}…)") @@ -455,15 +868,22 @@ def main(argv: list[str] | None = None) -> int: if done: print(f"resume {len(done)} claims already scored in {results}") - config = load_config() # ``interactive=False`` so an `ask` permission is denied and reported rather than # prompted for (product rule 4); ``--no-browser`` forces the browser step off - # outright, otherwise the config's own permission decides. + # outright and ``--browser`` on (asking for it is the consent, spec section 7.1); + # otherwise the config's own permission decides. # ``nli="default"`` is pinned: the published numbers describe the default install, # so the user's ``models.nli`` must not change what this script measures. + browser = True if args.browser else False if args.no_browser else None + # Built here, not where the key is checked: ``Engine.default`` owns it from this + # call on and closes it even when it fails, so nothing in between can leak it. + judge = judge_mod.Judge(judge_mod.build_client(config.judge, key)) if args.judge else None engine = Engine.default( - config, interactive=False, browser=False if args.no_browser else None, nli=NLI_PROFILE + config, interactive=False, browser=browser, nli=NLI_PROFILE, judge=judge ) + # The configured judge's name, taken before it can switch to its local fallback: + # the report names the judge the run was told to use, the table who answered. + judge_name = "" if judge is None else judge.name print(f"nli {NLI_PROFILE} profile (pinned; ignores models.nli)") searcher: Searcher | None = None if args.search: @@ -481,6 +901,7 @@ def main(argv: list[str] | None = None) -> int: searcher = engine.searcher # The judge the product would write queries with: the same rule ``verify`` uses. + # Under ``--judge`` it also gives its opinion of the escalated verdicts. judge = engine.judge if engine.escalate else None path = measured_path(judge) if searcher is not None else GOLD_PATH scored = 0 @@ -489,7 +910,9 @@ def main(argv: list[str] | None = None) -> int: if claim.id in done: continue if searcher is None: - stored.append(_decide_claim(engine, claim, sleep=args.sleep)) + stored.append( + _decide_claim(engine, claim, sleep=args.sleep, fresh=args.fresh, judge=judge) + ) else: # The gold URLs are the answer key; search mode reads what evidence # search would actually find, which is the §17.1 gate this run exists @@ -507,7 +930,16 @@ def main(argv: list[str] | None = None) -> int: found = replace( claim, text=searched.text, source_urls=searched.urls, non_urls=0 ) - stored.append(_decide_claim(engine, found, sleep=args.sleep, anonymous=True)) + stored.append( + _decide_claim( + engine, + found, + sleep=args.sleep, + anonymous=True, + fresh=args.fresh, + judge=judge, + ) + ) scored += 1 # After every claim, not at the end: a run this long is interrupted more # often than it finishes, and the file has to survive that. @@ -529,17 +961,33 @@ def main(argv: list[str] | None = None) -> int: return 2 finally: engine.close() + if judge is not None and judge.switched: + # The run's one switch notice, as the product prints it (spec section 11). + print(f"note {judge.switched}", file=sys.stderr) wanted = {claim.id for claim in claims} rows = [row for row in stored if row.claim_id in wanted] - report = render_report(score(rows), date=date.today().isoformat(), limit=args.limit, path=path) + report = render_report( + score(rows, judge=args.judge), + date=date.today().isoformat(), + limit=args.limit, + path=path, + browser=browser, + fresh=args.fresh, + judge=judge_name, + ) print() print(report) - suffix = "-search" if args.search else "" out = ( Path(args.out) if args.out - else Path("docs/eval") / f"{date.today().isoformat()}-averitec{suffix}.md" + else _report_path( + day=date.today().isoformat(), + search=args.search, + browser=args.browser, + judge=args.judge, + fresh=args.fresh, + ) ) out.parent.mkdir(parents=True, exist_ok=True) out.write_text(report, encoding="utf-8") diff --git a/src/proofpath/verify.py b/src/proofpath/verify.py index 4e49963..74558b6 100644 --- a/src/proofpath/verify.py +++ b/src/proofpath/verify.py @@ -1944,7 +1944,7 @@ class _Judged: cancelled: bool -def _escalates(result: ClaimResult) -> bool: +def escalates(verdict: Verdict) -> bool: """Whether this verdict is the judge's business (spec section 9 step 8). Three exclusions, each with a reason the product cannot do without: @@ -1957,8 +1957,10 @@ def _escalates(result: ClaimResult) -> bool: * Anything the models were confident about. The judge is an escape hatch for the low band, not a second pass over the whole document (spec section 11: 2-4 calls per paper, not one per claim). + + Public because ``scripts/eval_averitec.py`` measures the same band: an eval + that drew its own line would measure a judge the product never runs. """ - verdict = result.verdict if verdict.passage is None or verdict.reason.startswith(NUMERIC_REASON): return False return verdict.tier == "low" or verdict.label is Label.NEI @@ -1985,7 +1987,9 @@ def _judging( began = time.monotonic() emit(StageStart(name=JUDGING, by=judge.name)) - escalated = [(index, result) for index, result in enumerate(results) if _escalates(result)] + escalated = [ + (index, result) for index, result in enumerate(results) if escalates(result.verdict) + ] opinions: dict[str, JudgeOpinion] = {} fresh: dict[str, JudgeOpinion] = {} keys: dict[str, tuple[str, str]] = {} @@ -2006,7 +2010,7 @@ def _judging( opinions[ident] = stored continue passage = result.verdict.passage - if passage is None: # pragma: no cover - _escalates already refused these + if passage is None: # pragma: no cover - escalates already refused these continue ask.append( JudgeItem( diff --git a/tests/test_eval_averitec.py b/tests/test_eval_averitec.py index 2239931..f041e46 100644 --- a/tests/test_eval_averitec.py +++ b/tests/test_eval_averitec.py @@ -12,6 +12,7 @@ import sys from pathlib import Path from types import ModuleType, SimpleNamespace +from typing import Any import pytest @@ -411,3 +412,572 @@ def test_a_rejected_search_key_stops_the_run_at_the_first_claim( assert not out.exists() message = capsys.readouterr().err assert "TAVILY_API_KEY" in message and "rejected" in message + + +# --- per-source rows: the browser ablation and the judge's hypothetical column ------ + +Source = eval_averitec.Source +product_label = eval_averitec.product_label +judged_label = eval_averitec.judged_label +judge_decider = eval_averitec.judge_decider +without_browser = eval_averitec.without_browser + +GROQ = "groq openai/gpt-oss-120b" +LOCAL = "ollama qwen3.5:9b" + + +def _read( + label: str, + score: float = 0.5, + *, + step: int | None = 1, + tier: str = "medium", + escalated: bool = False, + judge_label: str | None = None, + judge_model: str | None = None, + url: str = LIVE, +) -> Any: + return Source( + url=url, + state=OK, + step=step, + label=label, + score=score, + tier=tier, + escalated=escalated, + judge_label=judge_label, + judge_model=judge_model, + ) + + +def _unread(state: str = UNREACHABLE, *, step: int = 1) -> Any: + return Source(url=LIVE, state=state, step=step, label=None) + + +def test_the_product_label_is_the_strongest_assertion_among_the_sources() -> None: + sources = (_read("SUPPORTED", 0.6), _read("REFUTED", 0.9), _read("NEI", 0.99)) + assert product_label(sources) == "REFUTED" + + +def test_on_a_tied_score_the_first_source_wins_as_it_always_did() -> None: + first = _read("SUPPORTED", 0.7, url="https://a.example/1") + second = _read("REFUTED", 0.7, url="https://b.example/2") + assert product_label((first, second)) == "SUPPORTED" + assert product_label((second, first)) == "REFUTED" + + +def test_the_product_label_is_nei_when_something_was_read_but_nothing_asserted() -> None: + assert product_label((_read("NEI"), _unread())) == "NEI" + + +def test_the_product_label_is_none_when_nothing_was_read() -> None: + assert product_label((_unread(), _unread(BLOCKED))) is None + assert product_label(()) is None + + +def test_a_confident_model_verdict_wins_over_the_judge() -> None: + sources = ( + _read("SUPPORTED", 0.8, tier="high"), + _read("NEI", tier="low", escalated=True, judge_label="REFUTED", judge_model=GROQ), + ) + assert judged_label(sources) == "SUPPORTED" + assert judge_decider(sources) == eval_averitec.DECIDED_BY_MODELS + + +def test_without_a_confident_verdict_the_judge_opinions_decide() -> None: + sources = ( + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read( + "REFUTED", 0.3, tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ + ), + ) + assert product_label(sources) == "REFUTED" # the product keeps the model's verdict + assert judged_label(sources) == "SUPPORTED" + assert judge_decider(sources) == GROQ + + +def test_a_tied_judge_vote_is_nei() -> None: + sources = ( + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read("NEI", tier="low", escalated=True, judge_label="REFUTED", judge_model=LOCAL), + ) + assert judged_label(sources) == "NEI" + # No vote won, so every answerer is named: together they left it at NEI. + assert judge_decider(sources) == f"{GROQ} + {LOCAL}" + + +def test_the_decider_is_the_model_whose_vote_won_not_every_model_that_answered() -> None: + sources = ( + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read("NEI", tier="low", escalated=True, judge_label="REFUTED", judge_model=LOCAL), + ) + assert judged_label(sources) == "SUPPORTED" + assert judge_decider(sources) == GROQ + + +def test_an_escalation_the_judge_never_answered_keeps_the_model_label_as_its_vote() -> None: + sources = (_read("REFUTED", 0.3, tier="low", escalated=True),) + assert judged_label(sources) == "REFUTED" + assert judge_decider(sources) == eval_averitec.NO_OPINION + + +def test_a_judge_that_says_nei_everywhere_leaves_a_read_claim_at_nei() -> None: + sources = ( + _read("REFUTED", 0.3, tier="low", escalated=True, judge_label="NEI", judge_model=GROQ), + ) + assert judged_label(sources) == "NEI" + + +def test_a_claim_with_nothing_escalated_or_read_has_no_judged_label() -> None: + assert judged_label((_unread(),)) is None + assert judge_decider((_unread(),)) == eval_averitec.NOTHING_ESCALATED + + +def test_without_browser_drops_only_what_the_browser_read() -> None: + browser = _read("SUPPORTED", 0.9, step=3) + cached = _read("REFUTED", 0.4, step=None) + plain = _read("NEI", step=2) + dropped = without_browser((browser, cached, plain)) + assert dropped[0].label is None + assert dropped[1:] == (cached, plain) + assert product_label(dropped) == "REFUTED" + + +# --- results files: one per run setup, the legacy name untouched --------------------- + + +@pytest.mark.parametrize( + ("search", "browser", "judge", "fresh", "name"), + [ + (False, False, False, False, "averitec_results.json"), + (True, False, False, False, "averitec_search_results.json"), + (False, True, True, True, "averitec_results-browser-judge-fresh.json"), + (True, True, False, True, "averitec_search_results-browser-fresh.json"), + (True, True, True, True, "averitec_search_results-browser-judge-fresh.json"), + (False, False, False, True, "averitec_results-fresh.json"), + ], +) +def test_every_run_setup_gets_its_own_results_file( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + search: bool, + browser: bool, + judge: bool, + fresh: bool, + name: str, +) -> None: + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path) + path = eval_averitec._results_path(search=search, browser=browser, judge=judge, fresh=fresh) + assert path == tmp_path / "datasets" / name + + +def test_two_search_setups_on_one_day_write_two_reports() -> None: + day = "2026-10-05" + sentence = eval_averitec._report_path( + day=day, search=True, browser=True, judge=False, fresh=True + ) + judged = eval_averitec._report_path(day=day, search=True, browser=True, judge=True, fresh=True) + assert sentence == Path("docs/eval/2026-10-05-averitec-search-browser-fresh.md") + assert judged == Path("docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md") + legacy = eval_averitec._report_path( + day=day, search=False, browser=False, judge=False, fresh=False + ) + assert legacy == Path("docs/eval/2026-10-05-averitec.md") + + +def test_rows_round_trip_with_their_sources(tmp_path: Path) -> None: + row = Row( + claim_id=7, + gold=SUPPORTED, + predicted="REFUTED", + states=(OK, UNREACHABLE), + urls=(LIVE, ARCHIVED), + sources=( + _read( + "REFUTED", + 0.3, + step=3, + tier="low", + escalated=True, + judge_label="SUPPORTED", + judge_model=GROQ, + ), + _unread(), + ), + ) + path = tmp_path / "results.json" + eval_averitec._save_rows(path, [row]) + assert eval_averitec._load_rows(path) == [row] + + +def test_a_legacy_results_file_without_sources_still_loads(tmp_path: Path) -> None: + path = tmp_path / "results.json" + path.write_text( + '[{"claim_id": 1, "gold": "Refuted", "predicted": null, "states": ["ok"], "urls": []}]', + encoding="utf-8", + ) + (row,) = eval_averitec._load_rows(path) + assert row.sources == () + + +# --- score and report: the ablation and the hypothetical judge ---------------------- + +JUDGED_ROWS = [ + # The browser read the only assertion; the judge agrees with the model. + Row( + claim_id=0, + gold=SUPPORTED, + predicted="SUPPORTED", + states=(OK,), + urls=(LIVE,), + sources=(_read("SUPPORTED", 0.9, step=3, tier="high"),), + ), + # The model is unsure and wrong; Groq's opinion would have been right. + Row( + claim_id=1, + gold=SUPPORTED, + predicted="REFUTED", + states=(OK,), + urls=(LIVE,), + sources=( + _read( + "REFUTED", + 0.3, + tier="low", + escalated=True, + judge_label="SUPPORTED", + judge_model=GROQ, + ), + ), + ), + # Read from the cache: the step is unknown, and the local judge got it wrong. + Row( + claim_id=2, + gold=REFUTED, + predicted="NEI", + states=(OK,), + urls=(LIVE,), + sources=( + _read( + "NEI", + step=None, + tier="low", + escalated=True, + judge_label="SUPPORTED", + judge_model=LOCAL, + ), + ), + ), + # Nothing read; nothing escalated. + Row( + claim_id=3, + gold=NEI, + predicted=None, + states=(UNREACHABLE,), + urls=(LIVE,), + sources=(_unread(),), + ), +] + + +def test_score_leaves_the_product_accuracy_alone() -> None: + result = score(JUDGED_ROWS, judge=True) + assert result.accuracy_3way == pytest.approx(2 / 4) + + +def test_score_drops_browser_reads_for_the_ablation() -> None: + ablation = score(JUDGED_ROWS).ablation + assert ablation is not None + assert ablation.read_sources == 3 + assert ablation.browser_sources == 1 + assert ablation.unknown_step == 1 + # Claim 0 loses its only source and falls to "nothing read" (NEI): 1 of 4 right. + assert ablation.accuracy_3way == pytest.approx(1 / 4) + + +def test_rows_without_sources_have_no_ablation_and_no_judged_column() -> None: + result = score(ROWS) + assert result.ablation is None + assert result.judged is None + + +def test_score_reports_the_hypothetical_judge_accuracy_and_who_decided() -> None: + judged = score(JUDGED_ROWS, judge=True).judged + assert judged is not None + # Claims 0 (models) and 1 (Groq) right, 2 (local) wrong, 3 NEI right. + assert judged.accuracy_3way == pytest.approx(3 / 4) + assert judged.escalated == 2 + assert judged.unanswered == 0 + assert judged.opinions == {GROQ: 1, LOCAL: 1} + assert judged.deciders[eval_averitec.DECIDED_BY_MODELS] == (1, 1) + assert judged.deciders[GROQ] == (1, 1) + assert judged.deciders[LOCAL] == (1, 0) + + +def test_a_judge_run_with_nothing_escalated_still_has_a_judged_column() -> None: + judged = score(ROWS[:1], judge=True).judged + assert judged is not None + assert judged.escalated == 0 + + +def test_the_report_states_the_run_setup() -> None: + text = render_report( + score(JUDGED_ROWS, judge=True), + date="2026-10-05", + limit=4, + browser=True, + fresh=True, + judge=GROQ, + ) + assert "browser step: allowed" in text + assert "fetch: live, cache bypassed" in text + assert f"judge: {GROQ}" in text + + +def test_the_ablation_bounds_the_pages_read_and_not_the_accuracy() -> None: + text = render_report(score(JUDGED_ROWS), date="2026-10-05", limit=4, browser=True) + assert "## Without the browser" in text + assert "lower bound on the pages such a run reads" in text + assert "not a bound on its accuracy" in text + assert "1 of 3 read sources" in text + assert "step that read them is unknown" in text # a cache read is said, not hidden + + +def test_the_judge_column_says_the_product_does_not_do_this() -> None: + text = render_report(score(JUDGED_ROWS, judge=True), date="2026-10-05", limit=4, judge=GROQ) + assert "## Judge (hypothetical)" in text + assert "never lets the judge change a verdict" in text + assert "**0.750**" in text + assert _md_line(GROQ, 1, 1, "1.000") in text # per deciding model + assert _md_line(LOCAL, 1, 0, "0.000") in text + + +def test_a_report_without_sources_or_judge_has_neither_section() -> None: + text = render_report(score(ROWS), date="2026-10-05", limit=6) + assert "## Without the browser" not in text + assert "## Judge (hypothetical)" not in text + + +def _md_line(*cells: object) -> str: + return "| " + " | ".join(str(c) for c in cells) + " |" + + +# --- _decide_claim: live fetch, step kept, the escalated verdicts asked -------------- + + +def _fetched( + url: str, *, step: int, text: str = "Some text. More text.", outcome: str = "ok" +) -> object: + from proofpath.fetch import Fetched, Outcome + + return Fetched( + url=url, + final_url=url, + step=step, + outcome=Outcome(outcome), + status=200, + content_type="text/html", + kind="html", + body=b"", + text=text, + notes=[], + from_cache=step == 0, + ) + + +class _Fetcher: + def __init__(self, pages: dict[str, object]) -> None: + self.pages = pages + self.calls: list[tuple[str, bool, bool]] = [] + + def fetch(self, url: str, *, anonymous: bool = False, use_cache: bool = True) -> object: + self.calls.append((url, anonymous, use_cache)) + return self.pages[url] + + +class _Judge: + def __init__(self, answers: dict[str, str], model: str = GROQ) -> None: + self.answers = answers # passage text -> label value + self.model = model + self.asked: list[object] = [] + + def review(self, items: list[object], **_: object) -> dict[str, object]: + from proofpath.judge import JudgeOpinion + from proofpath.models import Label + + self.asked.extend(items) + return { + item.id: JudgeOpinion(Label(self.answers[item.passage]), "why", self.model) # type: ignore[attr-defined] + for item in items + if item.passage in self.answers # type: ignore[attr-defined] + } + + +def _engine(fetcher: _Fetcher) -> object: + return SimpleNamespace( + fetcher=fetcher, + get_embedder=lambda: None, + get_scorer=lambda: None, + k=1, + thresholds=None, + ) + + +def _verdicts(monkeypatch: pytest.MonkeyPatch, by_url: dict[str, object]) -> None: + def decide(_claim: str, passages: list[object], *_a: object, **_k: object) -> object: + return by_url[passages[0].source_id] # type: ignore[attr-defined] + + monkeypatch.setattr(eval_averitec.pipeline, "decide", decide) + + +def _verdict(label: str, score: float, tier: str, url: str, reason: str = "") -> object: + from proofpath.models import Label, Passage, Verdict + + return Verdict(Label(label), score, tier, Passage(f"passage of {url}", url, 0), reason) # type: ignore[arg-type] + + +def test_a_fresh_run_climbs_past_the_cache_and_keeps_the_step( + monkeypatch: pytest.MonkeyPatch, +) -> None: + one, two = "https://a.example/1", "https://b.example/2" + fetcher = _Fetcher( + { + one: _fetched(one, step=3), + two: _fetched(two, step=1, outcome="UNVERIFIED (unreachable)", text=""), + } + ) + _verdicts(monkeypatch, {one: _verdict("SUPPORTED", 0.9, "high", one)}) + claim = averitec.Claim(id=1, text="A claim.", label="Supported", source_urls=(one, two)) + + row = eval_averitec._decide_claim(_engine(fetcher), claim, sleep=0, fresh=True) + + assert [call[2] for call in fetcher.calls] == [False, False] + assert row.predicted == "SUPPORTED" + assert [(s.step, s.label) for s in row.sources] == [(3, "SUPPORTED"), (1, None)] + + +def test_only_the_low_band_is_sent_to_the_judge(monkeypatch: pytest.MonkeyPatch) -> None: + low, high, numeric = "https://a.example/low", "https://b.example/high", "https://c.example/num" + fetcher = _Fetcher({url: _fetched(url, step=1) for url in (low, high, numeric)}) + _verdicts( + monkeypatch, + { + low: _verdict("REFUTED", 0.3, "low", low), + high: _verdict("SUPPORTED", 0.9, "high", high), + numeric: _verdict("REFUTED", 1.0, "low", numeric, reason="numeric mismatch: 3 vs 4"), + }, + ) + judge = _Judge({f"passage of {low}": "SUPPORTED"}) + claim = averitec.Claim( + id=2, text="A claim.", label="Supported", source_urls=(low, high, numeric) + ) + + row = eval_averitec._decide_claim(_engine(fetcher), claim, sleep=0, judge=judge) # type: ignore[arg-type] + + assert [item.passage for item in judge.asked] == [f"passage of {low}"] # type: ignore[attr-defined] + by_url = {s.url: s for s in row.sources} + assert by_url[low].escalated and by_url[low].judge_label == "SUPPORTED" + assert by_url[low].judge_model == GROQ + assert not by_url[high].escalated and not by_url[numeric].escalated + # The product's rule is unchanged: the strongest assertion, here the numeric 1.0. + assert row.predicted == "REFUTED" + assert row.predicted == product_label(row.sources) + + +def test_without_a_judge_nothing_is_asked_and_escalation_is_still_recorded( + monkeypatch: pytest.MonkeyPatch, +) -> None: + low = "https://a.example/low" + fetcher = _Fetcher({low: _fetched(low, step=1)}) + _verdicts(monkeypatch, {low: _verdict("REFUTED", 0.3, "low", low)}) + claim = averitec.Claim(id=3, text="A claim.", label="Refuted", source_urls=(low,)) + + row = eval_averitec._decide_claim(_engine(fetcher), claim, sleep=0) + + assert fetcher.calls[0][2] is True # not fresh: the cache may answer + (source,) = row.sources + assert source.escalated and source.judge_label is None + + +# --- main: the new flags --------------------------------------------------------------- + + +def test_browser_and_no_browser_cannot_both_be_asked_for( + capsys: pytest.CaptureFixture[str], +) -> None: + with pytest.raises(SystemExit) as exc: + eval_averitec.main(["--browser", "--no-browser"]) + assert exc.value.code == 2 + + +def test_a_judge_run_without_its_key_stops_before_the_dataset( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + from proofpath.config import Config + + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path) + monkeypatch.setattr(eval_averitec, "load_config", Config) + monkeypatch.setattr(eval_averitec.judge_mod, "resolve_api_key", lambda _name: None) + monkeypatch.setattr( + eval_averitec.averitec, + "ensure_downloaded", + lambda _cache: pytest.fail("the dataset was fetched before the key was checked"), + ) + + assert eval_averitec.main(["--judge"]) == 2 + assert "GROQ_API_KEY" in capsys.readouterr().err + + +# --- review round 1: a resume over an older file, the judge's role in search mode ----- + +# A row from a results file written before sources were kept, right as the product +# scored it, beside one new row whose only source the browser read. +MIXED_ROWS = [ + Row(claim_id=0, gold=SUPPORTED, predicted="SUPPORTED", states=(OK,), urls=(LIVE,)), + Row( + claim_id=1, + gold=REFUTED, + predicted="REFUTED", + states=(OK,), + urls=(LIVE,), + sources=(_read("REFUTED", 0.9, step=1, tier="high"),), + ), +] + + +def test_an_older_row_keeps_the_products_label_in_the_ablation() -> None: + ablation = score(MIXED_ROWS).ablation + assert ablation is not None + assert ablation.browser_sources == 0 + assert ablation.unrecorded == 1 + # No browser page was dropped, so nothing may move: 2 of 2, as the product scored. + assert ablation.accuracy_3way == pytest.approx(1.0) + + +def test_an_older_row_keeps_the_products_label_in_the_judged_column() -> None: + judged = score(MIXED_ROWS, judge=True).judged + assert judged is not None + assert judged.accuracy_3way == pytest.approx(1.0) + assert judged.unrecorded == 1 + assert judged.deciders[eval_averitec.UNRECORDED] == (1, 1) + + +def test_the_report_counts_the_rows_with_no_per_source_record() -> None: + text = render_report(score(MIXED_ROWS, judge=True), date="2026-10-05", limit=2, judge=GROQ) + assert "1 rows have no per-source record" in text + + +def test_in_search_mode_the_report_says_the_judge_wrote_the_queries() -> None: + result = score(JUDGED_ROWS, judge=True) + gold = render_report(result, date="2026-10-05", limit=4, judge=GROQ) + searched = render_report( + result, + date="2026-10-05", + limit=4, + judge=GROQ, + path=eval_averitec.measured_path(SimpleNamespace(name=GROQ)), + ) + assert "the verdicts are the models'" in gold + assert "writes the search queries" in searched + assert "writes the search queries" not in gold diff --git a/tests/test_verify.py b/tests/test_verify.py index e7b83b8..dcda3e7 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -27,7 +27,7 @@ from proofpath.browser import Answer, ConsentGate from proofpath.cache import Cache from proofpath.config import Config, ModelsConfig, Permission, Permissions -from proofpath.document import Claim, Document, Locator, PageError +from proofpath.document import Document, PageError from proofpath.entailment import Scorer, pick_onnx_file from proofpath.events import ( Cancelled, @@ -49,7 +49,6 @@ from proofpath.report import ( ABSTRACT_BASIS, JUDGE_DETAIL_PREFIX, - ClaimResult, Kind, Report, judge_detail, @@ -85,9 +84,9 @@ VERIFYING, Engine, Prepared, - _escalates, _parser_for, decide_all, + escalates, prepare, target_document, verify, @@ -2734,17 +2733,6 @@ def test_asking_for_a_summary_without_a_judge_is_a_programming_error() -> None: verify(draft(JUDGE_BODY, [REAL]), built, summarize=True) -def _result(verdict: Verdict) -> ClaimResult: - claim = Claim( - text="the method is faster", - locator=Locator(line=1), - cited_refs=(1,), - paragraph=0, - sentence=0, - ) - return ClaimResult(claim, 1, "doi:10.1/x", verdict, from_cache=False) - - PASSAGE = Passage("the method is faster", "doi:10.1/x", 0) @@ -2765,4 +2753,4 @@ def _result(verdict: Verdict) -> ClaimResult: def test_the_escalation_set_is_the_low_band_with_something_to_judge_against( verdict: Verdict, escalated: bool ) -> None: - assert _escalates(_result(verdict)) is escalated + assert escalates(verdict) is escalated From bd8bf4ecf6dd1896460b823bc102a5c9a7f8d1c6 Mon Sep 17 00:00:00 2001 From: Yigtwxx Date: Mon, 5 Oct 2026 15:19:06 +0300 Subject: [PATCH 2/2] docs(eval): AVeriTeC with the browser and the judge (2026-10-05) Three runs answer OPEN-ITEMS 20.10. With the gold URLs the score is 0.371, and the browser is worth about two claims. Search with sentence queries scores 0.371. Search with judge-written queries scores 0.483: 14 claims fixed, 4 broken, p = 0.031, the one gain that cleared the noise. The majority baseline is 0.708. The reports also record what the numbers do not flatter. Coverage is no longer the main loss: NEI on a source that was read is. As shipped, the judge never sees an NEI (20.13). Search calls far more false claims SUPPORTED than the cited sources do (20.15). Causes that were not measured are marked as hypotheses. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K46EEsxMvuAuM8wjvPNWLM --- CHANGELOG.md | 9 ++ README.md | 46 +++++++--- ...2026-10-05-averitec-browser-judge-fresh.md | 86 ++++++++++++++++++ ...026-10-05-averitec-search-browser-fresh.md | 56 ++++++++++++ ...-05-averitec-search-browser-judge-fresh.md | 87 +++++++++++++++++++ docs/superpowers/OPEN-ITEMS.md | 32 ++++++- site/src/data/measured.ts | 10 +-- 7 files changed, 307 insertions(+), 19 deletions(-) create mode 100644 docs/eval/2026-10-05-averitec-browser-judge-fresh.md create mode 100644 docs/eval/2026-10-05-averitec-search-browser-fresh.md create mode 100644 docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 3bcbf19..ab8417c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ All notable changes to this project are documented here. The format follows ## [Unreleased] ### Added +- **AVeriTeC eval: `--browser`, `--judge` and `--fresh`** (`scripts/eval_averitec.py`). + Every source now records the ladder step that read it and, under `--judge`, the + judge's opinion when its verdict was in the escalation band. The report adds two + numbers beside the product's: the accuracy with the browser's pages dropped, and a + hypothetical one in which the judge decides the low band. The product never does the + second, and the report says so. Each run setup gets its own results file and report. + Measured on 2026-10-05: gold URLs 0.371; search with sentence queries 0.371; search + with the judge writing the queries 0.483 (14 claims fixed, 4 broken, p = 0.031). The + majority baseline is 0.708. - **`proofpath update`** installs the latest release from PyPI, and **`proofpath update --check`** only says whether there is one. In the TUI they are `/update` and `/update --check`. Running the command is the consent, so nothing diff --git a/README.md b/README.md index 2b522e1..727897d 100644 --- a/README.md +++ b/README.md @@ -410,9 +410,21 @@ if it is not in English) before searching. If the judge is rate-limited, the loc fallback below takes over; only if that fails too is the plain sentence searched, and the report says so. -**This is experimental.** On AVeriTeC's search scenario, proofpath scores -[**0.404**](docs/eval/2026-09-29-averitec-search.md) against a 0.708 majority baseline, and every report from a -run that searched prints that line at the top. +**This is experimental.** On AVeriTeC's search scenario (2026-10-05, browser allowed), +proofpath scores [**0.371**](docs/eval/2026-10-05-averitec-search-browser-fresh.md) with +sentence queries and [**0.483**](docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md) +when the judge writes the queries, against a 0.708 majority baseline. The judge's queries +are the one gain that cleared the noise: on the same 89 claims they fixed 14 and broke 4 +(paired exact McNemar p = 0.031). The share of pages from fact-check sites did not change +(22.0 % against 22.5 %). + +**Search also asserts the wrong way far more often than the cited sources do.** With the +judge's queries, 25 of the 89 claims came back the wrong way, and 20 of those were false +claims called SUPPORTED. With sentence queries the figure was 29, against 11 when the gold +URLs are read. ("Wrong way" counts Supported read as REFUTED and Refuted read as SUPPORTED.) +One likely cause, not yet measured: a page found by searching for a claim often repeats it, +and the models may read the repetition as support. Both scores are still below the +baseline, and every report from a run that searched prints that line at the top. ## Optional LLM judge (v0.3) @@ -480,16 +492,24 @@ switch to, the run says so and tells you how to add one, on the same channels. | Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text | | Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) | | Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) | -| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.270 3-way accuracy against a 0.708 majority baseline — worse than always guessing "refuted"** ([details](docs/eval/2026-09-16-averitec.md)) | - -**The AVeriTeC row is the one to read before trusting this tool on a news claim.** A third -of those claims had no readable source at all: 32 of the source URLs needed the browser -step, 29 were unreachable, 14 were refused by `robots.txt`. On the claims that *did* have a -readable source the score is 0.361 — still below the baseline, and every one of the 19 -`Supported` claims was missed. The retrieval and entailment models were calibrated on -scientific abstracts, and a fact-check page is a different object: long, discursive, and -usually quoting the claim it debunks. Nothing was tuned after that measurement, and no -blocked URL was dropped from it. +| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.371 3-way accuracy from the gold source URLs (browser allowed) against a 0.708 majority baseline — still worse than always guessing "refuted"** ([details](docs/eval/2026-10-05-averitec-browser-judge-fresh.md)). With evidence search and a judge writing the queries: 0.483 ([details](docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md)). The first measurement, without the browser, was 0.270 ([2026-09-16](docs/eval/2026-09-16-averitec.md)) | + +**The AVeriTeC row is the one to read before trusting this tool on a news claim.** Coverage +is no longer the main loss. On 2026-10-05, 165 of the 200 gold source URLs were read, +and only 10 of those needed the browser step, worth about two claims, which is within the +noise of 89. The loss is in the reading: + +- 36 of the 89 scorable claims had a readable source and still came back NEI where the + gold label was Supported or Refuted; +- 11 were called the wrong way; +- 7 had nothing readable. + +The likely reason, not separately measured: the retrieval and entailment models were +calibrated on scientific abstracts, and a fact-check page is a different object (long, +discursive, and usually quoting the claim it debunks). +As shipped, the judge cannot help with those NEIs. An NEI verdict carries no passage, so it +is never escalated (rule 1); only 1 of the 165 read sources reached the judge. Nothing was +tuned after these measurements, and no blocked URL was dropped from them. What proofpath is good at is the academic path the other rows measure: finding out whether a cited paper exists, whether it was retracted, and whether its text says what the sentence diff --git a/docs/eval/2026-10-05-averitec-browser-judge-fresh.md b/docs/eval/2026-10-05-averitec-browser-judge-fresh.md new file mode 100644 index 0000000..a0718ae --- /dev/null +++ b/docs/eval/2026-10-05-averitec-browser-judge-fresh.md @@ -0,0 +1,86 @@ +# AVeriTeC dev — 2026-10-05 + +- claims: 100 (limit=100) +- measured path: gold source URLs, no search +- browser step: allowed +- fetch: live, cache bypassed +- judge: groq openai/gpt-oss-120b (its opinions are the hypothetical column below; the verdicts are the models') +- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json` (sha256 499793726b4a…) +- one run of the whole product: real claims, real source pages, real fetch ladder. + +## Headline + +3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims. + +The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330. + +## Per label + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 0 | 0.000 | +| Refuted | 63 | 28 | 0.444 | +| Not Enough Evidence | 7 | 5 | 0.714 | +| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 | + +## Source coverage + +| state | count | +|---|---| +| ok | 165 | +| UNVERIFIED (blocked, robots.txt) | 17 | +| UNVERIFIED (provider unavailable) | 10 | +| not a url | 9 | +| UNVERIFIED (blocked) | 4 | +| UNVERIFIED (reached, no text extracted) | 4 | + +76 of 200 source URLs are web.archive.org snapshots (38.0 %). + +## Without the browser + +10 of 165 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.348**. + +Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one. + +## Judge (hypothetical) + +The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label. + +3-way accuracy **0.371**, against 0.371 for the product. 1 sources were escalated; 0 got no opinion back. + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 0 | 0.000 | +| Refuted | 63 | 28 | 0.444 | +| Not Enough Evidence | 7 | 5 | 0.714 | + +| judge model | opinions | +|---|---| +| groq openai/gpt-oss-120b | 1 | + +| decided by | n | correct | accuracy | +|---|---|---|---| +| models (medium or high tier) | 41 | 28 | 0.683 | +| nothing escalated | 48 | 5 | 0.104 | + +## Notes + +- Same 100 dev claims as `2026-09-16-averitec.md` (0.270), but not the same web. That + run used `--no-browser` and read 120 sources; this one read 165 of 200, the browser + step 10 of them. Pages that were unreachable then answer now, so the two numbers + measure different snapshots of the same claims. +- **The browser is worth about two claims here.** Dropping the 10 pages it read gives + 0.348, against 0.371. At 89 claims the standard error is about ±0.05, so this is + within the noise. +- **The judge saw one source.** `pipeline.decide` never gives an NEI verdict a + passage, and `verify.escalates` sends nothing without one (rule 1). 105 of the 165 + read sources are NEI. The hypothetical column therefore equals the product. The judge + never switched to its local fallback. +- **Where the 56 wrong claims are.** 36 had a readable source and came back NEI where + the gold label was Supported or Refuted. 11 were called the wrong way: Refuted read as + Supported 5 times, Supported as Refuted 6 times. 7 had nothing readable, and 2 gold + NEI claims got a verdict. Coverage is no longer the main loss; the reading is. +- Of the 33 claims scored right, one is a gold-NEI claim with nothing readable, which + the score counts as NEI. 32 are verdicts or NEIs on sources that were read. +- Measured with the default NLI profile (pinned), `--fresh`, `--sleep 1.0`. + diff --git a/docs/eval/2026-10-05-averitec-search-browser-fresh.md b/docs/eval/2026-10-05-averitec-search-browser-fresh.md new file mode 100644 index 0000000..9cd4872 --- /dev/null +++ b/docs/eval/2026-10-05-averitec-search-browser-fresh.md @@ -0,0 +1,56 @@ +# AVeriTeC dev — 2026-10-05 + +- claims: 100 (limit=100) +- measured path: sentence queries, no judge +- browser step: allowed +- fetch: live, cache bypassed +- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json` (sha256 499793726b4a…) +- one run of the whole product: real claims, real source pages, real fetch ladder. + +## Headline + +3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims. + +The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330. + +## Per label + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 6 | 0.316 | +| Refuted | 63 | 26 | 0.413 | +| Not Enough Evidence | 7 | 1 | 0.143 | +| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 | + +## Source coverage + +| state | count | +|---|---| +| ok | 313 | +| UNVERIFIED (blocked, robots.txt) | 6 | +| UNVERIFIED (blocked) | 5 | +| UNVERIFIED (provider unavailable) | 1 | + +0 of 325 source URLs are web.archive.org snapshots (0.0 %). + +## Without the browser + +18 of 313 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.348**. + +Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one. + +## Notes + +- Evidence search with sentence queries (no judge), browser allowed, every page fetched + live. The 2026-09-29 run of the same path, without the browser, scored 0.404. The + three-claim gap sits inside the ±0.05 standard error at 89 claims, and the search + results themselves differ between the days. +- The browser read 18 of the 313 pages read. Dropping them gives 0.348. +- Coverage is close to complete: 313 of 325 pages read, 11 blocked (6 by `robots.txt`). +- **Wrong-way verdicts:** 29 of 89. 22 false claims were called SUPPORTED and 7 true + ones REFUTED, against 11 when the gold URLs are read. The count leaves out verdicts + given to gold-NEI claims. One likely cause, not yet measured: a page found by + searching for a claim often repeats it. +- This run is the baseline for `2026-10-05-averitec-search-browser-judge-fresh.md`, + which differs only in who writes the queries. + diff --git a/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md b/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md new file mode 100644 index 0000000..a67d6be --- /dev/null +++ b/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md @@ -0,0 +1,87 @@ +# AVeriTeC dev — 2026-10-05 + +- claims: 100 (limit=100) +- measured path: judge queries (groq openai/gpt-oss-120b) +- browser step: allowed +- fetch: live, cache bypassed +- judge: groq openai/gpt-oss-120b (writes the search queries; its opinions are the hypothetical column below) +- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json` (sha256 499793726b4a…) +- one run of the whole product: real claims, real source pages, real fetch ladder. + +## Headline + +3-way accuracy **0.483** vs majority baseline 0.708, over 89 of 100 claims. + +The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.430. + +## Per label + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 10 | 0.526 | +| Refuted | 63 | 31 | 0.492 | +| Not Enough Evidence | 7 | 2 | 0.286 | +| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 | + +## Source coverage + +| state | count | +|---|---| +| ok | 319 | +| UNVERIFIED (blocked, robots.txt) | 9 | +| UNVERIFIED (blocked) | 4 | + +0 of 332 source URLs are web.archive.org snapshots (0.0 %). + +## Without the browser + +17 of 319 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.483**. + +Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one. + +## Judge (hypothetical) + +The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label. + +3-way accuracy **0.483**, against 0.483 for the product. 1 sources were escalated; 0 got no opinion back. + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 10 | 0.526 | +| Refuted | 63 | 31 | 0.492 | +| Not Enough Evidence | 7 | 2 | 0.286 | + +| judge model | opinions | +|---|---| +| groq openai/gpt-oss-120b | 1 | + +| decided by | n | correct | accuracy | +|---|---|---|---| +| models (medium or high tier) | 71 | 41 | 0.577 | +| nothing escalated | 18 | 2 | 0.111 | + +## Notes + +- **The one gain that cleared the noise.** The only difference from + `2026-10-05-averitec-search-browser-fresh.md` (0.371) is who writes the search + queries: here the judge, `groq openai/gpt-oss-120b`. Paired over the same 89 claims, + the judge's queries were right where the sentence queries were wrong 14 times, and + the reverse 4 times (exact McNemar p = 0.031). `Supported` went from 6/19 to 10/19. +- **The share of fact-check pages did not change.** By a host/URL heuristic, they were + 22.0 % of the pages here and 22.5 % with sentence queries. That says the mix did not + shift, not that no better fact-check page was found. The checker's own site is + excluded in both, as in every search run. +- **Many claims still come back the wrong way.** 25 of the 89 claims came back the wrong + way: 20 false claims were called SUPPORTED, and 5 true ones REFUTED. Reading the gold + URLs gives 11. The count leaves out verdicts given to gold-NEI claims. One likely + cause, not yet measured: a page found by searching for a claim often repeats it. +- **One risk this run cannot rule out.** A model writing the queries may carry what it + remembers about a claim into them. The verdicts still come from the pages read, each + with its passage, but the choice of pages may not be blind. +- All queries were written by Groq. The judge never switched to its local fallback; + the run printed no switch notice. +- The browser read 17 of the 319 pages read; dropping them leaves the score at 0.483. + The hypothetical judge column equals the product: one source was escalated (see the + gold-URL run's note on NEI verdicts). +- Still below the 0.708 majority baseline, so the search stays experimental. + diff --git a/docs/superpowers/OPEN-ITEMS.md b/docs/superpowers/OPEN-ITEMS.md index 1a29bce..b8263f8 100644 --- a/docs/superpowers/OPEN-ITEMS.md +++ b/docs/superpowers/OPEN-ITEMS.md @@ -607,7 +607,7 @@ five packages. The user's decisions: | 20.7 | Skip model loads on a fully cached re-run | same as 10.2 | | 20.8 | Bibliography parsing that was measured but never shipped | two-column rejoin (9.10/12.1), multi-block headless fallback (11.11), more headings (16.5), `[1]-[3]` ranges (9.4) | | 20.9 | The judge's quote is never checked | `judge._collect` does not verify that the quoted span occurs in the passage. Checking it is a cheap way to enforce product rule 1 | -| 20.10 | Eval honesty | add a `--judge` flag to `scripts/eval_averitec.py`; do one run with the browser allowed; show the 0.404 search score in the README "Measured" table | +| 20.10 | Eval honesty | **done 2026-10-05**: `--browser`, `--judge`, `--fresh`; three runs and their results are under "20.10 results" below | | 20.11 | NLI bake-off result | **Winner: `large-fever` (MoritzLaurer DeBERTa-v3-large mnli-fever-anli-ling-wanli) × k=2 × `max`.** SciFact dev macro-F1 0.697, against 0.580 for the `base` × k=1 × `max` baseline and 0.608 for the best `base` row. Accuracy 0.697. AVeriTeC 3-way 0.438, or 0.475 on the readable subset, against a 0.708 majority baseline. Cost: 643 MB and 54 ms/pair, against 244 MB and 19 ms/pair. Full report: `docs/eval/2026-10-01-nli-bakeoff.md`. Next: (a) and (b) are settled by the accurate-NLI design (`docs/superpowers/specs/2026-10-01-accurate-nli-design.md`), which adds the winner as an opt-in "accurate" profile behind a consent download (spec §7.1); (c) stays open. (a) **settled** — rule-decided rows are left out of the tier walk (the numeric layer fired 4–6 times on train with 0 correct and pinned `high` at 1.000000); the accurate cuts are `high` 0.999142. (b) **settled** — the `medium` cut still sits on `decide`, and the report now says so itself (`NO_LOW_TIER`, default runs included). (c) **still open** — on AVeriTeC the winner is below `base` × k=2 (0.475 against 0.537), and `Supported` is still 2/19; the README recommends the profile for scientific sources, not for news | | 20.12 | Accurate-NLI follow-ups | (1) A shared `browser.read_terminal_answer` helper: `model_gate.ask_model_terminal` repeats `browser.ask_terminal`'s prompt loop and imports the private `_CHOICE_ANSWERS` to do it. (2) The `-q` `nli` line carries the fallback reason but not the advice. (3) Ctrl-C and `/stop` do not interrupt a running 643 MB download. (4) An "installed" but corrupt accurate model fails the run instead of falling back. (5) A multilingual NLI model is still deferred (out of scope in the accurate-NLI design); non-English claims still need the judge to translate | @@ -635,6 +635,36 @@ The user picked the recommended option on all four questions. - **Git.** One feature branch and one PR: the script, its tests, the `docs/eval` reports and the README "Measured" row. +### 20.10 results — 2026-10-05 + +| Run | 3-way | Without the browser | Judge (hypothetical) | +|---|---|---|---| +| Gold URLs, browser, judge | 0.371 | 0.348 | 0.371 | +| Search, browser, sentence queries | 0.371 | 0.348 | — | +| Search, browser, judge queries (Groq `gpt-oss-120b`) | **0.483** | 0.483 | 0.483 | + +All three are against a 0.708 majority baseline over 89 claims. Reports: +`docs/eval/2026-10-05-averitec-*.md`. + +- **Coverage is no longer the main loss.** 165 of 200 gold sources were read. The + browser was worth about two claims, within the ±0.05 noise. +- **The reading is the loss.** 36 of 89 claims had a readable source and came back NEI, + and 11 went the wrong way. +- **The judge's queries are the one gain that cleared the noise.** On the same claims + they fixed 14 and broke 4 (McNemar p = 0.031). The share of fact-check pages did not + change. +- **Search asserts the wrong way much more.** 25 of 89 claims (judge queries) and 29 of + 89 (sentence queries) came back the wrong way, mostly false claims called SUPPORTED. + The gold URLs give 11 (new item 20.15). +- **The judge cannot see NEIs** (new item 20.13). This is why the hypothetical column + equals the product. + +| # | Item | Note | +|---|---|---| +| 20.13 | The judge never sees an NEI | `pipeline.decide` returns NEI with `passage=None`, so `verify.escalates` never sends it (rule 1). Yet NEI-on-a-read-source is 36 of the 56 wrong claims on AVeriTeC. Measure first: give the judge the best passage of each NEI source, verify its quote (20.9), and score it as a hypothetical column. Run it without the fallback so it measures Groq alone. Only then decide whether the product should escalate NEI with its best passage | +| 20.14 | Web cuts | Fit `decide`/`medium` on AVeriTeC **train** and report them on dev, as the accurate model's cuts were fitted on SciFact train. This targets the same NEI slice without touching rule 1 | +| 20.15 | Search calls more false claims SUPPORTED | With search, 20–22 of the 63 false claims come back SUPPORTED, against 5 from the gold URLs. Hypothesis, not yet measured: a page found by searching for a claim often quotes it, and the models read the quote as support. Check that on the failing rows first. Candidate fix: drop passages that restate the claim (near-duplicate of the hypothesis) before entailment, and measure it on AVeriTeC train first. This bears on rule 3's spirit: a wrong SUPPORTED is the costliest error | + ## 21. `proofpath update` — 2026-10-05 The user asked for an update command like `claude update`. They picked the recommended diff --git a/site/src/data/measured.ts b/site/src/data/measured.ts index 11d22b1..ad19916 100644 --- a/site/src/data/measured.ts +++ b/site/src/data/measured.ts @@ -81,12 +81,12 @@ export const measures: readonly Measure[] = [ }, { id: 'averitec', - figure: '0.270', + figure: '0.371', reading: '3-way accuracy', - against: 'against a 0.708 majority baseline — worse than always guessing "refuted"', + against: 'against a 0.708 majority baseline — still worse than always guessing "refuted"', what: 'End to end on real web claims', - set: 'AVeriTeC dev, 100 claims', - doc: '2026-09-16-averitec.md', + set: 'AVeriTeC dev, 100 claims, browser allowed', + doc: '2026-10-05-averitec-browser-judge-fresh.md', weak: true, }, ]; @@ -98,4 +98,4 @@ export const accessDoc = '2026-09-11-coverage.md'; export const evalUrl = (doc: string): string => `${repo}/blob/main/docs/eval/${doc}`; export const averitecNote = - 'A third of those claims had no readable source at all: 32 URLs needed the browser step, 29 were unreachable, 14 were refused by robots.txt. On the claims that did have one the score is 0.361 — still below the baseline. The models were calibrated on scientific abstracts; a fact-check page is a different object.'; + 'Coverage is no longer the loss: 165 of 200 sources were read. 36 of 89 claims had a readable source and still came back NEI, and 11 went the wrong way. The models were calibrated on scientific abstracts; a fact-check page is a different object. With evidence search and the judge writing the queries it scores 0.483 (14 claims fixed, 4 broken, p = 0.031), the one gain beyond the noise — but 25 of 89 claims come back the wrong way there, 20 of them false claims called supported (likely cause, not yet measured: search finds pages that repeat the claim).';