From 5810918edc0d59763788678b34898498917c5155 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 02:41:59 +0000 Subject: [PATCH] docs: the benchmark corpus holds 15,926 events, not 16,146 README, BENCHMARK.md and the CHANGELOG all said the benchmark-corpus-2026-08-27 release holds 16,146 events. The manifest committed alongside it says otherwise: scripts/build-benchmark-corpus.sh writes each page's count from `wc -l` of its event file (one event per line), the manifest's hashes bind the release asset, and its ten counts sum to 15,926. The 16,146 was typed into the publishing commit's message and docs by hand, 220 high. The data and manifest are unchanged; only the prose was wrong. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CHTDKD1277KYSjDF1gpnLD --- BENCHMARK.md | 2 +- CHANGELOG.md | 2 +- README.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/BENCHMARK.md b/BENCHMARK.md index 39e9dd6..eb0db73 100644 --- a/BENCHMARK.md +++ b/BENCHMARK.md @@ -4,7 +4,7 @@ Refract can serve as deterministic ground truth for AI model evaluation. This do ## Ground-truth corpus -The published corpus is attached to the [`benchmark-corpus-2026-08-27` release](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27): one event-stream JSONL per benchmark page over a fixed window — 16,146 events, 82 MB uncompressed — with [`manifest.json`](benchmark/ground-truth/manifest.json) tracked here in the repo recording the Refract version, the window bounds, each page's last analyzed revision ID, event count, and SHA-256. The manifest hashes bind the release artifact to this commit. +The published corpus is attached to the [`benchmark-corpus-2026-08-27` release](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27): one event-stream JSONL per benchmark page over a fixed window — 15,926 events, 82 MB uncompressed — with [`manifest.json`](benchmark/ground-truth/manifest.json) tracked here in the repo recording the Refract version, the window bounds, each page's last analyzed revision ID, event count, and SHA-256. The manifest hashes bind the release artifact to this commit. The window is fixed (start pinned in the manifest, end pinned per page by `lastRevisionId`) rather than full-history, for two reasons: a corpus that changes when the pages change isn't a benchmark, and full-history analysis of the largest pages exceeds any fixed heap. To verify the corpus, run the manifest's window against each page with `--since --to ` and compare hashes — analysis is deterministic, so a match is byte-for-byte. diff --git a/CHANGELOG.md b/CHANGELOG.md index 5628897..b588970 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -58,7 +58,7 @@ Nothing has reached npm since 0.5.7, and 0.5.7's CLI does not start: it declares ## 0.5.15 (2026-08-27) ### Added -- **Benchmark ground-truth corpus**: 16,146 hash-pinned events across the ten benchmark pages, published as a release asset with an in-repo manifest (`BENCHMARK.md#ground-truth-corpus`). +- **Benchmark ground-truth corpus**: 15,926 hash-pinned events across the ten benchmark pages, published as a release asset with an in-repo manifest (`BENCHMARK.md#ground-truth-corpus`). ### Fixed - **Windowed analyses leaked the full page-move log**: a `--since` window opened with `page_moved` events from the page's earliest history. Moves are now bounded to the analyzed revision span (`windowPageMoves`). diff --git a/README.md b/README.md index 5d73ae5..cab818e 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,7 @@ Correlated 4 talk page discussions. … 195 more events ``` -No model is called, and the same source produces the same events on every run. A [hash-pinned ground-truth corpus](BENCHMARK.md#ground-truth-corpus) of 16,146 events across ten benchmark pages ships as a [release asset](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27), reproducible byte-for-byte from the manifest bounds. +No model is called, and the same source produces the same events on every run. A [hash-pinned ground-truth corpus](BENCHMARK.md#ground-truth-corpus) of 15,926 events across ten benchmark pages ships as a [release asset](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27), reproducible byte-for-byte from the manifest bounds. Refract ingests versioned sources (MediaWiki, text files), computes structural and semantic change events, tracks claims and citations across time, and emits structured provenance data that downstream systems can query, replay, and audit.