Skip to content

docs: the benchmark corpus holds 15,926 events, not 16,146 #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zz-plant merged 1 commit into from
Oct 8, 2026
Merged

docs: the benchmark corpus holds 15,926 events, not 16,146 #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Refract can serve as deterministic ground truth for AI model evaluation. This do

## Ground-truth corpus

The published corpus is attached to the [`benchmark-corpus-2026-08-27` release](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27): one event-stream JSONL per benchmark page over a fixed window — 16,146 events, 82 MB uncompressed — with [`manifest.json`](benchmark/ground-truth/manifest.json) tracked here in the repo recording the Refract version, the window bounds, each page's last analyzed revision ID, event count, and SHA-256. The manifest hashes bind the release artifact to this commit.
The published corpus is attached to the [`benchmark-corpus-2026-08-27` release](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27): one event-stream JSONL per benchmark page over a fixed window — 15,926 events, 82 MB uncompressed — with [`manifest.json`](benchmark/ground-truth/manifest.json) tracked here in the repo recording the Refract version, the window bounds, each page's last analyzed revision ID, event count, and SHA-256. The manifest hashes bind the release artifact to this commit.

The window is fixed (start pinned in the manifest, end pinned per page by `lastRevisionId`) rather than full-history, for two reasons: a corpus that changes when the pages change isn't a benchmark, and full-history analysis of the largest pages exceeds any fixed heap. To verify the corpus, run the manifest's window against each page with `--since <windowStart> --to <lastRevisionId>` and compare hashes — analysis is deterministic, so a match is byte-for-byte.

Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Nothing has reached npm since 0.5.7, and 0.5.7's CLI does not start: it declares
## 0.5.15 (2026-08-27)

### Added
- **Benchmark ground-truth corpus**: 16,146 hash-pinned events across the ten benchmark pages, published as a release asset with an in-repo manifest (`BENCHMARK.md#ground-truth-corpus`).
- **Benchmark ground-truth corpus**: 15,926 hash-pinned events across the ten benchmark pages, published as a release asset with an in-repo manifest (`BENCHMARK.md#ground-truth-corpus`).

### Fixed
- **Windowed analyses leaked the full page-move log**: a `--since` window opened with `page_moved` events from the page's earliest history. Moves are now bounded to the analyzed revision span (`windowPageMoves`).
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ Correlated 4 talk page discussions.
… 195 more events
```

No model is called, and the same source produces the same events on every run. A [hash-pinned ground-truth corpus](BENCHMARK.md#ground-truth-corpus) of 16,146 events across ten benchmark pages ships as a [release asset](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27), reproducible byte-for-byte from the manifest bounds.
No model is called, and the same source produces the same events on every run. A [hash-pinned ground-truth corpus](BENCHMARK.md#ground-truth-corpus) of 15,926 events across ten benchmark pages ships as a [release asset](https://github.com/refract-org/refract/releases/tag/benchmark-corpus-2026-08-27), reproducible byte-for-byte from the manifest bounds.

Refract ingests versioned sources (MediaWiki, text files), computes structural and semantic change events, tracks claims and citations across time, and emits structured provenance data that downstream systems can query, replay, and audit.

Expand Down
Loading