Skip to content

pgsum rows: build GenomeOS PGS evidence batches natively - #9

Merged
DylanCosto merged 3 commits into
mainfrom
rows-batches-20261022
Oct 22, 2026
Merged

DylanCosto merged 3 commits into
mainfrom
rows-batches-20261022

Conversation

@DylanCosto

Copy link
Copy Markdown
Owner

Summary

  • New command, pgsum rows: builds GenomeOS's per-term PGS evidence batches, byte for byte what GenomeOS's Python builder (pgs_pgsum_rows._build) writes:

    • the rows, merging the evidence rows with each term's source row, description and orientation from a GenomeOS catalog pack;
    • each batch document's digest;
    • the exact checkpoint delta.

    src/rows.rs lists every Python function it mirrors (canonical JSON with Python's float repr, decimal.Decimal arithmetic, describe, orientation, weight, the reducer).

  • Non-ASCII source lines: exit with status 3, so GenomeOS builds that run in Python. Python's text rules differ from ASCII ones only there.

  • evidence output unchanged: its rows now come from for_each_row, and a record met twice is recognised by its line instead of by re-serializing it. About 40% faster.

  • Allocator: mimalloc. With glibc malloc, rows built in one thread and freed in the other were about 3× slower.

  • Version: 0.6.0-dev. Tagging and releasing 0.6.0 is a separate step.

Checks (HG002, run D's inputs)

  • 354 sample batches (first two and last of all 126 report models) are identical to GenomeOS's Python builder: digest, delta and whole document bytes.
  • 417 batches (830,602 terms, two whole models) rebuilt natively match the digests in the case's signed batch chain.
  • evidence output is byte-identical to this branch's base (main 74ecb0a) on 100,000-term ranges of three models.
  • cargo test --release: 134 passed, 0 failed.
  • Speed: 200,000 terms in 4.4 s against about 28 s in Python (VM under load). A full 61 M-term rebuild against every signed batch runs tonight.

The GenomeOS side is DylanCosto/GenomeOS branch pgs-rows-rust-20261022. It uses rows when the pinned pgsum has it, rebuilds each run's first batch in Python as a check, and falls back to Python for older pgsum.

🤖 Generated with Claude Code

DylanCosto and others added 2 commits October 21, 2026 22:06
`pgsum rows` merges the `evidence` rows with the terms of a GenomeOS catalog pack and writes one JSON line per
batch (digest, exact checkpoint delta, optionally the document), byte for byte what GenomeOS's Python builder
writes (src/rows.rs lists the functions it mirrors). A non-ASCII source line exits 3 so GenomeOS builds that run in
Python. HG002, 126 models: 354 sample batches identical; 200,000 terms in 4.4 s against about 28 s in Python.

`evidence` keeps its output: its rows now come from `for_each_row`, and a record met twice is recognised by its line
(identical output on three HG002 models, about 40% faster). mimalloc is the allocator. Version 0.6.0-dev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…thon

The task's numbers are written back into the digested documents. serde_json's default parser can read a float one
unit in the last place off, and turns an integer beyond 64 bits into a float, so such a number would print differently
from Python (GenomeOS week review round 2, finding 3; latent: today's only float, 1.959963984540054, reads exactly).
Task::parse now reads each number token back and requires it to print to the same text; otherwise it exits with status
3 and GenomeOS builds the run in Python. No crate-wide parser change, so nothing else pgsum reads is affected.

HG002: the 354 sample batches are still identical to Python's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DylanCosto

Copy link
Copy Markdown
Owner Author

Added 498358a for review finding 3 (round 2): Task::parse reads every number token of the task back and requires it to print to the same text Python wrote; otherwise pgsum rows exits 3 and GenomeOS builds that run in Python. This covers a float serde_json reads 1 ULP off, or an integer beyond 64 bits. There is no crate-wide serde_json feature change. Tests: 135 passed. HG002: 354 of 354 sample batches still identical to Python.

CI's stable jobs failed at cargo fmt --check; clippy then flagged one
collapsible if. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DylanCosto
DylanCosto merged commit 3c829da into main Oct 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant