Skip to content

bench(nmse): first sealed GF16 vs bfloat16 NMSE manifest, from a t27 run harness (Closes #10019) - #10021

Merged
gHashTag merged 1 commit into
masterfrom
bench/nmse-pub02-first-manifest
Oct 17, 2026
Merged

bench(nmse): first sealed GF16 vs bfloat16 NMSE manifest, from a t27 run harness (Closes #10019)#10021
gHashTag merged 1 commit into
masterfrom
bench/nmse-pub02-first-manifest

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Closes #10019

What

PUB-02 asks for one sealed-toolchain NMSE manifest under docs/GF16_BFLOAT16_NMSE_PROTOCOL.md. This adds the procedure that produces one, written in t27, and the first manifest it produced.

  • specs/benchmarks/gf16_bfloat16_nmse_run.t27 (new, 19 tests): the L5 witness runs first and an abort writes no manifest; one xoshiro256pp stream seeded by splitmix64; D_NORM, D_LOG, D_RELU, D_PHI, D_DEEP; GF16 [S1|E6|M9] bias 31 with no subnormals, and bfloat16 with ieee or ftz subnormals, both round-to-nearest-even from f64; Neumaier-compensated sums; the manifest JSON. The input is a facts file on stdin, so there is no hand-written logic outside t27. seal_hash is FROZEN_HASH only when it equals the sha256 of the compiler.rs that was built, otherwise unsealed. x_extension.harness_sha256 records the sha256 of this spec.
  • bench/results/nmse-railway-x86_64-0x2545F4914F6CDD1D-2026-10-17.json: 10,000,000 samples per distribution, Railway lab x86_64, sealed, ieee.
  • docs/GF16_BFLOAT16_NMSE_PROTOCOL.md section 6.1: how the harness is driven, and the two readings the protocol leaves open, now declared in every manifest. docs/SCIENTIFIC_IMPROVEMENT_PLAN.md PUB-02: the manifest is recorded as a software run, not a silicon measurement. Its label stays target, because attaching it to a draft is still open.

Result (from the committed manifest)

Distribution NMSE GF16 NMSE BF16 ratio
D_NORM 1.727600840e-07 2.759105547e-06 6.261452527e-02
D_LOG 1.466388254e-07 2.352234957e-06 6.234021177e-02
D_RELU 1.727403167e-07 2.759058007e-06 6.260843965e-02
D_PHI 1.746953198e-07 2.793862619e-06 6.252824268e-02
D_DEEP 1.466442940e-07 2.348090535e-06 6.245257234e-02

The ratios sit near 1/16, which is what two extra mantissa bits predict. Across 50M samples, no sample hit a tie, a GF16 flush, a bf16 subnormal or an Inf: the D_LOG range 2^-10..2^10 stays inside both formats' normal range. So this run does not exercise the ties-to-even vs ties-toward-zero difference, or the subnormal policy.

How it was run (Railway lab only)

t27c gen-c specs/benchmarks/gf16_bfloat16_nmse_run.t27 > nm.c
# the lefthook.yml one-line main, not committed:
# int main(void){... nmse_manifest(stdin buffer) -> stdout; return n==0;}
cc -O2 -ffp-contract=off -o drv drv.c -lm
./drv < facts.txt > manifest.json

The facts are seed, samples, policy, a UTC timestamp, host_arch, the compiler string, FROZEN_HASH, the sha256 of bootstrap/src/compiler.rs, the sha256 of the harness, and the driver text. 10M x 5 takes 3.5 s.

Verification

  • t27c test-report (t27c built from master efe3c94, Zig 0.16.0): 19 pass, 0 fail, 0 vacuous. All 19 also pass through gen-c with a C test main, and a mutated KAT traps.
  • gen-c output is byte-identical between t27c 90fc25b and master.
  • Schema: 0 errors against schemas/nmse-protocol-v1.json (Draft 7, plus an RFC3339 check). Five schema-breaking mutants are all rejected: bad seal, missing results, zero ratio, bad RNG family, n below 1000.
  • Independent reference: a separate exact computation (frexp/ldexp rounding, fsum) on the same xoshiro stream. All 15 numbers agree within 2.3e-10 relative, at both n = 10,000 and n = 10,000,000. The splitmix64 and xoshiro256pp known-answer values in the tests were checked against an independent implementation.
  • x_extension.harness_sha256 = 30ba8c16...0b03, which is sha256sum specs/benchmarks/gf16_bfloat16_nmse_run.t27 at this commit.
  • Corpus ratchet (t27c suite --repo-root . --ratchet --corpus-only on master plus this spec): RATCHET CLEAN, 0 unexpected failures. The spec has no seal file, the same as the other new specs landed this week.

Not in this PR

  • Grounding cells 15, 16 and 24 of specs/t27f/reel_gf16_nmse.t27 against this manifest.
  • specs/api/tri_net_api.t27 still says bench/results/ does not exist. It is sealed, so changing it means re-sealing.

🤖 Generated with Claude Code

@gHashTag @claude
bench(nmse): first sealed GF16 vs bfloat16 NMSE manifest, from a t27 …
d8c2a1e 
…run harness (Closes #10019)

- specs/benchmarks/gf16_bfloat16_nmse_run.t27 (new): the PUB-02 run
  procedure in t27. L5 witness first (an abort writes nothing),
  xoshiro256pp seeded by splitmix64, the five protocol distributions,
  GF16 and bfloat16 round-to-nearest-even quantisers with a declared
  subnormal policy, Neumaier sums, and the manifest text. seal_hash is
  FROZEN_HASH only when it equals the built compiler.rs sha256, else
  "unsealed"; the manifest records the harness's own sha256. Driven by
  t27c gen-c and the lefthook.yml one-line stdin/stdout main; no
  hand-written code is committed. 19 tests.
- bench/results/nmse-railway-x86_64-0x2545F4914F6CDD1D-2026-10-17.json:
  10,000,000 samples per distribution on the Railway lab, sealed,
  ieee subnormals. NMSE(GF16)/NMSE(BF16) is 0.0623 to 0.0626 on all
  five distributions; 0 ties, flushes, subnormals or Infs observed.
- docs: protocol section 6.1 points at the harness and declares the two
  readings the protocol leaves open (D_PHI as mean/variance; ties-to-even
  against the frozen SSOT encoder's ties-toward-zero). PUB-02 records the
  manifest as a software run, not a silicon measurement.

Verified on the lab: 19/19 under t27c test-report (0 vacuous) and via
gen-c plus a C test main; 0 schema errors and 5/5 mutants rejected;
15/15 numbers within 2.3e-10 relative of an independent exact reference
on the same stream; corpus ratchet CLEAN with the spec in the corpus.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-17 16:55:26 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 49
PRs with All Checks Green 1
READY 0
FAILING 49
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 49 + 0 + 0 = 49, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=e769ef021713 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag enabled auto-merge (squash) October 17, 2026 16:55
@gHashTag
gHashTag merged commit cc80a30 into master Oct 17, 2026
24 of 29 checks passed
This was referenced Oct 17, 2026
Closed
Closed
Merged
gHashTag added a commit that referenced this pull request Oct 17, 2026
@gHashTag @claude
ci(dupe): bless the two deliberate put copies in the NMSE run harness (…
a19701a 
…Closes #10025) (#10027)

#10021 landed specs/benchmarks/gf16_bfloat16_nmse_run.t27 without moving
the duplicate-body ledger. Its put_lit and put_range repeat put_msg and
put_range in specs/policy/own_language.t27, so the Duplicate Body Ratchet
has been red on master since cc80a30.

The copies are deliberate. The harness lowers through t27c gen-c into one
C file, and the committed manifest records the spec's own sha256 as
x_extension.harness_sha256. Editing the spec to reuse the existing
functions would leave that manifest pointing at a file that is no longer
in the tree. So the ledger moves instead, written by
`python3 tools/dupe_scan.py --bless` on the Railway lab at cc80a30:
two added lines, put_lit 2 and put_range 2. No spec changes.

Verified on the lab: dupe_scan exits 1 before the bless and 0 after;
--self-test exits 0.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
gHashTag added a commit that referenced this pull request Oct 17, 2026
@gHashTag @claude
spec(t27f): reel cells 15, 16, 24 cite the sealed NMSE run, drawn as …
d3cb48f 
…earned drafts (Closes #10023)

Cells 15, 16 and 24 (steps 16, 17, 25) of specs/t27f/reel_gf16_nmse.t27 named 'the sealed run PUB-02 asks for does not exist' as their gap. PR #10021 landed it: specs/benchmarks/gf16_bfloat16_nmse_run.t27 and bench/results/nmse-railway-x86_64-0x2545F4914F6CDD1D-2026-10-17.json. The three cells now cite those artefacts and are grounded. They are research DELIVER steps and #10021 has 0 reviews, so earned_status gives DRAFT. Counts move to 15 published, 1 verified, 9 drafts, 2 gaps. A new test pins that a test pass or a live page does not lift them and a review would. The reel quotes no NMSE value and claims nothing about silicon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 17, 2026
Merged
Merged
gHashTag added a commit that referenced this pull request Oct 17, 2026
@gHashTag @claude
spec(t27f): reel cells 15, 16, 24 cite the sealed NMSE run, drawn as …
a04711c 
…earned drafts (Closes #10023) (#10031)

Cells 15, 16 and 24 (steps 16, 17, 25) of specs/t27f/reel_gf16_nmse.t27 named 'the sealed run PUB-02 asks for does not exist' as their gap. PR #10021 landed it: specs/benchmarks/gf16_bfloat16_nmse_run.t27 and bench/results/nmse-railway-x86_64-0x2545F4914F6CDD1D-2026-10-17.json. The three cells now cite those artefacts and are grounded. They are research DELIVER steps and #10021 has 0 reviews, so earned_status gives DRAFT. Counts move to 15 published, 1 verified, 9 drafts, 2 gaps. A new test pins that a test pass or a live page does not lift them and a review would. The reel quotes no NMSE value and claims nothing about silicon.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bench(nmse): the first sealed GF16 vs bfloat16 NMSE manifest, produced by a t27 run harness (PUB-02)

1 participant

@gHashTag