Skip to content

bench(nmse): first sealed GF16 vs bfloat16 NMSE manifest, from a t27 run harness (Closes #10019) #10021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gHashTag merged 1 commit into from
Oct 17, 2026
Merged

bench(nmse): first sealed GF16 vs bfloat16 NMSE manifest, from a t27 run harness (Closes #10019) #10021

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions bench/results/nmse-railway-x86_64-0x2545F4914F6CDD1D-2026-10-17.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"schema_version": "1.0",
"protocol_version": "1.0",
"seal_hash": "e769ef0217136b9bfdcb3dd783f02d8a8e23e91b70367416d02cdf61af4727e8",
"rng": { "family": "xoshiro256pp", "seed": "0x2545F4914F6CDD1D" },
"samples_per_distribution": 10000000,
"bf16_subnormal_policy": "ieee",
"identity_witness": { "phi_squared_residual_f64": 0.0, "trinity_residual_f64": 0.0 },
"results": {
"D_NORM": { "nmse_gf16": 1.727600840e-07, "nmse_bf16": 2.759105547e-06, "ratio": 6.261452527e-02 },
"D_LOG": { "nmse_gf16": 1.466388254e-07, "nmse_bf16": 2.352234957e-06, "ratio": 6.234021177e-02 },
"D_RELU": { "nmse_gf16": 1.727403167e-07, "nmse_bf16": 2.759058007e-06, "ratio": 6.260843965e-02 },
"D_PHI": { "nmse_gf16": 1.746953198e-07, "nmse_bf16": 2.793862619e-06, "ratio": 6.252824268e-02 },
"D_DEEP": { "nmse_gf16": 1.466442940e-07, "nmse_bf16": 2.348090535e-06, "ratio": 6.245257234e-02 }
},
"runner": { "host_arch": "x86_64", "compiler": "t27c gen-c (t27 90fc25b3b), cc (Debian 12.2.0-14+deb12u1) 12.2.0, flags -O2 with fp contraction off" },
"timestamp": "2026-10-17T16:42:23Z",
"x_extension": {
"harness": "specs/benchmarks/gf16_bfloat16_nmse_run.t27",
"harness_sha256": "30ba8c167d6ab27bc002c84223714922e5da12cc0e32a52135e0c17e634a0b03",
"driver": "t27c gen-c specs/benchmarks/gf16_bfloat16_nmse_run.t27, then the lefthook.yml one-line C main: stdin to nmse_manifest, result to stdout",
"stream": "one xoshiro256pp stream seeded by splitmix64(seed); D_NORM, D_LOG, D_RELU, D_PHI, D_DEEP drawn in that order, samples_per_distribution each",
"normal_sampler": "Box-Muller on 53-bit uniforms, both values of a pair used",
"d_phi_reading": "N(mean, variance): mean phi, standard deviation sqrt(1/phi)",
"d_deep_reading": "per sample: D_NORM with probability 0.7, else D_LOG",
"rounding": "both formats round to nearest, ties to even, once from f64; the frozen GF16 SSOT encoder rounds ties toward zero, which differs from this only on the samples counted in gf16_ties",
"gf16_range": "FORMAT-SPEC-001 GF16, no subnormals: |x| < 2^-31 and values rounding to 2^-31 flush to signed zero; E = 63 is Inf",
"accumulation": "Neumaier-compensated f64 sums; nmse = sum (x - Q(x))^2 / sum x^2",
"number_text": "ten significant digits, the last within one unit",
"counts": {
"D_NORM": { "gf16_ties": 0, "bf16_ties": 0, "gf16_flushed": 0, "bf16_subnormal": 0, "gf16_inf": 0, "bf16_inf": 0, "exact_zero_samples": 0, "sum_x2": 1.000167813e+07 },
"D_LOG": { "gf16_ties": 0, "bf16_ties": 0, "gf16_flushed": 0, "bf16_subnormal": 0, "gf16_inf": 0, "bf16_inf": 0, "exact_zero_samples": 0, "sum_x2": 3.777859984e+11 },
"D_RELU": { "gf16_ties": 0, "bf16_ties": 0, "gf16_flushed": 0, "bf16_subnormal": 0, "gf16_inf": 0, "bf16_inf": 0, "exact_zero_samples": 4999977, "sum_x2": 5.003426659e+06 },
"D_PHI": { "gf16_ties": 0, "bf16_ties": 0, "gf16_flushed": 0, "bf16_subnormal": 0, "gf16_inf": 0, "bf16_inf": 0, "exact_zero_samples": 0, "sum_x2": 3.236699811e+07 },
"D_DEEP": { "gf16_ties": 0, "bf16_ties": 0, "gf16_flushed": 0, "bf16_subnormal": 0, "gf16_inf": 0, "bf16_inf": 0, "exact_zero_samples": 0, "sum_x2": 1.135545964e+11 }
}
}
}
16 changes: 16 additions & 0 deletions docs/GF16_BFLOAT16_NMSE_PROTOCOL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,22 @@ The schema requires, at minimum:
A run that omits any required field is non-conforming and must not be
cited in TRI-NET documentation.

### 6.1 Reference run harness

`specs/benchmarks/gf16_bfloat16_nmse_run.t27` is the run procedure in t27:
witness first, the two quantisers, xoshiro256pp seeded by splitmix64, the
five distributions, and the manifest text. Its header says how it is
driven (`t27c gen-c` plus the one-line stdin/stdout main that
`lefthook.yml` already uses), and each manifest records the command in
`x_extension.driver`. Two readings this document leaves open are fixed
there and declared in every manifest's `x_extension`:

- `D_PHI` reads `N(phi, 1/phi)` as mean and **variance**, so the standard
deviation is `sqrt(1/phi)`.
- GF16 rounds ties to even, as section 3 asks. The frozen SSOT encoder
rounds ties toward zero instead. The two rules differ only on exact
ties, and the manifest counts those per distribution (`gf16_ties`).

---

## 7. Test obligations (L4)
Expand Down
4 changes: 2 additions & 2 deletions docs/SCIENTIFIC_IMPROVEMENT_PLAN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ generated file. When in doubt, omit the row.
| ID | Track | t27 deliverable | Label |
|----|-------|------------------|-------|
| PUB-01 | GoldenFloat family paper | `docs/WHITEPAPER/gf_paper_v3_imrad_draft.md` already exists in the repo. 2026 work: harden citations, run the NMSE protocol from PUB-02 on a fixed seed set, attach the manifest. | `target` |
| PUB-02 | NMSE manifest contributing to a publication | One sealed-toolchain `bench/results/nmse-*.json` produced under `docs/GF16_BFLOAT16_NMSE_PROTOCOL.md`. **No `Delta_dB` figure introduced before this manifest lands.** | `target` |
| PUB-02 | NMSE manifest contributing to a publication | One sealed-toolchain `bench/results/nmse-*.json` produced under `docs/GF16_BFLOAT16_NMSE_PROTOCOL.md`. **No `Delta_dB` figure introduced before this manifest lands.** First manifest landed 2026-10-17: `bench/results/nmse-railway-x86_64-0x2545F4914F6CDD1D-2026-10-17.json`, a sealed-toolchain software run of `specs/benchmarks/gf16_bfloat16_nmse_run.t27`. It is not a silicon measurement, and attaching it to a draft is still open. | `target` |
| PUB-03 | Tooling / spec-first methodology paper | A whitepaper-derivative doc that bundles `docs/TRI_NET_WHITEPAPER.md`, `STATUS.md`, `LINEUP.md`, `COMPETITORS.md` for an external review venue (workshop / preprint server). Draft only; submission is downstream. | `target` |

---
Expand Down Expand Up @@ -153,7 +153,7 @@ chip-repo CI jobs, not the Python SDK.
| `gen/verilog/` modules that pass simulation | `STATUS.md` 2.1 | 5/5 | `target` 5/5 maintained; new modules tracked |
| Sealed artefacts under `.trinity/seals/` | `STATUS.md` 2.1 | 729 (at audit time) | `target` strictly non-decreasing |
| Coq files with zero `Admitted` cited in line plans | `trios-coq/Physics/`, `coq/` | per `docs/NOW.md` ledger | `target` zero `Admitted` in any file cited by Zenodo v3 manifest |
| Conforming NMSE manifests under `bench/results/` | `docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` | 0 | `target` at least 1 sealed manifest for PUB-02 |
| Conforming NMSE manifests under `bench/results/` | `docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` | 1 sealed software run (`nmse-railway-x86_64-0x2545F4914F6CDD1D-2026-10-17.json`) | `target` at least 1 sealed manifest for PUB-02 |
| Schema files validating draft-07 | `schemas/` | 3 (numeric-format / tri-net-api / nmse-protocol) | `target` 3+ maintained; new schemas added only via constitutional / issue review |
| Documents passing `scripts/check_first_party_doc_language.py` | the gate itself | PASS (today) | `target` PASS maintained |

Expand Down
Loading
Loading