t27b against other compiler pipelines: October 2026 benchmark data for v0.2.0 #8817

Goal

Measure t27b against the other compiler pipelines on every axis that can be measured honestly, and publish the raw data with the method. The numbers feed the t27b v0.2.0 release notes and a blog post, so they must be reproducible, must cover the whole corpus where the axis allows it, and must show the axes where t27b loses as prominently as the ones where it wins.

Pipelines (all targeting aarch64-linux)

1. t27b (JIT under qemu-user; t27b build cross-compiled on the x86_64 host; --overflow trap and wrap).
2. t27c gen + zig 0.16 zig test, Debug, ReleaseSafe and ReleaseFast (LLVM).
3. t27c gen-c + aarch64-linux-gnu-gcc 12, -O0 and -O2.
4. t27c gen-c + clang 21.1.0 as shipped in zig 0.16 (zig cc), -O0 and -O2. The lab has no standalone clang.
5. t27c gen-rust + rustc 1.99 for aarch64: compile only, because gen-rust lowers no tests.

Axes

• A. Coverage and correctness over all of specs/ (generate, compile, run every test), and per-test disagreements between pipelines.
• B. Compile speed per spec and throughput, cold and warm, median of 5 and min, with the load recorded per sample; t27b measured natively as an x86_64 cross compiler and calibrated against its qemu run.
• C. Generated code: code size, qemu run time of every test, and executed guest instructions (a qemu TCG plugin), on a set chosen by a rule that does not look at t27b.
• D. Compiler resources: max RSS, binary size, dependency footprint.
• E. Determinism: each pipeline built twice in one directory and once in another.
• F. Trust and verification: signed receipts, DDC, JIT/interpreter cross-check, differential fuzzing, self-hosting share, with public sources for the other toolchains.

Deliverables

• docs/reports/t27b-bench-2026-10.json: raw numbers, machine info, load per sample, tool versions and hashes, commit sha.
• docs/reports/t27b-bench-2026-10.md: method, tables, caveats, and a "where t27b loses" section.

Rules

• Only data and prose are committed. The measurement scripts stay in lab scratch directories (owner rule "only t27", see policy: hand-written code in any language but t27 may only shrink (Closes #8227) #8236).
• Everything runs on the Railway t27c lab; nothing is built or timed on the owner's Mac.
• Prior work on the same sieve axes: docs/reports/2026-10-09-sieve-c6-c7.md (see sieve C6/C7 (G6): measure t27b JIT warmup; publish compile speed and code quality against a reference compiler #7596, epic EPIC: golden reference language -- the compiler sieve (TNF-style theorems), and t27 passes it #6488).

User Scenarios

• Given the t27b compiler and four other compiler pipelines targeting aarch64-linux, When the benchmark suite runs across all six axes (A–F) on the Railway t27c lab, Then two deliverables are produced: a raw JSON report and a Markdown report with method, tables, caveats, and a prominent "where t27b loses" section.
• Given the corpus of specs in specs/, When each pipeline generates, compiles, and runs every test, Then per-test disagreements between pipelines are recorded in the raw data.
• Given the requirement for reproducibility, When the measurement is repeated (twice in one directory, once in another), Then each pipeline's outputs are bit-for-bit identical across runs.

Requirements

• FR-001: The benchmark MUST execute all six axes (A–F) as defined in the issue across the five pipelines targeting aarch64-linux.
• FR-002: The raw data MUST be written to docs/reports/t27b-bench-2026-10.json including machine info, load per sample, tool versions and hashes, and commit sha.
• FR-003: The narrative report MUST be written to docs/reports/t27b-bench-2026-10.md with method, tables, caveats, and a dedicated "where t27b loses" section.
• FR-004: All measurements MUST run on the Railway t27c lab; nothing is built or timed on the owner's Mac.
• FR-005: The measurement scripts MUST remain in lab scratch directories (owner rule "only t27", see policy: hand-written code in any language but t27 may only shrink (Closes #8227) #8236); only data and prose are committed.
• FR-006: Coverage and correctness (Axis A) MUST be measured over all of specs/ with per-test disagreements recorded.
• FR-007: Compile speed (Axis B) MUST report cold and warm median-of-5 and min throughput per spec with load recorded per sample; t27b measured natively as x86_64 cross compiler and calibrated against qemu run.
• FR-008: Generated code metrics (Axis C) MUST include code size, qemu run time, and executed guest instructions (via qemu TCG plugin) on a spec set chosen by a rule that does not look at t27b.
• FR-009: Compiler resources (Axis D) MUST capture max RSS, binary size, and dependency footprint for each pipeline.
• FR-010: Determinism (Axis E) MUST verify each pipeline built twice in one directory and once in another produces identical outputs.
• FR-011: Trust and verification (Axis F) MUST document signed receipts, DDC, JIT/interpreter cross-check, differential fuzzing, self-hosting share, with public sources for other toolchains.

Success Criteria

• cat docs/reports/t27b-bench-2026-10.json | jq -e '.axes | has("A") and has("B") and has("C") and has("D") and has("E") and has("F")' exits 0
• grep -q "where t27b loses" docs/reports/t27b-bench-2026-10.md exits 0
• jq -e '.pipelines | length == 5' docs/reports/t27b-bench-2026-10.json exits 0
• jq -e '.determinism.all_identical == true' docs/reports/t27b-bench-2026-10.json exits 0
• 5 tests in specs/igla/coder/benchmark.t27 pass under all pipelines with recorded disagreements
• docs/reports/t27b-bench-2026-10.json contains tool_versions and commit_sha fields
• docs/reports/t27b-bench-2026-10.md references docs/reports/2026-10-09-sieve-c6-c7.md for prior work

Boundary

• docs/reports/t27b-bench-2026-10.json
• docs/reports/t27b-bench-2026-10.md
• docs/reports/2026-10-09-sieve-c6-c7.md
• specs/igla/coder/benchmark.t27