Skip to content

EPIC: t27b runs the whole corpus -- 100% of reference-passing specs, 0 mismatches, experiments on Railway #6063

Open
12 of 13 issues completed

Description

@gHashTag

Goal

t27b (the native AArch64 backend, cli/t27b) runs every corpus spec whose reference path (t27c gen + zig test) itself passes, with 0 mismatches against that reference. No skipped tests, no interpreter fallback: an unsupported construct is a counted blocker, not a pass.

Progress is the honest ratio t27b passes / reference passes, measured by t27b corpus, never estimated.

Where we are (2026-10-04)

PR What Files passing Mismatches
#5979 (merged) MVP 37 / 1174 0
#5992 invariant blocks run like tests 48 0
#6007 memory primitives (frame slots, loads/stores, rodata, bounds) 47 0
#6044 structs, pointers, struct results via hidden pointer, rodata struct consts 70 / 1185 0

Merge order: #5992 -> #6007 (retarget to master) -> #6044 (retarget to master). Open bug PR #5993 (zig signed %, Closes #5973) is CONFLICTING and needs a rebase.

Reference-path baseline: build_verify.t27 fails on the reference path too (33 vs 31), so it is outside the denominator.

Remaining blockers (t27b corpus --blockers, greedy unlock order)

  • Memory lane: str 389 files, []const u8 71, []const f32 30, fixed arrays [N]T, for over slices.
  • Scalar lane: EnumDecl 104, f64 53, ExprCast 26.
  • Unassigned: BenchBlock 48, std.mem.Allocator 30.

Sub-tasks

  • Railway lab: run the corpus off the Mac (cross-build t27b for aarch64-linux, run it under qemu-user; reference path natively), publish latest.json over HTTP -- no secrets in the service.
  • t27b JIT on Linux AArch64 (today it is gated to macOS: MAP_JIT / pthread_jit_write_protect_np).
  • Memory lane: fixed arrays, then strings, str, slices, for (the last PR of this lane carries Closes #5977).
  • Scalar lane: enums, f64, casts.
  • BenchBlock and Allocator.
  • README: coverage number from t27b corpus, not by hand.

Rules for every agent on this epic

  • One construct family per PR, Refs #<this epic> and its own issue; NOW entry per PR; L1-L7.
  • A PR states its before/after t27b corpus numbers and its mismatch count; a mismatch is a stop, not a skip.
  • Heavy builds and corpus runs go to the Railway lab, not the owner's Mac.
  • Progress reports are posted as comments here; a number in a report links the run that produced it.

Activity

  1. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward tick 2026-10-04T13:30Z

    Merged

    Open

    Numbers

    • Railway lab: there is no lab URL in this epic yet, and on this Mac railway CLI says Unauthorized. So I have no lab totals, mismatch list or top 5 blockers to report this tick.
    • Local reference-denominator run (t27b corpus specs --blockers --reference ... --list, pid 66006): 900 of 1185 files after 11284 s, still running. Its totals are not out yet.
    • Load average: 730 / 713 / 715.

    Next step / owner decision

  2. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward tick 2026-10-04T14:26Z

    Merged this tick: none.

    Open

    Railway lab

    Local reference run (pid 66006): 1100 of 1185 files after 14555 s, still running. I am not quoting totals until it finishes.

    Load average: 430 / 399 / 434.

    Next step

  3. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    t27b lab is live on Railway (#6071, PR #6098).

    T27_REF currently points at claude/t27b-railway-lab. Set it to master once #6098 merges.

  4. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward tick 2026-10-04T15:27Z

    Merged

    Open

    Lab (latest.json; ref claude/t27b-railway-lab f59fd76, which is the branch, not master)

    • t27b passes 47 / reference passes 646 of 1190 files. Mismatch 0, crash 0, timeout 0. Reference also has fail 25, timeout 3 and blocked 516.
    • Top first-rejecting constructs: StructDecl 393, string literal 308, EnumDecl 89, BenchBlock 35, str 31. The base predates feat(t27b): structs, field access, pointers and struct results #6044, so StructDecl is overstated.

    Load 171 / 213 / 225 (1/5/15 min)

    Next: merge #6098 and #6065 when they are green, then set T27_REF=master on the lab so the numbers are master's. After that: the shift PR, then the memory-lane PR.

  5. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward 2026-10-04T16:15Z (the owner asked for everything to be merged and for the Queen to get priority)

    Merged: #6098 (Railway lab, corpus --json with reference verdicts) and #6065 (integer casts). Each merged after validate, check-linked-issue and parse-ratchet passed. A sweep is now merging every open t27b PR once those three checks pass and it has no red check that is green on master. No --admin.

    Lab is on master: I set T27_REF=master. The first run was latest.json, commit dd3864d. That commit is before #6065; the lab polls master and will re-run on its own.

    • t27b passes 56 of the 648 specs the reference path passes (8.6%). It also passes 14 specs where the reference does not pass.
    • Mismatch 0, crash 0, timeout 0.
    • The single t27b failure is specs/fpga/verification/build_verify.t27, which fails on the reference path too ("1 of 11 tests fail").
    • Top first-rejecting constructs: str 390, []T 181, EnumDecl 104, [N]T 60, f64 53, BenchBlock 49, std.mem.Allocator 30, ExprCast 26 (casts landed in feat(t27b): compile integer casts natively, checked like @intCast #6065 after this run), << >> 24.

    Handed to the Queen (her bees run on Railway, not on the Mac):

    All three cards sit in her backlog. Their boundaries overlap in cli/t27b/src/lower.rs, so she will run them one at a time. Strings and slices (str, []T) are with the local memory-lane subagent. std.mem.Allocator needs a heap/runtime design decision first, so it has not been delegated.

  6. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward session report -- weak spots, prior art, tooling (2026-10-04 16:17Z)

    Done

    Doctor, live at 15:44Z (5 anomalies; since then the lab is on master)

    LAB-NOT-MASTER (lab on claude/t27b-railway-lab), LAB-BEHIND (f59fd76 vs master cedb826), PR-CONFLICTING #6098 (a merge-from-master subagent from the 15:27Z tick is alive, so no second one was sent), WORKTREE-DIRTY /tmp/t27b-pr (1 path, nobody inside, 233 min old; left untouched), RAILWAY-OLD-CLI. Load average 250 then, 42 at 16:17Z.

    Weak spots of the metric (from the sweep)

    1. "N of M" hides regressions that cancel out -> per-spec ledger (t27b: per-spec ratchet ledger + vacuous-pass count, so the coverage number can only move visibly #6115).
    2. Vacuous passes (asserts the compiler folds at compile time) -> count the asserts that actually run (t27b: per-spec ratchet ledger + vacuous-pass count, so the coverage number can only move visibly #6115).
    3. Bugs in the reference path itself (build_verify.t27) -> the reason reference-bug in the ledger.
    4. Overflow, shift and float-format semantics are not written down -> rules first, then mismatches count.
    5. Results under qemu emulation -> occasionally re-check natively.
    6. The denominator moves with master -> every report names its commit.

    Next (ranked, skill section 6)

    #6115 ratchet + vacuous -> auto-reduce each mismatch into a regression test (cvise, with "reference passes" in the interestingness test) -> grow tests/differential.rs into a whitelisted, checksum-ending fuzzer on the lab -> written UB rules -> a native re-check -> a per-function fallback as a second metric (the headline number stays strict).

    Three ways to work together for the next loop (owner picks)

    • A. Steward only (the default). The hourly tick keeps two coverage lanes (memory: strings and slices; scalar: shift, enums, f64), merges green PRs, and reports. The owner only answers questions on the epic. The slowest path, and the safest.
    • B. Steward + quality lane. One lane is given to t27b: per-spec ratchet ledger + vacuous-pass count, so the coverage number can only move visibly #6115 and then the reducer, so every later coverage PR is measured per spec and every mismatch becomes a test. Coverage speed drops for about a day; afterwards the number cannot silently lie.
    • C. Lab-first. feat(t27b): JIT on arm64 Linux, corpus --json/--runner, Railway lab #6098 is merged and T27_REF=master; next step is branch runs on the lab (each PR measured there before merge). All corpus runs move off the Mac (load 200-800), and lanes run three at once because only the lab builds. This needs one owner decision: allow a third lane.
  7. added a commit that references this issue on Oct 4, 2026
  8. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward report -- t27b tooling moves into t27 (2026-10-05)

    Owner's rule, now law

    "t27b and the tools that measure it are written in t27." The rule is now an AGENTS.md section, #6200 merged 6569e48 (Closes #6199). The port epic is #6198. Every hand-written line of t27b and its tooling is recorded as debt that only shrinks (counts in that section, wc -l at c532fca).

    First port slice: #6204 merged bae81ae (Closes #6202)

    • specs/tri/t27b/steward.t27 holds delta_code, delta_is_red and pct_tenths: 14 tests, 1 invariant.
    • t27c test-report: 14/14. t27b itself (t27b test --check): 14/14, 19 runtime asserts, the invariant held. t27b compiles a piece of its own tooling.
    • tri t27b delta and the status percentage now call gen/c/tri/t27b/steward.c (t27c gen-c output, byte-identical, never hand-edited) through ctypes. Python only maps the lab's strings to codes.
    • Same answers: the old and new delta --json over lab runs dd3864d -> e0f894f give the same 28 findings (identical sha256).
    • The mutation control caught a weak test: 56/648 and 75/657 come out the same whether you round or truncate. I added pct_rounds_to_nearest; the mutant now fails.
    • New CI step in loop-tools-gate: scripts/ci/test_the_t27b_steward_decides_in_t27.py.

    Also merged this session

    #6185 (honest ratio + delta), #6187 (?T; corpus ratchet retires 2 entries, cap 113 -> 111), #6188 (per-spec ledger docs/reports/t27b_expectations.json, tri t27b ratchet, pass_vacuous, lab ratchet step). The ledger was blessed from e0f894f: 75 pass / 582 not pass. With steward.t27 it is 76.

    Anomalies

    1. Lab run f033b20: the t27b corpus died with exit -9 after 64.8 s and wrote no JSON, so every t27b verdict reads not run. On the Mac, master c532fca runs the full corpus in 2.16 s, max RSS 14 MB, rc 0 (45 pass, 38 pass_vacuous, 1 test fail build_verify, 0 mismatch, 1194 files). It does not reproduce; most likely the container killed it. The lab is idle and has not run a newer master. Owner: trigger a run or redeploy the lab. The steward may not.
    2. My slip, rule Q12 again: adding a workflow step moved the census. It was re-blessed in the same PR. Skill rule Q15: steward logic is written as a .t27 spec first.
    3. Master is red on four non-required gates from other merges, not from t27b: types ratchet (BufgConfig/CounterConfig/CounterState from the openxc7 port specs), gate-topology (wasm-explorer.yml does not parse, Nothing builds bindings/wasm-explorer, which is how the artifact it replaces drifted in the first place #4498), duplicate-bodies, and the seal on kanban-card-chat (spec(automation): kanban-card-chat v7 -- the bee's own log on a running card, live #5680). All four reproduce on a clean 03c492f.

    Last honest number

    Lab e0f894f: t27b passes 75 of the 657 specs the reference passes (11.4%), mismatch 0; 47 of the 75 ran a test or invariant, 28 are compile-only. Run: https://t27b-lab-production.up.railway.app/latest.json

    Three ways to work together for the next loop (owner picks)

    Owner questions

    1. Should compile-only and pass_vacuous passes count toward 100%?
    2. A lab redeploy or run (anomaly 1).
    3. A, B or C.
  9. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward tick -- slice 3 merged, lab stuck at checkout

    Lab: needs the owner

    /latest.json (updated 17:55Z) is still the failed run at bae81ae. Checkout failed with could not fetch 737282229a... from promisor remote: the redeploy killed the previous run in the middle of a partial-clone fetch. The running container predates #6220, so it does not re-clone. One more redeploy of t27b-lab picks up #6220. After that, steps.checkout.clone should read recloned once and then kept. I am not redeploying: that is your decision.

    No lab numbers for master after #6065/#6229 exist yet. The last measured figure is still 56/648 at dd3864d.

    Remaining #6198 debt

    tri t27b doctor and status reading are still Python. Next slice: the anomaly codes go into steward.t27.

  10. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Correction + new lab numbers (18:40Z)

    Correction to my 18:18Z comment: I said the lab needed one more redeploy to recover. It did not. At 18:26Z the lab polled, saw the new master 88862a9, and its checkout succeeded on the old code (steps.checkout: ok, 4.6 s). The partial-clone failure was tied to the old target commit, not to the clone itself. #6220 is still a guard for the next time, but no redeploy is needed now. Lesson recorded in the skill: do not call the owner for a redeploy before the lab's next poll on a new commit.

    Lab run 88862a9 (finished 18:32:47Z, https://t27b-lab-production.up.railway.app/runs/88862a9499d239074622533ade539852fcba285b.json):

    Open:

  11. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward, 2026-10-04 20:15Z

    Merged: #6235 (ledger re-bless), #6242 c5406e6 (bless accounts for a cap rise; doctor alarms only when the reference passes).

    Coverage stack. Each PR is stacked on the one before. The numbers are local t27b corpus specs runs on macOS arm64, not the lab:

    PR family pass mismatch first-blocker files cleared
    #6247 [N]T fixed arrays 84 -> 161 0 --
    #6249 []T slices 161 -> 162 0 []T 136 -> 0
    #6254 plain enums (Closes #6146) 162 -> 165 0 EnumDecl 109 -> 0
    #6258 bench blocks (Closes #6179) 165 -> 169 0 BenchBlock 116 -> 0
    • Vacuous passes: pass_vacuous rose 38 -> 278 with feat(t27b): fixed-size arrays [N]T -- literals, indexing, .len, for (Closes #6245) #6247 and then stayed flat. These are files where no assert runs at run time. They are not coverage and are not in the pass column.
    • New passes: the reference agrees on all of them except one. server/api.t27 (in feat(t27b): plain enums as integer tags (Closes #6146) #6254) is not compiled by the reference: an unused struct names an undeclared type.
    • Bench (judgment call): a bench is compiled and never run, the same as the reference, which emits an uncalled fn. Prose-only bench bodies (22 files) now reject as StmtExpr instead of passing empty.
    • Next first blockers: f64 110, f32 70, ?T 47, std.mem.Allocator 44, [N]T leftovers 39, shifts 36.

    Gates fixed this round:

    t27c reference bugs filed: #6250 (x[i..] -> x[i], &[_]T{} -> &.{ _ }), #6255 (const D: E = .v; loses its value).

    Lab: the last lab run is still master 88862a9: t27b 84 / reference 708 (11.9%), mismatch 0. The stack is measured there after it merges. A ledger re-bless follows in its own PR.

    Merges run only through tri t27b ready = READY. Order: #6247 -> #6249 -> #6254 -> #6258; #6253 -> #6246.

    🤖 Generated with Claude Code

  12. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Steward, 2026-10-04 20:35Z

    Merged since the last report: #6247 (fixed arrays [N]T), #6253 (L2 gate is regen-aware, Closes #6226).

    Lab (https://t27b-lab-production.up.railway.app/runs/99b465ec2.json), as tri t27b status prints it:

    • ref master @ 99b465e, finished 2026-10-04T20:22:35Z
    • t27b passes 162 of the 710 specs the reference passes (22.8%), mismatch 0 (1247 files)
    • of those 162, all ran a test or invariant; 0 compile-only
    • not counted: 0 t27b passes where the reference fails, 1 frontend reject where it passes (specs/tri/agent/faculty_board.t27, a t27b parser/typecheck defect still to be filed)
    • tri t27b delta 77e5c44 -> 99b465e: in-reference passes 85/710 -> 162/710, checked 85 -> 162; GAINED only, no REGRESSED / NEW-MISMATCH / CHECK-LOST.
    • The lab is one commit behind master (82d8795, a spec port, no t27b change), so these numbers belong to 99b465e.

    Top 5 first blockers (files): type []T 136, EnumDecl 108, type f64 79, BenchBlock 70, type std.mem.Allocator 39. The open stack below covers the first, second and fourth.

    Open PRs (tri t27b ready):

    Load: 8.02 / 9.40 / 13.66.

    Next: merge #6249 -> #6254 -> #6258 in order as each reaches READY; then f64 (79) on the scalar lane and std.mem.Allocator (39) on the misc lane.

  13. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    t27b steward -- 2026-10-04 21:40Z

    Merged (all through tri t27b ready, gh pr merge --merge, required checks green):

    Lab -- https://t27b-lab-production.up.railway.app/runs/cfff5e45309e7803df22a2886ff791285dbed32a.json (master cfff5e4, finished 21:36:24Z):

    • files 1249, reference pass 712, t27b pass 170, of which 169 where the reference passes and 1 where it does not
    • mismatch 0, t27b fail 2, crash 0, timeout 0
    • previous measured master 88862a9: t27b 84 / reference 708; then b2f9dc5: 163 / 711

    Top blockers (first rejection): type f64 110, type f32 70, type ?T 47, std.mem.Allocator 44, [N]T leftovers 39, << >> 36.

    Next: f64 lane started on #6180 (one lower.rs lane). New issue #6268: 8 specs whose bench bodies Zig rejects once referenced (found by the bench lane; lazy analysis hides them).

    Slip learned (Q30): a watcher skipped the retarget of #6254 because one ready read failed; watchers now decide the retarget from gh pr view --json baseRefName.

  14. 148 remaining items

  15. gHashTag commented on Oct 10, 2026

    @gHashTag
    OwnerAuthor

    Steward tick 2026-10-10 ~09:40Z (claim pid 28907; loop skill t27b-loop, epic #8326)

    Lab line, run 1830512a2 (finished 09:21:37Z):

    t27b passes 1213 of the 1286 specs the reference passes (94.3%), mismatch 0 (1767 files); of those 1213 ran a test or invariant, 0 compile-only; not counted: 1 t27b pass where the reference fails (#8485); reference reused 1760 of 1767 from the cache.

    Landed this tick: #6620, #6623 (stale t27b test-block PRs, merged), #8471 (ledger rows for the 9 specs that landed since a97cb9e5; #8322 closed), #8484 (re-bless from 285512a21, pending). Queue cleanup: closed 7 PRs superseded by master with lab evidence (#6721, #6733, #6890, #6901, #6881, #8107, #8168) and their issues (#6470, #6471, #6476, #6479, #6481, #8060, #8061) plus #8359 (fixed by #8375).

    Remaining gap to 100%: 73 specs.

    Also in flight: #8181 (sieve C9 rebuild after master merge), #8217 (bench bodies, master re-merged, checks re-running), #8218 rebuilt clean as bless-by-key-3 (the old branch history swallowed unrelated commits — ~1495-file two-dot diff; will be closed in favor of the new PR), #8216 remains owner-only (gate file).

    Next tick: land the batches, re-bless the ledger from the new run, report the family queue.

  16. gHashTag commented on Oct 10, 2026

    @gHashTag
    OwnerAuthor

    Lab run on master c494cc5 finished (12:29Z): t27b 1254/1286 = 97.5% of reference-passing specs covered by real t27b passes (up from 1213/1286 = 94.3% this morning), mismatch 0, reference_disagree 0, jit/interp mismatch 0, crash 0.

    Gain comes from today's test-block batches (#8539 repairs, #8496, #8507, #8181, #6816, #8498, #8533, #8541) — real runtime asserts, not vacuous: t27b_pass_vacuous_where_reference_passes is down to 5.

    Remaining tail at this commit: 21 t27b_fail, 4 timeouts (5 retried), 2 pass-where-reference-does-not (outside the mismatch counter, #8485 class), 5 vacuous. The blocked-spec family map (i128 #7915, anytype #7878, anonymous struct #7886, generic #7890, Allocator #7917, @sizeof #7817, ExprArrayLiteral #7347/#8175 + #7382 in flight) is unchanged.

    Ledger re-bless follows once the reseal PR (#8564) merges and the next master run lands — trinet_node's j_go test now passes, so its row moves too.

  17. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    Final tick — t27b reference-parity, measured on master a9ad525 (native arm64 run of #9068's merge tree + targeted lab deltas, fresh reference cache):

    rows 1881   reference_pass 1435   t27b_pass 1407   gap 28   crash 0   per-test disagree 0
    

    Lanes landed this epic: #8965 (std.mem.trim/indexOf/indexOfScalar, std.math.approxEqAbs, @sqrt comptime fold — PR #9068) on top of the earlier stdlib/optional/cast lanes.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/compilerHIR/IR/codegen/optimizerepicowner-approved-foreignOwner-approved exception to the only-t27 rule: hand-written foreign code allowed in this PR

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions