fix(codegen): sample the corpus compile check from the baseline

[gHashTag]

Fixes #885. Part of #851, weak point W15. The generator crash it uncovered is #887.

Why

codegen-corpus on branch main was red 6 times from 09-24 to 10-04. Five of those failed on DOES NOT COMPILE: specs/tri/phi_utils_multi.tri with 0 spec(s) regressed:

• 36027587352;
• 36029076209;
• 37016445887;
• 37088537159;
• 37225448222, which is main's head 57de4a1f5.

The sixth, 36515512024, was the vsa_simd benchmark test.

Correction: an earlier version of this description said main was red 9 times. #885's table of 9 counts every ref: 5 on main and 4 on PR branches, all on phi_utils_multi. On branch main alone the API count (codegen.yml, branch=main) is the 6 above.

The compile sample was every 32nd entry of the run's clean list. phi_utils_multi sits at index 447, one before the slot at 448. Any one extra spec counted clean before it moves it into the sample, and its output genuinely does not compile.

Main's tool could count a spec clean without clean output in two ways:

• Missing file. zig ast-check prints error: unable to open file … and exits 1. That line has no : error: , and that string was all astCheckOk looked for.
• Empty file. zig ast-check passes a 0-byte file with exit 0, and so does zig test.

What changes

tools/codegen_corpus.zig:

• Sample from the baseline. Slot k is baseline entry k * stride, with the stride computed from the baseline length, so the sample depends on the commit and nothing else. A regressed entry is skipped, because it has already failed the gate.
• Strict clean. All three must hold:
  • the generator exits 0;
  • it leaves a non-empty regular file;
  • zig ast-check exits 0 and prints no : error: .
• The output file is deleted before each generation, so a stale file cannot stand in for a missing one.
• The output path mirrors the spec path. Flattening / to _ sent specs/storm/main.tri and specs/storm_main.tri to the same file.
• The counts are always printed, along with every REGRESSED, NEW CLEAN and NO OUTPUT spec by name.
• NO OUTPUT says why: the exit status or signal, no file or empty file, and the generator's crash line (a panic: line or a crash handler's Segmentation fault …), else its last error line, else its last line.
• Three unit tests:
  • parseBehaviours;
  • outPathFor is one-to-one;
  • describeFailure.

tools/codegen_corpus_baseline.txt, 788 → 769. The 19 entries removed never wrote a usable .zig:

• 18 are language: varlog specs. The generator aborts with SIGABRT after creating the file, leaving 0 bytes, and main's tool passed them through the empty file.
• specs/tri/long_context_system.vibee aborts and leaves no file in 59 of 60 runs.

phi_utils_multi moves from index 447 to 432. The slots are multiples of 32, so it is still not sampled.

.github/workflows/codegen.yml prints the runner's CPU model.

Every added if tests a named bool.

Measured

A/B, same four arms on one runner per tool (the exp/corpus-ab-* branches carry an identical helper and workflow):

arm	main's tool, 37243529377	this PR's tool, 37243526975
1 plain tree	green, 788	green, 769
2 one extra clean spec before phi_utils_multi	red, DOES NOT COMPILE phi_utils_multi	green, NEW CLEAN names it
3 a non-baseline spec before it cannot write	red, the same	green
4 a baseline spec (specs/algo/adam.tri) cannot write	green: hidden	red, REGRESSED: specs/algo/adam.tri

The earlier main-arm run 37242996093 gave the same four results.

Probe 37242067946 (exp/codegen-flake-ab): 60 passes over all 1033 specs with behaviours, removing the destination before each generation.

• All 19 removed entries end in signal 6. The 18 Verilog specs leave an empty file every time.

• Two specs vary between passes:

  spec	in baseline	result
  specs/tri/long_context_system.vibee	yes	aborts in 59 of 60 passes
  specs/tri/hdc_stream_classifier.vibee	no	aborts with no file in 3 of 60 passes; otherwise writes the same output

• Compile-all: 13 of the 769 written outputs pass ast-check and fail zig test. 9 of them lack the vsa module. None falls in the sample; the details are in vibee_gen aborts (SIGABRT) on 18 Verilog specs and intermittently on 2 Zig specs #887.

This PR's CI:

• 02b9958de (run 37243448031): 769 of 1033 specs with behaviours clean, baseline 769, 0 regressed, 0 new clean, 19 reported behaviours and left no file, and the sample compiled 24 of 24. Every NO OUTPUT read signal 6, no file, ???:?:?: 0x1571044 in ??? (???): a stack frame with no symbols, which says nothing. d959eb722 makes the panic line win.
• d959eb722 (run 37243963781): the same counts. All 18 Verilog specs read panic: programmer bug caused syscall error: FAULT, still labelled no file. long_context_system printed no panic line, so it still showed a frame.
• b937408ce (run 37244594555): the same counts, 0 regressed, and the sample is clean. The 18 now read signal 6, empty file, thread … panic: programmer bug caused syscall error: FAULT. long_context_system reads signal 6, no file, Segmentation fault at address 0x7fee7103b2a2.
• orphan-artefacts and doc-refs are red here. They fail on gft_mul: the product wire is 32 bits and GF-T32 needs 52 #510's research/arxiv_gft files and gft references, and are red on main at 57de4a1f5 as well (37225448159, 37225448197); fix(gates): green doc-refs and orphan-artefacts on main after #510 #872 fixes them. Of the 64 checks on b937408ce at about 23:48Z on 2026-10-04, those 3 failed (orphan-artefacts ran twice), 55 passed, 5 were skipped and 📊 Export Brain Metrics was still running.

Likely natural trigger on main (inference, not proven)

hdc_stream_classifier is not in the baseline, reports behaviours, and sorts before phi_utils_multi. When it aborts without a file, main's tool counts it clean. That shifts phi_utils_multi onto slot 448, which is arm 3 above, occurring naturally.

An abort rate of 3 in 60 is consistent with 5 phi-red runs out of 66. But main's tool printed no names on red, so no past run can confirm it. This PR removes the mechanism whatever the trigger was.

Not measured

• Why the two Zig-target specs abort. The Verilog path's abort is measured (an EFAULT panic in the write, vibee_gen aborts (SIGABRT) on 18 Verilog specs and intermittently on 2 Zig specs #887). long_context_system segfaults (run 37244594555), but where is not known; hdc_stream_classifier did not abort in any of this PR's 4 codegen runs (19 NO OUTPUT lines each).
• Whether the runner's CPU matters. The workflow now prints it; no comparison has been made.

Checked locally

• zig fmt --check passes.
• zig test passes 3/3 against tri_io/tri_proc/tri_env from origin/main.
• Four mutants of describeFailure, each caught by its own failing assertion:
  • taking the last line before an error line;
  • ranking an error line above a panic line;
  • ignoring crash-handler lines;
  • calling an empty file no file.

🤖 Generated with Claude Code

[github-actions]

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

[github-actions]

🧪 Brain Stress Test Results

Score: not measured (no stress test prints a Score: line)
Status: 🟢 PASS
Threshold: 270/300

X Stress test passed

[gHashTag]

#873 and #886 conflict with each other in tools/codegen_corpus.zig. Each merges clean onto main by itself, so neither PR's checks can see this. Whichever merges second will conflict.

How it was found: a trial merge of the 20 open loop PRs onto main 57de4a1, in the proposed order, done in git's object store only (git merge-tree --write-tree + commit-tree; no ref written). Result: 18 merged clean, 2 conflicted. One is #897 × #852 (tnf-cost-sweep.yml, already known). The other is this pair, and it was new. The check is now speed mergeorder in the loop skill.

A tested resolution is on exp/codegen-873-on-886 at 3d94723. It is #873's d02f21e replayed onto #886's b937408. The two PRs edit the same three places:

• sample and stride come from fix(codegen): sample the corpus compile check from the baseline #886 (the committed baseline). fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart #873's could_not_run is declared next to them.
• The failure message keeps fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart #873's four counts and uses fix(codegen): sample the corpus compile check from the baseline #886's denominator: picked.items.len, meaning the slots that were still clean and were compiled.
• The COULD NOT RUN message uses picked.items.len too. fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart #873 used sample, which also counts slots dropped as regressed.

Verified:

• Local, Zig 0.16.0, with the shim modules from main:
  • zig test tools/codegen_corpus.zig: 7/7, both PRs' tests, including their positive and negative controls;
  • zig build-exe links;
  • zig fmt --check is clean.
• CI, codegen.yml dispatched on the exp branch, run 37269657437, green:
  • 769 of 1033 specs clean against a baseline of 769;
  • 0 regressed;
  • 19 reported behaviours and wrote no usable file;
  • compile sample: 24 slots, stride 32, and all 24 compiled.

Not verified: no COULD NOT RUN or DOES NOT COMPILE line fired in that run, so the merged failure messages have run only in the unit tests, not in CI.

When the first of the two merges, the second can take 3d94723's tools/codegen_corpus.zig as its resolution.

Refs #851, #871, #885.

🤖 Generated with Claude Code

[tri-claw]

Summary by t27-bees

The PR fixes a flaky codegen-corpus gate by making the compile sample deterministic (drawn from the committed baseline instead of the run's live clean list), tightening the definition of "clean" (generator exit 0 + non-empty file + zig ast-check passes), deleting stale output files before each generation, and mirroring the spec path in the output to avoid collisions. It also removes 19 baseline entries that never produced usable output and adds diagnostic logging.

Worth a look

• tools/codegen_corpus.zig:176 – outPathFor now returns {out_dir}/{spec}.zig (e.g., .zig-cache/codegen-corpus/specs/tri/foo.tri.zig). The comment says "The generator creates the parent directory itself." If the generator does not create nested directories, every spec in a subdirectory will fail with "no file" and the gate will turn red.
• tools/codegen_corpus.zig:293–302 – generate deletes the destination file before invoking the generator (deleteFile catch {}). If the generator itself cannot create parent directories (see above), the deletion succeeds but the subsequent write fails, and the failure is recorded as "no file" — the gate will correctly report NO OUTPUT instead of silently passing a stale file.
• tools/codegen_corpus.zig:340–380 – describeFailure scans stderr for a panic line, a crash-handler line, or an error line. It assumes the last such line is the most informative. For a Zig panic the panic line appears before the stack trace, so last_crash captures it correctly; for a segfault the handler line appears before the frames. The fallback to the last non-empty line is reasonable.
• tools/codegen_corpus.zig:130–136 – Baseline loading builds both an ArrayList (for ordered sampling) and a StringHashMap (for O(1) regression checks). If the baseline file contains duplicate lines, the hashmap deduplicates but the arraylist does not, so base.items.len could exceed the unique count and the stride calculation base.items.len / sample would be slightly off. The baseline is maintained by the tool itself, so duplicates are unlikely.
• tools/codegen_corpus.zig:487–496 – astCheckOk now requires both exit code 0 and no : error: in stderr. Previously a missing file (exit 1, stderr "error: unable to open file…" without : error: ) was incorrectly treated as clean. This change is correct and closes the gap that let phi_utils_multi slip through.

---

_{Read: 3 files, +288 -48, the whole diff. Head 728130a. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.}