fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart

[gHashTag]

What

On main, zigTestOk in tools/codegen_corpus.zig drops the reason on both of its failure paths:

• catch return false reports a spawn error as "does not compile". So does stderr overflowing the 512 KiB cap.
• A nonzero exit frees r.stderr without printing it.

Because of this, the phi_utils_multi failure on main (run 37225448222) cannot be diagnosed from its log. Issue #871 has the details.

This PR:

path	before	after
zig test exits nonzero or is killed	DOES NOT COMPILE: <spec>, and nothing else	the same line, then how it ended and the last 40 lines of zig's stderr, prefixed |
zig test cannot be started, or its output overflows	DOES NOT COMPILE: <spec>	COULD NOT RUN zig test: <spec> (<ErrorName>), counted separately
verdict	fails on regressions or compile failures	unchanged for those. A sample with only could-not-run entries also fails, as CompileSampleNotRun rather than CodegenCorpusRegressed, because the compile check did not happen

The ast-check pass and the sample selection are not touched.

Measured

These are local runs on Zig 0.16.0, with the shim modules taken from main (src/tri/tri_proc.zig, tri_io.zig, tri_env.zig) and passed as -M modules:

1/4 codegen_corpus.test.lastLines keeps the end, where zig puts the cause...OK
2/4 codegen_corpus.test.a type error is DOES NOT COMPILE (positive control)...  DOES NOT COMPILE: bad.zig (ast-check passed it)
    zig test exited 1; last lines of its stderr:
    | .zig-cache/tmp/CkqJyOs5DYb14zaD/bad.zig:1:22: error: expected type 'u8', found '*const [2:0]u8'
    | test { const x: u8 = "no"; _ = x; }
    |                      ^~~~
OK
3/4 codegen_corpus.test.a file that compiles is compiled...OK
4/4 codegen_corpus.test.a missing compiler is COULD NOT RUN, not DOES NOT COMPILE (negative control)...  COULD NOT RUN zig test: unused.zig (FileNotFound)
OK
All 4 tests passed.

• On main, both of these cases would print the same single DOES NOT COMPILE line. That comes from reading lines 287 and 290–293; I did not run main's tool to confirm it.
• zig build-exe of the tool links cleanly.
• zig fmt --check is clean.

Not measured

• This PR's own CI run. codegen.yml runs zig build codegen-corpus on every PR to main, so it exercises the normal path end to end. Results follow in the checks.

• Why phi_utils_multi failed in run 37225448222. That becomes readable the next time the failure recurs, not before.

• The new tests do not run in CI. They are not wired to any build step. The tool imports named modules, so a step would need the same -M wiring that astcheck_ratchet has in build.zig. That is a separate change, left out to keep this diff to one file. To run them by hand:

  zig test --dep tri_io --dep tri_proc -Mroot=tools/codegen_corpus.zig \
    --dep tri_env --dep tri_io -Mtri_proc=src/tri/tri_proc.zig \
    -Mtri_io=src/tri/tri_io.zig -Mtri_env=src/tri/tri_env.zig -lc
  

Closes #871. Refs #851.

🤖 Generated with Claude Code

[github-actions]

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

[github-actions]

🧪 Brain Stress Test Results

Score: not measured (no stress test prints a Score: line)
Status: 🟢 PASS
Threshold: 270/300

X Stress test passed

[gHashTag]

#873 and #886 conflict with each other in tools/codegen_corpus.zig. Each merges clean onto main by itself, so neither PR's checks can see this. Whichever merges second will conflict.

How it was found: a trial merge of the 20 open loop PRs onto main 57de4a1, in the proposed order, done in git's object store only (git merge-tree --write-tree + commit-tree; no ref written). Result: 18 merged clean, 2 conflicted. One is #897 × #852 (tnf-cost-sweep.yml, already known). The other is this pair, and it was new. The check is now speed mergeorder in the loop skill.

A tested resolution is on exp/codegen-873-on-886 at 3d94723. It is #873's d02f21e replayed onto #886's b937408. The two PRs edit the same three places:

• sample and stride come from fix(codegen): sample the corpus compile check from the baseline #886 (the committed baseline). fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart #873's could_not_run is declared next to them.
• The failure message keeps fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart #873's four counts and uses fix(codegen): sample the corpus compile check from the baseline #886's denominator: picked.items.len, meaning the slots that were still clean and were compiled.
• The COULD NOT RUN message uses picked.items.len too. fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart #873 used sample, which also counts slots dropped as regressed.

Verified:

• Local, Zig 0.16.0, with the shim modules from main:
  • zig test tools/codegen_corpus.zig: 7/7, both PRs' tests, including their positive and negative controls;
  • zig build-exe links;
  • zig fmt --check is clean.
• CI, codegen.yml dispatched on the exp branch, run 37269657437, green:
  • 769 of 1033 specs clean against a baseline of 769;
  • 0 regressed;
  • 19 reported behaviours and wrote no usable file;
  • compile sample: 24 slots, stride 32, and all 24 compiled.

Not verified: no COULD NOT RUN or DOES NOT COMPILE line fired in that run, so the merged failure messages have run only in the unit tests, not in CI.

When the first of the two merges, the second can take 3d94723's tools/codegen_corpus.zig as its resolution.

Refs #851, #871, #885.

🤖 Generated with Claude Code

[tri-claw]

Summary by t27-bees

This PR splits the "does not compile" outcome into two cases: a real compile failure (nonzero exit) now prints the last 40 lines of stderr so the cause is visible, while a compiler that cannot be spawned or whose output overflows the 512 KiB cap is reported and counted separately as "COULD NOT RUN". A new CompileSampleNotRun error is returned when only the latter occurs.

Worth a look

• tools/codegen_corpus.zig:303–310 — The catch block treats any error from tri_proc.runIo as "could not run", including the max_output_bytes overflow. A genuine compile failure that emits >512 KiB of stderr would be misclassified as "could not run" and its error text would be dropped.
• tools/codegen_corpus.zig:193 — The error message format string \\{d} spec(s) regressed, {d} of {d} sampled outputs do not compile,\\{d} could not be compiled at all. is missing a space after the comma before the third {d}, producing run-together output like ...do not compile,5 could....
• tools/codegen_corpus.zig:338–360 — lastLines splits only on \n; if zig test ever emits \r\n (e.g., on Windows), the line count and tail selection could be off.
• tools/codegen_corpus.zig:324–335 — The switch (r.term) prints the exit code or signal, but r.stderr is freed by the defer on line 316 after this block runs, so the pointer is still valid here. Correct, but worth confirming the defer ordering survives future edits.
• tools/codegen_corpus.zig:207 — The new error.CompileSampleNotRun is returned when could_not_run > 0 and there are no regressions or compile failures. Callers of this tool (e.g., CI scripts) must handle this distinct error code; the diff does not show those call sites.

---

_{Read: 1 files, +105 -12, the whole diff. Head 5dde66e. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.}