codegen-corpus: compile sample moves with the run, so one commit is red and green (#851 W15) #885

codegen-corpus goes red for a reason the commit does not decide

Part of #851, weak point W15: a gate that fails or passes for a reason other than its claim.

What was measured

The red is always the same. I read the logs of the nine Codegen Validation failures from 09-24 to 10-04, on every ref: 5 on main and 4 on PR branches. All nine print the same two lines:

  DOES NOT COMPILE: specs/tri/phi_utils_multi.tri (ast-check passed it)
0 spec(s) regressed, 1 of 24 sampled outputs do not compile.

One commit, both verdicts. On e6eac0900, push run 37088537159 is red and schedule run 37090643092 is green. Main 57de4a1f5 is red on push (37225448222), while PR runs on the same spec tree are green.

The output really does not compile. Its compile errors, from the probe run below:

• compute_phi_power(n: u32) PhiResult has a stub body that falls off the end;
• the generated test calls encode_to_trits(allocator, "test"), but the function takes one parameter.

That is a generator defect and is tracked separately. The question here is why it is only sometimes sampled.

Why the sample moves

tools/codegen_corpus.zig compiles 24 outputs: every stride-th entry of this run's clean list, with stride = clean.len / 24.

• Green runs report 788 clean, so stride = 32. The picks are indices 0, 32, …, 736.
• phi_utils_multi is baseline line 448, which is index 447. The slot at 448 is one past it.
• If one spec that sorts before it counts as clean when the baseline does not hold it, phi_utils_multi moves to index 448 and is sampled.
• 0 spec(s) regressed means every baseline entry was still clean, so the red runs had at least 788 clean.
• Inference, not measured: the red runs had 789–791 clean, with exactly one extra entry before phi_utils_multi. The tool never prints its clean count on the failure path, so the red runs did not record it.

A false clean that this can come from (measured locally):

• vibee_gen prints Behaviors: N before it generates.
• When it cannot create or write the file, or when it produced 0 bytes, it leaves no file and exits 0 (writeGenerated(...) catch return).
• zig ast-check on a missing file prints error: unable to open file '…' for ast-check: FileNotFound and exits 1. On a directory it prints error: unable to load file … ReadFailed, also exit 1.
• Neither message contains : error: , which is the only thing astCheckOk looks for. The exit code is ignored, so the spec counts as clean.
• Measured later (probe 37242067946): a second hole. 18 baseline specs abort with SIGABRT after creating the output file and leave it empty, and zig ast-check passes an empty file with exit 0. The abort is vibee_gen aborts (SIGABRT) on 18 Verilog specs and intermittently on 2 Zig specs #887.

What does not explain it:

• Probe run 37241431527 generated all 1137 specs 5 times on one runner, which took 104.9 s.
  • 0 specs differ in exit code, behaviour count or output sha256.
  • Clean equals the baseline (788), with 0 extra and 0 lost.
  • Five repeats have little power against a rare flip, so this is not evidence that the generator is deterministic across runners.
• Same image ubuntu-24.04 20260927.320.1 on both runs of e6eac09. The red run was in westus2, the green one in westus.
• No randomness or clock use in the generator path: I grepped vibee_gen.zig, zig_codegen.zig, vibee_parser.zig and lang_generators.zig.
• vibee_gen is built for the native CPU (standardTargetOptions). A CPU-dependent outcome is possible and not tested.

Side defect found on the way

The output name flattened / to _, so specs/storm/main.tri and specs/storm_main.tri both wrote .zig-cache/codegen-corpus/specs_storm_main.tri.zig. Neither spec is in the baseline, so this is not the cause, but the compile sample could have tested one spec's output under the other's name.

Fix (PR to follow)

• The sample is drawn from the committed baseline: slot k is baseline entry k * stride. A run cannot move it.
• A spec counts as clean only if the generator exits 0, leaves a non-empty regular file, and zig ast-check exits 0 with no : error: .
• The output file is deleted before each generation. Its path mirrors the spec path.
• The clean count, regressed, new-clean and no-output specs are printed on both paths, by name. The next disagreement will then name its spec.
• The runner's CPU model is printed.

Done when

• PR merged and its own Codegen Validation run is green.

• CI A/B on throwaway branches (main 37243529377, fixed 37243526975; all three rows below came out as expected). Each arm runs main's tool and the fixed tool on the same tree:

  • plain;
  • one extra clean spec added before phi_utils_multi;
  • one non-baseline spec before it forced to leave no file;
  • one baseline spec forced to leave no file.

  Expected:

  arm	main's tool	fixed tool
  extra clean spec	red on phi_utils_multi	green, naming the new clean spec
  non-baseline spec leaves no file	red on phi_utils_multi	green, naming the spec with no output
  baseline spec leaves no file	green over the missing output	red, naming the regression

• A separate issue for the phi_utils_multi generator defects. They are listed in vibee_gen aborts (SIGABRT) on 18 Verilog specs and intermittently on 2 Zig specs #887 under "Also found", which is not yet an issue of their own.

Not measured

• The clean count of any red run. It was never printed.
• Whether the generator's output differs between CPU models.
• How many of the 788 baseline outputs compile. Measured in probe 37242067946: 19 never wrote a usable file (18 empty after SIGABRT, 1 missing; vibee_gen aborts (SIGABRT) on 18 Verilog specs and intermittently on 2 Zig specs #887), and 13 of the remaining 769 fail zig test. fix(codegen): sample the corpus compile check from the baseline #886 re-records the baseline as 769.

Also seen, not this issue

src/firebird/vsa_simd.zig:448, test "benchmark dot product speedup", asserts a wall-clock ratio > 0.5; its siblings use > 0.1. It made the 09-29 scheduled run red (36515512024). src/firebird/ is outside the direct-edit zones, so the threshold is the owner's call.

🤖 Generated with Claude Code