vulkan-shaders-gen: retry an empty shader compile instead of failing at link (potential fix for CI failures)

[christopherthompson81]

Problem

A shader whose compile produces no SPIR-V becomes an undefined reference at link, ~20 minutes into the build, naming a symbol in vendored code with nothing earlier in the log to explain it. The Nix (vulkan) job on #539 failed this way:

ld.bfd: ggml-vulkan.cpp:3964: undefined reference to
  `matmul_id_subgroup_nvfp4_f32_aligned_f16acc_cm1_len'

write_output_files() writes the declaration before it reads the artefact, then skips the definition if the file is empty, with no diagnostic:

hdr << "extern const uint64_t " << name << "_len;\n";      // always declared
...
std::string data = read_binary_file(path);
if (data.empty()) {
    continue;                                               // definition skipped
}

Compile success was judged by stderr alone, with no exit code, so a shader that merely warns is discarded while one that reports nothing and writes nothing is accepted.

Fix

Success is judged by the artefact — the SPIR-V must exist and be non-empty. An empty result is retried up to three times with short backoff; after the last attempt the shader is named and generation fails. Diagnostics alongside a usable artefact are treated as warnings and the shader is kept.

Why retry rather than just fail

The trigger does not reproduce locally. Investigating the #539 failure I built shaderc 2026.1 — the version this repo's Nix build uses, per the job log — and compiled the exact failing shader with the exact 17 defines: exit 0, no stderr, 27,700 bytes of valid SPIR-V. 120 repeat compiles under parallel load produced no anomalies. A full Vulkan build with that compiler and all four shader extensions enabled links cleanly: 2320 symbols declared, 2320 defined.

So the shader is fine and the compiler is fine; something environmental produced an empty file while reporting success. Retrying absorbs that. Failing loudly covers the case where it is not transient.

Verification

Against an injected failure, both directions:

• transient (first attempt empty, later attempts normal) — recovers, build links, exit 0:

shader matmul_id_subgroup_nvfp4_f32_aligned_f16acc_cm1 produced no SPIR-V; retrying (2/3)
[333/333] Linking CXX executable bin/audiocpp_cli

• permanent (always empty) — stops at generation naming the shader, exit 1, no link error:

shader matmul_id_subgroup_nvfp4_f32_aligned_f16acc_cm1 produced no SPIR-V; retrying (2/3)
shader matmul_id_subgroup_nvfp4_f32_aligned_f16acc_cm1 produced no SPIR-V; retrying (3/3)
cannot compile matmul_id_subgroup_nvfp4_f32_aligned_f16acc_cm1 after 3 attempts
shader generation failed; see errors above

Note on upstream

The same declaration-before-definition path exists in ggml-org/llama.cpp master. Upstream does already check the exit code from execute_command, which this tree's copy does not — worth a look when these files are next synced.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EkxqpYvUbjCpRDnFiNiVfx

[0xShug0]

@christopherthompson81 Thanks! I’ll keep it open for now until I get some feedback from the nix sub.

[christopherthompson81]

Moving this to draft so it isn't sitting in your review queue while you wait on feedback from the nix sub — no pressure either way, just ping me and I'll mark it ready again.