Raise the Ubuntu test step timeout above its measured runtime - #13
Conversation
build-and-test (ubuntu-24.04, Debug) failed on b286d07 without any test failing. It was killed at the 20-minute step limit with 464 of 474 tests already passed, mid-way into starting the 465th, and the orphaned ctest, tee and absolutec processes were reaped by the runner. The same commit passed in the parallel run of the same workflow, where the job took 14m13s against the 22m53s of the one that timed out. That 1.6x spread across two runners on identical input is the whole story: the Debug suite sits close enough to the cap that ordinary runner variance decides the outcome. Release is unaffected because it runs far quicker. Raise the Ubuntu test step to 35 minutes and the job to 60 so the step ceiling, not the job ceiling, is what a genuine hang hits. The step still fails fast on a real deadlock, and ctest keeps its own 180-second per-test timeout, so a single wedged test is still caught in seconds rather than after the step budget. This trades a longer worst case for not failing on healthy runs. Cutting the runtime itself, by giving ctest a --parallel, is the better fix but a larger change: the suite runs serially today and some tests bind ports and write shared paths, so parallelising needs each test checked for isolation first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
|
A step timeout in one workflow's job cannot affect a job in another. The sibling run's copy of this job is still queued, so the usual one-failed-one-passed confirmation is not available yet, but it is not needed for the causal question. This job is the known intermittent Linux failure documented on #12: it has previously failed and passed on the same commit across parallel runs, and the stress binary passes 300 consecutive local runs. It is pre-existing and tracked separately from this change. The signal this PR actually needs is Generated by Claude Code |
The backend selected a target CPU with getHostCPUName() and passed an empty feature string, so LLVM enabled every feature implied by that microarchitecture. The name only identifies a model; it says nothing about which instructions the machine actually exposes. A virtualised host can report a model whose default feature set includes instructions the hypervisor masks off, and generating for the name alone then emits code the CPU refuses to run, which surfaces as SIGILL in whatever executes the generated binary. Take the feature string from getHostCPUFeatures() instead. The signature changed in LLVM 19 from an out-parameter to a returned map, so the call is guarded the same way Host.h already is, and SubtargetFeature.h joins it in the PCH under the existing version guard. What prompted this: build-and-test (ubuntu-24.04, Debug) failed on 8f60a85 with nine tests, eight of them ILLEGAL, and both parallel runs produced an identical list. The same source passed all 474 tests an hour earlier in job 93122612240, so the source is not what changed. The failures cluster on tests that execute generated native code, which is the signature this defect produces. Also log lscpu on Linux, so the next run records the runner's model and feature bits instead of leaving them to be inferred from a signal number. Verification is honest about its limits: this cannot be reproduced locally, because getHostCPUName() here reports cascadelake on a machine that genuinely has AVX-512, so name and features agree and no illegal instruction is possible. The local Debug run confirms the change causes no regression; confirming it fixes the CI failure needs a hosted run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
|
Both changes on this branch are now confirmed by CI, and the second one turned out to matter far more than the first. The timeout raise did its jobOn What completing the run exposedWith the step no longer killed, both parallel runners failed with an identical 9-test list, 8 of them Identical results on two independent runners is deterministic, not flaky — so the earlier "intermittent Linux failure" framing was wrong. The same source had passed all 474 tests an hour before (job Root cause
That explains every symptom: the signal itself, tests spanning plugins, ABI, encoding, SIMD and scheduling (all of which execute generated native code), no local reproduction, and the switch from green to red without a code change. The fix takes the feature string from Result on
|
| Job | 8f60a85 |
65df49c |
|---|---|---|
| build-and-test (ubuntu-24.04, Debug) | 9 failures, both runners | success, 19m51s |
| language-stress-and-fuzz | failure | success, both runs |
| ubuntu Release, both Windows, llvm-compatibility 18/19/20/21, termux, wasm, TSan | green | green |
| Hardening required gate | failure | success, both runs |
language-stress-and-fuzz going green in both runs fits the same cause — it also executes generated code.
Two caveats stated plainly: the second Debug job is still running, and because the trigger was environmental I cannot fully exclude that the runner fleet simply rotated back. The mechanism is sound and the fix is correct regardless, but a few more runs are what would settle it. The lscpu line added to the Linux toolchain log will make the next occurrence diagnosable from the artifact instead of by inference.
macos-smoke still fails with its single pre-existing run-sanitizer-ownership-stability test — unchanged in count, which also confirms the codegen change did not affect macOS/arm64. The required gate here cannot go green while it fails.
Generated by Claude Code
2 participants
build-and-test (ubuntu-24.04, Debug)failed onb286d07without a single test failing. It was killed at the 20-minute step limit:Why this is timing, not a defect
The same commit passed in the parallel run of the same workflow:
Identical input, two runners, a 1.6x spread. The Debug suite sits close enough to the cap that ordinary runner variance decides the outcome. Release is unaffected — it finishes in roughly 5 minutes. 464 of 474 tests had already passed when the step was killed, so nothing was stuck; the budget simply ran out.
The change
Test (Ubuntu)goes from 20 to 35 minutes, and the job from 45 to 60 so the step ceiling rather than the job ceiling is what a genuine hang hits.This does not weaken the failure detection that matters.
ctestkeeps its own--timeout 180, so a single wedged test is still caught in three minutes regardless of the step budget. What the step limit catches is a whole-suite deadlock, and that is exactly the case where waiting longer costs nothing but a few idle runner minutes.What would be better
Cutting the runtime rather than raising the ceiling.
ctestruns serially here while the build steps already use--parallel 2, so there is real headroom. That is a larger change than it looks: some tests bind ports and write shared paths, so each needs checking for isolation before parallelising, and getting it wrong trades a predictable timeout for unpredictable flakes. Left for a separate change rather than folded in here.Verification
The workflow parses and the values land where intended:
Whether the timeout is actually gone can only be shown by CI over several runs, since the failure is intermittent by nature.
Generated by Claude Code