Fix CI failures blocking the required gate - #12
Conversation
Four independent failures kept "CI required gate" red: - OptimizationLevel was only declared in codegen.h, which pch.h includes under ABSOLUTE_HAS_LLVM. Absolute-Compiler.cpp uses it unconditionally, so every ABSOLUTE_ENABLE_LLVM=OFF build failed to compile the driver (build-and-test on windows-2022 Debug/Release and termux-host-contract). Move the enum to its own LLVM-free header, include it unconditionally, and put Absolute-CodeGen/include on the driver's include path so the header resolves without the CodeGen target. - llvm-compatibility ran in a bare ubuntu:24.04 container without zlib1g-dev. LLVM's exported LLVMSupport target lists ZLIB::ZLIB in its link interface, so find_package(LLVM) aborted at generate time for LLVM 18, 19 and 20. Install zlib1g-dev alongside libzstd-dev. - macos-smoke could not build Absolute-Runtime: Darwin's <ucontext.h> errors out unless _XOPEN_SOURCE is defined, leaving getcontext, makecontext and swapcontext undeclared. Define it for the Apple branch of scheduler_fiber.h. - windows-llvm-release timed out bootstrapping the ~1 GB LLVM SDK on a cache miss, and the always() diagnostics step then threw on the missing llvm-config.exe, failing the job at that step and hiding the timeout. Guard the lookup and give the bootstrap enough headroom. Verified locally by building the driver both with ABSOLUTE_ENABLE_LLVM=OFF and ON, and by reproducing the LLVM export failure through a missing zstd/zlib target. The Darwin and Windows paths are preprocessor- and workflow-only changes that cannot be compiled here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
Fixing the zlib gap in llvm-compatibility let those jobs get past CMake and reach the compiler, which exposed two API guards that never matched the versions the project claims to support (18 through 21). - LazyValueInfo::getPredicateAt returns Constant* rather than the removed Tristate enum since LLVM 19, but the guard in codegen.cpp opened at 21. LLVM 19 and 20 therefore took the old branch and failed with "no member named 'True' in 'llvm::LazyValueInfo'". - LLVM 20 added a ThinOrFullLTOPhase argument to the optimizer-last extension point callback, so the sanitizer pass registration in codegen_module.cpp no longer converted to the expected std::function. Split the registration per version and keep the pass setup shared. The remaining >= 21 guards were checked and are genuinely LLVM 21 API changes (CaptureInfo replacing NoCapture, Triple-based TargetMachine entry points), so they are left alone. Verified by installing LLVM 18, 19 and 20 locally and building Absolute-CodeGen and the absolutec driver against each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The LLVM and WASI bootstrap steps captured their output into a variable and only wrote it to the log after the command returned. When the LLVM step hit its timeout it produced no output at all for the full 25 minutes, so the job log showed only the timeout with no indication of how far the download had progressed. Pipe through Tee-Object instead, which writes as output arrives. This is a diagnostics change only. It does not address the underlying problem that a cold cache requires downloading an ~1 GB SDK inside the step timeout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The previous commit adapted the sanitizer callback at its call site, which was the wrong layer and broke LLVM 21: codegen_pch.h already ships an OptimizerLastEPCallback shim plus a macro that rewrites the registration, so call sites are meant to keep passing the two-argument form and let the shim add the LTO phase. Passing three arguments made the shim call the callback with two. The real defect is the shim's guard. LLVM 20 added the LTO-phase argument, but the shim only activated at 21, so LLVM 20 had no adapter at all. That guard also covered the unrelated Triple-based target APIs, which really are LLVM 21, and bundling them hid the mismatch. Split the block: the optimizer callback shim now activates at 20, the Triple helpers stay at 21, and codegen_module.cpp goes back to the plain two-argument lambda. LLVM 21 therefore returns to exactly the configuration that was passing before, now shared with LLVM 20. Verified by building absolutec against LLVM 18, 19 and 20 locally. LLVM 21 is not installable in this environment; it is covered by the restored call site being byte-identical to the previously passing one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
With the build failures fixed, these jobs reached their test phase for the first time and exposed two test-side assumptions. Four debug-info tests emit IR, link through --build-exe and inspect the result with the LLVM tools, but they were registered unconditionally. In any ABSOLUTE_ENABLE_LLVM=OFF configuration they failed with "LLVM backend is unavailable in this build", which is the configuration working as intended rather than a defect. Guard them with ABSOLUTE_ENABLE_LLVM, the pattern already used elsewhere in this file; the neighbouring semantic and diagnostic debug-info tests are frontend only and stay unguarded. This covers termux-host-contract and the windows-2022 build-and-test jobs, all of which configure the backend off. The debug-info IR assertion also required the llvm.dbg.declare intrinsic spelling. LLVM 19 prints debug info as records instead, so the check fails from 19 onward even though the emitted debug info is correct. Accept either spelling. Verified locally: the termux contract configuration goes from 4 failures to 178/178, the debug-info tests pass against both LLVM 18 (24 intrinsics) and LLVM 19 (23 records), and the full suite is 501/501 on LLVM 18. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The earlier fix defined _XOPEN_SOURCE inside scheduler_fiber.h, which tasks.cpp includes after about twenty standard library headers. That is enough to satisfy the #error in Darwin's <ucontext.h> and make the build succeed, but a feature macro only affects headers included after it: the Darwin headers libc++ had already pulled in were parsed without it. The declarations and the ucontext layout can therefore disagree within one translation unit, which is consistent with the Bus error and SIGTRAP the scheduler and async tests now hit on macOS. Set the macro from the command line for that source file instead, so it applies before any header, and pair it with _DARWIN_C_SOURCE so the stricter feature set does not hide BSD extensions from the rest of the unit. The header now states the requirement and fails loudly if it is missing, rather than silently defining it too late. This removes a real class of undefined behaviour. It is not confirmed to be the cause of the macOS failures: if they persist, the remaining explanation is that the ucontext routines are unusable on arm64 Darwin, which needs a different fiber backend and is a design decision rather than a fix. Verified on Linux, where the Apple branch is inert: tasks.cpp compiles and Absolute-Runtime links. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
|
Status on the two jobs that are still red, since both are now past the failures this PR set out to fix.
|
| Test | Signal |
|---|---|
runtime-scheduler-v2 |
Bus error |
runtime-scheduler-work-stealing, -fairness, -metrics, -stress |
SIGTRAP |
run-async, run-task-scheduling, run-task-group, run-async-methods, run-async-concurrency-advanced |
SIGTRAP |
run-exceptions, run-defer, run-std-http, run-std-concurrent-primitives, run-std-typed-channel |
SIGTRAP |
abi-linker-corpus-native, run-sanitizer-ownership-stability |
Failed |
Everything in the SIGTRAP group runs through the fiber scheduler. These tests have never executed on macOS in this branch's history — the job previously died at compile time — so there is no evidence they ever passed there.
Before attributing this to the platform I revisited my own fix. The first version defined _XOPEN_SOURCE inside scheduler_fiber.h, which tasks.cpp includes after roughly twenty standard library headers. That satisfies the #error in Darwin's <ucontext.h>, but a feature macro only affects headers included after it, so the Darwin headers libc++ had already pulled in were parsed without it — leaving the declarations and the ucontext_t layout potentially inconsistent inside one translation unit. That is consistent with a Bus error and SIGTRAP. The latest commit moves the definition to the build system for that source file so it applies before any header, paired with _DARWIN_C_SOURCE, and the header now fails loudly instead of defining the macro too late.
To be clear about what that is and is not: it removes a real class of undefined behaviour, but it is not confirmed to be the cause. If these tests still crash, the remaining explanation is that the ucontext routines are unusable on arm64 Darwin and the runtime needs a different fiber backend there — a design decision, not a fix, and out of scope for this PR.
windows-llvm-release — blocked, and not something I am fixing here
This one is not a code defect and I am deliberately not patching around it:
Cache not found for input keys: windows-llvm-18.1.8-sdk-v1
##[error]The action 'Bootstrap portable LLVM SDK' has timed out after 25 minutes.
Downloading the ~1 GB LLVM SDK exceeded both the original 15-minute timeout and the 25 minutes this PR raised it to. Because the job fails, actions/cache never saves, so the next run starts cold again — a self-sustaining loop. It passed earlier only against a warm cache that has since been evicted.
What this PR does contribute here is honesty: the diagnostics step no longer throws on the missing llvm-config.exe and fail the job at that step, and the bootstrap steps now stream their output instead of buffering it into a variable that is discarded on timeout — which is why the earlier run showed 25 minutes of complete silence followed by a misleading "not recognized as a name of a cmdlet" error.
Breaking the loop needs a provisioning decision — seeding or persisting the SDK cache, caching the partial download so curl --continue-at - can resume across runs, or pinning a smaller toolchain. That is a call for the repository owner.
Generated by Claude Code
|
Correction to my earlier comment on The download completes in 16 seconds at ~56 MB/s. The step then spends the entire remaining ~24.5 minutes inside The real bottleneck is decompressing and writing out the archive on
That reframes the fix away from timeouts and toward the extraction itself — for example excluding the The cache interaction I described still holds: because the job fails, Generated by Claude Code |
The streamed bootstrap output shows the ~1 GB SDK download finishing in 16 seconds at ~56 MB/s, after which the step spends the remaining 24.5 minutes inside tar.exe unpacking the .tar.xz before being killed. The comment added with the timeout increase claimed the step was killed mid-download, which is wrong and would point a reader at the wrong phase. Comment only; the 25-minute value is left as is. Raising it further would not help, since the overrun is in extraction rather than transfer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The streamed bootstrap output showed where windows-llvm-release actually spends its time: the ~1 GB SDK downloads in 16 seconds at ~56 MB/s, then tar.exe runs for the remaining 24.5 minutes and is killed. Real-time antivirus scanning of every file written is the usual cause of that ratio on CI runners, since the archive expands to several GB of mostly small headers. Add a best-effort Defender exclusion for the toolchains directory before extraction, and report how long extraction took. The exclusion is wrapped in a try/catch so a machine without Defender, or without the privileges to configure it, still extracts as before. The timing line makes the next run informative either way: if extraction is still slow, the number rules the antivirus theory out rather than leaving it open. Not verifiable from this environment, which has no Windows runner and no PowerShell to parse the script; CI is the test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
|
Second correction, this time to what I said about The signals line up across architectures. LLVM lowers a reached
Same defect class, different lowering, rather than two unrelated platform problems. Reaching an The fiber code is not arm64-specific. What I cannot do from here. The Linux failure is intermittent — two runs of the identical SHA I am not attempting a fix. Both the fiber scheduler and the proven-bounds elimination this branch introduces are plausible places to look, and choosing where to dig is better done by someone who can reproduce it. Generated by Claude Code |
|
Result of the Defender-exclusion commit on The exclusion was applied successfully — the cmdlet succeeded and printed the path — and extraction still consumed the entire remaining 24.8 minutes. The So the picture is now narrow and well established:
What remains are provisioning changes with real tradeoffs, none of which I can validate from this environment — there is no Windows runner here and no PowerShell to even parse the script, and each attempt costs a 25-minute CI cycle:
I am stopping here rather than iterating blind. Earlier in this PR a change I made without being able to run it broke a job that had been green, and repeating that pattern against a 25-minute feedback loop is not a good trade. The instrumentation is the durable part: whoever picks this up now gets a per-phase timing instead of 25 minutes of silence. Generated by Claude Code |
The instrumentation added in the previous commit settled where the time goes. The download finishes in 9 seconds at ~110 MB/s, the Defender exclusion is applied successfully, and tar.exe still consumes the entire remaining 24.8 minutes without returning. So neither transfer nor antivirus scanning explains it: bsdtar decoding this xz stream on a single thread does. 7-Zip ships on the hosted Windows images and decodes xz in parallel. Split extraction into an xz pass and a tar pass through it, timing each, and keep tar.exe as the fallback for any machine where 7-Zip is missing or the pass does not produce the expected archive. The Defender exclusion is retained: it was not the fix, but it is cheap and large file writes are genuinely slower under real-time scanning. The per-phase timings mean the next run distinguishes slow decompression from slow file creation, whichever way this lands. Not verifiable here: no Windows runner and no PowerShell to parse the script, so CI is the test. The fallback keeps the failure mode no worse than the current one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
Building the stress programs with --sanitize=address surfaced a leak that reproduces on every run of collection-snapshot-stress: 6048 bytes in 12 allocations, attributed to std.collections.Map.put and Set.add. The growth paths allocate a larger array, copy the live elements across, and then assign it with copy(...). That stores a duplicate in the field and leaves the freshly allocated local unreferenced and unreleased. The arithmetic matches exactly: six doublings from capacity 4 to 256 leak 8*(8+16+32+64+128+256) = 4032 bytes for the 8-byte KeyValuePair entries and 4*(...) = 2016 for the 4-byte set items, 6048 together, in 6 objects each. Assign with move(...) so the field takes ownership of the array that was just built. vector.abs already does exactly this at its own growth sites, so this brings map, set, deque and priority_queue in line with the idiom the standard library had already settled on rather than introducing a new one. A plain assignment is rejected by the analyzer, which names move(...) among the accepted forms. The remaining copy(...) uses are left alone: iterator snapshots, the copy-on-write in ensureUnshared, and the rehash locals in hash_map are all genuine copies. Verified with ASan: collection-snapshot-stress goes from 6048 bytes leaked to a clean exit. The full suite is 501/501 on LLVM 18. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
|
Correction, and a good one: the I previously reported that ASan on Linux did not reproduce the macOS failures, that my unified-defect hypothesis was unsupported, and that the cluster looked platform-specific and needed a Mac. That was wrong. The leak ASan found was the same defect. Both parallel macOS runs on 17 failures down to 1. All 14 SIGTRAPs and the Bus error are gone. The only change in that commit is The mechanism, inferred rather than directly observed: the growth paths built a new array, stored a duplicate of it in the field, and dropped the original. On Linux the duplicate kept everything reachable, so the damage surfaced only as the 6048-byte leak ASan reported. On macOS/arm64 the same mismanaged ownership left the collection referring to memory it did not own, which is exactly the profile of the SIGTRAP and Bus error crashes — a reached Worth noting for the record: The one remaining macOS failure, Generated by Claude Code |
Targets
agent/stabilize-foundationso that #10 can reach a greenCI required gate.Every red check on #10 was investigated to a named root cause. Fixing the first batch let several jobs reach stages they had never got to, which exposed further real defects, so this PR grew past its original four fixes. Nothing here is a test relaxation except where a test was asserting something the configuration cannot provide, which is called out explicitly below.
Build and portability defects
OptimizationLevelunavailable in every LLVM-free build.pch.hincludescodegen.honly underABSOLUTE_HAS_LLVM, butAbsolute-Compiler.cppusesOptimizationLevelunconditionally (lines 57, 137, 167–176, 1064–1066). EveryABSOLUTE_ENABLE_LLVM=OFFconfiguration failed to compile the driver:build-and-test (windows-2022, Debug/Release)witherror C2065: 'O3': undeclared identifier, andtermux-host-contractwitherror: 'OptimizationLevel' was not declared in this scope. The enum has no LLVM dependency, so it moves tooptimization_level.h, is included unconditionally, andAbsolute-CodeGen/includejoins the driver's include path — previously that path only arrived transitively from linking the CodeGen target. The CLI surface is unchanged:-O0..-O3parse whether or not the backend is compiled in.llvm-compatibilitymissingzlib1g-dev. The job runs in a bareubuntu:24.04container, and LLVM's exportedLLVMSupportlistsZLIB::ZLIBinINTERFACE_LINK_LIBRARIES, sofind_package(LLVM)aborted at generate time for LLVM 18, 19 and 20 atLLVMExports.cmake:73. LLVM 21 passed, which is why the failure looked selective.macos-smokecould not buildAbsolute-Runtime. Darwin's<ucontext.h>refuses to declaregetcontext/makecontext/swapcontextunless_XOPEN_SOURCEis defined. It is now defined in the Apple branch ofscheduler_fiber.h;tasks.cppis the only translation unit including that header and it pulls in no Darwin headers the stricter feature set would narrow, so the macro stays scoped to the header instead of the build files.LLVM version guards that never matched the supported range
The project documents support for LLVM 18 through 21. With the zlib gap closed, those jobs got far enough to disprove that.
LazyValueInfo::getPredicateAtreturnsConstant*rather than the removedTristateenum from LLVM 19 onward, but the guard incodegen.cppopened at 21, so LLVM 19 and 20 took the old branch and failed withno member named 'True' in 'llvm::LazyValueInfo'.The optimizer-last extension point gained a
ThinOrFullLTOPhaseargument in LLVM 20.codegen_pch.halready ships anOptimizerLastEPCallbackshim plus a macro that rewrites the registration, so call sites are meant to keep passing the two-argument form — but the shim only activated at 21, leaving LLVM 20 with no adapter. That same#ifalso covered the unrelated Triple-based target APIs, which genuinely are LLVM 21, and bundling the two hid the mismatch. The block is now split: the callback shim activates at 20, the Triple helpers stay at 21, and the call site keeps its plain two-argument lambda.Test-side assumptions
Four debug-info tests required the backend but were registered unconditionally.
debug-info-ir,build-debug-info,debug-info-artifactsandrun-debug-infoemit IR, link through--build-exeand inspect the result with the LLVM tools. In anABSOLUTE_ENABLE_LLVM=OFFbuild they failed with "LLVM backend is unavailable in this build" — the configuration behaving as designed, not a defect. They are now guarded byABSOLUTE_ENABLE_LLVM, the pattern already used elsewhere in that file. The neighbouring semantic and diagnostic debug-info tests are frontend only and stay unguarded, so no coverage is lost in a normal build.The debug-info IR assertion was LLVM-18-only. It required the
llvm.dbg.declareintrinsic spelling; LLVM 19 prints debug info as records instead, so the check failed from 19 onward even though the emitted debug info was correct. It now accepts either spelling.Workflow defects
windows-llvm-releasereported the wrong failure. TheRecord toolchain versionsstep runs underif: always()and invokedllvm-config.exeunconditionally. Since GitHub'spwshshell sets$ErrorActionPreference = 'Stop', a missing SDK made that diagnostics step fail the job itself and mask the real cause. It now checks for the binary first.The bootstrap steps discarded their own output. They captured into a variable and only wrote the log after the command returned, so a timeout left no diagnostics at all — the step produced nothing for 25 minutes. They now stream through
Tee-Object.Verification
LLVM 18, 19 and 20 were installed locally and
absolutecwas built against each. Additionally:ABSOLUTE_ENABLE_LLVM=OFFlinks the driver, the exact failure previously seen on Windows and Termux, and-O2is still accepted by that binaryllvm.dbg.declare) and LLVM 19 (23#dbg_declare)LLVMExports.cmake:73on the next missing targetLLVM 21 is not installable in this environment — CI runs it in
ubuntu:26.04. It is covered by the restored call site being identical to the previously passing one. The Darwin fix and the workflow changes cannot be exercised here either; CI is the check on those.Known blocker outside this PR
windows-llvm-releasecannot currently pass on a cold cache.Cache not found for input keys: windows-llvm-18.1.8-sdk-v1, and downloading the ~1 GB LLVM SDK exceeded both the original 15-minute and the raised 25-minute step timeout. Because the job fails,actions/cachenever saves, so the next run is cold again — a self-sustaining loop. It passed previously only on a warm cache that has since been evicted. This is a toolchain provisioning decision rather than a code defect and is deliberately left alone here.🤖 Generated with Claude Code
https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS