Skip to content

Fix CI failures blocking the required gate - #12

Merged
FuryBaM merged 10 commits into
agent/stabilize-foundationfrom
claude/local-language-benchmarks-9en3eo
Aug 8, 2026
Merged

Fix CI failures blocking the required gate#12
FuryBaM merged 10 commits into
agent/stabilize-foundationfrom
claude/local-language-benchmarks-9en3eo

Conversation

@FuryBaM

@FuryBaM FuryBaM commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Targets agent/stabilize-foundation so that #10 can reach a green CI required gate.

Every red check on #10 was investigated to a named root cause. Fixing the first batch let several jobs reach stages they had never got to, which exposed further real defects, so this PR grew past its original four fixes. Nothing here is a test relaxation except where a test was asserting something the configuration cannot provide, which is called out explicitly below.

Build and portability defects

OptimizationLevel unavailable in every LLVM-free build. pch.h includes codegen.h only under ABSOLUTE_HAS_LLVM, but Absolute-Compiler.cpp uses OptimizationLevel unconditionally (lines 57, 137, 167–176, 1064–1066). Every ABSOLUTE_ENABLE_LLVM=OFF configuration failed to compile the driver: build-and-test (windows-2022, Debug/Release) with error C2065: 'O3': undeclared identifier, and termux-host-contract with error: 'OptimizationLevel' was not declared in this scope. The enum has no LLVM dependency, so it moves to optimization_level.h, is included unconditionally, and Absolute-CodeGen/include joins the driver's include path — previously that path only arrived transitively from linking the CodeGen target. The CLI surface is unchanged: -O0..-O3 parse whether or not the backend is compiled in.

llvm-compatibility missing zlib1g-dev. The job runs in a bare ubuntu:24.04 container, and LLVM's exported LLVMSupport lists ZLIB::ZLIB in INTERFACE_LINK_LIBRARIES, so find_package(LLVM) aborted at generate time for LLVM 18, 19 and 20 at LLVMExports.cmake:73. LLVM 21 passed, which is why the failure looked selective.

macos-smoke could not build Absolute-Runtime. Darwin's <ucontext.h> refuses to declare getcontext/makecontext/swapcontext unless _XOPEN_SOURCE is defined. It is now defined in the Apple branch of scheduler_fiber.h; tasks.cpp is the only translation unit including that header and it pulls in no Darwin headers the stricter feature set would narrow, so the macro stays scoped to the header instead of the build files.

LLVM version guards that never matched the supported range

The project documents support for LLVM 18 through 21. With the zlib gap closed, those jobs got far enough to disprove that.

LazyValueInfo::getPredicateAt returns Constant* rather than the removed Tristate enum from LLVM 19 onward, but the guard in codegen.cpp opened at 21, so LLVM 19 and 20 took the old branch and failed with no member named 'True' in 'llvm::LazyValueInfo'.

The optimizer-last extension point gained a ThinOrFullLTOPhase argument in LLVM 20. codegen_pch.h already ships an OptimizerLastEPCallback shim plus a macro that rewrites the registration, so call sites are meant to keep passing the two-argument form — but the shim only activated at 21, leaving LLVM 20 with no adapter. That same #if also covered the unrelated Triple-based target APIs, which genuinely are LLVM 21, and bundling the two hid the mismatch. The block is now split: the callback shim activates at 20, the Triple helpers stay at 21, and the call site keeps its plain two-argument lambda.

Test-side assumptions

Four debug-info tests required the backend but were registered unconditionally. debug-info-ir, build-debug-info, debug-info-artifacts and run-debug-info emit IR, link through --build-exe and inspect the result with the LLVM tools. In an ABSOLUTE_ENABLE_LLVM=OFF build they failed with "LLVM backend is unavailable in this build" — the configuration behaving as designed, not a defect. They are now guarded by ABSOLUTE_ENABLE_LLVM, the pattern already used elsewhere in that file. The neighbouring semantic and diagnostic debug-info tests are frontend only and stay unguarded, so no coverage is lost in a normal build.

The debug-info IR assertion was LLVM-18-only. It required the llvm.dbg.declare intrinsic spelling; LLVM 19 prints debug info as records instead, so the check failed from 19 onward even though the emitted debug info was correct. It now accepts either spelling.

Workflow defects

windows-llvm-release reported the wrong failure. The Record toolchain versions step runs under if: always() and invoked llvm-config.exe unconditionally. Since GitHub's pwsh shell sets $ErrorActionPreference = 'Stop', a missing SDK made that diagnostics step fail the job itself and mask the real cause. It now checks for the binary first.

The bootstrap steps discarded their own output. They captured into a variable and only wrote the log after the command returned, so a timeout left no diagnostics at all — the step produced nothing for 25 minutes. They now stream through Tee-Object.

Verification

LLVM 18, 19 and 20 were installed locally and absolutec was built against each. Additionally:

  • ABSOLUTE_ENABLE_LLVM=OFF links the driver, the exact failure previously seen on Windows and Termux, and -O2 is still accepted by that binary
  • the termux contract configuration goes from 4 failures to 178/178
  • the debug-info tests pass against both LLVM 18 (24 llvm.dbg.declare) and LLVM 19 (23 #dbg_declare)
  • the full suite is 501/501 on LLVM 18
  • the ZLIB mechanism was reproduced directly: this machine had zlib but not zstd and CMake failed identically at LLVMExports.cmake:73 on the next missing target

LLVM 21 is not installable in this environment — CI runs it in ubuntu:26.04. It is covered by the restored call site being identical to the previously passing one. The Darwin fix and the workflow changes cannot be exercised here either; CI is the check on those.

Known blocker outside this PR

windows-llvm-release cannot currently pass on a cold cache. Cache not found for input keys: windows-llvm-18.1.8-sdk-v1, and downloading the ~1 GB LLVM SDK exceeded both the original 15-minute and the raised 25-minute step timeout. Because the job fails, actions/cache never saves, so the next run is cold again — a self-sustaining loop. It passed previously only on a warm cache that has since been evicted. This is a toolchain provisioning decision rather than a code defect and is deliberately left alone here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS

claude added 6 commits August 8, 2026 11:01
Four independent failures kept "CI required gate" red:

- OptimizationLevel was only declared in codegen.h, which pch.h includes
  under ABSOLUTE_HAS_LLVM. Absolute-Compiler.cpp uses it unconditionally,
  so every ABSOLUTE_ENABLE_LLVM=OFF build failed to compile the driver
  (build-and-test on windows-2022 Debug/Release and termux-host-contract).
  Move the enum to its own LLVM-free header, include it unconditionally,
  and put Absolute-CodeGen/include on the driver's include path so the
  header resolves without the CodeGen target.

- llvm-compatibility ran in a bare ubuntu:24.04 container without
  zlib1g-dev. LLVM's exported LLVMSupport target lists ZLIB::ZLIB in its
  link interface, so find_package(LLVM) aborted at generate time for
  LLVM 18, 19 and 20. Install zlib1g-dev alongside libzstd-dev.

- macos-smoke could not build Absolute-Runtime: Darwin's <ucontext.h>
  errors out unless _XOPEN_SOURCE is defined, leaving getcontext,
  makecontext and swapcontext undeclared. Define it for the Apple branch
  of scheduler_fiber.h.

- windows-llvm-release timed out bootstrapping the ~1 GB LLVM SDK on a
  cache miss, and the always() diagnostics step then threw on the missing
  llvm-config.exe, failing the job at that step and hiding the timeout.
  Guard the lookup and give the bootstrap enough headroom.

Verified locally by building the driver both with ABSOLUTE_ENABLE_LLVM=OFF
and ON, and by reproducing the LLVM export failure through a missing
zstd/zlib target. The Darwin and Windows paths are preprocessor- and
workflow-only changes that cannot be compiled here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
Fixing the zlib gap in llvm-compatibility let those jobs get past CMake
and reach the compiler, which exposed two API guards that never matched
the versions the project claims to support (18 through 21).

- LazyValueInfo::getPredicateAt returns Constant* rather than the removed
  Tristate enum since LLVM 19, but the guard in codegen.cpp opened at 21.
  LLVM 19 and 20 therefore took the old branch and failed with "no member
  named 'True' in 'llvm::LazyValueInfo'".

- LLVM 20 added a ThinOrFullLTOPhase argument to the optimizer-last
  extension point callback, so the sanitizer pass registration in
  codegen_module.cpp no longer converted to the expected std::function.
  Split the registration per version and keep the pass setup shared.

The remaining >= 21 guards were checked and are genuinely LLVM 21 API
changes (CaptureInfo replacing NoCapture, Triple-based TargetMachine
entry points), so they are left alone.

Verified by installing LLVM 18, 19 and 20 locally and building
Absolute-CodeGen and the absolutec driver against each.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The LLVM and WASI bootstrap steps captured their output into a variable
and only wrote it to the log after the command returned. When the LLVM
step hit its timeout it produced no output at all for the full 25
minutes, so the job log showed only the timeout with no indication of how
far the download had progressed. Pipe through Tee-Object instead, which
writes as output arrives.

This is a diagnostics change only. It does not address the underlying
problem that a cold cache requires downloading an ~1 GB SDK inside the
step timeout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The previous commit adapted the sanitizer callback at its call site, which
was the wrong layer and broke LLVM 21: codegen_pch.h already ships an
OptimizerLastEPCallback shim plus a macro that rewrites the registration,
so call sites are meant to keep passing the two-argument form and let the
shim add the LTO phase. Passing three arguments made the shim call the
callback with two.

The real defect is the shim's guard. LLVM 20 added the LTO-phase argument,
but the shim only activated at 21, so LLVM 20 had no adapter at all. That
guard also covered the unrelated Triple-based target APIs, which really
are LLVM 21, and bundling them hid the mismatch.

Split the block: the optimizer callback shim now activates at 20, the
Triple helpers stay at 21, and codegen_module.cpp goes back to the plain
two-argument lambda. LLVM 21 therefore returns to exactly the
configuration that was passing before, now shared with LLVM 20.

Verified by building absolutec against LLVM 18, 19 and 20 locally. LLVM 21
is not installable in this environment; it is covered by the restored call
site being byte-identical to the previously passing one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
With the build failures fixed, these jobs reached their test phase for the
first time and exposed two test-side assumptions.

Four debug-info tests emit IR, link through --build-exe and inspect the
result with the LLVM tools, but they were registered unconditionally. In
any ABSOLUTE_ENABLE_LLVM=OFF configuration they failed with "LLVM backend
is unavailable in this build", which is the configuration working as
intended rather than a defect. Guard them with ABSOLUTE_ENABLE_LLVM, the
pattern already used elsewhere in this file; the neighbouring semantic and
diagnostic debug-info tests are frontend only and stay unguarded. This
covers termux-host-contract and the windows-2022 build-and-test jobs, all
of which configure the backend off.

The debug-info IR assertion also required the llvm.dbg.declare intrinsic
spelling. LLVM 19 prints debug info as records instead, so the check fails
from 19 onward even though the emitted debug info is correct. Accept
either spelling.

Verified locally: the termux contract configuration goes from 4 failures to
178/178, the debug-info tests pass against both LLVM 18 (24 intrinsics) and
LLVM 19 (23 records), and the full suite is 501/501 on LLVM 18.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The earlier fix defined _XOPEN_SOURCE inside scheduler_fiber.h, which
tasks.cpp includes after about twenty standard library headers. That is
enough to satisfy the #error in Darwin's <ucontext.h> and make the build
succeed, but a feature macro only affects headers included after it: the
Darwin headers libc++ had already pulled in were parsed without it. The
declarations and the ucontext layout can therefore disagree within one
translation unit, which is consistent with the Bus error and SIGTRAP the
scheduler and async tests now hit on macOS.

Set the macro from the command line for that source file instead, so it
applies before any header, and pair it with _DARWIN_C_SOURCE so the
stricter feature set does not hide BSD extensions from the rest of the
unit. The header now states the requirement and fails loudly if it is
missing, rather than silently defining it too late.

This removes a real class of undefined behaviour. It is not confirmed to
be the cause of the macOS failures: if they persist, the remaining
explanation is that the ucontext routines are unusable on arm64 Darwin,
which needs a different fiber backend and is a design decision rather
than a fix.

Verified on Linux, where the Apple branch is inert: tasks.cpp compiles and
Absolute-Runtime links.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS

FuryBaM commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Status on the two jobs that are still red, since both are now past the failures this PR set out to fix.

macos-smoke — build fixed, tests exposed a deeper issue

BUILD: success now, so the _XOPEN_SOURCE compile break is resolved. The test phase reports 17 failures out of 501 (97% passing), and they cluster tightly:

Test Signal
runtime-scheduler-v2 Bus error
runtime-scheduler-work-stealing, -fairness, -metrics, -stress SIGTRAP
run-async, run-task-scheduling, run-task-group, run-async-methods, run-async-concurrency-advanced SIGTRAP
run-exceptions, run-defer, run-std-http, run-std-concurrent-primitives, run-std-typed-channel SIGTRAP
abi-linker-corpus-native, run-sanitizer-ownership-stability Failed

Everything in the SIGTRAP group runs through the fiber scheduler. These tests have never executed on macOS in this branch's history — the job previously died at compile time — so there is no evidence they ever passed there.

Before attributing this to the platform I revisited my own fix. The first version defined _XOPEN_SOURCE inside scheduler_fiber.h, which tasks.cpp includes after roughly twenty standard library headers. That satisfies the #error in Darwin's <ucontext.h>, but a feature macro only affects headers included after it, so the Darwin headers libc++ had already pulled in were parsed without it — leaving the declarations and the ucontext_t layout potentially inconsistent inside one translation unit. That is consistent with a Bus error and SIGTRAP. The latest commit moves the definition to the build system for that source file so it applies before any header, paired with _DARWIN_C_SOURCE, and the header now fails loudly instead of defining the macro too late.

To be clear about what that is and is not: it removes a real class of undefined behaviour, but it is not confirmed to be the cause. If these tests still crash, the remaining explanation is that the ucontext routines are unusable on arm64 Darwin and the runtime needs a different fiber backend there — a design decision, not a fix, and out of scope for this PR.

windows-llvm-release — blocked, and not something I am fixing here

This one is not a code defect and I am deliberately not patching around it:

Cache not found for input keys: windows-llvm-18.1.8-sdk-v1
##[error]The action 'Bootstrap portable LLVM SDK' has timed out after 25 minutes.

Downloading the ~1 GB LLVM SDK exceeded both the original 15-minute timeout and the 25 minutes this PR raised it to. Because the job fails, actions/cache never saves, so the next run starts cold again — a self-sustaining loop. It passed earlier only against a warm cache that has since been evicted.

What this PR does contribute here is honesty: the diagnostics step no longer throws on the missing llvm-config.exe and fail the job at that step, and the bootstrap steps now stream their output instead of buffering it into a variable that is discarded on timeout — which is why the earlier run showed 25 minutes of complete silence followed by a misleading "not recognized as a name of a cmdlet" error.

Breaking the loop needs a provisioning decision — seeding or persisting the SDK cache, caching the partial download so curl --continue-at - can resume across runs, or pinning a smaller toolchain. That is a call for the repository owner.


Generated by Claude Code

FuryBaM commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Correction to my earlier comment on windows-llvm-release. I attributed the timeout to the ~1 GB SDK download. That was wrong, and the streamed bootstrap output added in this PR is what disproved it:

12:36:45  Downloading LLVM SDK 18.1.8 (about 1 GB)...
12:37:02  100  936M  100  936M    0     0  56.0M      0  0:00:16  0:00:16 --:--:-- 55.9M
12:37:02  Extracting LLVM SDK...
13:01:41  ##[error]The action 'Bootstrap portable LLVM SDK' has timed out after 25 minutes.

The download completes in 16 seconds at ~56 MB/s. The step then spends the entire remaining ~24.5 minutes inside tar.exe -xf on the .tar.xz, and never reaches the extraction-verification step. So raising the timeout — which this PR also did, 15 → 25 minutes — was aimed at the wrong phase and would not have helped at any value.

The real bottleneck is decompressing and writing out the archive on windows-2022. Two things plausibly contribute, both independent of this repository's code:

  • tar.exe (bsdtar) decompresses xz single-threaded, and this archive expands to several GB across tens of thousands of small files.
  • Windows Defender real-time scanning of every extracted file, which is a known pathology for large toolchain extractions on GitHub's Windows runners.

That reframes the fix away from timeouts and toward the extraction itself — for example excluding the .absolute/toolchains path from Defender before extracting, decompressing the .xz with a multi-threaded tool and then untarring, or fetching a pre-extracted or .zip toolchain. I am not making that change here: I cannot exercise a Windows runner from this environment, and picking among those options is a provisioning decision for the repository owner.

The cache interaction I described still holds: because the job fails, actions/cache never saves the SDK, so every run starts cold and repeats the extraction.


Generated by Claude Code

claude added 2 commits August 8, 2026 13:03
The streamed bootstrap output shows the ~1 GB SDK download finishing in 16
seconds at ~56 MB/s, after which the step spends the remaining 24.5 minutes
inside tar.exe unpacking the .tar.xz before being killed. The comment
added with the timeout increase claimed the step was killed mid-download,
which is wrong and would point a reader at the wrong phase.

Comment only; the 25-minute value is left as is. Raising it further would
not help, since the overrun is in extraction rather than transfer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
The streamed bootstrap output showed where windows-llvm-release actually
spends its time: the ~1 GB SDK downloads in 16 seconds at ~56 MB/s, then
tar.exe runs for the remaining 24.5 minutes and is killed. Real-time
antivirus scanning of every file written is the usual cause of that ratio
on CI runners, since the archive expands to several GB of mostly small
headers.

Add a best-effort Defender exclusion for the toolchains directory before
extraction, and report how long extraction took. The exclusion is wrapped
in a try/catch so a machine without Defender, or without the privileges to
configure it, still extracts as before.

The timing line makes the next run informative either way: if extraction
is still slow, the number rules the antivirus theory out rather than
leaving it open.

Not verifiable from this environment, which has no Windows runner and no
PowerShell to parse the script; CI is the test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS

FuryBaM commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Second correction, this time to what I said about macos-smoke. I wrote that with the _XOPEN_SOURCE fix ruled out, "the remaining explanation is that the ucontext routines are unusable on arm64 Darwin". That is poorly supported, and the evidence now points somewhere more useful.

The signals line up across architectures. LLVM lowers a reached unreachable to brk #1 on arm64, which raises SIGTRAP, and to ud2 on x86-64, which raises SIGILL. What CI shows:

  • macOS (arm64): 17 failures out of 501, of which 14 are SIGTRAP and one is a Bus error, all clustered on the scheduler, async, exceptions, defer and channel tests
  • Linux (x86-64): collection-snapshot-stress dies with Illegal instruction (core dumped), exit 132, in the same family of concurrency tests

Same defect class, different lowering, rather than two unrelated platform problems.

Reaching an unreachable here implies UB, not a bounds failure. The only two CreateUnreachable sites are after Abort() in the assert builtin and after ExitFailure() in EmitOrExit. A genuine out-of-bounds access therefore prints "Array index out of bounds" and exits 1 — it does not trap. So a trap means the optimizer proved a path impossible, which is what LLVM does when the program has undefined behaviour on that path.

The fiber code is not arm64-specific. getcontext/makecontext/swapcontext are used on Linux too; Darwin only differs in needing _XOPEN_SOURCE to declare them. So "arm64 ucontext is broken" does not explain the Linux SIGILL, whereas a latent memory-safety bug in the scheduler or generated code explains both.

What I cannot do from here. The Linux failure is intermittent — two runs of the identical SHA 23be46b gave opposite results for build-and-test (ubuntu-24.04, Release) and language-stress-and-fuzz, and the stress binary passed 300 consecutive local runs — and I have no macOS runner. Pinpointing UB needs a reproduction, ideally the failing binary under ASan or TSAN on a runner where it actually fails.

I am not attempting a fix. Both the fiber scheduler and the proven-bounds elimination this branch introduces are plausible places to look, and choosing where to dig is better done by someone who can reproduce it.


Generated by Claude Code

FuryBaM commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Result of the Defender-exclusion commit on windows-llvm-release. It rules the antivirus theory out rather than confirming it:

13:09:26  Downloading LLVM SDK 18.1.8 (about 1 GB)...
13:09:35  100  936M  100  936M    0     0   103M      0  0:00:09  0:00:09 --:--:--  110M
13:09:36  Defender exclusion added for D:\a\Absolute\Absolute\.absolute\toolchains
13:09:36  Extracting LLVM SDK...
13:34:22  ##[error]The action 'Bootstrap portable LLVM SDK' has timed out after 25 minutes.

The exclusion was applied successfully — the cmdlet succeeded and printed the path — and extraction still consumed the entire remaining 24.8 minutes. The Extracted in Ns line never appeared because tar.exe never returned. The download, meanwhile, took 9 seconds at ~110 MB/s.

So the picture is now narrow and well established:

  • it is not the download, which is an order of magnitude faster than the timeout
  • it is not real-time antivirus scanning, which is now excluded for the target tree
  • tar.exe alone spends >24.8 minutes unpacking a 936 MB .tar.xz that expands to several GB of mostly small files

What remains are provisioning changes with real tradeoffs, none of which I can validate from this environment — there is no Windows runner here and no PowerShell to even parse the script, and each attempt costs a 25-minute CI cycle:

  • decompress with 7-Zip, which is present on the runners and multi-threads the xz decode, then untar as a second pass
  • avoid the general-purpose archive: fetch a pre-extracted toolchain, or a .zip variant, or a trimmed subset
  • seed the SDK cache once out of band so the cold path stops being exercised on every run

I am stopping here rather than iterating blind. Earlier in this PR a change I made without being able to run it broke a job that had been green, and repeating that pattern against a 25-minute feedback loop is not a good trade. The instrumentation is the durable part: whoever picks this up now gets a per-phase timing instead of 25 minutes of silence.


Generated by Claude Code

claude added 2 commits August 8, 2026 14:25
The instrumentation added in the previous commit settled where the time
goes. The download finishes in 9 seconds at ~110 MB/s, the Defender
exclusion is applied successfully, and tar.exe still consumes the entire
remaining 24.8 minutes without returning. So neither transfer nor
antivirus scanning explains it: bsdtar decoding this xz stream on a single
thread does.

7-Zip ships on the hosted Windows images and decodes xz in parallel. Split
extraction into an xz pass and a tar pass through it, timing each, and keep
tar.exe as the fallback for any machine where 7-Zip is missing or the pass
does not produce the expected archive. The Defender exclusion is retained:
it was not the fix, but it is cheap and large file writes are genuinely
slower under real-time scanning.

The per-phase timings mean the next run distinguishes slow decompression
from slow file creation, whichever way this lands.

Not verifiable here: no Windows runner and no PowerShell to parse the
script, so CI is the test. The fallback keeps the failure mode no worse
than the current one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS
Building the stress programs with --sanitize=address surfaced a leak that
reproduces on every run of collection-snapshot-stress: 6048 bytes in 12
allocations, attributed to std.collections.Map.put and Set.add.

The growth paths allocate a larger array, copy the live elements across,
and then assign it with copy(...). That stores a duplicate in the field and
leaves the freshly allocated local unreferenced and unreleased. The
arithmetic matches exactly: six doublings from capacity 4 to 256 leak
8*(8+16+32+64+128+256) = 4032 bytes for the 8-byte KeyValuePair entries and
4*(...) = 2016 for the 4-byte set items, 6048 together, in 6 objects each.

Assign with move(...) so the field takes ownership of the array that was
just built. vector.abs already does exactly this at its own growth sites,
so this brings map, set, deque and priority_queue in line with the idiom
the standard library had already settled on rather than introducing a new
one. A plain assignment is rejected by the analyzer, which names move(...)
among the accepted forms.

The remaining copy(...) uses are left alone: iterator snapshots, the
copy-on-write in ensureUnshared, and the rehash locals in hash_map are all
genuine copies.

Verified with ASan: collection-snapshot-stress goes from 6048 bytes leaked
to a clean exit. The full suite is 501/501 on LLVM 18.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SPqntJmyXNSmZNzuQWj5QS

FuryBaM commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Correction, and a good one: the move() change fixed macos-smoke.

I previously reported that ASan on Linux did not reproduce the macOS failures, that my unified-defect hypothesis was unsupported, and that the cluster looked platform-specific and needed a Mac. That was wrong. The leak ASan found was the same defect.

Both parallel macOS runs on e14ea98 now report:

99% tests passed, 1 tests failed out of 501
The following tests FAILED:
	337 - absolute.run-sanitizer-ownership-stability (Failed)

17 failures down to 1. All 14 SIGTRAPs and the Bus error are gone. The only change in that commit is copy(...) to move(...) in five collection growth paths, so the causal link is direct, and it reproduced across both runs.

The mechanism, inferred rather than directly observed: the growth paths built a new array, stored a duplicate of it in the field, and dropped the original. On Linux the duplicate kept everything reachable, so the damage surfaced only as the 6048-byte leak ASan reported. On macOS/arm64 the same mismanaged ownership left the collection referring to memory it did not own, which is exactly the profile of the SIGTRAP and Bus error crashes — a reached unreachable and a bad access. That also explains why the failures clustered on the scheduler, channel and async tests: those are the ones that drive the collections hard enough to trigger reallocation.

Worth noting for the record: vector.abs already used move() at its own growth sites, so map, set, deque and priority_queue were the outliers, not the other way round.

The one remaining macOS failure, run-sanitizer-ownership-stability, was already among the original 17 as a plain Failed rather than a signal crash, and it passes on Linux. It is a separate, pre-existing, macOS-specific problem and is now the only thing standing between this branch and a green CI required gate.


Generated by Claude Code

@FuryBaM
FuryBaM merged commit b286d07 into agent/stabilize-foundation Aug 8, 2026
27 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants