Skip to content

fix(ffi): the nightly sanitizers run on the arm runner - #921

Merged
kixelated merged 16 commits into
mainfrom
quest/m1/ffi-proofs/arm-sanitizers
Nov 20, 2026
Merged

kixelated merged 16 commits into
mainfrom
quest/m1/ffi-proofs/arm-sanitizers

Conversation

@kixelated

@kixelated kixelated commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m1/ffi-proofs/arm-sanitizers.md: the nightly's ffi-sanitize job on ubuntu-24.04-arm passes.

Cause

Not the suspects the quest named. The runner already has vm.mmap_rnd_bits = 28, and the clang/compiler-rt mismatch was not it either (a clang 23 shell still crashed). gdb on the failing api-c gave the backtrace:

atomic_load<u8>  (dst = compiler_builtins::aarch64_outline_atomics::HAVE_LSE_ATOMICS)
__aarch64_swp1_acq            <- compiler_builtins, built by -Zbuild-std with -Zsanitizer
__asan_init  (asan_inited_mutex)
_dl_init

On aarch64 the sanitizer runtimes use outline atomics: their own atomics call __aarch64_* helpers. -Zbuild-std -Zsanitizer=... builds compiler_builtins instrumented, and the link takes its __aarch64_* over the toolchain's. Its helper reads HAVE_LSE_ATOMICS through a shadow check, before __asan_init/__tsan_init has mapped the shadow, so the first program run segfaults with no output (TraceSwitchPart with a null thread state under TSan). x86_64 has no such helpers, hence only arm failed.

Fix

sanitized_glue (ffi/tests/fixtures.rs) adds -Ctarget-feature=-outline-atomics on aarch64 hosts. compiler_builtins then defines no __aarch64_*, so the link takes the toolchain's uninstrumented ones. Checked by cross-building for aarch64 locally: the flag takes the module's symbols from 8 swp1_acq references to 0, while +lse and -Ctarget-cpu=neoverse-n2 leave them. It is a no-op on x86_64, where the flag would warn.

Also in this PR:

  • ffi/README.md says why in one sentence.
  • Deletes the quest, and the proofs epic README, which completes once its last child is gone. quest/m1/README.md and quest/m1/ffi-export/README.md drop their links to it.

Validation

  • Ran on the arm runner, from this branch with only ffi-sanitize kept (nightly dispatch, run 44765819782): just ffi sanitize passed all 7 tests (api_asan, api_tsan, api_valgrind, api_swift_tsan, api_go_checked, api_python_debug, api_miri), 560 s. Before the fix: three SIGSEGVs, reproduced on runs 44750635586, 44751584607, 44753208891 (the earlier ones just ran the tests separately, which showed signal: 11).
  • just ci passes locally on x86_64 Linux. The workflow file and flake.nix are unchanged relative to main (the trimmed workflow and diagnostics were temporary commits, net-reverted; the branch history keeps them).
  • The nightly's ffi-sanitize will not run until the next scheduled nightly on main; no dispatch of the full workflow was made.

Public API / wire

None. Test harness only.

Decisions (taken without the maintainer, who was away)

  • Fix the cause in the harness (no sysctl, no x86 runner): nothing in the quest's plan holds, so the leading suspects were dropped.
  • -outline-atomics over keeping -Zbuild-std but uninstrumenting compiler_builtins, or dropping -Zbuild-std (which would leave std's non-generic code unchecked). It changes aarch64's atomics codegen in the glue under test from outline to inline LL/SC; the sanitizers see no difference in what they check.
  • Delete the proofs epic README. Open #906 adds quest/m1/ffi-proofs/loom-hold.md under it; whichever lands second needs the README back (a directory under quest/ without one fails quest check). Either keep the epic by restoring its README in perf(ffi): Kotlin's checked close in one JNI crossing #906, or move that quest.

Follow-ups

None added.

Models

Size: S
Implementer: Claude Code, claude-sonnet-5-5, unknown effort (arm B)

(Written by Claude Sonnet 5.5)

Review

Rewritten by quest review after each round.

  • Head: 3543971

  • Base: origin/main at 32e99fe

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1f9bc-7979-7081-a3a3-071f3142d113, round 1

  • Score: 9/10, No actionable defects were found in the focused aarch64 sanitizer workaround, but builds and tests remain unverified.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

Summary by CodeRabbit

  • Bug Fixes
    • Sanitizer builds on aarch64 now disable outline atomics, while builds on other architectures retain their existing target-feature settings.
  • Documentation
    • Updated project planning notes to reflect the current list of remaining work.
    • Removed documentation covering exporter proofs and ARM sanitizer investigations.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The sanitizer fixture disables outline atomics only on aarch64, and the ffi README documents this setting. Quest planning text changes its remaining-work lists, and two ffi-proofs documents are removed.

Changes: Sanitizer Build Configuration

Layer / File(s) Summary
Conditional outline-atomics flag
ffi/tests/fixtures.rs, ffi/README.md
The sanitizer fixture adds -Ctarget-feature=-outline-atomics on aarch64 and adds no target-feature flag on other architectures. The README documents the setting and its association with sanitizer initialization failure.

Quest Planning Documents

Layer / File(s) Summary
Proofs planning updates
quest/m1/README.md, quest/m1/ffi-export/README.md, quest/m1/ffi-proofs/README.md, quest/m1/ffi-proofs/arm-sanitizers.md
The planning text no longer lists the proofs epic as remaining work. The ffi-proofs overview and ARM sanitizer notes are deleted.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: pullfrog

Merge Risk: Minimal · up to 35439

This change only affects the sanitizer test harness on aarch64 and removes planning documents. It does not change the public API or the wire format, and no merge-blocking risk was identified.

Security architecture risk: Minimal · up to 35439

The ARM-specific flag changes how test libraries are compiled while preserving sanitizer instrumentation and failure checks. No introduced or worsened architecture-level security or failure-containment risk was established.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the fix and the area it touches (the nightly sanitizer job on the ARM runner), and the changeset matches that. The change adds an aarch64-only -Ctarget-feature=-outline-atomics flag …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the fix and the area it touches (the nightly sanitizer job on the ARM runner), and the changeset matches that. The change adds an aarch64-only -Ctarget-feature=-outline-atomics flag in sanitized_glue, documents the reason in ffi/README.md, and removes the completed sanitizer and proofs quest docs. The title does not mention the README or quest-doc cleanup, but it names the primary change. It also does not mention the flag or outline atomics, which is acceptable for a short title.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 01387940-9418-4fc2-91f7-93f3f706f998
Commits

Reviewing files that changed from the base of the PR and between 32e99fe and 3543971.

Files selected for processing: 6

  • ffi/README.md
  • ffi/tests/fixtures.rs
  • quest/m1/README.md
  • quest/m1/ffi-export/README.md
  • quest/m1/ffi-proofs/README.md
  • quest/m1/ffi-proofs/arm-sanitizers.md
Files with no reviewable changes: 2

  • quest/m1/ffi-proofs/README.md
  • quest/m1/ffi-proofs/arm-sanitizers.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

@kixelated kixelated changed the title The nightly sanitizers run on the arm runner fix(ffi): the nightly sanitizers run on the arm runner Nov 20, 2026
@kixelated
kixelated marked this pull request as ready for review November 20, 2026 16:14
@kixelated
kixelated enabled auto-merge November 20, 2026 16:20

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound — fixes the cause in the harness by building the sanitized glue without outline atomics on aarch64, so the link takes the toolchain's uninstrumented __aarch64_* helpers.

Reviewed changes — the full diff across 6 files.

  • sanitized_glue flag — ffi/tests/fixtures.rs appends -Ctarget-feature=-outline-atomics to RUSTFLAGS only under cfg!(target_arch = "aarch64"), so x86_64 is unchanged.
  • README note — ffi/README.md explains the workaround in one sentence.
  • Quest cleanup — deletes arm-sanitizers.md and the proofs epic README, and drops the links to them from quest/m1/README.md and quest/m1/ffi-export/README.md. A repo grep finds no dangling references to either.

The #906 interaction (it adds loom-hold.md under the deleted epic) is already called out in the PR body as a maintainer decision.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 24ff182 Nov 20, 2026
13 checks passed
@kixelated
kixelated deleted the quest/m1/ffi-proofs/arm-sanitizers branch November 20, 2026 16:27
@codspeed

codspeed Bot commented Nov 20, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.27%

1 warning

⚠️ 27 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 82 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ write 680.3 ms 906.6 ms -24.96%
❌ loop 8.8 s 11.5 s -23.95%
⚡ throughput[16384] 1.6 s 1.3 s +22.41%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing quest/m1/ffi-proofs/arm-sanitizers (3543971) with main (845b366)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (32e99fe) during the generation of this report, so 845b366 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant