Skip to content

[DO NOT MERGE] Reproduce flaky Rust parallel cargo test SIGSEGV - #2818

Closed
bdice wants to merge 8 commits into
NVIDIA:mainfrom
bdice:ci-repro-rust-parallel-segv
Closed

bdice wants to merge 8 commits into
NVIDIA:mainfrom
bdice:ci-repro-rust-parallel-segv

Conversation

@bdice

@bdice bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Reproduction PR for #2233 (Rust cargo test with default parallel test threads intermittently SIGSEGVs). Part of #2783.

CI-only changes:

  • pr.yaml reduced to conda-cpp-build (amd64, latest CUDA) and 4 copies of rust-build (2x rtxpro6000, 2x l4).
  • ci/stress_rust.sh loops the cuvs lib test binary with default, 8, 16, 32 and 64 test threads (some iterations under gdb), plus #[ignore]d multi-thread stress tests where each thread owns its own Resources, and a compute-sanitizer memcheck pass.

@copy-pr-bot

copy-pr-bot Bot commented Oct 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f4f7959

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 72a1e10

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 57efb6d

bdice added 4 commits October 11, 2026 00:40
The pinned host memory resource behind cuvsRMMHostAlloc/cuvsRMMHostFree
was a thread_local std::unique_ptr that was only created on the first
cuvsRMMHostAlloc call of each thread. Freeing a buffer on any other
thread dereferenced a null pointer. pinned_host_memory_resource is
stateless, so use a single process-wide instance instead.
adjust_centers() advanced a function-local static counter with a plain
read-modify-write. Concurrent builds (IVF-Flat, IVF-PQ, ...) from
several host threads, each with its own resources, raced on it.
@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d9e5779

@bdice

bdice commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Closing this reproducer: the parallel cargo test SIGSEGV did not reproduce in 6,700 iterations, so no fix is needed here. The related bug it found is fixed in #2832; tracked in #2783.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant