Skip to content

claim-test: concurrent mkdir CAS race fails intermittently on ubuntu CI #480

Closed

Description

@drchristensen

Problem

tests/fixtures/quest-log/claim-test.sh has a simultaneous-claims test (rounds 1–10) that launches two background claim-acquire processes and expects exactly one to exit 0 and the other to exit 6. The stub gh uses mkdir as its CAS primitive.

On ubuntu CI runners (azure eastus2, runner 20261021.618), both processes occasionally return exit 0 — the mkdir race fires — causing:

claim-test: round 1: exits 0:0, expected one 0 and one 6

Evidence

Two consecutive failures on PR #479 (only documentation and version-bump changes):

  • Run 40908299894, attempt 1: round 1 fails (exits 0:0)
  • Run 40908299894, attempt 2 (rerun --failed): round 2 fails (exits 0:0)

Last successful main run on same runner version: Oct 28 (run 40612584652), claim-test passed.

The test passes consistently on macOS CI and locally.

Root cause hypothesis

The stub gh does two separate steps to atomically claim: mkdir -p "$GH_STORE/quest-claim" (parent) then mkdir "$GH_STORE/$name" (leaf, atomic CAS). Under high concurrency or IO pressure, both background processes may complete their mkdir -p (parent, idempotent) before either completes the leaf mkdir, and the kernel mkdir(2) atomicity guarantee occasionally fails on this runner configuration. Alternatively, the two-step sequence creates a window where a third party could interfere.

Proposed fix

Use a single atomic mkdir -p + unique name or a (set -C; >file) pattern for the CAS instead of two separate mkdir calls. Or pre-create the parent directory in new_store() so both background processes skip the mkdir -p step and go directly to the atomic leaf mkdir.

Activity

  1. randomparity commented on Oct 29, 2026

    @randomparity
    Owner

    Scope — issue #480

    Tracking: blast radius — one test fixture; change hazards — none (shipped code unchanged; the fix restores the stub's intended atomicity); complexity — S; decompose — one PR; classification — trivial-bugfix; artifact lane — no-spec; review depth — single-pass; design denominator — not-applicable (no-spec).

  2. added
    type:bugIncorrect behavior; something is broken
    status:in-reviewadversarial review of the branch running; a PR may not be open yet
    and removed
    documentationImprovements or additions to documentation
    on Oct 29, 2026
  3. randomparity commented on Oct 29, 2026

    @randomparity
    Owner

    Trajectory — issue #480

    • Outcome: hand-off; PR ready for the maintainer to merge.
    • Branch/PR: feat/claim-cas-480, PR fix(test): make claim-test stub CAS independent of mkdir #481.
    • Guardrails: just ci green via pre-push hook at 915ae4b; CI green on ubuntu-latest (image ubuntu-26.04) and macos-latest.
    • Root cause: ubuntu-latest moved to 26.04, whose uutils mkdir 0.8.0 is non-atomic under a race; the stub CAS is now a bash noclobber (O_EXCL) create.
    • Review: single-pass, approve; 3 notes dispositioned (2 accepted-fixed, 1 rejected-with-evidence); security not triggered; deferrals none.
    • Handshake: MERGE-READY: fix(test): make claim-test stub CAS independent of mkdir #481 @ 915ae4b
  4. added
    status:awaiting-mergegreen + mergeable; human just clicks merge
    and removed
    status:in-reviewadversarial review of the branch running; a PR may not be open yet
    on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status:awaiting-mergegreen + mergeable; human just clicks mergetype:bugIncorrect behavior; something is broken

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions