Skip to content

fix(apps): create dependency locks exclusively - #10464

Merged
bolichen97 merged 2 commits into
mainfrom
fix/app-deps-lock-create-race
Sep 13, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
fix/app-deps-lock-create-race

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Concurrent first-time app dependency provisioning was observed to fail on macOS while creating .kirocrew-deps.lock: one directory-relative openat(O_CREAT) returned ENOENT before reaching the file lock, although the parent directory remained valid. The data-preserving uninstall path opens the same lock and had the same creation pattern.

Why it matters

The per-app lock should serialize dependency work, not reject an otherwise valid app start before locking. Both provisioning and uninstall must use compatible lock creation so neither retains the observed first-create failure mode.

What changed (motivation → approach → change)

Both lock openers now elect a creator with O_CREAT | O_EXCL. Only FileExistsError permits a second open, with no creation or truncation flags. Both attempts retain O_RDWR, O_NOFOLLOW where supported, and the same pinned directory descriptor (or the existing path-based fallback).

If the lock disappears before reopening, the operation fails closed rather than recreating it. The existing file-lock, install-stamp, uninstall rollback and trust-handling policies are unchanged. No broad retry, lock deletion, dependency-policy change or new shared abstraction is introduced.

Case Before After
Concurrent first lock creation 🟥 Observed ENOENT before locking on macOS 🟩 Exclusive creator election, then open the existing lock
Lock already exists 🟦 Reuse its inode 🟦 Reuse its inode without truncation
Uninstall opens the dependency lock 🟥 Retains the old creation pattern 🟩 Uses the same election as provisioning
Lock disappears before reopen 🟦 Existing error/rollback handling 🟨 Refuse recreation and use existing error/rollback handling
Lock is a symlink on an O_NOFOLLOW platform 🟦 Refused 🟦 Refused
First installer finishes 🟦 Waiter reuses its install stamp 🟦 Waiter reuses its install stamp

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

Every opener of this dependency lock now uses the same creator-election discipline.

Tests

  • Provisioning: fresh/existing locks in both pinned and fallback paths; unchanged contents; creation-free reopen; vanished-lock refusal; symlink target untouched.
  • Concurrent provisioning: a barrier aligns both first-create attempts; both callers succeed, installations never overlap and pip runs exactly once. The instrumentation preserves os.supports_dir_fd capability detection.
  • Uninstall: fresh/existing lock creation flags; the same descriptor on reopen; no truncation; user data preserved; a vanished lock refuses uninstall and leaves the app manifest and user data in place.
  • Three provisioning regression cases and three uninstall cases were each observed failing before their respective production fixes.

Validated locally with 62 related install/uninstall tests run serially (-n 0), flake8, isort, Python compilation and a backend wheel build. After incorporating the base-only frontend update, the entire PR patch was verified byte-for-byte identical to the reviewed patch; the same 62 serial tests and wheel build passed again. Full matrices run in CI, not on the local Mac.

Manual verification

The original production-path failure reproduced on clean main. Two independent native probes reproduced the concurrent directory-relative creation failure with valid descriptors for the same directory and no intervening deletion; exclusive creation followed by a creation-free reopen passed those probes. These observations are from one macOS host, not a claim about all Darwin releases or a linked upstream kernel defect. The deterministic tests check the lock-opening protocol independently of whether the scheduling race fires.

The original provisioning patch passed local GPT/Opus review. The uninstall follow-up passed a separate read-only verifier; the base synchronization did not change any PR patch bytes. No production services or live app installations were changed. Published-head cross-platform validation remains with CI.

Screenshots / video

N/A — no rendered UI change in this PR.

Related Issues

no linked issue: discovered as a mainline test failure while validating PR #10276; kept separate from that PR's path-resolution fix.

Pattern harvest

Rule candidate: review-prompt
Pattern: when fixing lock-file creation, inspect every opener of that inode; use exclusive creator election followed by a creation-free reopen without weakening pinned-directory or no-follow checks.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Use exclusive creation before opening an existing dependency lock.
Concurrent macOS openat calls could otherwise fail with ENOENT before
reaching flock. Preserve directory pinning and no-follow flags, and
fail closed if the lock disappears before reopening.

Force overlapping first creation in the regression test and verify
that the second installer reuses the first install's stamp.
@bolichen97
bolichen97 requested review from a team and cixuuz September 13, 2026 02:58
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Make concurrent first-time app dependency provisioning acquire the same lock reliably on macOS, so the second installer waits and reuses the first install's stamp.
Not a goal: Change dependency installation policy, directory pinning, symlink handling, or the path-resolution deadline work in #10276.

Validation: 42 dependency-install tests ran serially on head 517d77e9f22c41092caaff86a20f741770aca924, plus flake8, isort, compilation and a backend wheel build. The patch is unchanged (git range-diff reports =) from 013faa08b, which passed the full resolved local gates and both GPT/Opus reviews before the base-only updates. Full platform validation for the published head belongs to CI; no more full-suite runs are being launched on the local Mac.

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 99ab31c434c424b0f9f6915f05fa5e9ac89f5525 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 99ab31c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 99ab31c434c424b0f9f6915f05fa5e9ac89f5525: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 99ab31c434c424b0f9f6915f05fa5e9ac89f5525 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

A reproduced platform failure gets a strictly-safer protocol change — creator election, no truncation, fail-closed on vanish — pinned by tests at both openers and self-healing on retry.

Suggestions

  • The creator-election protocol is now hand-copied in backend.py and manager.py; this PR exists because those copies drifted once already. A tiny shared open_deps_lock(pin, path) helper would make future drift impossible instead of merely test-detected.

[DESIGN-REVIEWED] 99ab31c

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 99ab31c434c424b0f9f6915f05fa5e9ac89f5525 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The macOS ENOENT-on-concurrent-openat(O_CREAT) premise exists only as the description's word — no issue, probe artifact, or failing-on-base repro — and ~30 sibling lock openers share the pattern it indicts.

Not justified as shipped

  1. Provisioning lock creator election — unverified premise: the defect has "no linked issue" and the PR's own tests "check the lock-opening protocol independently of whether the scheduling race fires", so nothing checkable reproduces it.
  2. Uninstall lock creator election — unverified premise: rests entirely on the same uncheckable macOS observation.

What this change ships

Inventory (6 items) — 4 justified

Intent: stop concurrent first-time app dependency provisioning from failing on macOS before the per-app lock is even acquired — a FIX.

  1. Concurrent first-time provisioning on macOS no longer fails with ENOENT before locking — unverified premise (see above)
  2. Uninstall opens the same lock with the same creator election — unverified premise (see above)
  3. A lock that vanishes before reopen now fails the operation instead of recreating it — justified
  4. Spec §11.1 added in the same commit documenting the serialization — justified
  5. Concurrency test now bounds its joins and asserts pip runs exactly once — justified
  6. Protocol tests for both openers (flags, no truncation, symlink refusal) — justified

Watch

  • The core claim — "Concurrent openat(O_CREAT) of an absent file can return ENOENT on macOS" — is hedged by the author themself ("observations are from one macOS host, not a claim about all Darwin releases or a linked upstream kernel defect") and nothing in the PR reproduces it. If it is false, the change is harmless but baseless. Clears when: the native probe or the mainline failure log is linked, or an issue records the reproduction.
  • Breadth is unresolved either way: grep O_CREAT in src/ counts 32 core lock openers (43 with builtin apps) still using O_CREAT without O_EXCL — including the shared helper platform_compat.open_lock_file (platform_compat.py:705), gateway_lock.py:130, session_storage.py:1113, jsonl_util.py:463. If the race is not dir_fd-specific, this is a 2-of-32 point patch; if it is dir_fd-specific, the path-based fallback arm's election in both hunks rides beyond the observed defect. Clears when: the defect is pinned to dir_fd or not (probe/issue), and if general, the siblings are tracked in a linked issue.

[FIRST-PRINCIPLES-REVIEWED] 99ab31c

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 99ab31c434c424b0f9f6915f05fa5e9ac89f5525 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 99ab31c

Verdict parsed from the review's SHA-scoped output markers for commit 99ab31c434c424b0f9f6915f05fa5e9ac89f5525.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 99ab31c434c424b0f9f6915f05fa5e9ac89f5525: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 13, 2026
@bolichen97

bolichen97 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted — span=13dc1f97106a — require a public macOS issue or a deliberately failing CI run for the observed first-create race

The claim is limited to the observed macOS host, not a guarantee about every Darwin release or a diagnosed kernel source line. The original application test failed on both the feature tree and pristine main. Two independent investigations reproduced the native directory-relative open failure without application imports or directory removal. Their exclusive-create-then-reopen probes succeeded. These are local observations, not published upstream or macOS CI evidence. The directory descriptors remained valid and named the same inode.
The original flags are now rejected by deterministic fresh/existing-lock tests, which failed before the fix independently of whether the scheduling race fired. The concurrent production-path test additionally proves both callers succeed and pip runs once. The change uses standard exclusive creation and preserves the existing no-follow, pinned-directory and fail-closed behavior; it does not depend on attributing the observed syscall behavior to a particular Darwin/APFS implementation defect. Requiring another full CI experiment or an external issue does not change those code-level guarantees. Native Windows/Linux coverage remains with this PR's CI.

Use the same exclusive-create-then-open discipline for the uninstall
side of the shared dependency lock. Preserve directory pinning,
no-follow flags and data rollback on a vanished lock.

Include current main without rewriting published history. The five
lock-fix files are identical to the reviewed local 9dc90e41 commit;
62 targeted tests run serially and the backend wheel build pass.
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no

  • fixed — span=f377dfa4e16d — align the uninstall opener of the same dependency lock

Fixed in 99ab31c434c424b0f9f6915f05fa5e9ac89f5525: apps/manager.py now elects a creator with O_CREAT | O_EXCL, catches only FileExistsError, and reopens without creation/truncation flags. It retains the same pinned directory descriptor, O_RDWR, and O_NOFOLLOW where supported, matching provisioning without introducing another abstraction.
Three new uninstall cases failed before the fix. After it, 62 targeted install/uninstall tests passed serially, together with lint, import-order checks, compilation and a backend wheel build. The vanished-lock case refuses uninstall while preserving the installed app manifest and user data through the existing rollback path.
The focused read-only verifier passed local 9dc90e41ecff9cafa25f4b1bef184e99eafbcc79 with no tests executed. The entire PR diff is byte-for-byte identical after the base-only merge into the published fix SHA; the same 62 serial tests and wheel build passed on that tree. Published history was preserved, with two PR commits and no force push. Full cross-platform verification is left to the new CI run.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Native reproduction evidence for 99ab31c434c424b0f9f6915f05fa5e9ac89f5525

This is a standalone stdlib-only POSIX probe, not an application/test-suite run. I syntax-checked it and executed it once under a 15-second parent-process timeout. Cases and pairs run serially; only the two opens in each pair race, using one two-worker pool. Each pair starts with an absent lock in a new directory. The directory fd stays open until both calls return, and its device/inode are compared with the named directory before cleanup. There is no unlink/rename or application import during a pair.

Actual stdout:

{"system": "Darwin", "release": "25.6.0", "machine": "arm64", "python": "3.12.13", "pairs_per_case": 16}
{"relative": true, "exclusive": false, "outcomes": {"errno=2": 16, "ok": 16}, "eexist_reopens": 0, "parent_identity_stable": true}
{"relative": true, "exclusive": true, "outcomes": {"ok": 32}, "eexist_reopens": 16, "parent_identity_stable": true}
{"relative": false, "exclusive": false, "outcomes": {"ok": 32}, "eexist_reopens": 0, "parent_identity_stable": true}
{"relative": false, "exclusive": true, "outcomes": {"ok": 32}, "eexist_reopens": 16, "parent_identity_stable": true}

errno=2 is ENOENT. The original directory-relative pattern failed in every pair in this sample. Creator election removed the failures. The absolute-path control did not fail in this sample; that is not a proof that every absolute-path opener on every filesystem is immune, nor a diagnosis of a particular kernel source line. It is evidence for the bounded scope of this fix, not evidence that every textual O_CREAT match is defective.

Reproduction source (save as native_lock_probe.py in a disposable writable directory on macOS; it cleans up only its own newly created temporary directory):

"""Bounded native lock-create probe; no application imports or test framework."""
import collections
import concurrent.futures
import json
import os
import platform
import tempfile
import threading
from pathlib import Path


ROOT = Path(__file__).parent
PAIRS = 16
FLAGS = os.O_RDWR | getattr(os, "O_NOFOLLOW", 0)


def attempt(parent, dir_fd, relative, exclusive, barrier):
    path = "lock" if relative else str(parent / "lock")
    kwargs = {"dir_fd": dir_fd} if relative else {}
    barrier.wait(timeout=2)
    elected = False
    try:
        if exclusive:
            try:
                fd = os.open(path, FLAGS | os.O_CREAT | os.O_EXCL, 0o644, **kwargs)
            except FileExistsError:
                elected = True
                fd = os.open(path, FLAGS, **kwargs)
        else:
            fd = os.open(path, FLAGS | os.O_CREAT, 0o644, **kwargs)
        os.close(fd)
        return "ok", elected
    except OSError as exc:
        return "errno=" + str(exc.errno), elected


def main():
    print(json.dumps({"system": platform.system(), "release": platform.release(),
                      "machine": platform.machine(), "python": platform.python_version(),
                      "pairs_per_case": PAIRS}), flush=True)
    # Cases and pairs are serial; only the two opens within one pair race.
    with tempfile.TemporaryDirectory(prefix="lock-evidence-", dir=ROOT) as tmp:
        with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
            for relative in (True, False):
                for exclusive in (False, True):
                    counts = collections.Counter()
                    eexist = 0
                    stable = True
                    for i in range(PAIRS):
                        parent = Path(tmp) / f"{relative}-{exclusive}-{i}"
                        parent.mkdir()
                        dfd = os.open(parent, os.O_RDONLY | os.O_DIRECTORY)
                        before = os.fstat(dfd)
                        try:
                            barrier = threading.Barrier(2)
                            futures = [pool.submit(attempt, parent, dfd, relative,
                                                   exclusive, barrier) for _ in range(2)]
                            for future in futures:
                                result, elected = future.result(timeout=2)
                                counts[result] += 1
                                eexist += elected
                            after = os.fstat(dfd)
                            named = parent.stat()
                            stable &= ((before.st_dev, before.st_ino)
                                       == (after.st_dev, after.st_ino)
                                       == (named.st_dev, named.st_ino))
                        finally:
                            os.close(dfd)
                    print(json.dumps({"relative": relative, "exclusive": exclusive,
                                      "outcomes": dict(counts), "eexist_reopens": eexist,
                                      "parent_identity_stable": stable}), flush=True)


if __name__ == "__main__":
    main()

The invocation used a stdlib parent wrapper with subprocess.run([...], timeout=15, capture_output=True, text=True) and returned exit 0. No application source or PR head was changed. Full platform matrices remain in CI.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted — span=a9b242ac024e — require a shared dependency-lock opener in this fix

Both shipped owners of this inode now use the same short creator-election sequence, with regressions at both public entry points. The omission was the initial repair missing an existing caller, not two previously shipped copies of the new election algorithm diverging.
A shared helper is a reasonable refactoring option, but does not by itself prevent a future caller from bypassing it. The existing directory-pin/no-follow contracts and installer/uninstaller tests establish the behavior needed here. No remaining correctness defect requires another abstraction or a change to the backend/manager dependency boundary, so this PR keeps the minimal complete two-caller repair.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted — span=60b2c8948be2 — provisioning creator election has no checkable premise

The complete stdlib reproduction and actual output are now published in the evidence comment. On Darwin 25.6.0 arm64, 16 serial pairs using the old directory-relative O_CREAT pattern produced 16 ENOENT errors and 16 successes while the held/named parent identities remained unchanged; exclusive-create-then-reopen produced 32 successes.
This exercises native opens without application imports, pytest, directory removal, or instrumentation of os.open. It establishes the observed failure before acquisition that provisioning's pinned-directory opener encounters. The committed deterministic tests separately pin the protocol and safety properties; they are not being presented as proof of a universal kernel defect.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted — span=e23970fc41b0 — uninstall creator election has no checkable premise

The uninstall code opens the same .kirocrew-deps.lock through _data_pin.fd before acquiring the dependency lock. Its former directory-relative O_CREAT operation is exactly the failing operation isolated in the published native probe; this is not a different inode or an unrelated lock subsystem.
Consequently leaving uninstall unchanged would leave one owner using the reproduced failing creation path. Its new tests cover fresh/existing opens and preservation of the installed manifest/user data on a vanished lock. Matching the installer closes that concrete sibling without changing uninstall policy or rollback behavior.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted — span=54e7c89f8c30 — no probe or failure evidence is available on the PR

Native reproduction source and complete stdout now make the observation checkable. The bounded run compared directory-relative and absolute-path opens, with and without creator election, on fresh lock names. Only the old directory-relative case produced ENOENT in this sample; its parent directory stayed valid throughout.
The evidence identifies the host and Python version and distinguishes measured results from limits of inference. It does not claim an upstream issue, macOS CI reproduction, immunity of every absolute-path open, or a diagnosed Darwin source-code defect. Those qualifications preserve accuracy rather than retracting the observed failure.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted — span=2298b8884862 — textual O_CREAT siblings establish a repository-wide defect

The native control experiment distinguishes the relevant forms: old directory-relative opens had 16 ENOENT failures in 16 pairs, while the absolute-path control had 32 successful opens. This is bounded evidence, not a proof that no other form can ever fail.
I also read all four named examples: platform_compat.open_lock_file, GatewayLock.acquire, session_storage._mutation_lock, and jsonl_util's rotation lock call os.open with a path and no dir_fd. A grep count of O_CREAT is therefore not evidence that those sites share the demonstrated trigger. There is no established general 32-opener bug to defer or bulk-fix on that basis.
The path-based fallback remains in both app owners because it is the same logical dependency lock and the same create-or-open contract across platforms; it preserves no-truncation and now refuses recreation after a vanished lock. Using the same standard protocol there avoids adding a platform-specific policy branch. The PR claims the demonstrated app-lock repair, not repository-wide kernel-error coverage.

@cixuuz cixuuz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full exact-head diff and tests. The exclusive creator election preserves the pinned-dir/no-follow contract, avoids truncation, fails closed if the lock vanishes, and is covered at both lock owners plus the concurrent path.

@bolichen97
bolichen97 merged commit 9173117 into main Sep 13, 2026
79 checks passed
@bolichen97
bolichen97 deleted the fix/app-deps-lock-create-race branch September 13, 2026 05:37
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 13, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
Use exclusive creation before opening an existing dependency lock.
Concurrent macOS openat calls could otherwise fail with ENOENT before
reaching flock. Preserve directory pinning and no-follow flags, and
fail closed if the lock disappears before reopening.

Force overlapping first creation in the regression test and verify
that the second installer reuses the first install's stamp.
chenmingwei23 pushed a commit that referenced this pull request Sep 25, 2026
The darwin skip was a bisect step against the macOS deps-lock ENOENT race, which #10464 fixed with exclusive lock creation. Keep the win32 skip.

Closes #10704

Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants