Skip to content

fix: preserve concurrent first log appends on macOS - #11987

Merged
iamwhatever merged 1 commit into
mainfrom
fix/macos-log-append-create-race
Sep 19, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/macos-log-append-create-race

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Concurrent first appends can lose decision-log rows on macOS before the file lock is acquired. The failure was reproduced without pytest or mocks while investigating PR #11951's native Mac CI.

Why it matters

The caller catches the open error and warns, but the record never reaches disk. This is log data loss, not evidence of a memory leak.

What changed (motivation → approach → change)

src/kiro_crew/platform_log_append.py creates a new log exclusively. If the name already exists, it opens it without creation flags under the same pinned directory. This prevents Darwin's descriptor-relative nonexclusive-create race from dropping a writer's first record.

If that existing log disappears again before the second open, the ENOENT reaches the existing bounded directory-create retry like any other lost interleaving. The retry redoes the whole pinned sequence, so the row is still written, through the same exclusive no-follow create under the same resolved anchor. No private exception type, extra veto or new retry loop is added.

The resolve-once anchor, no-follow flags, regular/single-link checks, mode bits, file lock and Windows handle path are unchanged. No new lock, worker or setting is added.

test/test_platform_log_append.py covers first creation, existing files, link/hardlink refusal, parent-directory replacement, and a real existing log removed during handoff—with and without its directory. That case now asserts the row lands, the recreated leaf is created with O_CREAT | O_EXCL | O_NOFOLLOW at mode 0600, and every descriptor of the losing attempt is closed. The link-refusal test distinguishes one creation attempt from its two open calls. docs/system-specs/common/platform-compat.md states the handoff outcome.

Backwards compatibility

Compatible: valid existing logs retain their bytes, modes and serialization. Existing-file appends on POSIX cost one failed exclusive-create call. An existing leaf deleted by another actor during handoff is recreated by the bounded retry, which is what the base O_CREAT open did in that interleaving. Windows behavior is unchanged. This remains best-effort logging, not a crash-durable journal.

Tests

Verified on native macOS at e06aed2ccd5e3d0816719c5708e73840bd78039e, based on 8c6b3b272c3a (current main):

  • Append and decision-log files: 77 passed, 7 Windows-only skips.
  • New tests against the unchanged main implementation: 9 failed, 36 passed, 7 skipped.
  • Negative control: with the bounded retry reduced to one attempt, the three handoff-recreation cases fail; restored, they pass.
  • prove.py --base origin/main: PROVEN; restoring the main implementation trips assertions.
  • 39 applicable backend/static gates passed (12 frontend/deployment gates skipped: no such change); backend setup.py build_py passed, missing SPA assets reported.
  • No frontend build: no UI change.

Manual verification

Fresh stdlib-only native probe on this head: 100 rounds, four synchronized writers per fresh file, zero failed rounds; all four records retained each time. Files were fixture-owned and removed afterward.

Historical before/after on the original implementation: 40/100 failed rounds before, 0/100 after. That is historical evidence, not a fresh old-version run.

No live gateway, real log, production configuration or permission changed. Direct self-review only, as requested; no subagents or independent local model review claimed.

Review note

The requester authorized backing up each published head and SHA-leased rewrites to keep one commit within the two-commit CI limit. Local backups: keep/pr11987-before-rewrite-20260919 at 0bbfc6564c85d9d9f01d7087e50937cb2ef1db8b and keep/pr11987-head-301eef0a1-20260919 at 301eef0a11590eb03a410f92eb2e69ddee81ec5e.

First Principles on the previous head (span 1dced0e1bd62) found that refusing to recreate a disappeared existing leaf dropped the current row without a demonstrated harm from safe recreation. The requester ruled to allow safe recreation. The guard and its private exception are deleted; the previous head's test asserting non-recreation is replaced by one asserting exclusive, pinned recreation. That is the only behavior change from the previous head.

Current main includes theme repair #11978, the BSD-stat fixture repair, directory-create retry #12039 and two aws_control backup commits that import this module but do not touch it. No unrelated production fix is included.

The previous head's Windows shard 3 failed test_throttle_parks_two_runs_on_one_scope_and_wakes_by_capacity on a superseded run before that; it is not waived, and no scheduling code or assertion was changed. Fresh ordinary CI, current-head reviews, PR Readiness and all three native Mac shards are required. Rolling verification record.

Related Issues

Separate production repair for the log-loss blocker found during PR #11951. Its memory-reclamation tests are not included here.

Pattern harvest

Rule candidate: review-prompt

Pattern: a lock acquired after open cannot protect first creation. Test native concurrent writers on a fresh leaf, and let a lost handoff re-enter the existing bounded retry rather than adding a second failure path for the same interleaving.

Checklist

  • One Conventional Commit on current main
  • Related tests and applicable gates pass
  • Regression proof and direct self-review completed
  • Owning specification updated
  • No secrets, credentials or internal references in the diff
  • Fresh remote CI, reviews and native Mac lane complete

@bolichen97
bolichen97 requested a review from a team September 19, 2026 07:44
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Sep 19, 2026
@bolichen97
bolichen97 requested a review from buluoray September 19, 2026 07:44
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Preserve every cooperating writer's decision-log record when multiple callers first create the same daily file on macOS. Keep pinned-directory, no-follow, regular/single-link, bounded lock and rollback guarantees.
Not a goal: Change gateway/session lifecycle, memory reclamation, runtime settings, log retention, Windows handles, or repair unrelated CI failures. No merge or auto-merge requested.

This is the separately authorized production repair for #11951, not an extension of its test-only diff.

Verified at 48aa305e64f3e2d717ba5c434c47b3cc73864cbf, based on 8e27f52fc696:

  • 123 related native Darwin tests passed; 7 native Windows cases skipped.
  • 39 applicable backend/static gates passed; backend build passed without frontend assets (no UI changes).
  • prove.py --base origin/main: PROVEN; production-hunk reversal trips new assertions.
  • Real unmocked append_line, 4 synchronized writers and a fresh file per round: before 40/100 rounds failed with ENOENT/missing records; after 100/100 rounds contain all four records.
  • Raw-open controls in the post-fix experiment: nonexclusive O_CREAT still fails 98/100 rounds, exclusive-create/open-existing passes 100/100.
  • Direct self-review only, as explicitly requested. No subagents or independent local model review claimed.

The ci:macos label requests the repository's native Mac lane. A green ordinary PR Readiness result alone is not a claim that that separate lane passed.

@bolichen97 bolichen97 mentioned this pull request Sep 19, 2026
5 tasks done
@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of e06aed2ccd5e3d0816719c5708e73840bd78039e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e06aed2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e06aed2ccd5e3d0816719c5708e73840bd78039e: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of e06aed2ccd5e3d0816719c5708e73840bd78039e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the three native macOS shards are green at this head — Linux CI cannot exercise the Darwin create race this fixes.

[DESIGN-REVIEWED] e06aed2

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of e06aed2ccd5e3d0816719c5708e73840bd78039e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the three native Mac shards pass on this head — the author's own checklist leaves fresh remote CI unchecked.

What this change ships

Inventory (6 items) — 6 justified

Intent: FIX — stop concurrent first appends from silently losing decision-log rows on macOS.

  1. Concurrent first appends on macOS all reach disk (exclusive create, then a pinned non-creating open) — justified
  2. Existing-file appends on POSIX pay one extra failed exclusive-create syscall — justified
  3. An existing log deleted mid-handoff still gets this row, recreated through the existing bounded retry — justified
  4. platform-compat.md states the new create/open handoff — justified
  5. Link-refusal test's pin adapts from one open to one creation attempt across two opens — justified
  6. Five new tests pin the race, the handoff cases, and native four-writer concurrency — justified

Defect provenance is checkable: new tests fail on base (author reports 9 failures against unchanged main) and a linked native-Mac CI run. Grep of dir_fd= opens across src/ found zero other descriptor-relative nonexclusive O_CREAT sites — every other pinned create already uses O_EXCL — so no unfixed siblings. No new exception type, knob, retry loop or public surface; the vanished-leaf case reuses _CREATE_ATTEMPTS. The rewritten pin's semantic ("a refusal that is not a removal is not re-attempted") is preserved, adapted only to the two-call sequence.

[FIRST-PRINCIPLES-REVIEWED] e06aed2

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed e06aed2ccd5e3d0816719c5708e73840bd78039e — this comment is updated in place on each push.

Review details

I've independently traced the production change. The rewrite replaces a single non-exclusive O_CREAT leaf open with an O_CREAT | O_EXCL create plus an EEXIST fallback to a plain O_NOFOLLOW open under the same pinned parent_fd. Verifying the security-relevant properties against the code:

  • Symlink refusal preserved: O_CREAT|O_EXCL on any existing entry (including a symlink) returns EEXIST; the fallback retains O_NOFOLLOW, so a symlink yields ELOOP and is never followed. ELOOP is not FileNotFoundError, so _create_and_open's except BaseException: … raise propagates it without retry (line 238-240).
  • Vanished-leaf: EEXIST → plain open → ENOENT → FileNotFoundError → bounded retry re-runs the whole exclusive/pinned sequence; exhaustion raises FileNotFoundError with the full path (line 243-249).
  • Pin reuse: fallback open uses the same parent_fd from _pin_log_dir, so no re-resolution of the parent.
  • No fd leak: on EEXIST the first os.open raised before registering; only the fallback fd is pushed on stack; a failing fallback registers nothing and attempt.close() releases the pins.
  • No data-loss regression: existing regular file → EEXIST → plain open appends, identical to the old path; the fresh-file outcome only arises when another actor deletes the leaf concurrently, which the old non-exclusive O_CREAT produced identically.

The candidate list contained no candidates, and I could not ground any defect (a/b/c at 80+) on the changed lines.

No findings.

[OPUS-REVIEWED] e06aed2

Verdict parsed from the review's SHA-scoped output markers for commit e06aed2ccd5e3d0816719c5708e73840bd78039e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e06aed2ccd5e3d0816719c5708e73840bd78039e: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=d22d204b0653 — Item 1: the four named opens do not take the failing openat path

outcome: rebutted
I read all four sites at this head. SEL's chain sidecar, workflow allocate_run_id, memory_store_namespace_lock, and _open_store_use_lock each call os.open with a full path and NO dir_fd. This PR's log uses the descriptor-relative openat path.
A native stdlib-only matrix held one valid directory descriptor, created a fresh name per round, and synchronized four writers. It tested with and without dir_fd, and every combination of O_APPEND/O_NONBLOCK, while retaining O_RDWR|O_CREAT|O_NOFOLLOW.
Full-path open: 0/100 failed rounds with neither extra flag; 0/100 with O_APPEND; 0/100 with O_NONBLOCK; 0/100 with both (0/400 total).
Descriptor-relative openat: 96/100, 95/100, 95/100, and 96/100 failed rounds respectively (382/400 total), all ENOENT. No parent deletion, locks, pytest, or mocks were involved.
This does not prove those four subsystems can never fail. It does refute identifying them as instances of this measured defect solely by their flags. Their different open entry point is the omitted condition in the grep.
The existing closure of issue #10425 independently documents the same openat first-create defect and the same exclusive-create/open-existing protocol; see its maintainer's final comment. The originally proposed vanished-parent diagnosis there was disproved.
The PR description is being narrowed to descriptor-relative openat explicitly. No sibling code, permission rule or assertion is changed on the strength of the flag-only analogy.

Prior openat diagnosis. All probe files were temporary and removed when the experiment ended.

@bolichen97

bolichen97 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Current head: e06aed2ccd5e3d0816719c5708e73840bd78039e, one commit on main 8c6b3b272c3a.

The requester authorized backups plus SHA-leased force-pushes to this original PR branch. Local backups: keep/pr11987-before-rewrite-20260919 (0bbfc6564), keep/pr11987-head-301eef0a1-20260919 (301eef0a1), keep/pr11987-head-4273fb9a4-20260919 (4273fb9a4). Each lease pinned the exact prior remote SHA; no protected branch was pushed. The superseded head's in-flight runs were cancelled before each edit.

4273fb9a4 → e06aed2cc: test-only wording. GPT's one advisory finding on 4273fb9a4 (span e2218ec0de2f: a docstring still said "absence is not permission to retry") is fixed by renaming that test and rewriting its docstring to the bounded-retry behavior. No source, assertion or spec change; the production diff is byte-identical to 4273fb9a4.

Change since 301eef0a1, per the requester's ruling on span 1dced0e1bd62: the _ExistingLogDisappeared subtype and its isinstance branch are deleted; an existing leaf that vanishes between EEXIST and the non-creating open re-enters the existing bounded retry, which recreates it exclusively under the same pin with O_NOFOLLOW. test_existing_log_removed_during_handoff_is_recreated_exclusively asserts the row lands, flags EXCL / plain / EXCL, mode 0600, descriptors balanced; the "missing" competitor case asserts a third exclusive create.

Local verification on native macOS at e06aed2cc:

  • Append plus decision-log files: 77 passed, 7 Windows-only skips.
  • Unchanged main implementation with these tests: 9 failed, 36 passed, 7 skipped.
  • Negative control: _CREATE_ATTEMPTS reduced to 1 → the three recreation cases fail; restored → pass.
  • 39 applicable backend/static gates passed (12 frontend/deployment gates skipped: no such change); backend build_py passed.
  • prove.py --base origin/main: PROVEN. Native four-writer fresh-file probe: 100/100 rounds retained all four rows.
  • push_guard --require-single-on-base: SAFE; diff_signals --check-body: 0; disposition gate: ok, 0 violations.

Remote CI on e06aed2cc (REST check-runs, read 2026-09-19 ~19:15Z): 76 success, 10 skipped, 0 failed, 0 running.

  • All three native Mac shards: macOS Tests / Backend Tests (macOS) (1), (2), (3) SUCCESS.
  • Windows shard 3 SUCCESS — test_throttle_parks_two_runs_on_one_scope_and_wakes_by_capacity did not recur on this head (no scheduling code or assertion changed; the earlier red on 0bbfc6564 is recorded, not waived).
  • All Linux/Windows backend shards, Frontend Tests 1–4, E2E boot and offline lanes, security/scope/hygiene gates: SUCCESS.
  • GPT 5.6, Opus 4.8, Design, First Principles: all SUCCESS on this head with no open findings.
  • Commit statuses: PR Readiness success (18:54:42Z), AWS CodeBuild success. Label readiness: passed.

Base freshness: main is 1 commit ahead (src/kiro_crew/browser_cli/install.py + its test), no overlap with this PR's three files; no resync needed.

No subagents, live gateway changes, merge or auto-merge. mergeable_state: blocked reflects only the required human review. This PR is reported green and ready for maintainer review.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Sep 19, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 19, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 19, 2026
@bolichen97
bolichen97 force-pushed the fix/macos-log-append-create-race branch from 0bbfc65 to 301eef0 Compare September 19, 2026 17:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=1dced0e1bd62 — Item 2: existing-leaf disappearance needs an explicit behavior decision

outcome: needs-a-decision
Code inspection confirms the cost: fallback ENOENT becomes _ExistingLogDisappeared; _create_and_open closes the attempt and raises; decisions.log.append catches it, warns, and drops the current row.
The current tests prove refusal and descriptor release, but do not demonstrate a separate user harm from safe recreation. They cannot independently justify the contract they assert.
The production retention sweep removes old date-named files; normal append selects today's UTC filename. I found no demonstrated ordinary sweep/rotation scenario in this caller that requires the added refusal. I am not claiming that no external deletion scenario exists.
Opus's description of an audit-integrity choice does not itself establish that requirement. I will not rebut this concern just because the other review lane passed.
The active task explicitly requires that an observed existing log not be recreated. Removing this guard changes that accepted behavior, so I am asking the requester before changing it.
@bolichen97: may we allow the existing bounded retry after handoff ENOENT, while retaining exclusive creation, the resolve-once anchor, no-follow opens, regular/single-link checks, descriptor cleanup and the original lock? My recommendation is yes, with regression coverage proving the current row is retained and path refusals remain enforced.
No source/test changes, waiver or override made. CI is still running; the previous Windows failure is not waived, and all three native Mac shards remain required.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 19, 2026
@bolichen97
bolichen97 force-pushed the fix/macos-log-append-create-race branch from 301eef0 to 4273fb9 Compare September 19, 2026 18:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 19, 2026
Create the log exclusively and open an existing leaf under the same
directory pin, so Darwin's nonexclusive-create race cannot drop a
writer's first record. A leaf lost between EEXIST and the open is one
more lost interleaving: the existing bounded directory-create retry
redoes the whole pinned, exclusive, no-follow sequence, so the row is
still written. Test descriptor cleanup, the recreated leaf's flags and
mode, and native concurrent writes without weakening path checks.
@bolichen97
bolichen97 force-pushed the fix/macos-log-append-create-race branch from 4273fb9 to e06aed2 Compare September 19, 2026 18:34
@bolichen97

bolichen97 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=e2218ec0de2f — stale "absence is not permission to retry" docstring (test/test_platform_log_append.py:623 on 4273fb9)

outcome: fixed
The docstring and test name described the previous head's non-recreation contract; the code under test now redoes a vanished leaf through the bounded retry.
Renamed to test_create_loser_opens_the_winner_once_and_redoes_only_a_vanished_leaf and rewrote the docstring: a link is refused, never followed; only absence is retried, via the redo's own exclusive pinned create.
Test-only wording change; no source, assertion or spec change. 77 passed / 7 skipped, 39 gates pass, single commit e06aed2ccd5e3d0816719c5708e73840bd78039e on main 8c6b3b272c3a.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 19, 2026
@bolichen97
bolichen97 enabled auto-merge (rebase) September 19, 2026 19:35
@iamwhatever
iamwhatever merged commit 891e6de into main Sep 19, 2026
88 checks passed
@iamwhatever
iamwhatever deleted the fix/macos-log-append-create-race branch September 19, 2026 21:34
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants