Skip to content

test: reset the member eventlog service singleton between tests - #15534

Merged
buluoray merged 1 commit into
mainfrom
fix/eventlog-service-test-isolation-windows
Sep 30, 2026
Merged

buluoray merged 1 commit into
mainfrom
fix/eventlog-service-test-isolation-windows

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop the member eventlog service singleton from leaking across tests, which makes Backend Tests (Windows) shard 8 fail intermittently on unrelated PRs.

Backend Tests (Windows) (8) fails on rebased PRs (observed on #15316 head f0fbd50bab and #15476 head 54e672fc50) at:

  • test/test_members_dm_thread.py::TestMemberActivityRoute::test_the_read_never_asks_the_log_for_every_event
  • test/test_members_dm_thread.py::TestMemberActivityRoute::test_colliding_slugs_do_not_mix_histories

Both fail on the test's first line — assert record_activity(CREW, ...) returns False — not in the read logic the tests are about. main does not run Windows shards on push, so the breakage is only visible on PR boards.

Why it matters

Any PR rebased onto current main inherits a red Windows shard that has nothing to do with its own change, blocking otherwise-green PRs and burning reviewer/CI time chasing a phantom regression. The underlying isolation hole is shared by seven test files that use get_service(), so it can resurface on any shard-ordering change.

Not a goal

What changed (motivation → approach → change)

  • Symptom: the first record_activity in TestMemberActivityRoute returns False on Windows shard 8 only; the same tests pass on Linux CI and locally.
  • Root cause: eventlog.service.get_service() memoises one MemberEventLogService per process and rebuilds it only when the crew-log root moves. The root follows KIROCREW_HOME, which the autouse _isolate_kirocrew_home fixture repoints at a fresh tmp dir per test. A test that touches the service leaves a live singleton bound to its own home; the next test on the same xdist worker inherits it after that home is torn down. On POSIX get_service() rebuilds against the new home and nothing is observed; on Windows the leaked MemberLog holds open OS handles under the dead directory, blocking tmp-dir teardown and making the inheriting test's first write fail — record_activity catches the OSError and returns False. fix(members): order the Recent roster by the crew log (#15277) added test/test_members_roster_recency.py, whose TestRosterReadTakesTheFoldedRecency tests call get_service()/ensure/append without the module's set_service(None) reset, shifting pytest-split shard-8 ordering so the leaked service lands just before TestMemberActivityRoute on the same Windows worker.
  • Change: add an autouse _reset_member_eventlog_singleton fixture in test/conftest.py that clears the singleton at both ends of every test (set_service(None) on setup and teardown), matching the existing _restore_autonudge_singleton / reasoning-effort singleton floors. Each test now starts on a clean singleton bound to its own home, so the outcome is deterministic on both platforms.

The leak was verified directly: with the leaking predecessor running first, the inheriting test starts with get_service()._singleton.root pointing at the previous test's torn-down home (.../i0/1-kirocrew-home/...) while its own KIROCREW_HOME is .../i0/2-kirocrew-home — the exact cross-boundary bind this fixture removes.

Backwards compatibility

Compatible: test-only change. No production code, public API, or on-disk format is touched; the fixture only resets a process-global test singleton that production legitimately publishes and every test already isolates its home from.

Tests

  • New TestEventlogSingletonDoesNotLeakAcrossTests (two ordered tests, no local reset fixture) pins the cause on both platforms: the first builds a service bound to its own home and leaves it live; the second asserts service._singleton is None at its start (the conftest floor cleared it) and that a fresh read binds to its own home, not the leaked one. Verified it fails without the fixture (eventlog service leaked across the test boundary) and passes with it.
  • test/test_members_dm_thread.py, test_members_roster_recency.py, test_members_eventlog_wiring.py, test_members.py, test_eventlog_hooks.py, eventlog/test_eventlog.py, test_members_preview_speech_only.py — 561 passed together locally with the fixture in place.

Manual verification

N/A — unit coverage sufficient. The Windows-specific handle behaviour cannot be reproduced on the Linux dev box, so the pin asserts the platform-independent cause (a singleton crossing the test boundary) rather than the Windows-only symptom; the Windows shard confirms the symptom is gone.

Related Issues

Unblocks Windows shard 8 on PRs rebased onto main after #15277.

Pattern harvest

Rule candidate: review-prompt
Pattern: a process-global singleton memoised on a path/root that tests repoint per-test must be reset at the test boundary (autouse fixture), or it leaks the first test's binding into later tests on the same xdist worker — silent on POSIX, a torn-handle failure on Windows.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A
  • No secrets, credentials, or internal references in the diff

`eventlog.service.get_service()` memoises one `MemberEventLogService` per
process and rebuilds it only when the crew-log root moves. The root follows
`KIROCREW_HOME`, which the autouse `_isolate_kirocrew_home` fixture repoints at
a fresh tmp dir per test. A test that touches the service therefore leaves a
live singleton bound to its own home, and the next test on the same xdist
worker inherits it after that home is torn down.

On POSIX the next `get_service()` rebuilds against the new home and nothing is
observed. On Windows the leaked `MemberLog` still holds open OS handles under
the dead directory, which blocks the tmp-dir teardown and makes the inheriting
test's first write fail: `members.record_activity` catches the OSError and
returns False. That is the Windows-only red on
`TestMemberActivityRoute::test_the_read_never_asks_the_log_for_every_event`
and `::test_colliding_slugs_do_not_mix_histories` -- the first `record_activity`
line of each, which asserts a True return.

`fix(members): order the Recent roster by the crew log (#15277)` added
`test/test_members_roster_recency.py`, whose `TestRosterReadTakesTheFoldedRecency`
tests call `get_service()`/`ensure`/`append` without the module's
`set_service(None)` reset. That shifted the pytest-split shard-8 ordering so the
leaked service lands just before `TestMemberActivityRoute` on the same Windows
worker, surfacing a latent isolation hole that seven other test files share.

Fix: an autouse `_reset_member_eventlog_singleton` conftest fixture that clears
the singleton at both ends of every test, matching the existing
`_restore_autonudge_singleton` / reasoning-effort floors. A two-test pin in
`TestEventlogSingletonDoesNotLeakAcrossTests` locks the cause (a singleton
crossing the boundary) on both platforms; it fails without the fixture and
passes with it.
@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 30, 2026 18:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 20c436d9640005a75ca461717203b9b8c13a6177 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm Windows shard 8 is green on this PR's own CI run before merge — that run is the only witness to the actual symptom.

Suggestions

  • Mark TestEventlogSingletonDoesNotLeakAcrossTests with @pytest.mark.xdist_group so its two ordered tests share one worker; without it, -n auto --dist loadgroup can scatter them and the pin passes vacuously, unable to catch a future fixture removal.

[DESIGN-REVIEWED] 20c436d

Description read

[DESCRIPTION-READ] b1e58c2c82ce84e9345f4ff0052702a3c076f5bf68d98f7d737c6f6d1257631c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 20c436d9640005a75ca461717203b9b8c13a6177 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 20c436d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 20c436d9640005a75ca461717203b9b8c13a6177: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 20c436d9640005a75ca461717203b9b8c13a6177 — this comment is updated in place on each push.

Review details

The added leak guard cannot fail — its only assertion is satisfied by the floor's own setup reset before the test body runs.

FINDING — test/test_members_dm_thread.py:2607 — assert svc_mod._singleton is None is guaranteed by the autouse floor's own setup half (_svc.set_service(None), test/conftest.py:963) which runs before the test body, and the follow-up assert str(svc.root) != type(self)._bound_home holds unconditionally because get_service() rebuilds whenever _singleton.root != root and _isolate_kirocrew_home hands every test a distinct KIROCREW_HOME; deleting the teardown reset the docstring calls "the part that is not optional" leaves both new tests green → Fix: have an inner fixture (torn down before the outer conftest floor) capture svc_mod._singleton after the previous test finalizes and assert on that captured value, and drop the setup-side reset so the assertion has something to observe.

FINDING — test/test_members_dm_thread.py:2587 — _bound_home: str = "" passes state between two tests through a class attribute while the class carries no xdist_group mark, so under this repo's -n auto --dist loadgroup default (setup.cfg:376-389) the pair is normally scheduled onto different workers and the second test compares svc.root against the untouched "" default in its own process → Fix: mark the class @pytest.mark.xdist_group(name="eventlog_singleton_boundary") and make the second test fail rather than pass when _bound_home is still empty.

FINDING — test/conftest.py:955 — the cited witness is false: the docstring (and test_members_dm_thread.py:2575) blames test_members_roster_recency for leaving the service live, but that module's module-level autouse _fresh_eventlog fixture (test/test_members_roster_recency.py:142-147) calls set_service(None) at both setup and teardown for every test in it, including TestRosterReadTakesTheFoldedRecency → Fix: cite a module that genuinely leaves the singleton live (this file's own TestMemberActivityRoute, or test_crew_log_member_unit_reclaim.py's direct get_service() calls) or drop the named witness.

FINDING — test/conftest.py:937 — a new process-global floor lands without the same-commit spec update docs/system-specs/common/testing-conventions.md demands ("when you introduce a new process-global, ship its floor entry with it", line 4195), and it restores to a pristine None rather than to "what the test inherited" as every other entry in that list does → Fix: add the _reset_member_eventlog_singleton bullet beside _restore_autonudge_singleton and state why a pristine reset is correct here.

[OPUS-REVIEWED] 20c436d

Verdict parsed from the review's SHA-scoped output markers for commit 20c436d9640005a75ca461717203b9b8c13a6177.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 20c436d9640005a75ca461717203b9b8c13a6177: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 30, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I checked the change against source at 20c436d964:

  • kiro_crew.eventlog.service on main keeps a module-level _singleton behind get_service() / set_service(), so an autouse fixture calling set_service(None) at both setup and teardown clears exactly that singleton. Nothing in production code changes.
  • The new TestEventlogSingletonDoesNotLeakAcrossTests pair tests the cause (the singleton crossing a test boundary) rather than the Windows-only symptom, so it also guards against this on Linux.
  • All Windows shards are green on this head, shard 8 included. GPT 5.6 and Opus 5 found nothing blocking.

Non-blocking follow-up: #15547 hit this shard 8 red three times, on three different tests. Two are the TestMemberActivityRoute record_activity → False case this PR describes. The third was TestResumeGuards::test_concurrent_member_resume_does_not_duplicate_history ("history duplicated: 2 copies"), which doesn't obviously go through the event-log service. If it still fails on Windows after this merges, it has a separate cause.

@buluoray
buluoray merged commit 31a6511 into main Sep 30, 2026
97 checks passed
@buluoray
buluoray deleted the fix/eventlog-service-test-isolation-windows branch September 30, 2026 21:54
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 30, 2026
chenmingwei23 pushed a commit that referenced this pull request Oct 1, 2026
…e retirement

test/test_members_dm_thread.py flakes on shard 8 across multiple PR boards
after #15534 merged, on both Linux and Windows. The CI traceback anchors it:

    test_returns_recorded_entries_newest_first_with_allowlist_fields
    assert record_activity(CREW, 'dashboard_chat-1', 'persistent', via='chat')
    AssertionError: assert False

The write side returns False on the FIRST call in a fresh test, after earlier
tests ran on the same xdist worker.

Root cause. get_service() memoises one MemberEventLogService per process and
rebuilds it only when the crew-log root moves (which every test does and
production never does). The service sits in a REFERENCE CYCLE: its projection
registry holds set_on_change(service._on_change), a bound method back to the
service. So set_service(None) -- which the autouse _reset_member_eventlog_singleton
floor calls at each boundary -- drops the last external reference but does not
free the graph; the cycle keeps it alive until the cyclic collector runs. Each
cached MemberLog's CrewLog releases its write lease through a weakref.finalize
that fires only then, and holds its file descriptor until then.

On POSIX that deferral is invisible (an advisory lock on a torn-down directory
harms nothing, and the next acquire against a fresh path just works), which is
why the leak could not be reproduced on Linux by running the file, targeted
interleaves, or the full real shard-8 selection. On Windows the still-open
descriptor keeps a mandatory lock that pins the (torn-down) home directory, so
the directory cannot be removed and the next writer on the inheriting worker
fails -- record_activity returns False. #15534 reset the singleton REFERENCE at
both ends, which was necessary but not sufficient: the reference-cycle keeps the
handles alive regardless.

Fix (root, deterministic). Retirement now releases the handles NOW instead of
waiting for GC:

* CrewLog.release_ownership() runs its lease finalizer explicitly (idempotent,
  atomic via weakref.finalize) and detaches it.
* MemberLog.close() releases its cached CrewLog and unloads.
* MemberEventLogService.close() closes every cached log, clears the caches, and
  severs the registry -> bound-method -> service cycle.
* set_service() closes the outgoing service when it is replaced; get_service()
  closes the previous service on a home-change rebuild. Neither happens while an
  append is in flight on the same instance (the home moves only at a test
  boundary), so nothing races a writer that still needs the lease.

Tests (whole-file guard, POSIX-provable). TestEventlogServiceRetirementReleasesHandles
pins the CAUSE rather than the Windows-only symptom: after set_service(None) the
retired service holds no cached logs and its registry cycle is severed; a
home-change rebuild retires the previous service; close() is idempotent and reads
re-open on demand. Verified to FAIL without the set_service close wiring and pass
with it.

Verification: full test_members_dm_thread.py + eventlog/test_eventlog.py +
test_members.py + test_members_eventlog_wiring.py + test_eventlog_hooks.py +
test_crew_log_member_unit_reclaim.py + test_members_roster_recency.py = 576
passed / 1 skipped (-n0). test_members_dm_thread.py stressed at -n4 x8 = 182
passed each. black (py312) / isort / flake8 / mypy clean on the changed modules.
bolichen97 pushed a commit that referenced this pull request Oct 1, 2026
…dows (#15633)

* fix(acp): orphan alias-lock sweep deletes under its own handle on Windows

The orphan-lock sweep removes an empty <alias>.lock only while holding its
own exclusive lock on it: the held lock is the proof no launcher owns the
inode, so the unlink happens WITHOUT first releasing (releasing before the
unlink reopens the race the proof closes). On POSIX an open fd never blocks
an unlink, so this is correct and the two sweep tests pass.

On Windows a CRT descriptor (os.open) opens with FILE_SHARE_READ |
FILE_SHARE_WRITE and OMITS FILE_SHARE_DELETE, so while the sweep's own
verification handle lives, os.unlink of that same name fails with a sharing
violation (PermissionError / WinError 32). The sweep then declines to remove
genuine residue -- its own handle is what blocks the delete -- so
test_leftover_alias_lock_files_are_swept_only_when_provably_residue (the
residue file never unlinks) and test_a_lock_file_a_launcher_still_holds_
is_never_swept (the free sibling never unlinks, removed == 0 != 1) fail
Backend Tests (Windows) shard 2. main does not run Windows shards on push,
so this landed on main (e3309cd, #15150) and only surfaced on PRs rebased
onto it.

Fix at the root with a pin on both platforms: a new
platform_compat.open_lock_file_for_sweep opens the lock file RDWR, and on
Windows uses CreateFileW with FILE_SHARE_READ_WRITE_DELETE so the unlink
lands while the handle (and its msvcrt byte-range lock) is still held --
preserving the hold-across-unlink guarantee identically on both platforms.
OPEN_EXISTING never creates (the sweep only ever removes names already
present); no OPEN_REPARSE_POINT (the caller already refuses a reparse point
at the name and re-checks identity under the lock). POSIX keeps O_RDWR |
O_NOFOLLOW, exactly the open it was doing, named.

Tests + gates:
- TestOpenLockFileForSweep in test_platform_compat.py pins the POSIX
  unlink-while-open path and the POSIX symlink refusal (both skipped off
  POSIX, where their real os.unlink-while-open / os.symlink semantics do not
  hold), and -- via a faked WinDLL that runs everywhere -- that the Windows
  CreateFileW requests delete-sharing + GENERIC_READ|WRITE and that a wrap
  failure closes the native handle. The two shard-2 tests exercise the real
  sweep on POSIX and continue to pass.
- The extraction moved the sweep's by-name open behind
  open_lock_file_for_sweep, one hop past the link-screen gate's one-hop
  resolver reach -- which would silently drop _sweep_orphan_alias_locks from
  the ratchet (the hazard uncured, merely unseen). open_lock_file_for_sweep
  is added to the gate's PRIMITIVES (exactly as open_lock_file already is for
  the same reason), and test/link_screen_sites.py is regenerated so the site
  stays declared.

* fix(eventlog): release member log handles deterministically on service retirement

test/test_members_dm_thread.py flakes on shard 8 across multiple PR boards
after #15534 merged, on both Linux and Windows. The CI traceback anchors it:

    test_returns_recorded_entries_newest_first_with_allowlist_fields
    assert record_activity(CREW, 'dashboard_chat-1', 'persistent', via='chat')
    AssertionError: assert False

The write side returns False on the FIRST call in a fresh test, after earlier
tests ran on the same xdist worker.

Root cause. get_service() memoises one MemberEventLogService per process and
rebuilds it only when the crew-log root moves (which every test does and
production never does). The service sits in a REFERENCE CYCLE: its projection
registry holds set_on_change(service._on_change), a bound method back to the
service. So set_service(None) -- which the autouse _reset_member_eventlog_singleton
floor calls at each boundary -- drops the last external reference but does not
free the graph; the cycle keeps it alive until the cyclic collector runs. Each
cached MemberLog's CrewLog releases its write lease through a weakref.finalize
that fires only then, and holds its file descriptor until then.

On POSIX that deferral is invisible (an advisory lock on a torn-down directory
harms nothing, and the next acquire against a fresh path just works), which is
why the leak could not be reproduced on Linux by running the file, targeted
interleaves, or the full real shard-8 selection. On Windows the still-open
descriptor keeps a mandatory lock that pins the (torn-down) home directory, so
the directory cannot be removed and the next writer on the inheriting worker
fails -- record_activity returns False. #15534 reset the singleton REFERENCE at
both ends, which was necessary but not sufficient: the reference-cycle keeps the
handles alive regardless.

Fix (root, deterministic). Retirement now releases the handles NOW instead of
waiting for GC:

* CrewLog.release_ownership() runs its lease finalizer explicitly (idempotent,
  atomic via weakref.finalize) and detaches it.
* MemberLog.close() releases its cached CrewLog and unloads.
* MemberEventLogService.close() closes every cached log, clears the caches, and
  severs the registry -> bound-method -> service cycle.
* set_service() closes the outgoing service when it is replaced; get_service()
  closes the previous service on a home-change rebuild. Neither happens while an
  append is in flight on the same instance (the home moves only at a test
  boundary), so nothing races a writer that still needs the lease.

Tests (whole-file guard, POSIX-provable). TestEventlogServiceRetirementReleasesHandles
pins the CAUSE rather than the Windows-only symptom: after set_service(None) the
retired service holds no cached logs and its registry cycle is severed; a
home-change rebuild retires the previous service; close() is idempotent and reads
re-open on demand. Verified to FAIL without the set_service close wiring and pass
with it.

Verification: full test_members_dm_thread.py + eventlog/test_eventlog.py +
test_members.py + test_members_eventlog_wiring.py + test_eventlog_hooks.py +
test_crew_log_member_unit_reclaim.py + test_members_roster_recency.py = 576
passed / 1 skipped (-n0). test_members_dm_thread.py stressed at -n4 x8 = 182
passed each. black (py312) / isort / flake8 / mypy clean on the changed modules.

---------

Co-authored-by: kirocrew-agent <agent@kirocrew.local>
bolichen97 pushed a commit that referenced this pull request Oct 1, 2026
…dows (#15633)

* fix(acp): orphan alias-lock sweep deletes under its own handle on Windows

The orphan-lock sweep removes an empty <alias>.lock only while holding its
own exclusive lock on it: the held lock is the proof no launcher owns the
inode, so the unlink happens WITHOUT first releasing (releasing before the
unlink reopens the race the proof closes). On POSIX an open fd never blocks
an unlink, so this is correct and the two sweep tests pass.

On Windows a CRT descriptor (os.open) opens with FILE_SHARE_READ |
FILE_SHARE_WRITE and OMITS FILE_SHARE_DELETE, so while the sweep's own
verification handle lives, os.unlink of that same name fails with a sharing
violation (PermissionError / WinError 32). The sweep then declines to remove
genuine residue -- its own handle is what blocks the delete -- so
test_leftover_alias_lock_files_are_swept_only_when_provably_residue (the
residue file never unlinks) and test_a_lock_file_a_launcher_still_holds_
is_never_swept (the free sibling never unlinks, removed == 0 != 1) fail
Backend Tests (Windows) shard 2. main does not run Windows shards on push,
so this landed on main (e3309cd, #15150) and only surfaced on PRs rebased
onto it.

Fix at the root with a pin on both platforms: a new
platform_compat.open_lock_file_for_sweep opens the lock file RDWR, and on
Windows uses CreateFileW with FILE_SHARE_READ_WRITE_DELETE so the unlink
lands while the handle (and its msvcrt byte-range lock) is still held --
preserving the hold-across-unlink guarantee identically on both platforms.
OPEN_EXISTING never creates (the sweep only ever removes names already
present); no OPEN_REPARSE_POINT (the caller already refuses a reparse point
at the name and re-checks identity under the lock). POSIX keeps O_RDWR |
O_NOFOLLOW, exactly the open it was doing, named.

Tests + gates:
- TestOpenLockFileForSweep in test_platform_compat.py pins the POSIX
  unlink-while-open path and the POSIX symlink refusal (both skipped off
  POSIX, where their real os.unlink-while-open / os.symlink semantics do not
  hold), and -- via a faked WinDLL that runs everywhere -- that the Windows
  CreateFileW requests delete-sharing + GENERIC_READ|WRITE and that a wrap
  failure closes the native handle. The two shard-2 tests exercise the real
  sweep on POSIX and continue to pass.
- The extraction moved the sweep's by-name open behind
  open_lock_file_for_sweep, one hop past the link-screen gate's one-hop
  resolver reach -- which would silently drop _sweep_orphan_alias_locks from
  the ratchet (the hazard uncured, merely unseen). open_lock_file_for_sweep
  is added to the gate's PRIMITIVES (exactly as open_lock_file already is for
  the same reason), and test/link_screen_sites.py is regenerated so the site
  stays declared.

* fix(eventlog): release member log handles deterministically on service retirement

test/test_members_dm_thread.py flakes on shard 8 across multiple PR boards
after #15534 merged, on both Linux and Windows. The CI traceback anchors it:

    test_returns_recorded_entries_newest_first_with_allowlist_fields
    assert record_activity(CREW, 'dashboard_chat-1', 'persistent', via='chat')
    AssertionError: assert False

The write side returns False on the FIRST call in a fresh test, after earlier
tests ran on the same xdist worker.

Root cause. get_service() memoises one MemberEventLogService per process and
rebuilds it only when the crew-log root moves (which every test does and
production never does). The service sits in a REFERENCE CYCLE: its projection
registry holds set_on_change(service._on_change), a bound method back to the
service. So set_service(None) -- which the autouse _reset_member_eventlog_singleton
floor calls at each boundary -- drops the last external reference but does not
free the graph; the cycle keeps it alive until the cyclic collector runs. Each
cached MemberLog's CrewLog releases its write lease through a weakref.finalize
that fires only then, and holds its file descriptor until then.

On POSIX that deferral is invisible (an advisory lock on a torn-down directory
harms nothing, and the next acquire against a fresh path just works), which is
why the leak could not be reproduced on Linux by running the file, targeted
interleaves, or the full real shard-8 selection. On Windows the still-open
descriptor keeps a mandatory lock that pins the (torn-down) home directory, so
the directory cannot be removed and the next writer on the inheriting worker
fails -- record_activity returns False. #15534 reset the singleton REFERENCE at
both ends, which was necessary but not sufficient: the reference-cycle keeps the
handles alive regardless.

Fix (root, deterministic). Retirement now releases the handles NOW instead of
waiting for GC:

* CrewLog.release_ownership() runs its lease finalizer explicitly (idempotent,
  atomic via weakref.finalize) and detaches it.
* MemberLog.close() releases its cached CrewLog and unloads.
* MemberEventLogService.close() closes every cached log, clears the caches, and
  severs the registry -> bound-method -> service cycle.
* set_service() closes the outgoing service when it is replaced; get_service()
  closes the previous service on a home-change rebuild. Neither happens while an
  append is in flight on the same instance (the home moves only at a test
  boundary), so nothing races a writer that still needs the lease.

Tests (whole-file guard, POSIX-provable). TestEventlogServiceRetirementReleasesHandles
pins the CAUSE rather than the Windows-only symptom: after set_service(None) the
retired service holds no cached logs and its registry cycle is severed; a
home-change rebuild retires the previous service; close() is idempotent and reads
re-open on demand. Verified to FAIL without the set_service close wiring and pass
with it.

Verification: full test_members_dm_thread.py + eventlog/test_eventlog.py +
test_members.py + test_members_eventlog_wiring.py + test_eventlog_hooks.py +
test_crew_log_member_unit_reclaim.py + test_members_roster_recency.py = 576
passed / 1 skipped (-n0). test_members_dm_thread.py stressed at -n4 x8 = 182
passed each. black (py312) / isort / flake8 / mypy clean on the changed modules.

---------

Co-authored-by: kirocrew-agent <agent@kirocrew.local>
(cherry picked from commit 7460284)
bolichen97 added a commit that referenced this pull request Oct 1, 2026
…dows (#15633) (#15699)

* fix(acp): orphan alias-lock sweep deletes under its own handle on Windows

The orphan-lock sweep removes an empty <alias>.lock only while holding its
own exclusive lock on it: the held lock is the proof no launcher owns the
inode, so the unlink happens WITHOUT first releasing (releasing before the
unlink reopens the race the proof closes). On POSIX an open fd never blocks
an unlink, so this is correct and the two sweep tests pass.

On Windows a CRT descriptor (os.open) opens with FILE_SHARE_READ |
FILE_SHARE_WRITE and OMITS FILE_SHARE_DELETE, so while the sweep's own
verification handle lives, os.unlink of that same name fails with a sharing
violation (PermissionError / WinError 32). The sweep then declines to remove
genuine residue -- its own handle is what blocks the delete -- so
test_leftover_alias_lock_files_are_swept_only_when_provably_residue (the
residue file never unlinks) and test_a_lock_file_a_launcher_still_holds_
is_never_swept (the free sibling never unlinks, removed == 0 != 1) fail
Backend Tests (Windows) shard 2. main does not run Windows shards on push,
so this landed on main (e3309cd, #15150) and only surfaced on PRs rebased
onto it.

Fix at the root with a pin on both platforms: a new
platform_compat.open_lock_file_for_sweep opens the lock file RDWR, and on
Windows uses CreateFileW with FILE_SHARE_READ_WRITE_DELETE so the unlink
lands while the handle (and its msvcrt byte-range lock) is still held --
preserving the hold-across-unlink guarantee identically on both platforms.
OPEN_EXISTING never creates (the sweep only ever removes names already
present); no OPEN_REPARSE_POINT (the caller already refuses a reparse point
at the name and re-checks identity under the lock). POSIX keeps O_RDWR |
O_NOFOLLOW, exactly the open it was doing, named.

Tests + gates:
- TestOpenLockFileForSweep in test_platform_compat.py pins the POSIX
  unlink-while-open path and the POSIX symlink refusal (both skipped off
  POSIX, where their real os.unlink-while-open / os.symlink semantics do not
  hold), and -- via a faked WinDLL that runs everywhere -- that the Windows
  CreateFileW requests delete-sharing + GENERIC_READ|WRITE and that a wrap
  failure closes the native handle. The two shard-2 tests exercise the real
  sweep on POSIX and continue to pass.
- The extraction moved the sweep's by-name open behind
  open_lock_file_for_sweep, one hop past the link-screen gate's one-hop
  resolver reach -- which would silently drop _sweep_orphan_alias_locks from
  the ratchet (the hazard uncured, merely unseen). open_lock_file_for_sweep
  is added to the gate's PRIMITIVES (exactly as open_lock_file already is for
  the same reason), and test/link_screen_sites.py is regenerated so the site
  stays declared.

* fix(eventlog): release member log handles deterministically on service retirement

test/test_members_dm_thread.py flakes on shard 8 across multiple PR boards
after #15534 merged, on both Linux and Windows. The CI traceback anchors it:

    test_returns_recorded_entries_newest_first_with_allowlist_fields
    assert record_activity(CREW, 'dashboard_chat-1', 'persistent', via='chat')
    AssertionError: assert False

The write side returns False on the FIRST call in a fresh test, after earlier
tests ran on the same xdist worker.

Root cause. get_service() memoises one MemberEventLogService per process and
rebuilds it only when the crew-log root moves (which every test does and
production never does). The service sits in a REFERENCE CYCLE: its projection
registry holds set_on_change(service._on_change), a bound method back to the
service. So set_service(None) -- which the autouse _reset_member_eventlog_singleton
floor calls at each boundary -- drops the last external reference but does not
free the graph; the cycle keeps it alive until the cyclic collector runs. Each
cached MemberLog's CrewLog releases its write lease through a weakref.finalize
that fires only then, and holds its file descriptor until then.

On POSIX that deferral is invisible (an advisory lock on a torn-down directory
harms nothing, and the next acquire against a fresh path just works), which is
why the leak could not be reproduced on Linux by running the file, targeted
interleaves, or the full real shard-8 selection. On Windows the still-open
descriptor keeps a mandatory lock that pins the (torn-down) home directory, so
the directory cannot be removed and the next writer on the inheriting worker
fails -- record_activity returns False. #15534 reset the singleton REFERENCE at
both ends, which was necessary but not sufficient: the reference-cycle keeps the
handles alive regardless.

Fix (root, deterministic). Retirement now releases the handles NOW instead of
waiting for GC:

* CrewLog.release_ownership() runs its lease finalizer explicitly (idempotent,
  atomic via weakref.finalize) and detaches it.
* MemberLog.close() releases its cached CrewLog and unloads.
* MemberEventLogService.close() closes every cached log, clears the caches, and
  severs the registry -> bound-method -> service cycle.
* set_service() closes the outgoing service when it is replaced; get_service()
  closes the previous service on a home-change rebuild. Neither happens while an
  append is in flight on the same instance (the home moves only at a test
  boundary), so nothing races a writer that still needs the lease.

Tests (whole-file guard, POSIX-provable). TestEventlogServiceRetirementReleasesHandles
pins the CAUSE rather than the Windows-only symptom: after set_service(None) the
retired service holds no cached logs and its registry cycle is severed; a
home-change rebuild retires the previous service; close() is idempotent and reads
re-open on demand. Verified to FAIL without the set_service close wiring and pass
with it.

Verification: full test_members_dm_thread.py + eventlog/test_eventlog.py +
test_members.py + test_members_eventlog_wiring.py + test_eventlog_hooks.py +
test_crew_log_member_unit_reclaim.py + test_members_roster_recency.py = 576
passed / 1 skipped (-n0). test_members_dm_thread.py stressed at -n4 x8 = 182
passed each. black (py312) / isort / flake8 / mypy clean on the changed modules.

---------


(cherry picked from commit 7460284)

Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com>
Co-authored-by: kirocrew-agent <agent@kirocrew.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants