Skip to content

fix(crew-log): let the eager seal finish before a delete removes the log - #35705

Merged
bolichen97 merged 1 commit into
mainfrom
fix/34376-ledger-remove-race
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/34376-ledger-remove-race

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Permanently deleting a chat removes its crew log every time, instead of sometimes leaving it on disk until the retention sweep.

test/test_remove_slot_for_history_key_crew_log.py::TestSessionLedgerOnPermanentDelete::test_a_ledger_the_emitter_itself_opened_is_still_removed fails in CI on pull requests that cannot reach it (Linux, macOS and Windows backend shards, #34376): after _remove_slot_for_history_key the session's crew log is still on disk (assert not self._exists("acp-emitted")). It kept failing after #34569, which fixed a different hold on the same lease.

Why it matters

That test is the production shape of a permanent delete: the gateway that deletes the chat is also the one writing its log. When the race is lost in a real gateway, the deleted conversation's crew log, which holds its message bodies, stays on disk (excluded from its slot) until the retention sweep collects it. And every unrelated pull request that draws the red pays a rerun.

Not a goal

  • Changing how remove_unit claims a unit's lease, or how the eager folder seals a closed unit, for any caller.
  • Waiting without a bound: a fold that does not settle in time still leaves the unit to retention, as a flush that times out does today.
  • The other removals (retention sweep, trash, crew member removal): none of them removes a session unit right after writing its close.

What changed (motivation → approach → change)

A delete now removes the chat's crew log even when the log was closed a moment before.

A permanent delete writes session/closed, flushes the emitter so that entry lands and the emitter lets go of its handle, then claims the unit's lease sole to remove the files. Since #33967 the same close also wakes the eager folder, which seals the unit's savepoints (seal_session_folds) under a shared hold of that lease. The flush does not wait for that fold, so a claim that landed mid-seal answered owned and the log stayed.

_remove_session_crew_logs (dashboard/handlers/sessions_parts/slot_teardown.py) now calls eager.drain after the flush, with its own 2 s bound, so the claim comes after the folder has settled, which covers the seal the close woke. crew-log-core.md and the eager.drain docstring name this third caller.

Backwards compatibility

Compatible: the delete makes the same removal, and its response can come back at most 2 s later while the folder settles; nothing it accepts, returns or keeps changes.

Removes nothing: the change only orders an existing removal after a background fold; the units removed and the units kept are the same.

Tests

No test file changes: the existing test_a_ledger_the_emitter_itself_opened_is_still_removed is the regression test.

Determinism proof (testing-conventions, Proving a determinism fix), parent = ee7a026dd0:

Run Parent ee7a026dd0 This head
Forced condition (below), -n0, 5 runs 5 red (assert not self._exists("acp-emitted"); the sole claim was refused while the seal held the lease) 5 green
Alone at -n0 10 runs: 2 red 20 runs: 20 green
-n 4 --dist loadgroup --max-worker-restart=2 with test/test_crew_log_retention.py and test/test_remove_slot_for_history_key.py, 10 runs 10 green (175 tests each)
This file plus test/test_crew_log_retention.py, shuffled, seed 1874 (94 ids) green

The forced condition is a throwaway -p plugin, not committed. Once the close is emitted, the delete's sole lease claim waits until the close-time seal (checkpoint.save(final=True)) holds its shared lease. The seal keeps that lease until the delete has either made its claim or entered eager.drain, then 0.3 s more, so this head goes green only if drain really waits for the seal. The plugin logs each step, so a run where the condition never fired cannot count as evidence.

Proof: not provable -- other: prove.py re-runs only changed test files and this fix changes none; the unchanged flaky test is proven by the forced condition above

Manual verification

N/A — unit coverage sufficient: the change is one bounded wait, off the event loop, before the delete's removal, and the forced condition above drives exactly that interleaving.

Screenshots / video

N/A — backend only; nothing a user sees changes.

Related Issues

Closes #34376

Sibling sweep

Swept: git grep -nE "remove_unit(|stage_unit(" -- src/
Sibling: src/kiro_crew/dashboard/handlers/sessions_parts/slot_teardown.py:509 -- changed here
Sibling: src/kiro_crew/crew_log/store_retention.py:151 -- not affected -- the sweep removes only units whose close is older than the retention window, and an owned unit is retried by the next pass
Sibling: src/kiro_crew/eventlog/service.py:1312 -- not affected -- member units are not session units, so the close-time session seal never holds their lease
Sibling: src/kiro_crew/session_storage_parts/crew_staging.py:82 -- not affected -- the trash stages only sessions untouched for MIN_RECLAIM_AGE_DAYS, long after any close fold

Pattern harvest

Not generalizable: the delete funnel is the only path that writes a session unit's close and removes the unit in the same breath; every other removal acts on units closed long before.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 12, 2026 01:34
@bolichen97
bolichen97 requested a review from buluoray November 12, 2026 01:34
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d85cb63d61e6614b1933c3b34eb245e61575315a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d85cb63d61e6614b1933c3b34eb245e61575315a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the 2 s wait comes after the flush, runs off the event loop, and a timeout leaves the log to the retention sweep as before.

Notes

  • eager.drain waits on the folder's global _settled < _queued counters, not on this one unit. On a busy gateway the delete can spend the full 2 s waiting on other sessions' folds, and then still lose the race to the retention sweep. If that shows up in practice, the fix is a per-unit settle, not a longer wait.

[DESIGN-REVIEWED] d85cb63

Description read

[DESCRIPTION-READ] f22f00e8ba43a700a3d2d25efae4db21a93e85b341e30c862d11b08de695d4c2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d85cb63d61e6614b1933c3b34eb245e61575315a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The change is one bounded call to an existing barrier, eager.drain, and it fixes the race at the mechanism level: the delete's sole lease claim now waits until the close-time seal has finished.

Notes

  • Item 1 — symptom-level (accepted): the real cause is that a sole claim in remove_unit gives up on a short shared seal hold instead of waiting for it. That fix is excluded by ## Not a goal ("Changing how remove_unit claims a unit's lease"), and the description names it, so this is a deferral, not a gap.

What this change ships

Inventory (3 items) — 2 justified

Intent: when someone permanently deletes a chat, its crew log should be removed every time and not left on disk until retention runs. This is a FIX.

  1. The delete now waits up to 2 s for the eager folder to settle before it removes the crew log — symptom-level (cause deferred under Not a goal)
  2. A delete's response can now take up to 2 s longer (shares the existing _CREW_LOG_TEARDOWN_FLUSH_SECONDS bound) — justified
  3. The spec crew-log-core.md and the eager.drain docstring now name the delete funnel as a third caller — justified

[FIRST-PRINCIPLES-REVIEWED] d85cb63

Description read

[DESCRIPTION-READ] f08f0aadb34605dbe0dc405b6d01510bcde8d187f1d8c8f036feb83058fd28bd

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d85cb63d61e6614b1933c3b34eb245e61575315a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d85cb63

Verdict parsed from the review's SHA-scoped output markers for commit d85cb63d61e6614b1933c3b34eb245e61575315a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d85cb63d61e6614b1933c3b34eb245e61575315a false-positive: <reason, 20 to 500 characters>

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Forced-condition evidence for the ## Tests table: the throwaway pytest plugin (not committed) and what it logged.

Command, run 5 times in a worktree at the parent ee7a026dd0 and 5 times at this head 9980f977b5:

PYTHONPATH=<plugin dir> FORCE_LOG=<log file> python -m pytest -n0 -p no:cacheprovider -p _force_seal_race -q \
  'test/test_remove_slot_for_history_key_crew_log.py::TestSessionLedgerOnPermanentDelete::test_a_ledger_the_emitter_itself_opened_is_still_removed'

Log of one parent run (all 5 the same; each failed with assert not self._exists("acp-emitted") at line 158; the eager.drain entry is the test teardown, after the refused claim):

seal lease=held
sole claim after close: seal seen=True refused=CrewLogError
seal released by delete progress=True
delete entered eager.drain

Log of one run at this head (all 5 passed and none logged a refused claim; in four the delete entered eager.drain just before the seal took its lease rather than just after; the last eager.drain entry is the test teardown):

seal lease=held
delete entered eager.drain
seal released by delete progress=True
delete entered eager.drain
_force_seal_race.py
"""Throwaway forcing plugin for #34376 (never committed).

Forces the race the diagnosis names: the delete's SOLE lease claim lands while the
eager worker's close-time seal (``checkpoint.save(final=True)``) holds the unit's
NON-SOLE lease.

* Once ``session/closed`` has been emitted, the delete's sole claim first waits (5 s
  bound) until the seal holds its lease.
* The seal, once it holds the lease, keeps it until EITHER the delete has made its
  claim OR the delete has entered ``eager.drain`` (5 s bound), and then 0.3 s more.

Parent (no eager.drain before the claim): the claim lands while the seal is still
blocked holding the lease -> REMOVE_OWNED -> the ledger stays -> RED on every run.
Fix (eager.drain first): the seal lets go 0.3 s after the delete starts waiting; the
claim is made only once drain returns, so it is green only if drain really waited for
the seal (a drain that returned at once would claim inside those 0.3 s and go red).

FORCE_LOG names a file that records what happened, so a run whose condition never
fired cannot pass for evidence.
"""

import os
import threading
import time

_tl = threading.local()
_closed = threading.Event()
_seal_holding = threading.Event()
_released_by = threading.Event()  # the delete claimed, or entered eager.drain


def _note(line: str) -> None:
    path = os.environ.get("FORCE_LOG")
    if path:
        with open(path, "a", encoding="utf-8") as fh:
            fh.write(line + "\n")


def pytest_configure(config):
    from kiro_crew.crew_log import checkpoint as cp
    from kiro_crew.crew_log import eager
    from kiro_crew.crew_log import emit
    from kiro_crew.crew_log import store_removal

    orig_save, orig_hold = cp.save, cp._hold
    orig_acquire = store_removal.acquire_lease
    orig_closed = emit.on_session_closed
    orig_drain = eager.drain

    def save(*a, **k):
        _tl.final = bool(k.get("final"))
        try:
            return orig_save(*a, **k)
        finally:
            _tl.final = False

    def hold(handle, *a, **k):
        lease = orig_hold(handle, *a, **k)
        if getattr(_tl, "final", False):
            _note(f"seal lease={'held' if lease is not None else 'refused'}")
            if lease is not None:
                _seal_holding.set()
                released = _released_by.wait(5.0)
                _note(f"seal released by delete progress={released}")
                time.sleep(0.3)
        return lease

    def acquire(*a, **k):
        if k.get("sole") and _closed.is_set():
            seen = _seal_holding.wait(5.0)
            try:
                return orig_acquire(*a, **k)
            except Exception as exc:
                _note(f"sole claim after close: seal seen={seen} refused={type(exc).__name__}")
                raise
            finally:
                _released_by.set()
        return orig_acquire(*a, **k)

    def drain(*a, **k):
        if _closed.is_set():
            _note("delete entered eager.drain")
            _released_by.set()
        return orig_drain(*a, **k)

    def on_session_closed(*a, **k):
        _closed.set()
        return orig_closed(*a, **k)

    cp.save, cp._hold = save, hold
    store_removal.acquire_lease = acquire
    emit.on_session_closed = on_session_closed
    eager.drain = drain

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d85cb63d61e6614b1933c3b34eb245e61575315a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.

[GPT-REVIEWED] d85cb63

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d85cb63d61e6614b1933c3b34eb245e61575315a false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 12, 2026 01:48
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Approve. This is a small, bounded fix for a race in permanent chat delete. Closing a session writes session/closed, and that wakes the eager folder, which seals the session's savepoints under a shared hold of the session log's write lock (its lease). The delete's exclusive ("sole") lock claim could land mid-seal, get refused as owned, and leave the chat's crew log on disk until the retention sweep. The PR adds a call to eager.drain (with its own 2 s limit) after emit.flush and before the claim.

I checked that the fix works:

  • note_commit runs on the log's writer thread inside the append job (emit_state.py:213), so the close's wake is already counted in _queued before flush returns.
  • The seal (seal_session_folds) runs inside _fold_batch, and _settle(taken) runs in a finally after it (eager.py:550-556). So once drain sees _settled >= _queued, the seal has let go of the lock.
  • The delete runs _remove_session_crew_logs through asyncio.to_thread, so the extra wait of up to 2 s does not block the event loop.
  • If drain times out, the outcome is the same as on main: the claim is refused as owned and the log is left to retention. Nothing fails open and

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 9980f977b5 · 39 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 9980f97. A permanent chat delete now waits, for at most 2 seconds and off the event loop, for the eager folder to finish the seal that the chat's close woke, and only then claims the crew log's lease to remove it. Before this, a claim that landed while that seal held a shared lease was refused, and the deleted chat's crew log stayed on disk until the retention sweep. The order is sound: the close's wake is queued inside the append job, so the flush returns only after it is counted, and the seal runs inside the fold batch before the batch is counted as settled. Removing the new wait turned the existing regression test red in 5 of 20 runs; with it the test passed 20 of 20. Every wait is bounded, nothing new holds a lock across a wait, and a timeout falls back to the earlier behaviour (the log is left to retention), so no path deletes more than before or blocks forever. Follow-ups, not blocking: the drain waits for the whole folder, not only this unit, so on a busy gateway it can use its full 2 seconds while other sessions keep queuing wakes; and the forced-interleaving plugin that proves the fix deterministically is not committed, so CI only catches a regression by chance. Decisive lines: src/kiro_crew/dashboard/handlers/sessions_parts/slot_teardown.py:390, src/kiro_crew/crew_log/eager.py:1176, src/kiro_crew/crew_log/eager.py:779, src/kiro_crew/crew_log/eager.py:546, src/kiro_crew/crew_log/emit_state.py:213, src/kiro_crew/crew_log/store_removal.py:166

Permanently deleting a chat flushed the emitter so the session/closed
entry landed, then claimed the unit's lease sole to remove its crew
log. That close also wakes the eager folder, which seals the unit's
savepoints under a shared hold of the same lease. A sole claim landing
mid-seal answered owned, so the deleted conversation's log stayed on
disk until the retention sweep, and the test that pins the production
shape failed whenever the seal won the race.

The funnel now waits for the eager folder to settle, with its own 2 s
bound like the flush, before it claims the lease.

Closes #34376
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/handlers/updates.py and test/test_update_boot_apply.py since this head's CI ran (ee7a026 -> 169429f); CI re-running.

Both test reds on 9980f97 were main's, and #35625 fixed them: test_updates_handlers_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers (Linux shard 6, Windows shard 4) and test_wait_discipline_ratchet.py::test_the_scan_matches_known_sites (Linux shard 5, Windows shard 2). The rebase was clean and the patch is unchanged (git range-diff: =). The new head is d85cb63.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026
@bolichen97
bolichen97 merged commit 6d8fb8d into main Nov 12, 2026
97 checks passed
@bolichen97
bolichen97 deleted the fix/34376-ledger-remove-race branch November 12, 2026 11:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_ledger_the_emitter_itself_opened_is_still_removed leaves the session ledger on disk when run alone

3 participants