Repository navigation
fix(history): exclude concurrent consolidations across processes - #21159
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: stop
[FIRST-PRINCIPLES-REVIEWED] db248d4 Description read[DESCRIPTION-READ] a2d870f5bb6cd52d6da63db2bec066aba13d9e295a14bf8012df1b31cf5ec781 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the [DESIGN-REVIEWED] db248d4 Description read[DESCRIPTION-READ] b14653b6dc9c3c157aff3b2a73276c40e29a77ba280d69512742874d7bb42b4f That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Claim a durable, renewable session lease before consolidation. Compare the holder PID, process identity and monotonic heartbeat across processes; fence every publication, marker write and failure charge with the acquisition token under the transcript lock. An unpersisted or unavailable claim refuses the pass without advancing caller completion bookkeeping. Carried onto current main (sub-message slicing, retry accounting, the consolidation owner split): the lease lives in its own owner, history_consolidation_lease, bound on HistoryConsolidator through the facade. The sub-offset advance is fenced like the whole-message marker, and the failure charge plus the abandon marker run in ONE fenced lock hold, so a stale holder can no longer cap the new holder's span. The lease records only the monotonic heartbeat; a record without one reads as free. Publication receipts are left out of this change. (cherry picked from commit a4491e3) Co-authored-by: Premshay <28099628+Premshay@users.noreply.github.com> Co-authored-by: Claude Code <noreply@anthropic.com>
0197d04 to
db248d4
Compare
chenmingwei23
left a comment
There was a problem hiding this comment.
Approved: readiness: passed, no conflicts, checks green.
Problem / Motivation
Goal: Two processes never run a history consolidation over the same session span at the same time, so a span is billed and written to memory once.
HistoryConsolidator._runningis an in-memory set. It stops two consolidation passes over one session inside a single process and says nothing about another one — andkirocrew consolidateIS another process, running against the same transcripts while the gateway's 60s idle sweep is live. Both processes snapshot the same span, both spend a provider turn on it, and both write the history entry, preferences and lessons that turn produced.Why it matters
Nothing catches it afterwards.
mark_consolidatedis idempotent, so the durable offset ends up correct and no error is raised: the cost is a duplicate billed turn and doubled memory writes for the same span, with no signal that it happened.kirocrew consolidate --allagainst a live gateway is enough to hit it.Not a goal
_runningset, which stays as the cheap scheduling short-circuit.What changed (motivation → approach → change)
This replaces fork PR #9900 by @Premshay, carried onto current main with Premshay as commit author. The design is theirs; the changes below are what the rebase and the open review findings needed.
Where it lives. Main split
HistoryConsolidatorinto owner modules (#20868). The lease is a new owner,history_consolidation_lease.py(constants,_lease_holder_is_live, and the lease/fence methods), bound on the class through thehistory_consolidation.pyfacade like the other owners._LeaseLostMidRun,_CONSOLIDATION_BUSYand_no_pass_ransit beside the other run sentinels inhistory_consolidation_span.py;_consolidate's changes are inhistory_consolidation_pass.py.test_history_consolidation_facade.py's pinned surface and member table list the new names and signatures.Lease.
_consolidateclaims a durable lease on the session's metadata line throughConversationLog.update_metadata_if(compare-and-set under the existing cross-processflock) BEFORE its authoritative snapshot, so a busy attempt reads nothing and bills nothing. Theflockis held only for the compare-and-set, never across the LLM turn. The holder is identified by PID +platform_compat.get_process_start_id+ the newprocess_incarnation_id(reboot-unique); an unknown identity is never read as a mismatch. A heartbeat on the cross-process monotonic clock is renewed every 5 min;_CONSOLIDATION_LEASE_CEILING_SECS(1 h) bounds silence, not pass length.update_metadata_if(require_write=True)refuses a write skipped on a legacy row-first line, so an unpersisted claim is never granted; other callers keep the old result.Fence. The acquisition token travels with the run. Every publication (
_publication_hold_checked), the snapshot, every marker write (_lease_fence, now also covering main's sub-offset advance from #17965) and the failure accounting re-read the token under the transcript lock and raise_LeaseLostMidRunwhen a peer took over. A fenced pass returns_CONSOLIDATION_REFUSED; a busy pass returns_CONSOLIDATION_BUSY. Callers test_no_pass_ran(...), so the idle-sweep throttle, the preference offset and the CLI success flag stay put for both.Failure charge fenced with the abandon marker (GPT 6.1 fenced finding at
history_consolidation.py:1305on #9900). The charge used to be written before any ownership check. With four failures recorded, a stale holder whose lease was taken would write the fifth, then have its abandon marker refused, leaving the new holder's span capped._note_failed_attemptnow checks the token, records the charge and, at the cap, writes the abandon marker inside ONE fenced lock hold. On a lost lease nothing is charged, no abandon notice fires, and the pass is refused.Removed from the fork version (First Principles / Design concerns on #9900):
preferences.md,projects.mdand the daily history files. They changed user-edited file formats, leaked a source comment into model context (Design), and only covered a takeover after an hour of silence. The fence alone keeps a stale holder from writing.consolidation_lease_at, a wall-clock field with zero readers, and the branch that kept a record with no monotonic heartbeat forever. Every writer stamps the heartbeat, so such a record is malformed and reads as free.Blockers listed by @NicholasRBowers on 2026-09-30, and how each is resolved
time.monotonic()only; no wall-clock stamp is written. Pinned bytest_wall_clock_correction_does_not_expire_live_lease.snapshot_for_consolidation— the lease is acquired first and the snapshot re-checks the token under the same lock hold. Pinned bytest_busy_pass_does_not_snapshot_or_copy,test_snapshot_follows_peer_completion_before_acquisition.history.md— rebuilt on current main (including feat(memory): sub-chunk an over-budget consolidation head with a durable sub-offset #17965 sub-message slicing and fix(history): isolate consolidation sessions #19733); every new marker path (_advance_history_marker, head retirement, member-receipt replay) is fenced.settingsRegistrydrift) — not in this diff; this branch starts from current main._LEASE_ATfield and the retain-forever legacy branch are removed, and receipts are dropped.get_process_start_idisNoneon Windows —Noneis "cannot tell", never a mismatch; the holder falls back to pid-exists plus the monotonic ceiling. Pinned bytest_an_unknowable_identity_is_not_read_as_a_mismatch.The Design lane's alternative (a per-session OS try-lock held across the provider pass) is not adopted: holding a lock across the provider turn is an explicit Not-a-goal of the original PR.
Backwards compatibility
Compatible: the lease fields are new optional keys on the metadata line;
update_metadata_ifkeeps its old result unless a caller passesrequire_write=True;_update_metadata_lockednow returns a bool that existing callers ignore. The CLI's skip line now reads "no completed pass recorded" instead of "skipped (consolidation retry backoff)", since a skip can also be a busy lease.Removes nothing: no key, record, path or public function is removed or renamed; a consolidation that would previously run concurrently with a peer is refused instead.
Tests
test/test_history_consolidation_lease.py(new, from fix(history): exclude concurrent consolidations across processes #9900, adapted): who holds a lease (no PID, missing/invalid heartbeat, dead, recycled PID, unknown identity, reboot, ceiling), heartbeat renewal and loss, take/refuse/release, busy pass costs nothing, lease released on every exit path including cancellation, a taken-over lease publishes nothing, the fence reads the line and not the cache.test_empty_result_after_takeover_charges_nothing[0|4](with four prior failures, the new holder's span keeps 4 attempts and stays eligible),test_a_lost_lease_refuses_the_charge_and_the_abandon_marker,test_a_held_lease_charges_and_abandons_in_one_hold.test/consolidation_lease_helpers.py: a mock log whose metadata writes read back, used by the mock-log consolidation tests (including main's newertest_member_memory_algorithm,test_memory_write_switchesand the sub-slice member test).test/test_history_consolidation_facade.py: surface and member inventory updated for the new owner and the two changed signatures.TMPDIR=/tmp/r9900 PYTHONPATH=src pytest -n 8over every test file that touches consolidation, plus the facade-inventory, wait-discipline, platform-compat and privacy-mode suites — 4876 passed, 35 skipped. black (baselined), isort, flake8, mypy, file-line-cap, feature-map and subprocess-encoding gates pass on the changed set.Manual verification
N/A — the race is between two processes on a shared metadata line; the compare-and-set, liveness, ceiling and fence paths are covered by tests that use real transcripts on disk.
Related Issues
Fixes #21158
Replaces #9900
Pattern harvest
Rule candidate: review-prompt
Pattern: an in-process guard (a set, a flag, an in-memory lock) used as mutual exclusion over durable state another process can reach; and accounting written before an ownership check on a span a peer can take over.
Checklist
docs/system-specs/modules/history.md,docs/system-specs/common/platform-compat.md)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md. Original work by @Premshay in #9900.