Skip to content

fix(history): exclude concurrent consolidations across processes - #21159

Merged
chenmingwei23 merged 1 commit into
mainfrom
replace/9900
Oct 16, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
replace/9900

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Two processes never run a history consolidation over the same session span at the same time, so a span is billed and written to memory once.

HistoryConsolidator._running is an in-memory set. It stops two consolidation passes over one session inside a single process and says nothing about another one — and kirocrew consolidate IS another process, running against the same transcripts while the gateway's 60s idle sweep is live. Both processes snapshot the same span, both spend a provider turn on it, and both write the history entry, preferences and lessons that turn produced.

Why it matters

Nothing catches it afterwards. mark_consolidated is idempotent, so the durable offset ends up correct and no error is raised: the cost is a duplicate billed turn and doubled memory writes for the same span, with no signal that it happened. kirocrew consolidate --all against a live gateway is enough to hit it.

Not a goal

  • Replacing the in-process _running set, which stays as the cheap scheduling short-circuit.
  • A maximum duration for a consolidation pass; the ceiling bounds only a holder that stops renewing.
  • Holding a lock across the provider turn.
  • Deduplicating outputs a holder committed before it lost its lease (publication receipts). That is separate work.

What changed (motivation → approach → change)

This replaces fork PR #9900 by @Premshay, carried onto current main with Premshay as commit author. The design is theirs; the changes below are what the rebase and the open review findings needed.

Where it lives. Main split HistoryConsolidator into owner modules (#20868). The lease is a new owner, history_consolidation_lease.py (constants, _lease_holder_is_live, and the lease/fence methods), bound on the class through the history_consolidation.py facade like the other owners. _LeaseLostMidRun, _CONSOLIDATION_BUSY and _no_pass_ran sit beside the other run sentinels in history_consolidation_span.py; _consolidate's changes are in history_consolidation_pass.py. test_history_consolidation_facade.py's pinned surface and member table list the new names and signatures.

Lease. _consolidate claims a durable lease on the session's metadata line through ConversationLog.update_metadata_if (compare-and-set under the existing cross-process flock) BEFORE its authoritative snapshot, so a busy attempt reads nothing and bills nothing. The flock is held only for the compare-and-set, never across the LLM turn. The holder is identified by PID + platform_compat.get_process_start_id + the new process_incarnation_id (reboot-unique); an unknown identity is never read as a mismatch. A heartbeat on the cross-process monotonic clock is renewed every 5 min; _CONSOLIDATION_LEASE_CEILING_SECS (1 h) bounds silence, not pass length. update_metadata_if(require_write=True) refuses a write skipped on a legacy row-first line, so an unpersisted claim is never granted; other callers keep the old result.

Fence. The acquisition token travels with the run. Every publication (_publication_hold_checked), the snapshot, every marker write (_lease_fence, now also covering main's sub-offset advance from #17965) and the failure accounting re-read the token under the transcript lock and raise _LeaseLostMidRun when a peer took over. A fenced pass returns _CONSOLIDATION_REFUSED; a busy pass returns _CONSOLIDATION_BUSY. Callers test _no_pass_ran(...), so the idle-sweep throttle, the preference offset and the CLI success flag stay put for both.

Failure charge fenced with the abandon marker (GPT 6.1 fenced finding at history_consolidation.py:1305 on #9900). The charge used to be written before any ownership check. With four failures recorded, a stale holder whose lease was taken would write the fifth, then have its abandon marker refused, leaving the new holder's span capped. _note_failed_attempt now checks the token, records the charge and, at the cap, writes the abandon marker inside ONE fenced lock hold. On a lost lease nothing is charged, no abandon notice fires, and the pass is refused.

Removed from the fork version (First Principles / Design concerns on #9900):

  • V1 publication receipts in preferences.md, projects.md and the daily history files. They changed user-edited file formats, leaked a source comment into model context (Design), and only covered a takeover after an hour of silence. The fence alone keeps a stale holder from writing.
  • consolidation_lease_at, a wall-clock field with zero readers, and the branch that kept a record with no monotonic heartbeat forever. Every writer stamps the heartbeat, so such a record is malformed and reads as free.

Blockers listed by @NicholasRBowers on 2026-09-30, and how each is resolved

  1. Wall-clock lease freshness (suspend/resume, NTP can expire a live holder) — heartbeat age is time.monotonic() only; no wall-clock stamp is written. Pinned by test_wall_clock_correction_does_not_expire_live_lease.
  2. Lease taken after snapshot_for_consolidation — the lease is acquired first and the snapshot re-checks the token under the same lock hold. Pinned by test_busy_pass_does_not_snapshot_or_copy, test_snapshot_follows_peer_completion_before_acquisition.
  3. Base conflict with fix(history): keep incognito and temporary transcripts, learn nothing #13760 and history.md — rebuilt on current main (including feat(memory): sub-chunk an over-budget consolidation head with a durable sub-offset #17965 sub-message slicing and fix(history): isolate consolidation sessions #19733); every new marker path (_advance_history_marker, head retirement, member-receipt replay) is fenced.
  4. Unrelated test reds (Windows timing flake, settingsRegistry drift) — not in this diff; this branch starts from current main.
  5. First Principles "delete the heartbeat" vs GPT's fix — renewal kept: removing it reopens expiry of a long, working pass. The smaller option taken instead: the zero-reader _LEASE_AT field and the retain-forever legacy branch are removed, and receipts are dropped.
  6. get_process_start_id is None on Windows — None is "cannot tell", never a mismatch; the holder falls back to pid-exists plus the monotonic ceiling. Pinned by test_an_unknowable_identity_is_not_read_as_a_mismatch.

The Design lane's alternative (a per-session OS try-lock held across the provider pass) is not adopted: holding a lock across the provider turn is an explicit Not-a-goal of the original PR.

Backwards compatibility

Compatible: the lease fields are new optional keys on the metadata line; update_metadata_if keeps its old result unless a caller passes require_write=True; _update_metadata_locked now returns a bool that existing callers ignore. The CLI's skip line now reads "no completed pass recorded" instead of "skipped (consolidation retry backoff)", since a skip can also be a busy lease.

Removes nothing: no key, record, path or public function is removed or renamed; a consolidation that would previously run concurrently with a peer is refused instead.

Tests

  • test/test_history_consolidation_lease.py (new, from fix(history): exclude concurrent consolidations across processes #9900, adapted): who holds a lease (no PID, missing/invalid heartbeat, dead, recycled PID, unknown identity, reboot, ceiling), heartbeat renewal and loss, take/refuse/release, busy pass costs nothing, lease released on every exit path including cancellation, a taken-over lease publishes nothing, the fence reads the line and not the cache.
  • New for the GPT finding: test_empty_result_after_takeover_charges_nothing[0|4] (with four prior failures, the new holder's span keeps 4 attempts and stays eligible), test_a_lost_lease_refuses_the_charge_and_the_abandon_marker, test_a_held_lease_charges_and_abandons_in_one_hold.
  • test/consolidation_lease_helpers.py: a mock log whose metadata writes read back, used by the mock-log consolidation tests (including main's newer test_member_memory_algorithm, test_memory_write_switches and the sub-slice member test).
  • test/test_history_consolidation_facade.py: surface and member inventory updated for the new owner and the two changed signatures.
  • Local: TMPDIR=/tmp/r9900 PYTHONPATH=src pytest -n 8 over every test file that touches consolidation, plus the facade-inventory, wait-discipline, platform-compat and privacy-mode suites — 4876 passed, 35 skipped. black (baselined), isort, flake8, mypy, file-line-cap, feature-map and subprocess-encoding gates pass on the changed set.

Manual verification

N/A — the race is between two processes on a shared metadata line; the compare-and-set, liveness, ceiling and fence paths are covered by tests that use real transcripts on disk.

Related Issues

Fixes #21158
Replaces #9900

Pattern harvest

Rule candidate: review-prompt
Pattern: an in-process guard (a set, a flag, an in-memory lock) used as mutual exclusion over durable state another process can reach; and accounting written before an ownership check on a span a peer can take over.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (docs/system-specs/modules/history.md, docs/system-specs/common/platform-compat.md)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md. Original work by @Premshay in #9900.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision db248d437fb89bce8411bd1abf15bba436b110f6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of db248d437fb89bce8411bd1abf15bba436b110f6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

_CONSOLIDATION_BUSY is a second sentinel that no production code tells apart from _CONSOLIDATION_REFUSED, and _no_pass_ran exists only to treat the two alike.

Not justified as shipped

  • Item 3 — zero consumers: git grep _CONSOLIDATION_BUSY -- src finds only its definition, its return in _consolidate and its re-exports. All 5 callers test _no_pass_ran(...), so busy and refused are handled identically everywhere.
    Subtraction: return _CONSOLIDATION_REFUSED when the lease is busy, delete _CONSOLIDATION_BUSY and _no_pass_ran, and keep the existing is not _CONSOLIDATION_REFUSED checks at the 5 call sites.
    Clears when: a production caller acts differently on busy than on refused, or the deletion is made.
  • Item 4 — oversized: on every platform where process_incarnation_id answers, it carries the same start identity as _LEASE_START (Linux: start ticks plus boot id; macOS: libproc microseconds; Windows: FILETIME). So the lease stores and compares the same identity twice.
    Subtraction: keep one identity field and drop the other comparison branch in _lease_holder_is_live. If you keep only _LEASE_START, also drop the new public process_incarnation_id alias.
    Clears when: one identity field remains, or a platform is named where the two fields actually differ.

What this change ships

Inventory (7 items) — 5 justified

Intent: stop kirocrew consolidate and the gateway sweep from billing and writing the same span twice. This is a FIX.

  1. A pass claims a lease on the session's metadata line before it reads the span — justified
  2. Every durable write and failure charge checks the lease token first — justified
  3. A new "busy" refusal sentinel, plus a _no_pass_ran helper — zero consumers that tell it apart
  4. Two holder-identity fields, plus a new platform_compat.process_incarnation_id — oversized
  5. A heartbeat renews the lease, with a 1 h ceiling on silence — justified
  6. update_metadata_if re-reads from disk for every caller, and gains require_write — justified
  7. The CLI skip line now reads "no completed pass recorded" — justified

[FIRST-PRINCIPLES-REVIEWED] db248d4

Description read

[DESCRIPTION-READ] a2d870f5bb6cd52d6da63db2bec066aba13d9e295a14bf8012df1b31cf5ec781

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of db248d437fb89bce8411bd1abf15bba436b110f6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the owner.cancel() call on a lost lease never cancels a caller's task when that caller awaits _consolidate inline instead of in its own task.

[DESIGN-REVIEWED] db248d4

Description read

[DESCRIPTION-READ] b14653b6dc9c3c157aff3b2a73276c40e29a77ba280d69512742874d7bb42b4f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed db248d437fb89bce8411bd1abf15bba436b110f6 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] db248d4

Verdict parsed from the review's SHA-scoped output markers for commit db248d437fb89bce8411bd1abf15bba436b110f6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable db248d437fb89bce8411bd1abf15bba436b110f6: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of db248d437fb89bce8411bd1abf15bba436b110f6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] db248d4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt db248d437fb89bce8411bd1abf15bba436b110f6: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 16, 2026
@iamwhatever
iamwhatever marked this pull request as ready for review October 16, 2026 09:42
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 16, 2026 09:42
Claim a durable, renewable session lease before consolidation. Compare the
holder PID, process identity and monotonic heartbeat across processes; fence
every publication, marker write and failure charge with the acquisition token
under the transcript lock. An unpersisted or unavailable claim refuses the pass
without advancing caller completion bookkeeping.

Carried onto current main (sub-message slicing, retry accounting, the
consolidation owner split): the lease lives in its own owner,
history_consolidation_lease, bound on HistoryConsolidator through the
facade. The sub-offset advance is fenced like the whole-message marker, and the failure
charge plus the abandon marker run in ONE fenced lock hold, so a stale holder
can no longer cap the new holder's span. The lease records only the monotonic
heartbeat; a record without one reads as free. Publication receipts are left
out of this change.

(cherry picked from commit a4491e3)

Co-authored-by: Premshay <28099628+Premshay@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 16, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 16, 2026

@chenmingwei23 chenmingwei23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved: readiness: passed, no conflicts, checks green.

@chenmingwei23
chenmingwei23 merged commit 61bd116 into main Oct 16, 2026
114 of 122 checks passed
@chenmingwei23
chenmingwei23 deleted the replace/9900 branch October 16, 2026 15:40
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

History consolidation can run twice on one session across processes

3 participants