Skip to content

fix(history): exclude concurrent consolidations across processes - #9900

Closed
Premshay wants to merge 1 commit into
kirodotdev:mainfrom
Premshay:upstream/consolidation-cross-process-lease
Closed

Premshay wants to merge 1 commit into
kirodotdev:mainfrom
Premshay:upstream/consolidation-cross-process-lease

Conversation

@Premshay

@Premshay Premshay commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Two processes never run a history consolidation over the same session span at the same time, so a span is billed and written to memory once.

HistoryConsolidator._running is an in-memory set. It stops two consolidation passes over one session inside a single process and says nothing about another one — and kirocrew consolidate IS another process, running against the same transcripts while the gateway's 60s idle sweep is live.

Both processes snapshot the same span, both spend a provider turn on it, and both write the history entry, preferences and lessons that turn produced.

Why it matters

Nothing catches it afterwards. mark_consolidated is idempotent, so the durable offset ends up correct and no error is raised: the cost is a duplicate billed turn and doubled memory writes for the same span, with no signal that it happened. The _running guard reads like exclusion at every call site, which is what makes the gap easy to keep.

The collision is not exotic — kirocrew consolidate --all against a live gateway is enough, and every session whose idle sweep fires inside that window is exposed.

Not a goal

  • Replacing the in-process _running set, which stays as the cheap scheduling short-circuit.
  • A maximum duration for a consolidation pass; the ceiling bounds only a holder that stops renewing.
  • Holding a lock across the provider turn.

What changed (motivation → approach → change)

_consolidate takes a durable lease on the session's metadata line, immediately after the retry-eligibility gate and before anything can be billed. It is claimed through ConversationLog.update_metadata_if, whose guard runs under the same cross-process flock as the write, so two processes arriving together cannot both pass it.

No new locking machinery. The compare-and-set, the lock and the metadata line all already existed for the consolidation retry accounting — which is state about the same span, read under the same lock on every pass. The lease is four more fields on that line.

The flock is held only for the compare-and-set, never across the LLM turn. Holding it for the pass would block every append to the session and freeze the conversation behind a provider call. That makes the lease a durable record rather than a held lock, so it needs its own answer to a holder that dies mid-pass.

Liveness, not a timeout. _lease_holder_is_live reads the recorded PID plus platform_compat.get_process_start_id — already this repository's idiom for "still the process that wrote this record" (session_pid._pid_start_token), in-process on every platform that answers it, and different across a PID reuse. A crashed holder therefore stops excluding peers on the very next attempt rather than after a wait. None from that lookup means "cannot tell" and is never read as a mismatch, so a Windows holder degrades to pid-exists plus the ceiling instead of being falsely stolen from.

_CONSOLIDATION_LEASE_CEILING_SECS (1 h) bounds silence when the persisted monotonic heartbeat is comparable with the current clock. It is not a maximum pass duration. Legacy/malformed heartbeats and future monotonic stamps with inconclusive process identity retain a live PID until release, exit, identity mismatch or a usable heartbeat; the one-hour bound does not cover those cases. Expiry permits a peer to acquire the lease, and the previous holder is cancelled when its renewal detects the replacement.

The token identifies one ACQUISITION, not the holder. That is what makes a second attempt from the same process refuse, and what stops a pass that overran the ceiling from clearing the claim that replaced it on its way out. The lease fields are deliberately not in _CONSOLIDATION_META_KEYS: those are dropped when a span is marked consolidated, and the lease is still held at that point — the memory, lesson and history writes all come after it.

Every write is fenced on that token. A holder whose renewals fail past the ceiling keeps working until its next renewal notices, so the expiry that lets a peer claim the span also opens a window where both passes are running over the same messages — and both reach publication. The acquisition token therefore travels with the run: _publication_hold_checked re-reads it inside the transcript lock the publication is about to take (dropping this process's cached metadata line first, the same precaution update_metadata_if takes for its guard) and raises _LeaseLostMidRun when the line names another acquisition. _mark_consolidated_fenced holds that same lock across the check and the marker rewrite, so a stale holder cannot advance the marker either, and the abandon marker in _note_failed_attempt is fenced the same way so it cannot retire messages the new holder is reading. A fenced pass returns _CONSOLIDATION_REFUSED, charges no attempt and leaves the span to the new holder — including when an earlier output already committed, because the pass does not own what it would write next.

_CONSOLIDATION_BUSY is a second instance of the sentinel class _CONSOLIDATION_REFUSED already uses. Callers now test with _no_pass_ran instead of against one instance, because what they act on is the shared part: no prompt was sent, nothing was billed, no window was covered — so the idle sweep's throttle, maybe_consolidate's preference offset and the CLI's success flag all have to stay put. Testing one instance is exactly how a new refusal reason reads as a completed pass; the preference offset advancing over a window nothing extracted is the case with teeth, and it has a test.

Heartbeat age is monotonic. Every claim and renewal stamps time.monotonic(), and liveness measures silence on that clock, so a forward or backward wall-clock step (NTP, VM resume) cannot release a working holder. platform_compat.process_incarnation_id adds a reboot-unique identity (Linux start ticks plus boot UUID, macOS libproc start, Windows creation FILETIME), so a PID/start pair reused after a reboot is not mistaken for the holder. A record with no monotonic stamp (legacy) keeps a live holder until it releases or exits rather than guessing elapsed time from the wall clock.

The lease is taken before the authoritative snapshot, so a busy attempt reads nothing and a winner snapshots a span no peer can still be consolidating. Acquisition is shielded from cancellation, and a token written during a cancellation is recovered so finally releases it. A refusal after billing (a source change, a lost lease) returns the same no-pass sentinel, and kirocrew consolidate reports "no completed pass recorded" for it.

Deliberately out of scope

The lease does not replace _running. The in-process set stays as the cheap scheduling short-circuit that avoids taking the lock at all, the same role the entry points' retry_eligible pre-checks play for the backoff.

Pattern harvest

Rule candidate: an in-process guard is not mutual exclusion over durable state. Where a resource is reachable by a second process — a file, a metadata line, a billed external call keyed on one — a set, a flag or a lock held in one interpreter's memory reads like exclusion at every call site while providing none. The tell is a guard whose lifetime is the process and whose subject is on disk.

Two consequences worth carrying: the exclusion has to live where the contention does (here, the same locked metadata line the state already occupies, not a new lock file), and a durable claim that outlives its holder needs a liveness answer, because a record cannot release itself. Prefer an identity the platform can verify (PID plus process start time) over a duration guess; keep a ceiling only as the backstop for what that identity cannot answer.

Tests

New test/test_history_consolidation_lease.py (55 tests): who still holds a lease (missing/​unparsable record, running holder, dead holder, recycled PID, unknowable identity, ceiling), taking and dropping it (claim, refuse, release, stale-token release, dead-holder takeover), what a refused pass costs (no provider call, span untouched, no attempt charged, holder's lease survives, preference window not marked covered), that the lease does not outlive its pass (completed and raising), and that a taken-over lease publishes nothing: the pass is refused, append_history and mark_consolidated are never reached, the span stays for its new holder, and a peer write hidden behind this process's cached metadata line is still seen. test/consolidation_lease_helpers.py gives the mock-log fixtures a metadata record the lease compare-and-set can read back, so a fixture that grants a lease it never records stops reading as a stale holder.

The fence tests were also run against the pre-fix revision and fail there — the stale holder completes its pass and publishes — so that coverage is non-vacuous too.

Both exclusion guards confirmed non-vacuous by disabling the gate and observing test_a_held_session_never_reaches_the_provider and test_a_prefs_window_is_not_marked_covered fail.

pytest -q -k "history or consolidat or cli or memory" --ignore=test/test_transcribe.py — 7543 passed, 31 skipped. One unrelated pre-existing failure in the run, test_webex_client.py::TestReadLoopFrameDispatch::test_malformed_binary_frame_does_not_crash_the_loop, confirmed failing on main without this change. Repo gates pass on the changed set: black ratchet, sync-io-in-async, comment-history, docs lint.

Manual verification

N/A — the race is between two processes on a shared metadata line, and the compare-and-set, liveness and ceiling paths are all covered above.

Related Issues

Touches consolidate_now's refusal handling and the CLI's skip message, which #9742 also edits. The two are logically independent — this one is about exclusion, that one about prompt size — and whichever lands second will be rebased.

Checklist

  • Existing tests pass and regression coverage is included.
  • Self-review completed; code follows project style guidelines.
  • Documentation updated where applicable.
  • No secrets, credentials, or internal references in the diff.

Contribution License Agreement

N/A — template placeholder; no CLA wording is supplied.

🤖 Generated with Claude Code

@Premshay
Premshay requested review from a team and cixuuz September 10, 2026 12:27
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 10, 2026
@github-actions

github-actions Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of a4491e3200dbf969756d4071b72913b7cbe30fdd via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The PR rebuilds process-death detection by hand: PID, start id, incarnation, monotonic heartbeat, 1 h ceiling, token fencing and receipts. A per-session OS try-lock does that by itself.

Watch

  • A hand-built lease where an existing OS lock would do. The description rejects a lock file because the claim "has to survive a restart to be worth taking". But surviving its holder is exactly what forces the rest of the design: _lease_holder_is_live, process_incarnation_id, the 300 s heartbeat (each one a full-transcript rewrite in _update_metadata_locked), _LeaseLostMidRun fencing at every write, and the receipt subsystem for takeovers. platform_compat.try_acquire_lock already exists on every platform. Held on a separate per-session .consolidate.lock for the length of the pass, it never blocks transcript appends, the kernel releases it when the process dies, and two holders can never overlap. So no fencing is needed, and the half the description calls "Liveness, not a timeout" goes away entirely.
    Clears when: the description says why a non-blocking try_acquire_lock on a per-session lock file held for the pass is not enough, or the lease is replaced by one.
  • Receipts live in documents users edit and the model reads. append_history writes <receipt>\n<!-- consolidation-source: N digest -->, but _without_publication_receipts strips only \n<receipt>\n. The source comment therefore still reaches read_recent_history, which feeds both model context and the dashboard editor. write_today_history then saves the user's edit without the receipt, so after any edit to today's history a later takeover's consolidation_history_prefix finds no receipt and appends the entry again. This also changes the format of preferences.md, projects.md and the daily history files, which the visible description never mentions.
    Clears when: the receipts move out of user-edited content (or the strip covers both lines and survives an edit), and the description explains the memory-file format change.

Suggestions

  • If the lease stays, ship the receipt and dedup machinery in its own PR. Cross-process exclusion does not need it, and its persisted format deserves its own review.

[DESIGN-REVIEWED] a4491e3

Description read

[DESCRIPTION-READ] 5f4883c4a6d4346f679e40dde1e78887b7d2e544fed9c1f1a7627dffb311d5db

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of a4491e3200dbf969756d4071b72913b7cbe30fdd via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The lease, its fence and its liveness test fix the cause. A second mechanism, digest receipts written into users' memory files, rides along, and one wall-clock lease field has no reader.

Not justified as shipped

  • Item 4 — rides along, oversized: the description says "No new locking machinery … The lease is four more fields on that line." The diff also adds a persisted format, <!-- consolidation-publication: … --> comments with up to 128 kept, inside preferences.md, projects.md and the daily history files. It adds consolidation_history_prefix and strips the comments on three read paths. The lease's fence already refuses stale writes. Receipts only cover a takeover after renewals have been silent for over an hour, and the base-documented append_history "carries no receipt" defect, which is a separate fix.
    Subtraction: remove _with_publication_receipts, _without_publication_receipts, consolidation_history_prefix and the publication_id/source_messages parameters from this PR. The fence is enough for the Goal.
    Clears when: receipts ship in their own PR, or a test shows the fence alone writes a span to memory twice.
  • Item 7 — zero consumers: _LEASE_AT is written in three places and read nowhere (counted 0 production readers of _LEASE_AT). The no-monotonic "legacy" branch handles a record that no base or diff writer produces (consolidation_lease appears 0 times on base).
    Subtraction: delete _LEASE_AT and the KeyError/non-finite "retain forever" branch.
    Clears when: both are removed, or an existing reader is named.

What this change ships

Inventory (10 items) — 6 justified

Intent: stop the CLI and the gateway from both billing and writing memory for the same session span — FIX.

  1. A second process can no longer consolidate a session span that another process is already consolidating — justified
  2. The holder renews its claim every 5 minutes and is cancelled if a peer takes the claim over — justified
  3. Every memory write and marker write is refused once the claim has been lost — justified
  4. Digest receipts stored inside history, preferences and projects files — rides along, oversized
  5. The guarded metadata write always rereads disk and can require a real write — justified
  6. process_incarnation_id: process identity that stays unique across reboots — justified
  7. Wall-clock heartbeat field and the legacy-record branch — zero consumers
  8. CLI now prints "no completed pass recorded" instead of the backoff-only message — justified
  9. A session with no transcript file now reports "already in flight" instead of completed — undeclared
  10. Info logs added for skips of non-persistent and incognito sessions — rides along

[FIRST-PRINCIPLES-REVIEWED] a4491e3

Description read

[DESCRIPTION-READ] 37554f0d212ed000ecd8868c0ba34eb78a92d6f9ac2f39d64b963906abbb4a89

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed a4491e3200dbf969756d4071b72913b7cbe30fdd via the fork AI-review pipeline; updated in place on each push.

1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/history_consolidation.py:1305 -- A stale holder can permanently cap the replacement’s span

except _LeaseLostMidRun:
Four recorded failures + lease takeover before the replacement’s retry gate -> the old provider returns empty -> _note_failed_attempt persists attempt five before checking ownership -> abandonment is refused, leaving the marker unchanged and retry_eligible rejecting that span even after backoff expires.
Anchor: residual/crash-data-loss-corruption
Fix: Fence the failure-accounting increment under the transcript lock before recording it; propagate ownership loss as _CONSOLIDATION_REFUSED without charging the replacement’s span.
[BLOCK-MERGE] a4491e3
[GPT-REVIEWED] a4491e3

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 is the only finding, and it is fenced. My ruling is FLAG: it keeps blocking, and the only way to clear it is still an /ai-review override from a repository writer. There were no adjudicable findings, so the first footer is all zeros.

F1: I confirmed the failure path. _note_failed_attempt charges the attempt through record_consolidation_failure with no lease check (history_consolidation.py:943, history.py:3484). The check only runs later, at the abandonment-marker write (diff hunk at 1300–1311). The cap is 5 (history_consolidation.py:67).

The conditions needed are rare, all at once:

  • The span already has four recorded failures.
  • The old holder is alive but has sent no heartbeat for 3600s. Renewals run every 300s and a transient error retries, so that means about 12 straight failed renewals (diff _lease_holder_is_live / _heartbeat_consolidation_lease).
  • A peer takes the lease, and the stale holder's provider then returns empty.

The harm is LOW to MEDIUM and contained. At the cap the span would have been dropped without a memory pass anyway. The only extra damage is a missing marker, and the base already documents and accepts that exact state for a failed abandonment-marker write (history.py:3371-3375). It clears on new messages for a span the attempt read to the end (history.py:3404-3411), or on a rotation or rewrite (history.py:3392-3400). No data is corrupted and nothing touches credentials or governance.

[ADJUDICATION] a4491e3200dbf969756d4071b72913b7cbe30fdd total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] a4491e3200dbf969756d4071b72913b7cbe30fdd

[ADJUDICATION-FENCED] a4491e3200dbf969756d4071b72913b7cbe30fdd fenced=1 flagged=1
FLAG F1 src/kiro_crew/history_consolidation.py:1305 -- Needs four prior failures, plus a live holder silent past the 3600s ceiling despite 300s retried renewals, plus a peer takeover and then an empty result; the outcome is the base's already-documented failed-abandonment-marker cap (history.py:3371), which clears on growth, rotation or rewrite, and the span would have been abandoned at the cap anyway, so nothing is corrupted.
[GPT-ADJUDICATED-FENCED] a4491e3200dbf969756d4071b72913b7cbe30fdd

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override recorded by a repository writer, who must independently verify a rationale before recording it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop. (This lane's comment deliberately carries no override command.)

  • F1 src/kiro_crew/history_consolidation.py:1305 — Needs four prior failures, plus a live holder silent past the 3600s ceiling despite 300s retried renewals, plus a peer takeover and then an empty result; the outcome is the base's already-documented failed-abandonment-marker cap (history.py:3371), which clears on growth, rotation or rewrite, and the span would have been abandoned at the cap anyway, so nothing is corrupted.

@github-actions

github-actions Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed a4491e3200dbf969756d4071b72913b7cbe30fdd via the fork AI-review pipeline; updated in place on each push.

Review details

No findings block this merge. Three advisory findings follow.

FINDING — src/kiro_crew/memory.py:97 — append_history writes receipt + "\n" and then <!-- consolidation-source: N <hex> -->\n, but content.replace(f"\n{receipt}\n", "") removes only the receipt. Every consolidated V1 entry read through _guarded_entry (_read_recent_history_uncached → model context, the activity index, the dashboard history editor) therefore ends with entry<!-- consolidation-source: … -->, and a dashboard save writes that orphaned line to disk for good → Fix: strip the receipt together with an optional following consolidation-source line, for example with re.sub(r"\n<!-- consolidation-publication: [0-9a-f]{64} -->\n(?:<!-- consolidation-source: [0-9]+ [0-9a-f]{64} -->\n)?", "", content).

FINDING — src/kiro_crew/history_consolidation.py:1305 — the except _LeaseLostMidRun: arm logs "Not marking abandoned consolidation" and then falls through to callback = self.on_abandoned. A stale holder that hits the attempt cap after a peer took its lease charges the peer's span, because record_consolidation_failure is not fenced. It also makes Slack _notify_consolidation_abandoned report messages as abandoned when no marker was written → Fix: return from that arm, and check _lease_is_current before record_consolidation_failure.

FINDING — src/kiro_crew/memory.py:514 — full = _with_publication_receipts(...) is passed unchanged to self._index_file(self._projects_file, full) and to _warn_projects_over_cap(full), and the early-return path indexes current with its receipts too. Up to 128 receipts of about 90 bytes each (roughly 11 KB of hex comments) end up in FTS search results. They also count toward a cap warning ("sessions will not see its end") that sessions never hit, because their reads strip the receipts → Fix: run the index and cap check on _without_publication_receipts(...), and append receipts only to the bytes written to disk.

[OPUS-REVIEWED] a4491e3

@Premshay

Copy link
Copy Markdown
Contributor Author

Both fenced findings addressed — one fixed, one I'd like to argue against. New head: 668fa3ce0.

F1 — lease expires during a supported pass (history_consolidation.py:70). Fixed.

The finding is right, and the ceiling's own comment gave the game away: it claimed an hour was "far past any real pass", which is a statement about duration when the thing it needs to bound is silence. A holder that is alive and working past the ceiling is exactly the pass that most needs the exclusion — a long transcript on a slow provider — and it was the one that lost it.

_LEASE_AT is now a heartbeat rather than an acquisition stamp. _heartbeat_consolidation_lease refreshes it every _CONSOLIDATION_LEASE_RENEW_SECS (5 min) for as long as the pass runs, started right after acquisition and cancelled by the pass's own finally. The ceiling keeps its value and its purpose: a wedged holder stops renewing, so it is still bounded, and the interval is twelve times narrower than the ceiling so a starved loop or a slow metadata write can miss several beats without costing the lease.

Two details worth calling out, both pinned by tests:

  • The renewal is token-guarded exactly like the release, so a pass whose lease was already taken cannot stamp its own liveness onto the peer's claim.
  • The heartbeat is cancelled before the release, not after — otherwise a beat landing between them could backdate a peer's fresh acquisition.

I chose renewal over the alternative the finding offers ("cover the maximum supported consolidation duration") because there is no such maximum to cover. Widening the ceiling to fit the worst transcript we can imagine only moves the boundary; it does not stop it from cutting across a live pass.

F2 — fail-open on lease-write failure (:581). I'd like to keep this as it is.

This one I think should stay, and the adjudication above reaches the same conclusion — it flags F2 as an edge case on the grounds that the fail-open is a deliberate, documented tradeoff.

Returning None unless the lease is durably acquired trades a bounded, rare cost for an unbounded one. The lease exists to prevent a duplicate provider turn, which costs spend and some doubled memory — the marker write is idempotent, so the offset stays correct either way. Failing closed on an unwritable metadata line converts every transient IO error into a session that never consolidates at all, and there is no retry that recovers it: consolidation is what stops the transcript growing without bound, so a session wedged this way degrades until someone notices by hand.

The two also need very different conditions. Fail-open costs anything only when a metadata write fails and a cross-process peer is consolidating the same session in the same window. Fail-closed costs a wedged session on the metadata write failure alone.

If you'd still like this closed, the shape I'd suggest is not a fail-closed return but a retry with backoff around update_metadata_if, so a transient error is absorbed rather than converted into either failure mode. Happy to add that if you want it — I've left it out for now because it is a new mechanism for a case the current code already handles in the safer direction.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 10, 2026
@Premshay
Premshay force-pushed the upstream/consolidation-cross-process-lease branch from 668fa3c to f0f4c18 Compare September 10, 2026 17:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 10, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 11, 2026
@Premshay
Premshay force-pushed the upstream/consolidation-cross-process-lease branch from f0f4c18 to 1d371ad Compare September 13, 2026 21:09
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 13, 2026
@Premshay
Premshay force-pushed the upstream/consolidation-cross-process-lease branch from 9f21ca8 to c5b1892 Compare September 17, 2026 15:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been idle since 2026-09-26 with a blocking review and a conflicting base. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: GPT blocks because lease freshness is measured on the wall clock, so an ordinary forward step (suspend/resume, NTP) lets a peer take a demonstrably live holder's lease and both processes consolidate the same span. The stated fix is to measure heartbeat age with a cross-process monotonic timestamp and keep wall time only as a reboot fallback, plus re-snapshotting after acquisition since the lease is currently taken after snapshot_for_consolidation. The base conflict is core logic: main's #13760 rewrote the same refusal/incognito region of _consolidate that this PR edits, and history.md overlaps too. Both test reds look unrelated to this diff — a Windows timing flake in test_decisions_memory_recall_reachable.py and a frontend settingsRegistry base-drift failure that only fails Coverage Gate closed — and should clear on rebase.

One tension I will resolve rather than escalate: First Principles asks to delete the heartbeat/renewal subsystem outright, which pulls against GPT's fix, and deleting it reopens the earlier-round finding the heartbeat was added for. That is exactly the repair-versus-remove comparison our PR procedure already defines, so I will rule on it, take the smaller complete option that keeps the goal intact, and post a disposition either way. I will also flag that get_process_start_id returns None on Windows, so the monotonic change has to keep the pid-plus-ceiling fallback honest.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@Premshay
Premshay force-pushed the upstream/consolidation-cross-process-lease branch from c1bd966 to 01da51d Compare October 1, 2026 09:27
@Premshay
Premshay requested a review from a team as a code owner October 1, 2026 09:27
@Premshay
Premshay requested review from CrysisDeu and removed request for cixuuz October 1, 2026 09:27
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 1, 2026
@Premshay
Premshay force-pushed the upstream/consolidation-cross-process-lease branch from 01da51d to 0ab49b4 Compare October 1, 2026 09:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@Premshay

Premshay commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Current-head verification at 0ab49b42e2f49c3b937d71ae9ed6f395292b26e0:

  • 176 lease, retry and consolidation CLI tests pass locally on Python 3.12 with two workers. Both TestShutdownRefillWindow tests also pass. The macOS shutdown-test failure was not reproduced on this Linux host; this is not evidence that the macOS failure is fixed.
  • The historical hygiene log is stale: the current body contains both Goal: and Not a goal.
  • The design review correctly identifies an overstatement in the description. I corrected only that paragraph under What changed; the frozen goal sections are unchanged. The implementation already pins the retained-holder trade in test_unknown_reboot_identity_and_future_stamp_retain_holder and documents it in the history spec. A prior-boot future stamp with unknown identity can retain exclusion beyond one hour. I have not substituted wall-clock expiry or represented that retention policy as newly approved by a maintainer.
  • Cross-process comparability is a documented Python guarantee, not an inference from the epoch: time.monotonic has been the same clock for all processes since Python 3.5, including macOS since 3.10; this repository requires Python >=3.12. An undefined reference point does not prevent comparisons between processes in one boot. https://docs.python.org/3.12/library/time.html#time.monotonic
  • Removing renewal would reopen the already-addressed long-running-pass expiry case. I have not adopted the proposed lease redesign or added publication fencing after the existing adjudication downgraded that finding.

No source or branch change. Remaining review decision: accept the explicitly documented retention/expiry trade, or specify a different recovery policy. macOS verification of the shutdown flake remains outstanding.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 1, 2026
@Premshay

Premshay commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Pushed c96f96e40ac8ffdb27c2d47192ef05ba8a10f581 (two PR commits). This repairs the shutdown test failure without changing consolidation retention policy or production code.

The current macOS shard-4 log fails TestShutdownRefillWindow::test_a_tick_that_spawned_after_shutdown_began_kills_its_own_child with killed == []. Its class-wide ensure_spawned mock sets the test executor's shutdown flag even when called for a different executor's child. Another live pool's reaper can therefore stop this executor before its own tick reaches the mocked spawn.

I reproduced that exact assertion deterministically on Linux by injecting a second executor's ensure_spawned call immediately after the class patch. It fails on both the old PR head and clean parent d8fa6a527e927504b399246332cd243d7b450e2f; it passes after scoping the shutdown and kill-recording callbacks to this executor's children. Foreign children use the original implementation.

All 71 subprocess-pool tests pass (Python 3.12, two workers), as does the deterministic interleaving probe. Changed-file Black/isort/flake8, Comment History Gate and diff-check pass. The previously completed 176 lease/retry/CLI tests were not rerun because their code is unchanged. Native macOS confirmation remains outstanding; Linux reproduction is not a claim of a green macOS run.

PR Hygiene is already SUCCESS in the refreshed check list (run 36925900749, job 110584077664). The reported missing Goal/Not-a-goal failure belongs to the older run 36845089299. No further description or policy change was made.

@Premshay

Premshay commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

@NicholasRBowers — thanks for the offer. The blocker has moved since your assessment: the wall-clock-freshness finding is addressed at the current head via the monotonic lease stamp, and the remaining GPT 6.1 finding — the metadata CAS reporting success when its locked write skips a row-first line — is now fixed at head (8bd8167: the CAS fails when the write is skipped, with a regression test). No further pushes should be needed from your side. One gate remains: the Issue Gate wants an issue of this repository declared, and there is none — this exclusion work follows the consolidation changes already merged. Could you waive issue-gate, or point me at the issue it should reference?

@Premshay

Premshay commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

New head 70ba439d4 fixes the privacy-mode regression (squashed to one commit for the one-commit check).

Root cause: the lease CAS made update_metadata_if return False when the locked write skipped a legacy message-first first line. That was right for the lease — an unpersisted claim must be refused — but it also changed the answer every other caller gets, and the privacy-mode pin reads the header back: a skipped write on a message-first transcript must still answer True, because the read-back is the only observable and persist_failed misdiagnoses it.

The fix splits the contract instead of choosing a side: update_metadata_if keeps the legacy True for a skipped write by default, and the three lease sites (acquire, renew, release) opt into require_write=True, where that no-op returns False exactly as before. I checked all other update_metadata_if callers — none pass the flag, so they are back on the base behaviour the pin carries.

Verified at this head: 143 passed across test_messaging_privacy_mode.py (92) and test_history_consolidation_lease.py (51), including the previously failing test_a_message_first_transcript_refuses_and_publishes_nothing.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision a4491e3200dbf969756d4071b72913b7cbe30fdd touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

Claim a durable, renewable session lease before consolidation. Compare the
holder PID, process identity and monotonic heartbeat across processes; fence
every publication and marker write with the acquisition token under the
transcript lock. An unpersisted or unavailable claim refuses the pass without
advancing caller completion bookkeeping.

A holder can lose its lease after publishing but before acknowledging the
transcript. Commit V1 publication receipts in the same atomic replacement as
their outputs, and consult them on takeover. Validate the published history
prefix and retain its boundary when new messages arrive. Preserve document
receipts through owner edits and repair derived indexes on replay.

Exercise takeover, interrupted publication, midnight rollover and source
changes. Bound heartbeat cancellation checks with completion signals and
isolate clock patches from the shared standard-library module.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@Premshay

Premshay commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Separate OS lock versus the frozen lease design (span=ff6e91e07bf3)

needs-a-decision: Maintainers, may this PR replace its metadata lease with a separate per-session OS try-lock held across the provider pass? Such a lock need not block transcript appends and automatically releases on process death. The frozen Not-a-goal explicitly excludes holding a lock across the provider turn, so adopting that simpler design requires an explicit human scope amendment. Otherwise the lease stays and failure accounting/abandonment must be atomically fenced. No data-loss concern is waived.

AI-authored by kirocrew-steward (Codex).

@Premshay

Copy link
Copy Markdown
Contributor Author

AI-authored by kirocrew-steward (Codex). Audit disposition ata4491e3200dbf969756d4071b72913b7cbe30fdd: data-loss repair and design ruling required; not review-ready.

The stale-holder finding is legitimate: failure accounting happens before ownership is fenced, and _LeaseLostMidRun falls through to on_abandoned. A retained-lease repair must fence ownership and failure increment atomically and suppress abandonment on lease loss. Memory receipts must not leak source comments into context/indexing/cap warnings; the same finding appears at both history-read and projects-write sites.

Compare receipt removal with crash/takeover behavior before editing the persisted format. The actual Windows CRLF receipt tests and V1 prompt test red need reproduction/classification. The separate OS-lock alternative is put to maintainers because it conflicts with the frozen Not-a-goal. No data-loss deferral, override or clearance is claimed.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @Premshay for this work. The cross-process consolidation lease, its liveness test and the write fence are your design, and they now continue in #21159, with you as the commit author.

Main moved a lot under this PR (sub-message slicing in #17965, and the HistoryConsolidator owner split in #20868), so #21159 carries the change onto current main. It also:

  • fences the failure charge and the abandon marker in one lock hold, which clears the GPT 6.1 finding at history_consolidation.py:1305;
  • leaves out the publication receipts and the unused wall-clock lease field, per the First Principles and Design concerns;
  • lists how each blocker from the 2026-09-30 assessment is resolved.

All checks on #21159 are green. Tracking issue: #21158. I'm closing this one in favour of #21159; your branch is left as it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants