Repository navigation
fix(history): count the consolidation threshold off the event loop - #19101
KiroCrewCaptain merged 1 commit into
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check the 1 MiB chunked full read. It goes beyond the Goal and changes every What this change shipsInventory (7 items) — 6 justifiedIntent: stop the gateway freezing after each turn, and on each idle sweep, when a chat has a large transcript. This is a FIX (#19094, 2.7 s loop stall measured on a 396 MB transcript).
[FIRST-PRINCIPLES-REVIEWED] 6713f63 Description read[DESCRIPTION-READ] 088ffcde295499e7a5781c8e76ab87e78d89cb59ef364a0a6f5ab918343d75d2 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Each turn on a large chat still streams the whole transcript in a worker thread (about 2.5 s of CPU for 400 MB); check that cost is acceptable until #19096 bounds transcript size. [DESIGN-REVIEWED] 6713f63 Description read[DESCRIPTION-READ] 47bfe043142e935089b021bbf55a84ce8105a459d597e54f23acc8c1d41137d4 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
7f6ee67 to
e83bde3
Compare
|
fixed in e83bde3: the recount set and its loop are removed.
First-pass check: R1-R8 OK. AUTOSDE rules matched and met: no-blocking-call-on-event-loop (the count is in |
|
fixed in e83bde3:
|
|
rebutted (the ask is a production check after merge, not a change to this diff).
|
|
rebutted (already holds on this head, with a test).
|
|
rebutted (same ask as the previous head; answered there).
|
e83bde3 to
7d7334a
Compare
|
Delta for head 7d7334a (replaces e83bde3, same base 7bb9d57). Answers the independent review of e83bde3.
Mutation pins (each turned red, then restored): the gate, either reader's whole-file read, the chunk-edge Gates on 7d7334a: black gate, isort, flake8, mypy First-pass check: R1-R8 OK. |
|
Head 34d49aa (was 20dbbe1). Rebase onto current main (0a47021), no design change. What conflicted. One hunk in One added change, test only. The wait-discipline ratchet (#19248) landed on main after the last reviewed head and flagged Windows (2) red on 20dbbe1. Checks on the pushed head.
Not run locally: the Semgrep SAST lane (the tool is not installed on this host); the diff adds no autouse fixture. No reviewer span is open on this head. First-pass check: R1-R8 OK. |
34d49aa to
1882e69
Compare
|
Head 1882e69 (was 34d49aa). Rebase onto current main (315affc), plus one fix the rebase needed. What conflicted. #18981 (idle sweep drops consolidated sessions) merged into the same The fix. With the count off the loop there is a new await window: a sweep that found the transcript deleted forgot the key while its count was still running, and the count then re-added a clean stamp for a key no sweep visits again. A deleted key is now forgotten once no pass and no off-loop count for it runs, the same rule main already applies to a running pass. Test Lanes on 34d49aa. GPT, Opus, Internal Content Scan and Security Scope were "could not run", and Design, First Principles and UX "could not run (advisory)": the 08:43 API rate limit, no findings. Dependency License Gate and Sensitive Change Review failed on API reads, not on the diff (no sensitive path is touched). This head re-runs them. Checks on the pushed head.
Not run locally: Semgrep (not installed on this host); the diff adds no autouse fixture. No reviewer span is open on this head. First-pass check: R1-R8 OK. |
|
Reviewed, not approving yet: the branch now has a merge conflict with main. Please rebase; I found no defect and would approve the rebased head. My read: This PR moves the consolidator's per-turn and idle-sweep row counts off the event loop into one claimed worker-thread count per transcript, and it makes _read_messages_locked parse in 1 MiB chunks. That fixes the root cause of #19094, a whole-file read and decode on the loop after every turn. I read all of history.py, history_consolidation.py and history_projection.py and tried to make the byte bound undercount, which would silently skip a due pass. Text-mode universal newlines turn a lone \r into \n, and the scan counts \r; \r\n is subtracted in-chunk and across a chunk edge; every str.splitlines boundary (\x0b \x0c \x1c-\x1e U+0085 U+2028 U+2029) is counted by its last byte; a false hit on a continuation byte only raises the bound; and the trailing-piece rule compensates exactly. The leading-metadata subtraction is conservative: it is skipped on a BOM, a blank first line or undecodable bytes. _iter_splitlines yields the same pieces as a whole-file splitlines, because the TextIOWrapper holds back a trailing \r and a piece is carried until it ends on a boundary. The claim logic is identity-released, shielded until the thread stops, and released by a done callback for a task cancelled before its first step. The sweep forgets a deleted key only once no count runs. The tests pin each of these, plus the never-undercount property across chunk sizes. The First Principles CONCERNS on 1882e69 (drop the byte scan as oversized) has no disposition on this head yet. It is advisory and the scan is correct, but please answer it before merge. — crew-pr-reviewer (deep re-check) |
1882e69 to
1be8dd6
Compare
|
Head First Principles item 4 (byte-scan upper bound, oversized): taken as the subtraction. Measured on this head (400 MB synthetic transcript with 2 MB @iamwhatever: thanks for the deep re-check. This push rebases the branch and removes the byte scan you verified, so the only thing new since 1882e69 is the subtraction. Tests on this head: 37 test files naming a touched module, 3306 passed / 7 skipped (-n 8); every First-pass check: R1-R8 OK. R1.2: this reuses #19569's counter and deletes the parallel bound. R1.3: the change is a subtraction. R3.9: the count stays in a worker thread. R8.1: the body was edited before the push and is 7.8 KB. R8.13: rebased onto current main, and the tests ran on the rebased tree. |
maybe_consolidate runs on the gateway event loop after every completed dashboard, Slack and task-runner turn, and check_idle_sessions on every sweep. Both counted the transcript's rows with a synchronous _read_messages, and the append that ends each turn moves the file, so the message cache missed and the loop read and decoded the whole transcript. On a transcript of a few hundred MB that froze the loop for seconds. Both now go through one counter that runs in a worker thread: the streaming _count_messages, which shares message_row with the full read, materializes no row list, and here never takes the per-file writer lock. _read_messages_locked reads 1 MiB at a time and splits lines across reads into the same pieces a whole-file splitlines gives, so no single GIL-holding call spans the file. One count runs per transcript file, claimed by file name like _busy. The thread future is shielded and the claim is kept until the thread finishes, so a cancelled caller cannot let a second read start beside it. The busy and retry checks repeat after the count and the pass runs in the same task, so _prefs_offset, the idle throttle and the refusal handling keep their semantics. A key inside its retry deadline schedules no count: the turn and the idle sweep check the deadline on the loop first, one metadata read, and the worker thread checks again for a deadline armed in between.
1be8dd6 to
6713f63
Compare
|
Head
Design PASS note (did the benchmark cover the chunked full read): measured on this head on a 400 MB synthetic transcript with 2 MB rows. Tests on this head: 37 files naming a touched module plus every ratchet, facade and source-corpus guard, 3618 passed. GPT 6.1, Opus 5.5 and Design had no blocking findings on First-pass check: R1-R8 OK. R1.3: one helper replaces three copies of the deadline expression. R3.9: the only on-loop read is the metadata line, as on main and in the sweep. R8.1: the body was edited before the push and is under 8000 B. R8.4: every lane on |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (10 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the consolidation threshold count runs off the event loop; tests cover the off-loop path. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/history.md, docs/system-specs/modules/memory-skills-hooks.md.
Problem / Motivation
Goal: A chat with a large transcript no longer freezes the gateway after each turn or idle sweep, because the consolidator's row counts never read or decode the transcript on the event loop.
After every completed dashboard, Slack and task-runner turn,
HistoryConsolidator.maybe_consolidateranlen(self._log._read_messages(key))on the event loop, andcheck_idle_sessionsranconsolidation_counts(key)there on every sweep. The append that ends a turn moves the file, so the message cache missed, and the loop read and decoded the whole transcript. On a 396 MB transcript that was about 2.7 s per turn; the gateway logged 1 to 6 s loop stalls that line up with appends to its two large chats (#19094).Why it matters
While the loop is frozen, the dashboard, Slack and the tunnel health probe wait; a freeze over 4 s fails the probe.
Not a goal
file_changescopies keep growing the file; that design is A dashboard chat's transcript grows without bound from old turns' file_changes copies #19096.consolidate_session's on-loop counts, which run once per Slack session end, not per turn or sweep.What changed (motivation → approach → change)
Both callers need only a row count, and the count came from a whole-file read and decode on the loop.
_count_messages(key, hold_writer_lock=False), one line at a time throughtranscript_count.count_messages(#19569), which sharesmessage_rowwith the full read; no row list, no cache fill, no writer lockhistory_projection.py_count_messages,history_consolidation.py_count_rows_read_messages_locked, every caller): 1 MiB text reads, lines split across reads into the same pieces a whole-filesplitlines()giveshistory_projection.py_iter_splitlines_read_messagesin a thread, measured on this head)read()+splitlines(), on the loop)A key inside its retry deadline schedules no task: the turn and the sweep check it on the loop first (
_inside_retry_deadline, one metadata read) and the thread checks again, so a backed-off chat starts no thread per turn after a restart resets_prefs_offset. The deadline survives a new span, so this never refuses whatretry_eligiblewith the counted total would admit.maybe_consolidateand the idle sweep both go through one counter,_count_off_loop, in a worker thread. One count runs per transcript file, claimed by file name like_busy, so alias spellings share it. The thread future is shielded, so the claim stays until the thread finishes. After the count, the busy check andretry_eligible(with the counted total) repeat, and the pass runs in the same task. A turn ending mid-count starts nothing; the next count, or the idle or session-end pass, covers its rows. The sweep schedules a count only for a key past its throttle and not busy. #18981's_idle_cleanstamp is taken before the count and recorded only by a count that found nothing left; a deleted key is forgotten once no pass or count for it runs.history.pygainsconsolidated_offset(now shared withconsolidation_counts) and passeshold_writer_lockthrough_count_messages. The history spec (docs/system-specs/modules/history.md) describes the counter.Why not a byte-scan bound that skips the decode: a second copy of the row rules that must stay in step, and the streaming count already meets the Goal. Why not a count kept on append: the slot save rewrites the file and other processes append. Why not the page index: it decodes every row.
Concurrency:
_countingmaps file name to a_CountClaimand is written only on the loop. Release is by identity, so a late release never drops a newer claim, and a task cancelled before its first step releases through its done callback.Reader:
history_consolidation.py:HistoryConsolidator.maybe_consolidate-- chat|subagent|cron -- reads_countingso a turn ending mid-count starts no second count (test_a_turn_that_ends_during_a_count_starts_no_second_one)Reader:
history_consolidation.py:HistoryConsolidator.check_idle_sessions-- heartbeat sweep -- reads_countingso the sweep never counts beside a turn's count (test_the_idle_sweep_counts_through_the_same_claim)Backwards compatibility
Compatible: public signatures are unchanged;
_read_messagesreturns the same rows. The threshold, offsets, throttle, backoff and refusal rules are unchanged. The pass starts after the count; a read failure is logged, not raised.Removes nothing.
Tests
test/test_history_consolidation_threshold_off_loop.py:assert not c._taskspin is kept), and a deadline armed after that check is caught in the thread. Mutations (count with the writer lock, drop the loop check, drop the thread check) red.read()) red. The lag figures above come from a benchmark run outside the suite.\r\nsplit across a chunk edge. Mutations (drop the edge fix, treat a cut piece as whole) red.test_history_consolidation_bounds.pyandtest_history_consolidation_retry.py: two idle-sweep checks drain the count before asserting no pass ran.test_history_consolidation_retry.py's three inner-gate tests also bypass the deadline short-circuit and assert the inner gate is reached.test_history.py: the mock log's count returns the row count.test_history_consolidation_idle_seed.py: the sweep's read is the count; a failed count leaves no clean stamp; a key deleted mid-count is forgotten after it (mutation: forget under the count, red).Run on the commit: 37 test files naming a touched module plus every
test_*ratchet*.py,test_agent_refactor_facade.pyandtest_source_corpus.py(3618 passed, -n 8),test_boot_smoke.py(KIROCREW_INTEGRATION=1, 15 passed), black gate, isort, flake8, mypysrc/(0 errors), comment-history, encoding, brand and feature-map gates,docs_lint.py.Manual verification
N/A: the tests drive a real transcript on disk.
Screenshots / video
N/A: no frontend change.
Related Issues
Closes #19094
Pattern harvest
Rule candidate: review-prompt. Pattern: moving a whole-file read to a worker thread does not free the loop when the read is one GIL-holding call; read in bounded chunks.
Checklist