Skip to content

fix(history): count the consolidation threshold off the event loop - #19101

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/19094-consolidate-count-off-loop
Oct 13, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/19094-consolidate-count-off-loop

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A chat with a large transcript no longer freezes the gateway after each turn or idle sweep, because the consolidator's row counts never read or decode the transcript on the event loop.

After every completed dashboard, Slack and task-runner turn, HistoryConsolidator.maybe_consolidate ran len(self._log._read_messages(key)) on the event loop, and check_idle_sessions ran consolidation_counts(key) there on every sweep. The append that ends a turn moves the file, so the message cache missed, and the loop read and decoded the whole transcript. On a 396 MB transcript that was about 2.7 s per turn; the gateway logged 1 to 6 s loop stalls that line up with appends to its two large chats (#19094).

Why it matters

While the loop is frozen, the dashboard, Slack and the tunnel health probe wait; a freeze over 4 s fails the probe.

Not a goal

What changed (motivation → approach → change)

Both callers need only a row count, and the count came from a whole-file read and decode on the loop.

Step Where Cost on a 400 MB transcript (heartbeat max lag)
Streaming exact count in a worker thread: _count_messages(key, hold_writer_lock=False), one line at a time through transcript_count.count_messages (#19569), which shares message_row with the full read; no row list, no cache fill, no writer lock history_projection.py _count_messages, history_consolidation.py _count_rows 2.5-2.8 s wall, 9-54 ms max lag over 5 runs (the worst is one 2 MB row decoded in one call)
Full reads (_read_messages_locked, every caller): 1 MiB text reads, lines split across reads into the same pieces a whole-file splitlines() gives history_projection.py _iter_splitlines 2.0 s wall, 8-21 ms max lag (_read_messages in a thread, measured on this head)
Before this PR (whole-file read() + splitlines(), on the loop) 2.7 s loop freeze per turn

A key inside its retry deadline schedules no task: the turn and the sweep check it on the loop first (_inside_retry_deadline, one metadata read) and the thread checks again, so a backed-off chat starts no thread per turn after a restart resets _prefs_offset. The deadline survives a new span, so this never refuses what retry_eligible with the counted total would admit.

maybe_consolidate and the idle sweep both go through one counter, _count_off_loop, in a worker thread. One count runs per transcript file, claimed by file name like _busy, so alias spellings share it. The thread future is shielded, so the claim stays until the thread finishes. After the count, the busy check and retry_eligible (with the counted total) repeat, and the pass runs in the same task. A turn ending mid-count starts nothing; the next count, or the idle or session-end pass, covers its rows. The sweep schedules a count only for a key past its throttle and not busy. #18981's _idle_clean stamp is taken before the count and recorded only by a count that found nothing left; a deleted key is forgotten once no pass or count for it runs.

history.py gains consolidated_offset (now shared with consolidation_counts) and passes hold_writer_lock through _count_messages. The history spec (docs/system-specs/modules/history.md) describes the counter.

Why not a byte-scan bound that skips the decode: a second copy of the row rules that must stay in step, and the streaming count already meets the Goal. Why not a count kept on append: the slot save rewrites the file and other processes append. Why not the page index: it decodes every row.

Concurrency: _counting maps file name to a _CountClaim and is written only on the loop. Release is by identity, so a late release never drops a newer claim, and a task cancelled before its first step releases through its done callback.

Reader: history_consolidation.py:HistoryConsolidator.maybe_consolidate -- chat|subagent|cron -- reads _counting so a turn ending mid-count starts no second count (test_a_turn_that_ends_during_a_count_starts_no_second_one)
Reader: history_consolidation.py:HistoryConsolidator.check_idle_sessions -- heartbeat sweep -- reads _counting so the sweep never counts beside a turn's count (test_the_idle_sweep_counts_through_the_same_claim)

Backwards compatibility

Compatible: public signatures are unchanged; _read_messages returns the same rows. The threshold, offsets, throttle, backoff and refusal rules are unchanged. The pass starts after the count; a read failure is logged, not raised.

Removes nothing.

Tests

test/test_history_consolidation_threshold_off_loop.py:

  • Below the threshold the streaming count runs off the loop and parses nothing into a row list or the cache. A key in its retry deadline schedules no task (main's assert not c._tasks pin is kept), and a deadline armed after that check is caught in the thread. Mutations (count with the writer lock, drop the loop check, drop the thread check) red.
  • Every full read of a 12 MB transcript asks for at most one chunk. The heartbeat test, asserted by schedule, not stopwatch (D7); mutation (whole-file read()) red. The lag figures above come from a benchmark run outside the suite.
  • Chunked rows equal the whole-file parse, and the count equals them, on 11 fixtures plus 8 leading-line variants at chunk sizes 1, 2, 3, 5 and 4096; a file with no trailing newline; each multi-byte boundary and \r\n split across a chunk edge. Mutations (drop the edge fix, treat a cut piece as whole) red.
  • A cancelled count keeps its claim until the thread stops and no second count starts (mutation: no shield, red); two spellings of one file share one claim (mutation: key by session key, red); a cancel before the first step releases; a late release keeps a newer claim.
  • Idle sweep: no parse on the loop, the pass runs and sets the throttle, it shares the turn's claim, a key inside its retry deadline schedules nothing, a refused pass sets no throttle.
  • Threshold, other-writer, mid-count, busy, refusal, unreadable-file, raised-pass.

test_history_consolidation_bounds.py and test_history_consolidation_retry.py: two idle-sweep checks drain the count before asserting no pass ran. test_history_consolidation_retry.py's three inner-gate tests also bypass the deadline short-circuit and assert the inner gate is reached. test_history.py: the mock log's count returns the row count. test_history_consolidation_idle_seed.py: the sweep's read is the count; a failed count leaves no clean stamp; a key deleted mid-count is forgotten after it (mutation: forget under the count, red).

Run on the commit: 37 test files naming a touched module plus every test_*ratchet*.py, test_agent_refactor_facade.py and test_source_corpus.py (3618 passed, -n 8), test_boot_smoke.py (KIROCREW_INTEGRATION=1, 15 passed), black gate, isort, flake8, mypy src/ (0 errors), comment-history, encoding, brand and feature-map gates, docs_lint.py.

Manual verification

N/A: the tests drive a real transcript on disk.

Screenshots / video

N/A: no frontend change.

Related Issues

Closes #19094

Pattern harvest

Rule candidate: review-prompt. Pattern: moving a whole-file read to a worker thread does not free the loop when the read is one GIL-holding call; read in bounded chunks.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 11, 2026 13:03
@jeeshofone
jeeshofone requested a review from bolichen97 October 11, 2026 13:03
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 6713f634ed8318b157a749c24e68e9889b550975 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 6713f634ed8318b157a749c24e68e9889b550975 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check the 1 MiB chunked full read. It goes beyond the Goal and changes every _read_messages caller, not only the consolidator's counts.

What this change ships

Inventory (7 items) — 6 justified

Intent: stop the gateway freezing after each turn, and on each idle sweep, when a chat has a large transcript. This is a FIX (#19094, 2.7 s loop stall measured on a 396 MB transcript).

  1. The row count maybe_consolidate takes after each turn now runs in a worker thread, through the existing streaming transcript_count.count_messages. It no longer does a full read on the loop — justified
  2. The idle sweep's consolidation_counts read moves off the loop through the same counter — justified
  3. Only one count runs per transcript file. A turn that ends during a count starts no second one. A cancelled count keeps its claim until its thread finishes — justified
  4. A key inside its retry backoff schedules no count task. This takes the place of the old retry_eligible pre-check on the loop and keeps the assert not c._tasks pin — justified
  5. Every full transcript read (_read_messages_locked) now reads 1 MiB at a time instead of one whole-file read()/splitlines() — rides along: declared, with the GIL hold measured at 0.45–0.67 s
  6. _count_messages(hold_writer_lock=False): the off-loop count no longer blocks writers on the loop (1 caller passes False; the idle-seed scan keeps the default) — justified
  7. consolidated_offset is pulled out of consolidation_counts and has 2 consumers (consolidation_counts, _count_rows) — justified

[FIRST-PRINCIPLES-REVIEWED] 6713f63

Description read

[DESCRIPTION-READ] 088ffcde295499e7a5781c8e76ab87e78d89cb59ef364a0a6f5ab918343d75d2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 6713f634ed8318b157a749c24e68e9889b550975 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Each turn on a large chat still streams the whole transcript in a worker thread (about 2.5 s of CPU for 400 MB); check that cost is acceptable until #19096 bounds transcript size.

[DESIGN-REVIEWED] 6713f63

Description read

[DESCRIPTION-READ] 47bfe043142e935089b021bbf55a84ce8105a459d597e54f23acc8c1d41137d4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 6713f634ed8318b157a749c24e68e9889b550975 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6713f63

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 6713f634ed8318b157a749c24e68e9889b550975 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6713f63

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@jeeshofone
jeeshofone force-pushed the fix/19094-consolidate-count-off-loop branch from 7f6ee67 to e83bde3 Compare October 11, 2026 13:37
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

fixed in e83bde3: the recount set and its loop are removed.

Upheld. A turn that ends while a count is in flight now starts nothing. The next turn's count reads a fresh len(rows) against _prefs_offset, so its rows are included.
A quiet chat loses nothing either: the idle and session-end passes run _consolidate(include_history=True), whose prompt asks for preferences_update / projects_update over the same unconsolidated tail.
test_a_turn_that_ends_during_a_count_starts_no_second_one pins it: one task during the held count, and the following turn runs the pass at 31. Dropping the in-flight check turns it red.

First-pass check: R1-R8 OK. AUTOSDE rules matched and met: no-blocking-call-on-event-loop (the count is in asyncio.to_thread and takes no lock the loop waits on), tests-are-deterministic (held reads release on events, no sleeps or upper time bounds), an-invariant-line-needs-a-mutation-pin (in-flight check, busy re-check, retrieved raise, refusal offset, lock-free read and finally release each turn a test red when removed), a-refusal-is-not-a-commit (offset unchanged on refusal), no-sync-store-call-from-a-coroutine, no-test-side-effects (tmp_path logs only).

@jeeshofone

Copy link
Copy Markdown
Contributor Author

fixed in e83bde3: _threshold_pass takes the raise from _consolidate.

Upheld. _consolidate logs with logger.exception and releases _running in its own finally before re-raising, so the counting task now returns on Exception with the offset unchanged, as the old done-callback's fut.exception() did. Cancellation still propagates.
test_a_failed_pass_is_retrieved_and_leaves_the_offset_alone installs a loop exception handler and asserts the task finished with no exception and nothing was reported unretrieved. Removing the catch turns it red.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

rebutted (the ask is a production check after merge, not a change to this diff).

The JSON parse in a worker thread still holds the GIL while it decodes, but CPython switches threads every 5 ms (sys.getswitchinterval), so the loop keeps running between slices instead of waiting out the whole 2.7 s parse. The read also no longer holds the per-file writer lock, so an on-loop _locked caller does not block behind it.
The gateway's own event-loop heartbeat: lag warning is the measure. We will compare it on the large chat once a build with this change runs there and report the result on #19094.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

rebutted (already holds on this head, with a test).

A dropped call only skips a count; it records nothing. The next maybe_consolidate for the key counts the whole file again against _prefs_offset, and test_a_turn_that_ends_during_a_count_starts_no_second_one shows that count running the pass at 31.
With no next turn, check_idle_sessions and consolidate_session read consolidation_counts themselves and run _consolidate(include_history=True), whose prompt asks for preferences_update and projects_update over the unconsolidated tail, so the dropped turn's rows reach preference and project extraction there.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

rebutted (same ask as the previous head; answered there).

The decode still holds the GIL while it runs, but CPython hands it back every 5 ms switch interval, so the loop is no longer stopped for the whole parse, and the read no longer holds the writer lock an on-loop _locked would wait on.
The gateway's event-loop heartbeat: lag warning on the large chat is the check; we will report it on #19094 once a build with this change runs there.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Oct 11, 2026
@jeeshofone
jeeshofone force-pushed the fix/19094-consolidate-count-off-loop branch from e83bde3 to 7d7334a Compare October 11, 2026 16:24
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Delta for head 7d7334a (replaces e83bde3, same base 7bb9d57). Answers the independent review of e83bde3.

  • B1 (loop still lagged 228/502/669 ms): fixed. A chunked byte scan in the worker thread counts every boundary str.splitlines() honours, plus one (history_projection.py splitlines_piece_upper_bound). It never undercounts, so when the bound minus _prefs_offset is below 30 the count returns with no decode, which is the usual turn. The exact path is chunked too: _read_messages_locked reads 1 MiB at a time and splits lines across reads into the same pieces a whole-file splitlines() gives (_iter_splitlines). That keeps _read_messages semantics exactly, so there is no second row reader. Bench (bench_loop_lag.py, 5 ms heartbeat, warm cache, max lag): 133 MB 217 ms before, 1.6 ms scan, 8.4 ms exact; 301 MB 502 ms before, 1.0 ms scan, 10.1 ms exact; 400 MB 670 ms before, 1.4 ms scan, 11.2 ms exact. The heartbeat property is tested by schedule: every read of a 12 MB transcript asks for at most one chunk (a wall-clock upper bound would break tests-are-deterministic D7). Bound >= exact and chunked rows == whole-file rows on the reviewer's fixtures plus every-boundary, at chunk sizes 1, 2, 3, 5 and 4096, with \r\n and each multi-byte boundary split across a chunk edge.
  • S1 (idle sweep parsed on the loop): fixed. check_idle_sessions no longer calls consolidation_counts. A non-seeded key past its throttle and retry deadline goes through the same _count_off_loop with need=1 against last_consolidated, and the history pass runs in that task. The idle sweep's on-loop work is now one metadata read. The reviewer's sweep repro shows 0 on-loop parses and 1 off-loop. This overlaps fix(history): drop consolidated sessions from the idle sweep #18981 at that call site: its drop of fully consolidated keys keys off the on-loop unconsolidated < 1, which now lives in _idle_pass after the off-loop count. I kept this change minimal there and left a note on fix(history): drop consolidated sessions from the idle sweep #18981. Docstring at history_projection.py _read_messages corrected.
  • N1: stale "marks a recount" comment gone (the test was rewritten). The body says the next turn or the idle/session-end pass covers a mid-count turn.
  • N2: fixed. The thread future is shielded and the claim is released only when the thread is done, by identity. The reviewer's test_r2 (adjusted to start a count at the threshold) passes with peak 1. Mutation (no shield) turns test_a_cancelled_count_keeps_its_claim_until_the_thread_stops red.
  • N3: fixed. The claim is keyed by self._log._path(key).name, as _busy is. Mutation (key by session key) turns test_two_spellings_of_one_transcript_share_one_claim red.

Mutation pins (each turned red, then restored): the gate, either reader's whole-file read, the chunk-edge \r\n fix, the +1, the carried piece, the shield, the file-name key, release-by-identity, the pre-start release, the idle unconsolidated < 1 check and the idle retry-deadline pre-check.

Gates on 7d7334a: black gate, isort, flake8, mypy src/kiro_crew/ 0 errors, every runnable scripts/check_*.py (subprocess encoding, sync-io, comment, changelog and decision history against origin/main, harness parity from the merge-base), docs_lint, semgrep 1.78 with the repo rules, boot smoke 15/15, and 234 test files (touched-module importers, ratchets, facade, error-code contract) at -n 4.

First-pass check: R1-R8 OK.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 11, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 34d49aa (was 20dbbe1). Rebase onto current main (0a47021), no design change.

What conflicted. One hunk in HistoryConsolidator.__init__ in history_consolidation.py. Main's sub-chunking work (#17965) widened the on_abandoned comment and type to Callable[..., None]; this PR adds _counting directly above it. Resolved by keeping both: main's on_abandoned text and type unchanged, the _counting claim map ahead of it. Every other file merged cleanly. The diffstat is unchanged (8 files, +1185/-73).

One added change, test only. The wait-discipline ratchet (#19248) landed on main after the last reviewed head and flagged _until in test_history_consolidation_threshold_off_loop.py (a 0.01 s sleep in a loop, 0 -> 1). _until now calls kiro_crew.testing.wait.async_wait_until; the callers and what they assert are unchanged.

Windows (2) red on 20dbbe1. test_stt_engine.py::test_idle_eviction_frees_a_model_that_ran_a_sized_decode, tracked as #18835. This PR touches no STT file. #18835 was closed by #19151 (stt: hand the model over to the pool so unload frees it at once), which this rebase picks up.

Checks on the pushed head.

  • 1066 passed across the consolidation, idle-sweep, history and projection test files, plus 2723 passed across 219 neighbouring files and all test_*ratchet*.py files (-n 4). The only failure was the wait-discipline ratchet above; it passes after the fix.
  • test_boot_smoke.py with KIROCREW_INTEGRATION=1: 15 passed.
  • mypy on src/kiro_crew: 0 errors. isort, flake8, black gate, comment-history and subprocess-encoding gates are clean.
  • Loop lag, same benchmark as the PR body, warm cache, 5 ms heartbeat. Skip path: 1.2 ms (400 MB) and 2.5 ms (133 MB). Exact decode: 12.1 ms (400 MB) and 7.8 ms (133 MB). maybe_consolidate skip path 2.7 ms sync and check_idle_sessions 2.4 ms sync. The old head measured the same way on this host gave the same shape (decode 11.5 ms at 133 MB), so the rebase did not move it.

Not run locally: the Semgrep SAST lane (the tool is not installed on this host); the diff adds no autouse fixture.

No reviewer span is open on this head. First-pass check: R1-R8 OK.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@jeeshofone
jeeshofone force-pushed the fix/19094-consolidate-count-off-loop branch from 34d49aa to 1882e69 Compare October 13, 2026 00:39
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 1882e69 (was 34d49aa). Rebase onto current main (315affc), plus one fix the rebase needed.

What conflicted. #18981 (idle sweep drops consolidated sessions) merged into the same check_idle_sessions region. On main it stamps _idle_clean from an on-loop consolidation_counts. Here the sweep no longer counts on the loop, so the stamp moves into _idle_pass: taken on the loop before the off-loop count, recorded only by a count that found nothing left, dropped by a count that found rows, and left alone by a count that was skipped (retry deadline, failed read). The memory-skills-hooks.md paragraph and the idle-seed tests now say and pin that.

The fix. With the count off the loop there is a new await window: a sweep that found the transcript deleted forgot the key while its count was still running, and the count then re-added a clean stamp for a key no sweep visits again. A deleted key is now forgotten once no pass and no off-loop count for it runs, the same rule main already applies to a running pass. Test test_a_deleted_session_is_forgotten_once_its_count_has_ended; mutation (forget under the count) red.

Lanes on 34d49aa. GPT, Opus, Internal Content Scan and Security Scope were "could not run", and Design, First Principles and UX "could not run (advisory)": the 08:43 API rate limit, no findings. Dependency License Gate and Sensitive Change Review failed on API reads, not on the diff (no sensitive path is touched). This head re-runs them.

Checks on the pushed head.

  • 15,395 passed across 249 test files importing a touched module plus every test_*ratchet*.py and test_agent_refactor_facade.py (-n 8). One failure, test_auto_improvement_members.py::test_assignments_build_real_context_from_only_their_private_store, came from the local pytest basetemp sitting under a managed state directory; it passes with a basetemp outside it and is not in this diff.
  • test_boot_smoke.py with KIROCREW_INTEGRATION=1: 15 passed.
  • mypy src/kiro_crew: 0 errors. Black gate, isort, flake8, comment-history, subprocess-encoding, brand and feature-map gates from the merge-base, and docs_lint.py: clean. The description check passes for both callers.

Not run locally: Semgrep (not installed on this host); the diff adds no autouse fixture.

No reviewer span is open on this head. First-pass check: R1-R8 OK.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 13, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed, not approving yet: the branch now has a merge conflict with main. Please rebase; I found no defect and would approve the rebased head. My read: This PR moves the consolidator's per-turn and idle-sweep row counts off the event loop into one claimed worker-thread count per transcript, and it makes _read_messages_locked parse in 1 MiB chunks. That fixes the root cause of #19094, a whole-file read and decode on the loop after every turn. I read all of history.py, history_consolidation.py and history_projection.py and tried to make the byte bound undercount, which would silently skip a due pass. Text-mode universal newlines turn a lone \r into \n, and the scan counts \r; \r\n is subtracted in-chunk and across a chunk edge; every str.splitlines boundary (\x0b \x0c \x1c-\x1e U+0085 U+2028 U+2029) is counted by its last byte; a false hit on a continuation byte only raises the bound; and the trailing-piece rule compensates exactly. The leading-metadata subtraction is conservative: it is skipped on a BOM, a blank first line or undecodable bytes. _iter_splitlines yields the same pieces as a whole-file splitlines, because the TextIOWrapper holds back a trailing \r and a piece is carried until it ends on a boundary. The claim logic is identity-released, shielded until the thread stops, and released by a done callback for a task cancelled before its first step. The sweep forgets a deleted key only once no count runs. The tests pin each of these, plus the never-undercount property across chunk sizes. The First Principles CONCERNS on 1882e69 (drop the byte scan as oversized) has no disposition on this head yet. It is advisory and the scan is correct, but please answer it before merge. — crew-pr-reviewer (deep re-check)

@jeeshofone
jeeshofone force-pushed the fix/19094-consolidate-count-off-loop branch from 1882e69 to 1be8dd6 Compare October 13, 2026 03:53
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 13, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 1be8dd67c (rebased onto main de2d143dc, which clears the merge conflict). One commit; the body was edited before this push.

First Principles item 4 (byte-scan upper bound, oversized): taken as the subtraction. splitlines_piece_upper_bound, _boundary_count, _RARE_BOUNDARY_BYTES, _BOUNDARY_END_BYTES, _leads_with_metadata, _METADATA_PROBE_BYTES, both _message_count_upper_bound layers and _NOTHING_DUE are deleted; git grep finds none of them. _count_rows now returns the exact count from ConversationLog._count_messages(key, hold_writer_lock=False), the streaming count main gained in #19569. It shares message_row with _read_messages_locked, so no second copy of the row semantics remains in this diff, and it builds no row list and fills no cache. hold_writer_lock is the only change to that function.

Measured on this head (400 MB synthetic transcript with 2 MB file_changes rows, heartbeat every 5 ms, 3 runs): 2.6-2.8 s wall in the worker thread, 21-54 ms max loop lag. The worst case is one 2 MB row decoded in a single call. Before this PR, a turn froze the loop for about 2.7 s. The body's table now has these figures.

@iamwhatever: thanks for the deep re-check. This push rebases the branch and removes the byte scan you verified, so the only thing new since 1882e69 is the subtraction.

Tests on this head: 37 test files naming a touched module, 3306 passed / 7 skipped (-n 8); every test_*ratchet*.py, test_agent_refactor_facade.py, test_source_corpus.py, wait-discipline and test_boot_smoke.py, 311 passed. mypy src/ reports 0 issues. Black gate, isort, flake8, comment-history and subprocess-encoding gates pass. Mutations: counting with the writer lock turns test_the_count_does_not_hold_the_writer_lock_while_it_reads red, and dropping the retry-deadline check turns test_a_turn_inside_the_retry_deadline_counts_nothing red.

First-pass check: R1-R8 OK. R1.2: this reuses #19569's counter and deletes the parallel bound. R1.3: the change is a subtraction. R3.9: the count stays in a worker thread. R8.1: the body was edited before the push and is 7.8 KB. R8.13: rebased onto current main, and the tests ran on the rebased tree.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 13, 2026
maybe_consolidate runs on the gateway event loop after every completed
dashboard, Slack and task-runner turn, and check_idle_sessions on every
sweep. Both counted the transcript's rows with a synchronous
_read_messages, and the append that ends each turn moves the file, so the
message cache missed and the loop read and decoded the whole transcript.
On a transcript of a few hundred MB that froze the loop for seconds.

Both now go through one counter that runs in a worker thread: the
streaming _count_messages, which shares message_row with the full read,
materializes no row list, and here never takes the per-file writer lock.
_read_messages_locked reads 1 MiB at a time and splits lines across reads
into the same pieces a whole-file splitlines gives, so no single
GIL-holding call spans the file.

One count runs per transcript file, claimed by file name like _busy. The
thread future is shielded and the claim is kept until the thread
finishes, so a cancelled caller cannot let a second read start beside it.
The busy and retry checks repeat after the count and the pass runs in the
same task, so _prefs_offset, the idle throttle and the refusal handling
keep their semantics.

A key inside its retry deadline schedules no count: the turn and the
idle sweep check the deadline on the loop first, one metadata read, and
the worker thread checks again for a deadline armed in between.
@jeeshofone
jeeshofone force-pushed the fix/19094-consolidate-count-off-loop branch from 1be8dd6 to 6713f63 Compare October 13, 2026 04:33
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 6713f634e answers the First Principles item 7 raised on 1be8dd67c (a backed-off key starts a task and a thread on every turn, and the pin was rewritten). Taken as stated.

  • maybe_consolidate now checks the retry deadline on the loop before _claim_count, with one metadata read, and returns. A backed-off key schedules no task. The three deadline sites (the turn, the sweep and the worker thread) share one helper, HistoryConsolidator._inside_retry_deadline. It reads only the deadline, which survives a new span, so it never refuses a key that retry_eligible with the counted total would admit. The check in _count_rows stays as the backstop for a deadline armed between the loop check and the thread.
  • The original pin is back. test_maybe_consolidate_is_refused_inside_the_backoff again asserts not c._tasks with main's docstring, and test_a_turn_inside_the_retry_deadline_counts_nothing asserts the same. The new test_a_deadline_armed_after_the_turn_check_is_caught_in_the_thread pins the backstop.
  • This also exposed a vacuous premise on the old head. The three inner-gate tests in TestConsolidateIsTheEligibilityChokePoint lie through retry_eligible, but the in-thread deadline check refused before _consolidate ran, so the inner gate they exist to test was never reached. They now bypass _inside_retry_deadline too and assert retry_eligible was called twice, so the inner gate's refusal really runs.
  • Mutations: with the loop check removed, the two task-pin tests go red. With the thread check removed, the backstop test goes red. All six new or changed tests fail on 1be8dd67c's source.

Design PASS note (did the benchmark cover the chunked full read): measured on this head on a 400 MB synthetic transcript with 2 MB rows. _read_messages in a thread took 2.0 s wall with 8-21 ms max loop lag. The streaming count took 2.5-2.8 s with 9-54 ms. The body's table carries both.

Tests on this head: 37 files naming a touched module plus every ratchet, facade and source-corpus guard, 3618 passed. test_boot_smoke.py with KIROCREW_INTEGRATION=1, 15 passed. mypy src/ 0 issues. Black gate, isort, flake8, comment-history, encoding, brand, feature-map and docs_lint.py pass.

GPT 6.1, Opus 5.5 and Design had no blocking findings on 1be8dd67c.

First-pass check: R1-R8 OK. R1.3: one helper replaces three copies of the deadline expression. R3.9: the only on-loop read is the metadata line, as on main and in the sweep. R8.1: the body was edited before the push and is under 8000 B. R8.4: every lane on 1be8dd67c was read and is answered here.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 13, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 13, 2026 07:09

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (10 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the consolidation threshold count runs off the event loop; tests cover the off-loop path. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/history.md, docs/system-specs/modules/memory-skills-hooks.md.

@KiroCrewCaptain
KiroCrewCaptain merged commit 45e6a6b into kirodotdev:main Oct 13, 2026
107 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

maybe_consolidate re-reads the whole transcript on the event loop after every turn, freezing the gateway for seconds

3 participants