Repository navigation
perf(gateway): stop the session search indexer pinning the GIL - #12708
Conversation
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Motivation claims gateway GIL relief the base already ships via the spawned index worker ( Not justified as shipped
What this change shipsInventory (5 items) — 4 justifiedIntent: stop session-search indexing from degrading gateway service under live-session load — a FIX (perf).
[FIRST-PRINCIPLES-REVIEWED] 72e46e2 |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the fleet symptom persists on post-#12751 main — the child worker already ended gateway GIL pinning; this PR's remaining wins are lock contention and O(n²) I/O. Suggestions
[DESIGN-REVIEWED] 72e46e2 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/history_index.py:156 -- |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed |
07bbad7 to
35b7785
Compare
|
Addressed the blocking finding in 35b7785. The review is right that |
35b7785 to
4a4ed39
Compare
|
4a4ed39 addresses the Design Review watch item: Note for CI watchers: Backend shard 4 (Linux + Windows) and the downstream Coverage Gate are red on every merge ref that includes current main — that's #12713 (slot-reader enumeration test vs the |
4a4ed39 to
7efeee3
Compare
7efeee3 to
10ba010
Compare
…quiet window The worker tests append a session and expect the spawned child to index it in the same breath. The backfill pass defers a transcript modified inside the quiet window as one still being written, and a monkeypatch does not reach a spawned interpreter, so the tests move the file mtime back instead, which is the one signal the deferral reads that crosses the process boundary.
b152c76 to
72e46e2
Compare
|
Shard 7 red on Re-pushed |
|
Disposition for the First Principles CONCERNS on |
…otdev#12708) * perf(gateway): stop the session search indexer pinning the GIL Two of the three stacked causes behind gateway event-loop stalls under live-session load: cjk_inventory walked every character of the folded transcript in Python, calling is_cjk_char (a generator over six ranges) per character. Dedupe the whole string first with dict.fromkeys -- a C-level pass bounded by the number of distinct characters, never transcript length -- then classify only the survivors. 4.9M-char corpora, median of 5: mixed 2.91s -> 0.103s (28x), ASCII-only 3.15s -> 0.069s (46x), CJK-heavy 0.32s -> 0.29s. Output byte-identical; memory stays bounded by distinct codepoints (a findall projection would materialize one element per CJK occurrence). backfill_index re-indexed any session whose stat changed, so a session being written was re-read and re-folded whole on every turn -- O(n^2) over its life. Defer sessions modified within a 120s quiet window, reported as a new deferred count rather than in remaining: the caller's busy cadence keys on remaining, and a deferral cannot be serviced by coming straight back, so a fleet of live sessions idles at the 60s pace instead of pinning the 2s one. Deferred sessions are indexed by a later pass once quiet; until then search scans them directly under the existing un-vouched-for superset rule. * test(history-index): settle the seeded transcripts past the backfill quiet window The worker tests append a session and expect the spawned child to index it in the same breath. The backfill pass defers a transcript modified inside the quiet window as one still being written, and a monkeypatch does not reach a spawned interpreter, so the tests move the file mtime back instead, which is the one signal the deferral reads that crosses the process boundary. --------- Co-authored-by: Wael Dimassi <ydimawae@amazon.com> Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com>
Problem / Motivation
Under normal fleet load (~53 live chat sessions) the gateway's event loop stalls: repeated
event-loop heartbeat: lag 1.0-6.6s (loop was blocked), a 47 s session creation, andresolver-budget timeouts on plain reads, with the process GIL-pinned at ~1.1 cores across 143
threads. A py-spy
--gilprofile puts 28% of GIL-holding samples in the session search indexer:cjk_inventory/is_cjk_char. Restarting reproduces within 10 minutes -- steady-state load, notaccumulated state.
Why it matters
On the current base the indexer already runs in a spawned child (#12751,
dashboard/server.py_kick_session_search_index), so the gateway event loop no longer competes with it for the GIL.What remains is the child's own cost: the pure-Python hot loop burns 28-46x more CPU than it needs,
and every live session is re-read in full on every turn (O(n^2) over its lifetime), so the child
never catches up on a large live transcript and holds that session's flock for longer than the
gateway's 10 s
historywrite budget (observed: 40Failed to save slotin one hour on one 98 MBlive session). This PR is the child-side CPU and index-catch-up fix.
What changed (motivation → approach → change)
Issue #12706 identifies three stacked causes. This PR takes the two mechanical ones:
(a)
cjk_inventorywalked every character in Python. It looped over the whole foldedtranscript calling
is_cjk_char, which builds a generator over six ranges per character.cjk_inventorynow dedupes the whole string first --dict.fromkeys(folded), a C-level pass whoseresult is bounded by the number of distinct characters, never by transcript length -- and then
classifies only those survivors through
is_cjk_char(a few thousand calls at most). First-seenorder is preserved by
dict.fromkeys, and filtering distinct characters in encounter order yieldsexactly what the old walk yielded. Index-time only; the query path does not call it.
This deliberately does not use
_CJK_PATTERN.findall, which the issue sketched:findallmaterializes one list element per CJK occurrence, a transient allocation proportional to the
whole document on CJK-heavy transcripts (the fork AI review flagged this, correctly) -- and it also
benchmarks slower than the old walk on exactly that corpus shape. The dedupe-first form is bounded
and faster on every corpus shape measured.
Benchmarks (this branch, CPython 3.12.13,
time.perf_counter, median of 5 runs, outputs verifiedbyte-identical before timing):
(b) Live sessions were re-indexed on every turn.
backfill_indexre-indexed any session whosestat changed; a session being written changes every turn, and each change re-reads, re-folds and
re-walks the whole transcript -- O(n^2) over the session's life, and with ~53 live sessions the pass
never caught up.
backfill_indexnow defers any session whose file changed within_INDEX_QUIET_WINDOW_SECS(120 s), reporting it in a newdeferredcount rather than inremaining: the caller's pass cadence keys onremaining(2 s busy / 60 s idle), and a deferralcannot be serviced by coming straight back -- only by waiting out the window -- so a fleet of
always-live sessions idles at the 60 s pace instead of pinning the 2 s one (this addresses the
Design Review's watch item). A deferred session is indexed by a later pass once quiet; until then
it is simply not vouched for, so search scans it directly -- the same superset guarantee un-indexed
sessions already have; freshness semantics (mtime/size/dev/ino) are unchanged. The deferral happens
while building
pending, before any read, so it also composes with the docstring's acknowledgedbudget-overrun concession: the very-large-because-live session is skipped before it can overrun the
pass budget.
Out of scope, deliberately: cause (c) -- moving the indexer out of the gateway process -- shipped
separately in #12751 and is already on the base this PR is measured against. Also untouched per the
issue: pulse-board
logindex.py, the 6 GB RSS.Backwards compatibility
Compatible:
cjk_inventoryoutput is byte-identical (asserted by test and verified on the 4.9Mbenchmark corpus), so existing index rows and new rows agree;
backfill_indexkeeps its returnshape and only shifts when a changing session is indexed, within the freshness rules callers
already handle (an un-vouched-for session is scanned, as today).
Tests
New
test/test_session_index_hot_path.py(runs as a named file with-n0):cjk_inventoryequivalence against the old per-character walk, kept in the test as the oracle:empty, ASCII-only, CJK-only, mixed, duplicate characters (first-seen order), astral Extension B,
astral non-CJK (emoji), every
_CJK_RANGESedge +/- 1 codepoint, boundaries interleaved withASCII, and a lone surrogate. Plus explicit first-seen-order and pure-ASCII-is-empty assertions.
deferred, notremaining, not indexed -- deferred-only backfills take the idle pause);a quiet session is indexed; a deferred session is indexed once quiet; an indexed session that
changes again goes back behind the window.
Existing suites: the two test files that append-then-backfill in the same breath
(
test_session_search_index_equivalence.py,test_session_storage_search_index.py) gain anautouse fixture zeroing the quiet window, with a comment saying why. All search-touching suites
pass locally: 1,273 passed with
-n0acrosstest_session_index_hot_path.py,test_session_storage_search_index.py,test_session_search_index_equivalence.py,test_history_index.py,test_search_sessions_cost.py,test_history.py,test_search_chat_history.py,test_session_search_redaction_seam.py,test_history_composition_contract.py,test_history_cache_fill_race.py,test_discord_sessions.py,test_teams_sessions.py,test_telegram_parity.py,test_source_provider_plugin.py,test_stale_owner_session_signal.py,test_open_slots_persistence.py,test_instances_search_federation.py,test_instances.py(2 pre-existing environmental ssh failures in
TestSshTunnelMultiplexingreproduce identicallyon clean
origin/main).Manual verification
Benchmark above run manually on this branch (script builds the 4.9M-char corpus, asserts
byte-identical output, then times both implementations). The full fleet-load acceptance in the
issue (py-spy profile clean, no heartbeat-lag lines) is owned by #12751 together with this PR;
unit coverage plus the benchmark pin everything this PR claims.
Related Issues
Addresses causes (a) and (b) of #12706; cause (c) landed in #12751.
Pattern harvest
Rule candidate: review-prompt
Pattern: per-character Python classifier loop over a whole document where a compiled regex
projection of the same character ranges already exists in the codebase.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)