Skip to content

perf(gateway): stop the session search indexer pinning the GIL - #12708

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
dimwael:perf/cjk-inventory-compiled-pattern
Sep 23, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
dimwael:perf/cjk-inventory-compiled-pattern

Conversation

@dimwael

@dimwael dimwael commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Under normal fleet load (~53 live chat sessions) the gateway's event loop stalls: repeated
event-loop heartbeat: lag 1.0-6.6s (loop was blocked), a 47 s session creation, and
resolver-budget timeouts on plain reads, with the process GIL-pinned at ~1.1 cores across 143
threads. A py-spy --gil profile puts 28% of GIL-holding samples in the session search indexer:
cjk_inventory / is_cjk_char. Restarting reproduces within 10 minutes -- steady-state load, not
accumulated state.

Why it matters

On the current base the indexer already runs in a spawned child (#12751, dashboard/server.py
_kick_session_search_index), so the gateway event loop no longer competes with it for the GIL.
What remains is the child's own cost: the pure-Python hot loop burns 28-46x more CPU than it needs,
and every live session is re-read in full on every turn (O(n^2) over its lifetime), so the child
never catches up on a large live transcript and holds that session's flock for longer than the
gateway's 10 s history write budget (observed: 40 Failed to save slot in one hour on one 98 MB
live session). This PR is the child-side CPU and index-catch-up fix.

What changed (motivation → approach → change)

Issue #12706 identifies three stacked causes. This PR takes the two mechanical ones:

(a) cjk_inventory walked every character in Python. It looped over the whole folded
transcript calling is_cjk_char, which builds a generator over six ranges per character.
cjk_inventory now dedupes the whole string first -- dict.fromkeys(folded), a C-level pass whose
result is bounded by the number of distinct characters, never by transcript length -- and then
classifies only those survivors through is_cjk_char (a few thousand calls at most). First-seen
order is preserved by dict.fromkeys, and filtering distinct characters in encounter order yields
exactly what the old walk yielded. Index-time only; the query path does not call it.

This deliberately does not use _CJK_PATTERN.findall, which the issue sketched: findall
materializes one list element per CJK occurrence, a transient allocation proportional to the
whole document on CJK-heavy transcripts (the fork AI review flagged this, correctly) -- and it also
benchmarks slower than the old walk on exactly that corpus shape. The dedupe-first form is bounded
and faster on every corpus shape measured.

Benchmarks (this branch, CPython 3.12.13, time.perf_counter, median of 5 runs, outputs verified
byte-identical before timing):

corpus (4.9M chars) old (per-char walk) new (dedupe first) speedup
mixed ASCII/CJK incl. astral Ext B (6.1 MB utf-8) 2.9083 s 0.1031 s 28x
ASCII-only (common case) 3.1526 s 0.0692 s 46x
CJK-heavy (worst case for dedup) 0.3184 s 0.2891 s 1.1x

(b) Live sessions were re-indexed on every turn. backfill_index re-indexed any session whose
stat changed; a session being written changes every turn, and each change re-reads, re-folds and
re-walks the whole transcript -- O(n^2) over the session's life, and with ~53 live sessions the pass
never caught up. backfill_index now defers any session whose file changed within
_INDEX_QUIET_WINDOW_SECS (120 s), reporting it in a new deferred count rather than in
remaining: the caller's pass cadence keys on remaining (2 s busy / 60 s idle), and a deferral
cannot be serviced by coming straight back -- only by waiting out the window -- so a fleet of
always-live sessions idles at the 60 s pace instead of pinning the 2 s one (this addresses the
Design Review's watch item). A deferred session is indexed by a later pass once quiet; until then
it is simply not vouched for, so search scans it directly -- the same superset guarantee un-indexed
sessions already have; freshness semantics (mtime/size/dev/ino) are unchanged. The deferral happens
while building pending, before any read, so it also composes with the docstring's acknowledged
budget-overrun concession: the very-large-because-live session is skipped before it can overrun the
pass budget.

Out of scope, deliberately: cause (c) -- moving the indexer out of the gateway process -- shipped
separately in #12751 and is already on the base this PR is measured against. Also untouched per the
issue: pulse-board logindex.py, the 6 GB RSS.

Backwards compatibility

Compatible: cjk_inventory output is byte-identical (asserted by test and verified on the 4.9M
benchmark corpus), so existing index rows and new rows agree; backfill_index keeps its return
shape and only shifts when a changing session is indexed, within the freshness rules callers
already handle (an un-vouched-for session is scanned, as today).

Tests

New test/test_session_index_hot_path.py (runs as a named file with -n0):

  • cjk_inventory equivalence against the old per-character walk, kept in the test as the oracle:
    empty, ASCII-only, CJK-only, mixed, duplicate characters (first-seen order), astral Extension B,
    astral non-CJK (emoji), every _CJK_RANGES edge +/- 1 codepoint, boundaries interleaved with
    ASCII, and a lone surrogate. Plus explicit first-seen-order and pure-ASCII-is-empty assertions.
  • Quiet-window behavior: a freshly written session is deferred (reported deferred, not remaining, not indexed -- deferred-only backfills take the idle pause);
    a quiet session is indexed; a deferred session is indexed once quiet; an indexed session that
    changes again goes back behind the window.

Existing suites: the two test files that append-then-backfill in the same breath
(test_session_search_index_equivalence.py, test_session_storage_search_index.py) gain an
autouse fixture zeroing the quiet window, with a comment saying why. All search-touching suites
pass locally: 1,273 passed with -n0 across test_session_index_hot_path.py,
test_session_storage_search_index.py, test_session_search_index_equivalence.py,
test_history_index.py, test_search_sessions_cost.py, test_history.py,
test_search_chat_history.py, test_session_search_redaction_seam.py,
test_history_composition_contract.py, test_history_cache_fill_race.py,
test_discord_sessions.py, test_teams_sessions.py, test_telegram_parity.py,
test_source_provider_plugin.py, test_stale_owner_session_signal.py,
test_open_slots_persistence.py, test_instances_search_federation.py, test_instances.py
(2 pre-existing environmental ssh failures in TestSshTunnelMultiplexing reproduce identically
on clean origin/main).

Manual verification

Benchmark above run manually on this branch (script builds the 4.9M-char corpus, asserts
byte-identical output, then times both implementations). The full fleet-load acceptance in the
issue (py-spy profile clean, no heartbeat-lag lines) is owned by #12751 together with this PR;
unit coverage plus the benchmark pin everything this PR claims.

Related Issues

Addresses causes (a) and (b) of #12706; cause (c) landed in #12751.

Pattern harvest

Rule candidate: review-prompt
Pattern: per-character Python classifier loop over a whole document where a compiled regex
projection of the same character ranges already exists in the codebase.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) -- N/A: no doc describes the backfill cadence or the inventory mechanism
  • No secrets, credentials, or internal references in the diff

@dimwael
dimwael requested review from a team and patrigao September 22, 2026 04:35
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 22, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 72e46e284c271eb12bdb5a666a7fe75685e44642 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Motivation claims gateway GIL relief the base already ships via the spawned index worker (server.py:3195); this PR's real win is child-side CPU and index catch-up.

Not justified as shipped

  • Intent gap — premise risk: "the indexer's pure-Python hot loop competes with the event loop for the GIL" and "cause (c) — moving the indexer out of the gateway process — … is left to a follow-up" are both contradicted by the base: dashboard/server.py:3190-3196 already runs indexing in a spawned child (citing the same py-spy 28% profile), and backfill_index has exactly one production caller, the child's loop (history_index_worker.py:230 — grep backfill_index, src). The code items stand on the harm this base still has (O(n²) re-reads, measured 28–46x CPU); the stated harm does not.
    Clears when: the title/description claims the child-side win, or the author names a path on this base that still runs backfill_index in the gateway process.

What this change ships

Inventory (5 items) — 4 justified

Intent: stop session-search indexing from degrading gateway service under live-session load — a FIX (perf).

  1. Building a transcript's CJK inventory costs 28–46x less CPU, output byte-identical — justified
  2. A session written within the last 120 s is skipped by the index pass until quiet; search scans it directly meanwhile — justified
  3. The backfill report gains a deferred count kept out of remaining, so deferred-only passes wait at the idle pace (zero src consumers of the count, but the contract is pre-recorded in _pause_for's docstring, history_index_worker.py:180-185) — justified
  4. Two search-index suites zero the quiet window via autouse fixture — justified
  5. Index-worker tests backdate transcripts past the window (_settle) — undeclared (test accommodation the description never mentions)

[FIRST-PRINCIPLES-REVIEWED] 72e46e2

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 72e46e284c271eb12bdb5a666a7fe75685e44642 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the fleet symptom persists on post-#12751 main — the child worker already ended gateway GIL pinning; this PR's remaining wins are lock contention and O(n²) I/O.

Suggestions

  • Retitle/refresh the description: "cause (c) … left to a follow-up" is stale — perf(gateway): move the session search indexer out of the gateway process #12751 already landed it on main, and perf(gateway): stop … pinning the GIL now names a mechanism the base tree no longer has; the accurate claim (indexer child stops re-reading live transcripts every 2 s pass and stops contending for _locked(key) with gateway writers) is stronger and true.

[DESIGN-REVIEWED] 72e46e2

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 72e46e284c271eb12bdb5a666a7fe75685e44642 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/history_index.py:156 -- "a few thousand calls at most" contradicts classification once per distinct character -> Fix: say “once per distinct character.”
[GPT-REVIEWED] 72e46e2

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 72e46e284c271eb12bdb5a666a7fe75685e44642 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 72e46e2

@dimwael
dimwael force-pushed the perf/cjk-inventory-compiled-pattern branch from 07bbad7 to 35b7785 Compare September 22, 2026 04:53
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@dimwael

dimwael commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the blocking finding in 35b7785. The review is right that findall materializes one list element per CJK occurrence — proportional to the whole document on CJK-heavy transcripts, where the old walk only ever held the bounded distinct set. Rather than streaming finditer (which benchmarks at 1.43 s on a 4.9M-char CJK-heavy corpus, 4.5x slower than the old walk it replaces), the new shape dedupes first: dict.fromkeys(folded) is a C-level pass bounded by distinct characters, and is_cjk_char then classifies only those survivors. Bounded memory on every input, and faster than both the old walk and every regex variant on every corpus shape measured (mixed 28x, ASCII 46x, CJK-heavy 1.1x — where findall/finditer were 3–4.5x slower than the old walk). Output byte-identical; equivalence suite unchanged and green. PR body benchmark table updated.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@dimwael
dimwael force-pushed the perf/cjk-inventory-compiled-pattern branch from 35b7785 to 4a4ed39 Compare September 22, 2026 05:35
@dimwael

dimwael commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

4a4ed39 addresses the Design Review watch item: deferred is now its own count in the backfill_index report instead of being folded into remaining, so _kick_session_search_index idles at the 60 s pause when the only remainder is deferred — a fleet of always-live sessions no longer pins the 2 s busy cadence. The deferred-only → no-busy-signal case is pinned by test_a_freshly_written_session_is_deferred_not_indexed (asserts deferred == 1 and remaining == 0).

Note for CI watchers: Backend shard 4 (Linux + Windows) and the downstream Coverage Gate are red on every merge ref that includes current main — that's #12713 (slot-reader enumeration test vs the _switch_target_busy refactor), unrelated to this diff and reproduced on pristine main. Will re-roll once the fix lands.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@dimwael
dimwael force-pushed the perf/cjk-inventory-compiled-pattern branch from 4a4ed39 to 7efeee3 Compare September 22, 2026 06:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@dimwael
dimwael force-pushed the perf/cjk-inventory-compiled-pattern branch from 7efeee3 to 10ba010 Compare September 22, 2026 07:10
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 23, 2026
…quiet window

The worker tests append a session and expect the spawned child to index it
in the same breath. The backfill pass defers a transcript modified inside the
quiet window as one still being written, and a monkeypatch does not reach a
spawned interpreter, so the tests move the file mtime back instead, which is
the one signal the deferral reads that crosses the process boundary.
@chenmingwei23
chenmingwei23 force-pushed the perf/cjk-inventory-compiled-pattern branch from b152c76 to 72e46e2 Compare September 23, 2026 07:21
@chenmingwei23

Copy link
Copy Markdown
Contributor

Shard 7 red on b152c76a4e was inherited from the base, not from this PR: test_crew_log_exc_info_sites.py was failing on the merge ref because #12757 added crew_log/session_tree_projection.py after this branch's base and the guard's VETTED set was only updated in #13007 (18f9984b04). None of the six files this PR touches are involved.

Re-pushed 72e46e284c with an identical tree so CI builds a fresh merge ref against current main. Locally on main 18f9984b04 + this diff, -n0: the four owned test files plus test_crew_log_exc_info_sites.py -> 86 passed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

Disposition for the First Principles CONCERNS on 72e46e284c: agreed on the premise. The description was written before #12751 merged; on the current base the indexer already runs in a spawned child (dashboard/server.py:3190-3196), so this PR no longer relieves the gateway event loop directly. What it does on this base: cuts the child's cjk_inventory CPU by 28-46x and stops the O(n^2) re-read of transcripts that are still being written, which is what lets the child catch up (the 98 MB live session that was starving history writes of its flock is the concrete case). No code change needed; @dimwael, if you edit the description to say "child-side CPU and index catch-up" rather than "GIL relief", the finding clears.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@bolichen97
bolichen97 merged commit 5aed824 into kirodotdev:main Sep 23, 2026
162 of 165 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 23, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…otdev#12708)

* perf(gateway): stop the session search indexer pinning the GIL

Two of the three stacked causes behind gateway event-loop stalls under
live-session load:

cjk_inventory walked every character of the folded transcript in Python,
calling is_cjk_char (a generator over six ranges) per character. Dedupe
the whole string first with dict.fromkeys -- a C-level pass bounded by
the number of distinct characters, never transcript length -- then
classify only the survivors. 4.9M-char corpora, median of 5: mixed
2.91s -> 0.103s (28x), ASCII-only 3.15s -> 0.069s (46x), CJK-heavy
0.32s -> 0.29s. Output byte-identical; memory stays bounded by distinct
codepoints (a findall projection would materialize one element per CJK
occurrence).

backfill_index re-indexed any session whose stat changed, so a session
being written was re-read and re-folded whole on every turn -- O(n^2)
over its life. Defer sessions modified within a 120s quiet window,
reported as a new deferred count rather than in remaining: the caller's
busy cadence keys on remaining, and a deferral cannot be serviced by
coming straight back, so a fleet of live sessions idles at the 60s pace
instead of pinning the 2s one. Deferred sessions are indexed by a later
pass once quiet; until then search scans them directly under the
existing un-vouched-for superset rule.

* test(history-index): settle the seeded transcripts past the backfill quiet window

The worker tests append a session and expect the spawned child to index it
in the same breath. The backfill pass defers a transcript modified inside the
quiet window as one still being written, and a monkeypatch does not reach a
spawned interpreter, so the tests move the file mtime back instead, which is
the one signal the deferral reads that crosses the process boundary.

---------

Co-authored-by: Wael Dimassi <ydimawae@amazon.com>
Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants