Repository navigation
perf(dashboard): cache the tool-policy read by agents-directory revision - #12722
Conversation
|
Intent: Make GET /api/session-tool-policy answer an unchanged agents directory from one scandir instead of re-reading and re-validating every spec (twice) per request, removing the second-largest GIL holder in the gateway profile while keeping every response, refusal and audit event identical. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the agents directory can never live on a network mount — a stale NFS attribute cache defeats the stat revision beyond the 2-second racy window. Suggestions
[DESIGN-REVIEWED] ce858d4 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. The single candidate against
I could not re-derive a concrete-in-practice input plus an observable wrong outcome at the required bar, and found no other grounded defect in the caching logic (the before/after revision guard, racy-window skip, symlink/Windows/overflow disables, and shared spec-generation invalidation are all consistent). [OPUS-REVIEWED] ce858d4 |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the 4,096-entry cap on a real large install: past it the fix silently switches off, and big directories are exactly who this PR is for. What this change shipsInventory (9 items) — 9 justifiedIntent: stop an unchanged agents directory from costing ~1 s of GIL-holding spec parsing on every session-tool-policy request — a FIX (measured py-spy profile, issue #12721, tests verified red on base).
The existing stat-signature cache ( [FIRST-PRINCIPLES-REVIEWED] ce858d4 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/dashboard/handlers/sessions.py:3610 -- replacing |
15528ca to
3fad276
Compare
|
|
|
|
|
|
Note on the deferral above: the task and |
3fad276 to
a127bcc
Compare
a127bcc to
2504a38
Compare
|
|
28f964f to
172e911
Compare
|
|
|
Re-pushed the identical tree as 2c1f072: on 172e911 the CI run's |
172e911 to
2c1f072
Compare
|
Pushed 1974dad: fixes the Windows shard failures on 2c1f072, which were this PR's own tests assuming the memo engages on a platform where it is deliberately off. The platform decision is a module constant ( |
2c1f072 to
1974dad
Compare
|
|
|
|
Pushed d768ffb (test-only): |
|
Pushed 73b00c0 (test-only): the fake |
|
GET /api/session-tool-policy resolved an agent's managedToolPolicy by parsing every spec in the agents directory to match its declared name, then, on a miss, parsing every spec again strictly to prove the miss honest. Each pass ran the hardened reader per file (symlink resolve, sensitive-path fence, pinned open, JSON parse). With ~2000 installed specs a request cost 750-965 ms on a worker thread, most of it holding the GIL, and the py-spy GIL profile attributed 6% of all GIL-holding samples to this chain; the stalls fed the loop-lag signal that pinned the adaptive subagent cap at 1 on an idle host. The read is now answered from a per-directory cache keyed on a stat-only revision: the directory's mtime plus (name, mtime, ctime, size) of every spec entry. An unchanged directory costs one scandir. Only resolved answers are cached; unreadable and ambiguous refusals are re-derived on every call so the fail-closed semantics and their per-request audit are unchanged.
|
|
|
Problem / Motivation
GET /api/session-tool-policycosts 750-965 ms per request on a host with ~2000 installed agent specs. Managed MCP servers call it at session start, so it runs on ordinary traffic. The read is offloaded to a worker, but it is pure Python holding the GIL, and a py-spy--gilprofile of the live gateway attributes 6% of all GIL-holding samples (17 of 275) to this one call chain.Why it matters
Those GIL stalls surface as event-loop lag. The adaptive concurrency controller reads the lag as pressure and pinned the subagent cap at 1 on an idle 128-core host, so every user with a large agents directory runs fewer subagents than the machine allows, and every other request on the loop waits behind the scan.
What changed (motivation → approach → change)
The handler resolves an agent's
managedToolPolicyby declared name, parsing every spec in the directory through the hardened reader, and when nothing declares the name it parses every spec a second time so that "no match" is honest. Nothing is remembered between requests, so an unchanged directory pays 2N hardened reads per call.The fix memoizes the resolved answer per agents directory and agent name in
src/kiro_crew/dashboard/handlers/sessions.py, keyed on a stat-only revision: the directory mtime,(name, mtime_ns, ctime_ns, size, inode, device, mode)of every spec-suffixed entry, and the parsed-spec generation thatagent_discovery.clear_list_agents_cache()already bumps, exposed asspec_cache_generation()insrc/kiro_crew/agent_discovery.py. Every in-process writer that clears the catalog caches therefore also invalidates this memo, with no writer changes and no callback registry.The memo steps aside wherever metadata cannot prove freshness, and the read then costs exactly what it costs on main: a symlinked spec entry (its metadata cannot see edits to the target; the stat itself is
follow_symlinks=False); Windows (st_ctime_nsis creation time there;_TOOL_POLICY_MEMO_ENABLED); any entry whose mtime or ctime is younger than two seconds (_TOOL_POLICY_RACY_WINDOW_NS, the rule git applies to racy index entries, so a same-tick rewrite is never served stale); more than_TOOL_POLICY_REVISION_MAX_ENTRIES(4,096) entries, with one warning per process. An answer is stored only when the revision is identical before and after the read, so a write landing mid-read is never memoized under the old revision. At most_TOOL_POLICY_CACHE_MAX_AGENTS(256) names are retained per directory.Only resolved answers are cached.
ManagedToolPolicyUnreadableandAmbiguousAgentSpecErrorare re-derived on every call, so the fail-closed refusal and its per-request audit are byte-for-byte what they were; the readers and the path resolver are untouched. Scope: this covers the profiled route,GET /api/session-tool-policy; the KAS session-start projection inacp/kas_agents.pystill parses per call and is tracked on #12721.Backwards compatibility
Compatible: the route's responses, status codes and audit events are unchanged for every input; the only difference is that an unchanged directory is not re-read.
Tests
test/test_session_tool_policy_cache.py(new), each verified RED on main and GREEN here:test_an_unchanged_directory_is_answered_without_reading_any_spec-- a second call opens no spec file.test_an_in_place_edit_is_seen_and_the_new_policy_is_served-- an edited spec is read again and the new policy returned.test_a_no_policy_answer_is_cached_until_a_spec_appears--Noneis cached; a spec added afterwards is found.test_a_refusal_is_re_derived_on_every_call-- an unreadable spec refuses and reads on every call.test_the_cache_is_per_agent_and_per_directory.test_the_revision_sees_content_and_permission_changes-- content/size on every platform, mode on POSIX.test_the_memo_is_disabled_when_the_platform_cannot_prove_freshness-- with the memo switched off the route still answers correctly and reads on every call.test_an_edit_during_the_read_is_not_memoized-- a spec rewritten while the uncached read is in flight leaves nothing in the cache; the next call serves the new policy.test_a_directory_past_the_entry_cap_is_not_memoized-- above the entry cap there is no revision, the route still answers, and exactly one warning is logged.test_a_fresh_spec_is_not_memoized_until_the_racy_window_expires-- a just-written spec yields no revision; once the clock passes the window it does.test_a_symlink_entry_is_seen_without_creating_one-- the platform-independent counterpart: a fake directory entry reportingis_symlink()yields no revision.test_the_catalog_invalidation_point_drops_cached_policy_answers_too-- with no filesystem change the revision is stable, andclear_list_agents_cache()alone changes it.test_the_revision_ignores_files_the_spec_scans_ignore.test_the_route_serves_a_repeat_request_without_spec_reads-- end to end through the handler.test/test_session_tool_policy_off_loop.py,test_agent_discovery.py,test_agent_capabilities.py,test_member_memory_api.py,test_mcp_call_site_auth_coverage.pyandtest_agent_spec_hardened_reads.pystill pass.Manual verification
N/A -- unit coverage sufficient: the regression is a count of reads at the real filesystem seam, which the tests measure directly.
Related Issues
Part of #12721 -- the issue stays open for the KAS session-start sibling (
acp/kas_agents.py) and the deferred follow-ups recorded there.Pattern harvest
Rule candidate: review-prompt
Pattern: a per-request handler that scans a whole directory of files to find one entry, with no revision-keyed memo, on a directory that is user-installed and can hold thousands of entries.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Part of #12721 -- the issue stays open for the KAS session-start sibling (
acp/kas_agents.py) and the deferred follow-ups recorded there.