Repository navigation
perf(skills): memoise the skills catalog behind GET /api/skills - #7031
Conversation
bede830 to
ec57bc3
Compare
ec57bc3 to
20af770
Compare
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of The base confirms everything the diff relies on: Design-Verdict: PASS Burst-scoped single-flight with nothing retained: solves the measured redundant-scan defect without creating any invalidation obligation, and is trivially reversible. [DESIGN-REVIEWED] e1cbd43 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. The sole candidate requires a gap-free stream of overlapping [OPUS-REVIEWED] e1cbd43 |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of All evidence checked against the base: the removed hunk matches base First-Principles-Verdict: PASS A measured redundant-scan defect fixed by coalescing that stores nothing, with the invalidation-owing cache variant already subtracted out during its own review history. What this change shipsIntent: stop N simultaneous skill-menu opens from each paying their own filesystem scan — a FIX.
The mechanism reuses the established #4800 shape (per-key [FIRST-PRINCIPLES-REVIEWED] e1cbd43 |
20af770 to
38f80a5
Compare
96c0697 to
86e0833
Compare
86e0833 to
cb44619
Compare
cb44619 to
89d0e0b
Compare
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
|
Both audit items verified at head Item 1 (#7062) — satisfied, nothing owed here. The visible-hang half is closed on main and this PR does not touch it. At head, Item 2 (#7067) — the re-measure was owed, and the numbers were stale in code. You were right that the root-cause work landing changes the cost-benefit. Re-measured
The original 60.1% share and the 41/69/168s contended samples were withdrawn rather than restated, because they came from a live capture under production session churn that cannot be reproduced here, and those profile rows are inclusive frame shares that do not sum to 100 — so arithmetic on them is not valid. The description already marked them withdrawn; the code comment above the assembly still asserted them as fact, which made the two surfaces contradict each other. As of this sha that comment no longer carries them, and the per-path staleness rationale it held now lives in The residual claim is deliberately narrower than the original: the memo dedupes concurrent assemblies of a ~0.6s idle scan and is a bounded symptom mitigation, not a cure. Whether that still justifies a 3s staleness window with redaction ~2.2× cheaper is a judgement for a human reviewer, and it is being weighed rather than assumed. REBASE, recommended on both items — done. This head's parent is main tip |
Key on ($HOME, project dir) gated on loader identity: a project-only key collapsed every slotless caller onto one entry and served a neighbour's catalog. Coalesce concurrent readers on LoopBoundLock plus a re-check under it, as api_system does, and invalidate at the chokepoints a caller awaits a read-after-write on (22 sites to 15, behind one get_or_assemble entry point).
bolichen97
left a comment
There was a problem hiding this comment.
Tech Lead review — approved.
Correctness of the invalidation question (the one that matters here): there is no invalidation obligation, because nothing is retained. The handoff, the waiter count, and the per-key LoopBoundLock are all popped in the same finally when remaining == 0, so a read that is not concurrent with another always re-scans on-disk state. Verified the base's recorded "no result cache" default survives: test_a_later_sequential_read_reassembles asserts n == 2 on two sequential reads plus empty bookkeeping, and test_a_key_lock_does_not_outlive_its_burst bounds the registry. No mutator anywhere owes this code a call, which is the property the earlier stored-TTL revisions could not offer.
Key is real, not decorative. _get_skills returns a cached instance on both branches (context_builder.skills, else state._standalone_skills), so (loader, project) is stable across requests and the join actually fires in production rather than only in tests. Per-key rather than global is right: assembly is not reliably sub-second, and test_different_keys_assemble_in_parallel pins that two projects still scan concurrently.
No shared-mutable hazard. The rows are handed out un-copied, and the only downstream consumer is the ?agent= comprehension in api_skills, which builds a new list and never mutates an entry in place before json_response. The two-order parametrized end-to-end test pins that a filtered result can never be what gets shared.
Locking discipline. _catalog_lock (a threading.Lock) is only ever held across synchronous dict operations and never across an await; the increment precedes lock acquisition and the decrement follows release, so a key's lock cannot be evicted while a waiter still references it. Cancelling a leader publishes nothing, which is the stated reason the epoch stamp was correctly deleted.
Residual, accepted as non-blocking: under a hypothetically gapless overlapping request stream the waiter count never returns to 0 and _catalog_handoff[key] would persist unrefreshed. Fast-path readers return in microseconds on a browser-triggered endpoint, so the drain empties almost immediately; I reach the same conclusion the Opus lane did.
Scope and hygiene: 3 files, one commit, spec updated in the same commit, no new module and no new dependency, no auth/trust/sandbox/secret surface touched. All 4 AI lanes clean on this exact sha; SAST, Loop-Bound Locks Gate, PR Scope, PR Hygiene, Testpaths Coverage all pass. Both relationship-audit REBASE items were verified and discharged by the author.
One non-blocking nit for a follow-up, not worth another round: the test_skills_catalog_cache.py module docstring still says "readers of different projects serialize on the one global assembly lock," which is a leftover from the pre-per-key revision and contradicts both the implementation and test_different_keys_assemble_in_parallel directly below it.
What this buys, stated first
Concurrent readers of
GET /api/skillsstop each running their own filesystem scan.At 8-way concurrency the redundant-scan rate goes 0.0% → 87.5% (measured, table below):
eight simultaneous skill-menu opens cost one assembly instead of eight.
Nothing is stored and there is no expiry. A read that is not part of a concurrent burst
always scans current on-disk state, so the base's recorded default — "No result cache: the
endpoint always reflects current on-disk state, so freshly created/installed skills appear
immediately" — is preserved, not reversed. No mutation path anywhere in the codebase owes
this change a call.
What it costs, stated with it: a reader that arrives while an assembly is already running
shares that assembly's rows, so it can be served data read up to one scan earlier. That is the
irreducible cost of coalescing at all, and it is bounded by one assembly for that key — the
assembly lock is per key, so no reader ever waits behind an unrelated project's scan. Outside
such a burst nothing is shared.
This revision is substantially smaller than earlier ones, by subtraction. Earlier
revisions also shipped a 3s stored entry. Review established that the store contributed none
of the measured win while obliging every present and future catalog mutator to invalidate it —
19
invalidate_skills_catalog()calls across 9 modules, a mutation-driven generation counter,an exemption kwarg for background sweeps, and a silent-staleness defect class for any path
that forgot one (one such defect was found in review, in the conductor-skill toggle). The
store bought a repeat read within 3s, which for a project-scoped chat the frontend's 5-minute
skillsCacheStaleTimealready covered. The store and the entire obligation were removedtogether. The PR now touches 3 files where it previously touched 14, and adds no new module.
Problem / Motivation
GET /api/skillscan take tens of seconds on a busy gateway. The assembly work itself isnot the problem — measured standalone on the affected host:
~/.aimtree censused/api/skillsend-to-end when the gateway is idleThe same endpoint on the same host, returning the same payload, was also observed at 41s, 69s
and 168s within twenty minutes. Those contended figures are WITHDRAWN as motivation. The
regime that produced them was largely event-loop contention from history flush, which
#7056/#7067 substantially fixed; the real fix landed elsewhere. What remains, and what this
PR addresses, is the redundant-scan half that neither of those PRs touches: under concurrency
every reader paid for its own scan.
Why it matters
Opening the
$skill-picker menu in several chats at once, or one chat re-opening it while aprevious request is still assembling, multiplied a ~0.6s filesystem scan by the number of
readers. That is the measured defect: 0% redundant-scan elimination at 8-way concurrency
before this change.
The endpoint is a victim rather than the cause of the original hang, and this PR is scoped to
the redundant-scan half only. It does not claim to fix the contention.
What changed
The coalescing protocol lives in
_assemble_skills_cataloginsrc/kiro_crew/dashboard/handlers/prompts.py, alongside its only consumer:LoopBoundLockvalues in a registry, the shape Slack auto-title exception masking + ~40 sibling module-global asyncio.Lock instances lack loop tracking (split from #4789) #4800established for exactly this): a fast path off the lock, then the lock, then a re-check under
it. The re-check is the join — readers queued behind the leader take its rows instead of
each scanning. Per key rather than global, so readers of DIFFERENT projects still scan in
parallel as the base did; assembly is not reliably sub-second (it can run seconds long on
large skills×agents catalogs), so a shared lock would have given a multi-project burst worse
tail latency than the base. The registry entry is created and dropped under the same lock as
the waiter count that bounds its life, so it cannot outlive the burst.
same key is still inside the call, and are dropped when the last one leaves. There is no
expiry to get wrong, and nothing survives the burst.
(loader, project dir), so two readers holdingdifferent loaders get different keys and cannot collide on one entry.
$HOMEis not in thekey — it takes one value per process, so it distinguished nothing.
?agent=is deliberately not part of the key: it is applied downstream as a comprehensionover the assembled rows.
Two mechanisms present in earlier revisions are gone, and both removals were review findings.
An epoch stamp guarded against rows from a superseded assembly being served; the handoff is
published only after an await returns and dies with its burst, so a cancelled leader publishes
nothing and that state was unreachable (its own test had to forge it), while the epoch being
global meant a neighbouring project's
assembly invalidated a live handoff and forced a redundant scan. And the protocol briefly lived
in its own leaf module, justified by mutators outside the dashboard needing module-scope
access to the invalidator — with no invalidator there are no such callers, so it folded into
its single consumer and the injected-assembly indirection went with it.
Scope — this is a bounded mitigation of the redundant-scan half
The contended samples are withdrawn and the root cause is out of scope. What is claimed is
the measured concurrency result below and nothing more.
Deliberate limitations, documented in the code
assembleis awaited inline, socancelling the leader cancels the assembly and the next waiter starts over. That costs a
repeated ~0.6s scan in the disconnect case and buys the removal of an in-flight task map,
with its shields, done-callbacks and cancellation bookkeeping.
Measured hit rate
Measured against a counting assembler, so "shared one assembly" is asserted by the assembler
not running rather than by a latency measurement.
Tests
test/test_skills_catalog_cache.py, 7 tests:object
the waiter bookkeeping are empty afterwards
keys and are served distinct objects
global lock cannot be reintroduced without failing here
reader leaves, so it cannot grow per project visited
orders, so a join that ever shared the FILTERED result fails whichever asked second
Follow-ups this does not cover
partly addressed by perf(security): gate the credential scan behind a cheap superset pre-filter #7056/perf(security): cut the per-run decode and scan work in redaction passes 2 and 3 #7067.