Skip to content

perf(skills): memoise the skills catalog behind GET /api/skills - #7031

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
rnoack1:fix/skills-catalog-server-cache
Sep 6, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
rnoack1:fix/skills-catalog-server-cache

Conversation

@rnoack1

@rnoack1 rnoack1 commented Aug 30, 2026 •

Copy link
Copy Markdown
Contributor

What this buys, stated first

Concurrent readers of GET /api/skills stop each running their own filesystem scan.
At 8-way concurrency the redundant-scan rate goes 0.0% → 87.5% (measured, table below):
eight simultaneous skill-menu opens cost one assembly instead of eight.

Nothing is stored and there is no expiry. A read that is not part of a concurrent burst
always scans current on-disk state, so the base's recorded default — "No result cache: the
endpoint always reflects current on-disk state, so freshly created/installed skills appear
immediately" — is preserved, not reversed. No mutation path anywhere in the codebase owes
this change a call.

What it costs, stated with it: a reader that arrives while an assembly is already running
shares that assembly's rows, so it can be served data read up to one scan earlier. That is the
irreducible cost of coalescing at all, and it is bounded by one assembly for that key — the
assembly lock is per key, so no reader ever waits behind an unrelated project's scan. Outside
such a burst nothing is shared.

This revision is substantially smaller than earlier ones, by subtraction. Earlier
revisions also shipped a 3s stored entry. Review established that the store contributed none
of the measured win while obliging every present and future catalog mutator to invalidate it —
19 invalidate_skills_catalog() calls across 9 modules, a mutation-driven generation counter,
an exemption kwarg for background sweeps, and a silent-staleness defect class for any path
that forgot one (one such defect was found in review, in the conductor-skill toggle). The
store bought a repeat read within 3s, which for a project-scoped chat the frontend's 5-minute
skillsCacheStaleTime already covered. The store and the entire obligation were removed
together. The PR now touches 3 files where it previously touched 14, and adds no new module.

Problem / Motivation

GET /api/skills can take tens of seconds on a busy gateway. The assembly work itself is
not the problem — measured standalone on the affected host:

measurement result
capability-package skill rows (AIM roots) 0.33s, 3 consecutive runs, 275 rows
whole ~/.aim tree censused 13,607 dirs / 25,475 files in 1.70s
/api/skills end-to-end when the gateway is idle 0.55–0.62s

The same endpoint on the same host, returning the same payload, was also observed at 41s, 69s
and 168s within twenty minutes. Those contended figures are WITHDRAWN as motivation. The
regime that produced them was largely event-loop contention from history flush, which
#7056/#7067 substantially fixed; the real fix landed elsewhere. What remains, and what this
PR addresses, is the redundant-scan half that neither of those PRs touches: under concurrency
every reader paid for its own scan.

Why it matters

Opening the $ skill-picker menu in several chats at once, or one chat re-opening it while a
previous request is still assembling, multiplied a ~0.6s filesystem scan by the number of
readers. That is the measured defect: 0% redundant-scan elimination at 8-way concurrency
before this change.

The endpoint is a victim rather than the cause of the original hang, and this PR is scoped to
the redundant-scan half only. It does not claim to fix the contention.

What changed

The coalescing protocol lives in _assemble_skills_catalog in
src/kiro_crew/dashboard/handlers/prompts.py, alongside its only consumer:

  • One assembly lock per key (LoopBoundLock values in a registry, the shape Slack auto-title exception masking + ~40 sibling module-global asyncio.Lock instances lack loop tracking (split from #4789) #4800
    established for exactly this): a fast path off the lock, then the lock, then a re-check under
    it. The re-check is the join — readers queued behind the leader take its rows instead of
    each scanning. Per key rather than global, so readers of DIFFERENT projects still scan in
    parallel as the base did; assembly is not reliably sub-second (it can run seconds long on
    large skills×agents catalogs), so a shared lock would have given a multi-project burst worse
    tail latency than the base. The registry entry is created and dropped under the same lock as
    the waiter count that bounds its life, so it cannot outlive the burst.
  • A waiter-scoped handoff: the leader's rows are offered only while another reader for the
    same key is still inside the call, and are dropped when the last one leaves. There is no
    expiry to get wrong, and nothing survives the burst.
  • The loader is part of the key: the key is (loader, project dir), so two readers holding
    different loaders get different keys and cannot collide on one entry. $HOME is not in the
    key — it takes one value per process, so it distinguished nothing.

?agent= is deliberately not part of the key: it is applied downstream as a comprehension
over the assembled rows.

Two mechanisms present in earlier revisions are gone, and both removals were review findings.
An epoch stamp guarded against rows from a superseded assembly being served; the handoff is
published only after an await returns and dies with its burst, so a cancelled leader publishes
nothing and that state was unreachable (its own test had to forge it), while the epoch being
global meant a neighbouring project's
assembly invalidated a live handoff and forced a redundant scan. And the protocol briefly lived
in its own leaf module, justified by mutators outside the dashboard needing module-scope
access to the invalidator — with no invalidator there are no such callers, so it folded into
its single consumer and the injected-assembly indirection went with it.

Scope — this is a bounded mitigation of the redundant-scan half

The contended samples are withdrawn and the root cause is out of scope. What is claimed is
the measured concurrency result below and nothing more.

Deliberate limitations, documented in the code

  • No work preservation when the leader disconnects. assemble is awaited inline, so
    cancelling the leader cancels the assembly and the next waiter starts over. That costs a
    repeated ~0.6s scan in the disconnect case and buys the removal of an in-flight task map,
    with its shields, done-callbacks and cancellation bookkeeping.
  • Mid-burst sharing is the one staleness admitted, bounded by a single assembly for that key.

Measured hit rate

pattern redundant scans eliminated
8 simultaneous reads, before 0.0% (8 reads → 8 assemblies)
8 simultaneous reads, after 87.5% (8 reads → 1 assembly)

Measured against a counting assembler, so "shared one assembly" is asserted by the assembler
not running rather than by a latency measurement.

Tests

test/test_skills_catalog_cache.py, 7 tests:

  • the join — 8 concurrent readers produce exactly 1 assembly and all receive the same
    object
  • nothing is retained — a later, non-concurrent read reassembles, and both the handoff and
    the waiter bookkeeping are empty afterwards
  • a different loader does not join — the loader is part of the key, so two loaders are two
    keys and are served distinct objects
  • different keys assemble in parallel — two projects DO assemble concurrently, so a single
    global lock cannot be reintroduced without failing here
  • a key lock does not outlive its burst — the per-key registry is empty once the last
    reader leaves, so it cannot grow per project visited
  • cross-agent isolation end-to-end — two agents driven through the real endpoint in both
    orders, so a join that ever shared the FILTERED result fails whichever asked second

Follow-ups this does not cover

@rnoack1
rnoack1 requested review from a team and pepmach August 30, 2026 16:11
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 30, 2026
@rnoack1
rnoack1 force-pushed the fix/skills-catalog-server-cache branch from bede830 to ec57bc3 Compare August 30, 2026 17:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 30, 2026
@rnoack1
rnoack1 force-pushed the fix/skills-catalog-server-cache branch from ec57bc3 to 20af770 Compare August 30, 2026 17:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@github-actions

github-actions Bot commented Aug 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed e1cbd43b93d592aebf036d0513011b0c41fdf5b8 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e1cbd43

@github-actions

github-actions Bot commented Aug 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of e1cbd43b93d592aebf036d0513011b0c41fdf5b8 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

The base confirms everything the diff relies on: threading/asyncio are already imported in prompts.py, LoopBoundLock is the established module-global lock shape, and the replaced hunk is exactly the base's inline assembly. The design has been pre-litigated in the description — the stored-TTL cache, the invalidation obligation, the epoch stamp, and the separate module were all removed, leaving a burst-scoped single-flight with no retention, which preserves the base's recorded "no result cache" freshness default and creates no mutation-side obligations. The staleness admitted is bounded to one assembly per key and documented in the docstring plus the spec (updated in the same commit). Per-key locks keep cross-project parallelism, pinned by a test; the change is fully reversible (delete one function, restore the inline block). The strongest alternative (per-key shared asyncio.Task) is explicitly weighed against leader-cancel restart cost. No unaccounted hunks; description matches the diff bidirectionally.

Design-Verdict: PASS

Burst-scoped single-flight with nothing retained: solves the measured redundant-scan defect without creating any invalidation obligation, and is trivially reversible.

[DESIGN-REVIEWED] e1cbd43

@github-actions

github-actions Bot commented Aug 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed e1cbd43b93d592aebf036d0513011b0c41fdf5b8 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

The sole candidate requires a gap-free stream of overlapping /api/skills requests where the in-flight waiter count for one (loader, project) key never returns to 0. That is the only condition under which _catalog_handoff[key] persists unrefreshed. On this browser-triggered skill-menu endpoint, requests are discrete user actions (a burst of tabs, then a gap); fast-path readers complete in microseconds, so the post-assembly drain empties the waiter count almost immediately and the handoff is popped at remaining == 0, after which the next reader reassembles current on-disk state. The triggering condition (a) is speculative "sustained gapless load," which the discovery pass itself could not confirm — it fails the "occurs in practice" bar. No crash, data loss, or boundary breach; the only harm is a minor, self-clearing menu staleness that never materializes without the speculative trigger.

[OPUS-REVIEWED] e1cbd43

@github-actions

github-actions Bot commented Aug 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of e1cbd43b93d592aebf036d0513011b0c41fdf5b8 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All evidence checked against the base: the removed hunk matches base prompts.py:2071-2096 exactly, LoopBoundLock and the per-key registry shape it cites are real and widely used (_EDIT_LOCKS in slack/outbound.py:70, _REPO_LOCKS in handlers/worktree.py:176, _identity_locks in auth/refresh.py:40 — so this reuses an established mechanism rather than inventing one), the frontend's skillsCacheStaleTime claim is real (website/src/lib/skillsCache, used in SkillPickerMenu.tsx:99), and _get_skills (handlers/_shared.py:383) confirms the loader is a genuine input to the assembly, so keying on it is keying on the computation's inputs. No generic single-flight helper exists in the tree to duplicate. Final review:

First-Principles-Verdict: PASS

A measured redundant-scan defect fixed by coalescing that stores nothing, with the invalidation-owing cache variant already subtracted out during its own review history.

What this change ships

Intent: stop N simultaneous skill-menu opens from each paying their own filesystem scan — a FIX.

  1. Eight simultaneous skill-menu opens cost one scan, not eight — justified, measured (0% → 87.5%)
  2. A reader joining a burst may get rows one scan old — declared cost, bounded, docstring-pinned
  3. Different projects still scan in parallel (per-key lock) — justified, test-pinned
  4. A non-concurrent read still scans fresh; no mutator owes an invalidation — justified, preserves base's recorded default
  5. Spec section in memory-skills-hooks.md — undeclared in description, but mandated by the same-commit spec-sync rule
  6. Seven-test suite pinning join, lifetime, and cross-agent isolation — rides with the fix, justified

The mechanism reuses the established #4800 shape (per-key LoopBoundLock registry: slack/outbound.py:70, handlers/worktree.py:176, auth/refresh.py:40 — counted, 3+ existing instances of the pattern, none a second spelling of this capability since none coalesces reads). The frontend's 5-minute skillsCacheStaleTime cannot do this job: it dedupes per client per query key, while ?agent= varies the query key and the backend key deliberately omits it. Root-cause framing is honest — the contended field samples are withdrawn, the residual defect is measured by construction, and the stored-cache alternative was tried and deleted for a named, counted reason (19 invalidation call sites it would have obliged).

[FIRST-PRINCIPLES-REVIEWED] e1cbd43

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Aug 30, 2026
@rnoack1
rnoack1 force-pushed the fix/skills-catalog-server-cache branch from 20af770 to 38f80a5 Compare August 30, 2026 18:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@rnoack1
rnoack1 force-pushed the fix/skills-catalog-server-cache branch from 96c0697 to 86e0833 Compare August 30, 2026 22:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Aug 30, 2026
@rnoack1
rnoack1 force-pushed the fix/skills-catalog-server-cache branch from 86e0833 to cb44619 Compare August 30, 2026 23:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@rnoack1
rnoack1 force-pushed the fix/skills-catalog-server-cache branch from cb44619 to 89d0e0b Compare August 30, 2026 23:58
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • This PR is PARTIALLY_COVERED with PR #7062. Coverage is explicitly incomplete; this finding is not a completion or closure claim. Recommended action for PR #7031: REBASE. The visible-hang half of the original report is closed on main; the server-side redundant-scan half this PR removes is not, so the merged work covers part of the motivation and none of the change. Files: website/src/api/client.ts.
  • This PR is PARTIALLY_COVERED with PR #7067. Coverage is explicitly incomplete; this finding is not a completion or closure claim. Recommended action for PR #7031: REBASE. The root-cause work the PR defers to has landed, so the 60.1%/41-168s figures that carry its entire cost-benefit argument were measured against a slower main than exists now. The memo is not implemented anywhere on main, so nothing is duplicated — but the numbers should be re-taken on current main before merge, since a much smaller contended latency changes how much new staleness the 3s window is worth buying. Files: src/kiro_crew/security.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@rnoack1

rnoack1 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Both audit items verified at head 67818c7ec. One was already satisfied; the other was genuinely outstanding and is fixed in this sha.

Item 1 (#7062) — satisfied, nothing owed here. The visible-hang half is closed on main and this PR does not touch it. At head, website/src/api/client.ts:36 declares export const SKILLS_TIMEOUT_MS = 15_000, and :2665 wires it via withDeadline(SKILLS_TIMEOUT_MS, signal, …) on the skills call; website/src/test/apiSkillsDeadline.test.ts pins it. client.ts is not among this PR's 13 files, so the client deadline needs no change from this branch. The audit's read of the split is correct: the server-side redundant-scan half is what this PR removes.

Item 2 (#7067) — the re-measure was owed, and the numbers were stale in code. You were right that the root-cause work landing changes the cost-benefit. Re-measured redact_credentials on current main against the pre-landing tree, byte-identical inputs, positive-controlled by the presence/absence of the bytes.translate pass:

payload before current main speedup
prose 177.9 µs 52.7 µs 3.4×
base64 blob 130.1 µs 36.7 µs 3.5×
binary-ish 574.5 µs 179.0 µs 3.2×
mixed 1283.5 µs 589.7 µs 2.18×

The original 60.1% share and the 41/69/168s contended samples were withdrawn rather than restated, because they came from a live capture under production session churn that cannot be reproduced here, and those profile rows are inclusive frame shares that do not sum to 100 — so arithmetic on them is not valid. The description already marked them withdrawn; the code comment above the assembly still asserted them as fact, which made the two surfaces contradict each other. As of this sha that comment no longer carries them, and the per-path staleness rationale it held now lives in docs/system-specs/modules/memory-skills-hooks.md instead.

The residual claim is deliberately narrower than the original: the memo dedupes concurrent assemblies of a ~0.6s idle scan and is a bounded symptom mitigation, not a cure. Whether that still justifies a 3s staleness window with redaction ~2.2× cheaper is a judgement for a human reviewer, and it is being weighed rather than assumed.

REBASE, recommended on both items — done. This head's parent is main tip 824ab57e8, resolved as a union with the kiro-user/ / kiro-workspace/ write-refusal change that landed in the same file: both its constant and its two call sites are intact alongside this PR's additions.

Key on ($HOME, project dir) gated on loader identity: a project-only key collapsed every slotless caller onto one entry and served a neighbour's catalog.

Coalesce concurrent readers on LoopBoundLock plus a re-check under it, as api_system does, and invalidate at the chokepoints a caller awaits a read-after-write on (22 sites to 15, behind one get_or_assemble entry point).

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tech Lead review — approved.

Correctness of the invalidation question (the one that matters here): there is no invalidation obligation, because nothing is retained. The handoff, the waiter count, and the per-key LoopBoundLock are all popped in the same finally when remaining == 0, so a read that is not concurrent with another always re-scans on-disk state. Verified the base's recorded "no result cache" default survives: test_a_later_sequential_read_reassembles asserts n == 2 on two sequential reads plus empty bookkeeping, and test_a_key_lock_does_not_outlive_its_burst bounds the registry. No mutator anywhere owes this code a call, which is the property the earlier stored-TTL revisions could not offer.

Key is real, not decorative. _get_skills returns a cached instance on both branches (context_builder.skills, else state._standalone_skills), so (loader, project) is stable across requests and the join actually fires in production rather than only in tests. Per-key rather than global is right: assembly is not reliably sub-second, and test_different_keys_assemble_in_parallel pins that two projects still scan concurrently.

No shared-mutable hazard. The rows are handed out un-copied, and the only downstream consumer is the ?agent= comprehension in api_skills, which builds a new list and never mutates an entry in place before json_response. The two-order parametrized end-to-end test pins that a filtered result can never be what gets shared.

Locking discipline. _catalog_lock (a threading.Lock) is only ever held across synchronous dict operations and never across an await; the increment precedes lock acquisition and the decrement follows release, so a key's lock cannot be evicted while a waiter still references it. Cancelling a leader publishes nothing, which is the stated reason the epoch stamp was correctly deleted.

Residual, accepted as non-blocking: under a hypothetically gapless overlapping request stream the waiter count never returns to 0 and _catalog_handoff[key] would persist unrefreshed. Fast-path readers return in microseconds on a browser-triggered endpoint, so the drain empties almost immediately; I reach the same conclusion the Opus lane did.

Scope and hygiene: 3 files, one commit, spec updated in the same commit, no new module and no new dependency, no auth/trust/sandbox/secret surface touched. All 4 AI lanes clean on this exact sha; SAST, Loop-Bound Locks Gate, PR Scope, PR Hygiene, Testpaths Coverage all pass. Both relationship-audit REBASE items were verified and discharged by the author.

One non-blocking nit for a follow-up, not worth another round: the test_skills_catalog_cache.py module docstring still says "readers of different projects serialize on the one global assembly lock," which is a leftover from the pre-per-key revision and contradicts both the implementation and test_different_keys_assemble_in_parallel directly below it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants