Skip to content

perf(crew): match a Bot's tool approvals by index, not all-pairs - #37691

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/cap-approvals-scale
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/cap-approvals-scale

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Bot with hundreds of tool approvals opens, previews and saves its Abilities and Approvals panes instead of timing out.

A Bot with many tool approvals (many @server/tool entries in allowedTools) shows "Couldn't load this Bot's approvals." and "Couldn't load this Bot's capabilities." The capabilities GET gives up after 20 s and answers 503 capabilities_unavailable. Measured on main with a tool-only Bot and no skills: 408 approvals take 34 s, 520 take 66 s, 1020 take 509 s. #37626 speeds up the skill side of the same read and leaves this tool side as it is.

Why it matters

The people who added the most tools to a Bot are the ones who cannot see, switch off or remove them. Preview and save run the same read again, so a remove costs two slow reads, and the save holds the agents config lock for the whole read.

Not a goal

What changed (motivation → approach → change)

Symptom → cause. worker_summary in src/kiro_crew/mate_capabilities.py checks, for every added row, whether its Remove would make the row go away. Each check rebuilds the worker's effective tools and approvals, then tests every listed ref against every entry with entry_reaches. That is rows × refs × entries: about 10^9 calls for 1000 approvals.

Change. A new ReachIndex in src/kiro_crew/agent_sdk/mcp_refs.py answers any(entry_reaches(entry, ref) for entry in entries) with the same result, without trying every entry. An entry with no glob characters can only match its own case-folded spelling, so it is a set lookup. An entry whose server name has no glob can only match refs on that server, so it is tried only for those. Server globs, bare globs and anything else still go through entry_reaches itself. The per-row check now uses the index and keeps its list order, so a ref it adds still reaches later refs, as before. Preview and save still build their view with one read each; that read is now fast, so they are left as they are.

Tool-only Bot main this PR
4 servers × 100 approvals 34 s 0.5 s
10 × 50 66 s 1.0 s
10 × 100 509 s 2.5 s

Backwards compatibility

Compatible: every row, state and Remove target is the same; only how many matches run changes.

Removes nothing: no field, row, key or function is removed or tightened.

Tests

  • test_reach_index_answers_what_trying_every_entry_answers and test_reach_index_agrees_on_every_subset_and_after_adds pin ReachIndex.reaches to any(entry_reaches(...)) over exact refs, @server, @server/, @server/*, tool globs, [...] classes, server globs (@*, @kirocrew-*), *, bare names and globs, case and ß/SS folding, @, @/x and the empty string, on 300 seeded random subsets built partly by add. Dropping the bare-verb lookup fails them.
  • test_get_preview_and_put_do_not_match_every_approval_against_every_other: a Bot file with 1000 approvals over 10 servers. get, preview and put each list every added row as removable, the preview reports the removal, the save writes it, and entry_reaches runs fewer than 20 × 1000 times per call. It counts work instead of timing it; on the parent it does not finish before the 120 s pytest timeout.
  • Existing worker_summary, listed-skills and redaction tests pass unchanged.
  • Determinism: 3/3 repeats at -n0, plus a run at -n 4 --dist load with the worker-summary file, all green.

Proof: not provable -- other: the new scale test runs past the 120 s pytest timeout on the parent instead of failing an assertion

Manual verification

A tool-only Bot of 4 × 50 approvals was summarised by worker_summary on the parent and on this change: the full JSON output was byte-identical. The branch also merges cleanly with #37626, and the combined tree passes the worker-summary, listed-skills, redaction, scale and entry_reaches tests (143) and reads the 1020-approval Bot in 2.6 s.

Related Issues

Part of #37625

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 96d50a1cc98ecea95e806ac53e231b638b0a21d4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 96d50a1cc98ecea95e806ac53e231b638b0a21d4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a backend-only change. It indexes the same entry_reaches predicate instead of replacing it, and tests check the index against the old all-pairs answer.

Notes

  • ReachIndex relies on facts about how entry_reaches behaves today: case folding never adds a / or a glob character, and only these glob characters exist. If entry_reaches or _TOOL_GLOB_CHARS changes, the index can return a different answer without anything failing loudly. Keep the subset-agreement test next to entry_reaches, and grow it whenever the glob grammar grows.

[DESIGN-REVIEWED] 96d50a1

Description read

[DESCRIPTION-READ] 516c34d496606e5e6aeb49f3c1ad2497599ae3523bf91a2b4a3d140fab6a6059

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 96d50a1cc98ecea95e806ac53e231b638b0a21d4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the index removes the measured cubic matching cost at the code that causes it, returns the same answers as before, and has a real consumer.

What this change ships

Inventory (3 items) — 3 justified

Intent: let someone who added hundreds of tool approvals to a Bot open, preview and save its Abilities and Approvals panes without a timeout. This is a FIX.

  1. A Bot with about 1000 approvals now loads its Abilities and Approvals panes in seconds instead of hitting the 503 timeout. — justified
  2. Previewing and saving a Remove on that Bot now finish quickly, and the save holds the agents config lock for much less time. — justified
  3. New ReachIndex in agent_sdk/mcp_refs.py that gives the same answer as entry_reaches over every entry; its one caller is worker_summary._effective. — justified

[FIRST-PRINCIPLES-REVIEWED] 96d50a1

Description read

[DESCRIPTION-READ] b6489d18adc10f024032efca7574af19ce148cd162de6a3a3dd71949b6744d4e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 96d50a1cc98ecea95e806ac53e231b638b0a21d4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 96d50a1

Verdict parsed from the review's SHA-scoped output markers for commit 96d50a1cc98ecea95e806ac53e231b638b0a21d4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 96d50a1cc98ecea95e806ac53e231b638b0a21d4 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 96d50a1cc98ecea95e806ac53e231b638b0a21d4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 96d50a1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 96d50a1cc98ecea95e806ac53e231b638b0a21d4 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

ReachIndex gives the same answer as any(entry_reaches(e, ref)). Glob-free entries reduce to equality before or after case folding, so set lookups give the same result. The bare-entry verb lookup gives the same result because casefold works per character and never adds or removes '/'. Entries with a literal server part can only match refs on that server, raw or folded, so bucketing them by folded server name is safe. Server globs and bare globs still go through entry_reaches. The caller keeps main's progressive, list-ordered expansion: set.update(generator) on main also added each ref as it was produced. The index is built per call and nothing persists, and entry_grants (the strict trust check) is untouched. One new test checks the index against the all-pairs answer; a scale test counts entry_reaches calls through get, preview and save. CI is green.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 96d50a1cc9 · 246 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 15, 2026 22:25

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head the review checked that ReachIndex gives the same answer as any(entry_reaches(...)) for each bucket (exact set, per-server bucket, fallback scan). An independent seeded fuzz over case folding, glob classes, @ and @/x found 0 mismatches. The caller keeps main's list-order expansion, entry_grants is untouched, and the PR's tests plus the sibling worker_summary and listed-skills suites passed locally (132 passed). Removing the bare-verb lookup reds the equivalence tests, as the body says.
Done well: the index never re-implements the matching rule, and the scale test bounds a deterministic call count instead of wall time.
Optional:

  • Deleting index.add(ref) in mate_capabilities.py reds no test. A small case where a spec ref is reached only through an earlier one (@notes/a? then @notes/a* then @notes/abc) would pin it.
  • In the scale test, counted could call pytest.fail as soon as the count passes MAX_MATCHES. A regression would then fail by name instead of through the 120 s timeout.
  • The scale test takes about 12.5 s. A smaller server count would still show the gap at a fraction of the runtime.

@KiroCrewCaptain
KiroCrewCaptain merged commit d4f0a82 into main Nov 15, 2026
98 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/cap-approvals-scale branch November 15, 2026 22:25
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants