Skip to content

perf(crew): offer a Bot approval's Remove without rebuilding the file per row - #37773

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/37743-mate-scale-timeout
Nov 16, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/37743-mate-scale-timeout

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Opening a Bot that has about a thousand tool approvals is fast again, and its scale test stops timing out in CI.

test/test_mate_capabilities_scale.py::test_get_preview_and_put_do_not_match_every_approval_against_every_other hit the 120 s pytest timeout on a 4-core Linux runner (Backend Tests (3.12, 4)) at line 74, the service.get("big") read. On a fast local host the same test took 14.4 s.

Why it matters

The Abilities and Approvals panes read GET /api/agents/<bot>/capabilities, which gives up after 20 s. A Bot with hundreds of approvals could not open its panes, and the scale test fails PRs that do not touch this code.

Not a goal

  • Changing which rows carry a Remove, or what Remove does.
  • Speeding up preview or put beyond what the shared read already costs.

What changed (motivation → approach → change)

Symptom: one get read of a Bot with 1,000 approvals took 14.4 s locally and over 120 s on CI.

Root cause: worker_summary decides each added row's Remove by building a candidate file without that row, then re-running _materialize, worker_additions and _effective on it. That is one whole-file rebuild per row, so the read is quadratic in approvals: the new probe counts 1,021 worker_additions calls for one read. #37691 indexed the reach matching inside each rebuild but kept the per-row rebuild.

Change: a glob-free row that no other effective entry reaches gets its Remove directly, without the rebuild. Removing an entry only shrinks the file's additions, and the reach pass is monotone, so a key that nothing else reaches surely disappears. The new OtherReach in agent_sdk/mcp_refs.py, beside ReachIndex, answers whether any effective entry other than the key reaches it. Globbed entries go through ReachIndex. Glob-free entries are looked up by the key's case-folded pattern and by its verb, then re-checked with entry_reaches. Glob rows, rows something else reaches, and the skills, resources and server sections keep the existing rebuild path.

Result: the scale test's call phase went from 14.4 s to 2.9 s locally.

Backwards compatibility

Compatible: the Remove shown for each row is unchanged. The shortcut only answers for rows where the rebuild would give the same answer, and every other row still takes the rebuild path.

Removes nothing: the change adds a shortcut in front of an existing check and drops no rows, keys or functions.

Tests

  • test/test_mate_capabilities_scale.py: counts mate_worker.worker_additions calls during one get read and bounds them at a tenth of the approval count. Main makes 1,021 calls here. This is a call count, not a wall-clock bound.
  • test/test_mate_capabilities_remove_differential.py::test_every_added_row_offers_the_remove_the_rebuild_offers: for a file with globbed entries, case variants, @srv/tool beside bare tools, a slash-bearing bare entry, a reach-pass-derived entry, a base-server selection and 300 plain rows, every added row's Remove is the same with the shortcut on and with it forced off (every key read as globbed, so every row takes the rebuild). It also asserts that the shortcut answered more than 300 rows, so the comparison is not vacuous.
  • test/test_mate_capabilities_remove_differential.py::test_other_reach_matches_trying_every_other_entry: OtherReach.reached_by_another equals any(entry_reaches(e, key) for e in entries if e != key) for every glob-free key, including a slash-bearing bare key with a bare-verb reacher. This fails if the verb lookup is limited to @ keys again.
  • The existing test_a_glob_entry_withholds_remove_on_a_tool_it_still_grants cases in test/test_mate_capabilities_worker_summary.py still pass, so a row that a glob still grants keeps no Remove.

Proof: PROVEN test/test_mate_capabilities_scale.py::test_get_preview_and_put_do_not_match_every_approval_against_every_other (head 4ae29f7)

Manual verification

N/A: unit coverage is sufficient. The change is backend-only and the scale test exercises the same get, preview and put path the panes call.

Related Issues

Fixes #37743

Sibling sweep

Swept: git grep -n "copy.deepcopy(file_spec)" -- src/
Siblings: none -- worker_summary's _remove_ref is the only per-row whole-file rebuild of a Bot's spec

Pattern harvest

Not generalizable: the per-row rebuild was specific to how worker_summary decides Remove, and the scale test already guards it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

… per row

worker_summary checked each added row's Remove by rebuilding the file's
additions without that row, so a read made one whole-file rebuild per
approval: 1021 rebuilds for a thousand approvals. That kept the scale
test near the 120 s pytest timeout on 4-core CI runners. A glob-free
entry that no other effective entry reaches surely disappears when
removed, so its Remove is offered directly; anything else still takes
the rebuild path.

Fixes kirodotdev#37743
@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners November 16, 2026 03:01
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 4ae29f79dea3ba7017756c71f41e7d371b29a931 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 4ae29f79dea3ba7017756c71f41e7d371b29a931 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the shortcut runs only when the rebuild must give the same answer, and a differential test compares the two on every row.

Notes

  • The shortcut assumes worker_additions never derives a glob-free key from a non-entry source (for example a server's autoApprove). Add one line to _remove_ref naming that assumption, so a future derived entry is checked against this shortcut.

[DESIGN-REVIEWED] 4ae29f7

Description read

[DESCRIPTION-READ] 81a273733b051349e0fef483bc179a7fc5c99fd21a1b29b1e5903419c6727e83

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 4ae29f79dea3ba7017756c71f41e7d371b29a931 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This is a performance fix for issue #37743, aimed at its cause: the read rebuilt the whole file once per row. The rows that still rebuild are listed in the description.

Notes

  • Item 2 — one consumer, generalized: grant_fold makes OtherReach the third holder of a fold rule that ReachIndex.reaches (mcp_refs.py:339) still writes out inline as (_canonical_grant_pattern(ref) or ref).casefold(). Only mate_capabilities.worker_summary imports the module's new names, and it imports glob_free and OtherReach, not grant_fold. Subtraction: make grant_fold private, and have ReachIndex.reaches call it so the inline copy goes.

What this change ships

Inventory (4 items) — 3 justified

Intent: open a Bot that has about a thousand approvals without the capabilities read timing out. FIX.

  1. A Bot's tool or approval rows get their Remove without rebuilding the whole file per row — justified
  2. New names in mcp_refs other code can call: OtherReach, glob_free, grant_fold — undeclared (glob_free, grant_fold); one consumer, generalized
  3. The scale test now caps whole-file rebuilds at a tenth of the approval count — justified
  4. A differential test checks that the shortcut and the rebuild give every row the same Remove — justified

[FIRST-PRINCIPLES-REVIEWED] 4ae29f7

Description read

[DESCRIPTION-READ] e47dc94696630ca6265a9785aacfcd794e117f0d43da1cbb404ee0fa4d512e10

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 4ae29f79dea3ba7017756c71f41e7d371b29a931 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4ae29f7

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 4ae29f79dea3ba7017756c71f41e7d371b29a931 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4ae29f7

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This change adds a shortcut in worker_summary._remove_ref. A tools or allowedTools row that has no glob, and that no other effective entry reaches, gets its Remove directly instead of a whole-file rebuild. I traced worker_additions, _materialize, _apply_operations and the reach pass in _effective. Removing a stored entry only shrinks the additions, the key's only source is its own file entry, and the reach pass can only grow the set. So the shortcut answers only rows where the rebuild gives the same Remove. Whole-server selectors count as globbed and still rebuild, so the existing tests with a stored '*' are unaffected. OtherReach's lookups by folded pattern and by verb cover every way entry_reaches can match a key with no glob, and every hit is checked again. When the summary falls back on a read failure, every row takes the old rebuild path. The new tests cover the behaviour: the scale test counts rebuild calls, which fails on main, and a comparison test checks every row with the shortcut on and off.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 4ae29f79de · 170 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 16, 2026 05:15

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on head 4ae29f7. We read the full diff and checked the shortcut's soundness against worker_additions and the reach pass in _effective. A removed glob-free key that no other entry reaches cannot come back. We ran the three affected test files with -n0 (29 passed). Reverting the shortcut reds the scale test (1021 < 100) and the differential test. Six mutations of OtherReach each red 2 to 8 tests. A 20k-case random comparison against trying every other entry found 0 mismatches. The branch merges clean with current main.

Done well: the differential test runs every row with the shortcut on and with it forced off. It also asserts that the fast path answered more than the bulk rows, so it cannot pass vacuously. The new bound counts rebuild calls instead of timing them, so a loaded runner cannot flake it.

Optional:

  • grant_fold has one consumer, and ReachIndex.reaches (mcp_refs.py:374) still writes the same fold inline. Making it _grant_fold and calling it from there would leave one copy of the rule.
  • The glob_free(key) guard in _remove_ref (mate_capabilities.py:551) is redundant: a globbed key always takes the rebuild anyway. You could drop it, or add a comment saying it only skips a lookup.

@KiroCrewCaptain
KiroCrewCaptain merged commit ef1564a into kirodotdev:main Nov 16, 2026
94 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_get_preview_and_put_do_not_match_every_approval_against_every_other times out on a 4-core Linux shard

2 participants