Skip to content

fix(skills): say a scoped skill_search miss is out of scope, not absent - #19299

Merged
bolichen97 merged 1 commit into
mainfrom
fix/skill-search-scoped-no-match-17559
Oct 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/skill-search-scoped-no-match-17559

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: when skill_search finds nothing in an agent's mapped skill scope, the message says so, instead of reading as "this skill is not installed".

A session whose agent declares skill:// resources searches only those (session_skill_globs). A miss returned:

No skills matched 'babysit'. Try broader keywords, browse action='list', or load a known full key with action='read'.

babysit was installed. It was simply outside the agent's mapping. Agents read the message as "the skill does not exist" and either gave up or tried more keywords, none of which can match.

Replaces #17559 by @Premshay, carried onto a main-repo branch so CI can run with repository permissions. The commit keeps @Premshay as author, with a Co-authored-by: trailer.

Why it matters

The message is the only signal the agent gets, and it points the wrong way: "try broader keywords" cannot succeed for a skill outside the mapping. One session searched for babysit three times and got the same message each time, with the skill installed throughout.

Not a goal

  • Changing what a mapped scope can find. It stays bounded; test_unmapped_custom_agent_still_gets_nothing is untouched.
  • The unscoped message.

What changed (motivation → approach → change)

  • Symptom: a scoped miss reads like "not installed". Root cause: the tool's response did not know the search was scoped. Change:
  • api_skills search and list responses carry scoped: true when the session's agent maps skill:// resources, false when unscoped. It is a bit, not a count: the mapping is a list of fnmatch globs, so one wildcard entry such as skill://~/.kiro/skills/*/SKILL.md maps many skills and no glob count is a skill count.
  • Why a new scoped and not the catalog's agent_scoped: they answer different questions. agent_scoped (catalog branch, ?agent=) is true only when the listing was filtered, and an agent with an empty mapping keeps the unfiltered legacy listing, so it reads false. On the search/list/read branch an empty mapping is not unfiltered: only == [] finds nothing (test_unmapped_custom_agent_still_gets_nothing). That search was restricted, so scoped is true. Reusing agent_scoped would give one name two meanings.
  • skill_search says so on a scoped miss: No skills matched 'babysit'. Search covers only this agent's skill:// resources, so an installed skill outside them is not found; browse action='list' to see the mapped set.

Backwards compatibility

Compatible: scoped is an added response field. Unscoped sessions and the CLI path get the old message.

Removes nothing: the diff only adds a response field and a more specific miss message.

Tests

  • New: test_skill_mapping_and_lazy_default.py::TestScopedDiscovery::test_gateway_wildcard_mapping_reports_scope_without_a_count: one team/*/SKILL.md glob over 3 skills; asserts scoped is True. Fails on 2a9e805.
  • New: test_mcp_core_coverage.py::TestSkillSearch::test_no_matches_in_a_mapped_scope_names_the_scope: the scoped miss names the scope and states no count. Fails on 2a9e805.
  • Extended: test_gateway_list_and_exact_read_share_the_session_mapping asserts scoped for a mapped and a cleared mapping.
  • test/test_mcp_core_coverage.py and test/test_skill_mapping_and_lazy_default.py pass on this branch (219 passed).

Manual verification

N/A — unit coverage sufficient: the change is a response field and a message string, both asserted directly.

Related Issues

Fixes #19294

Replaces #17559

Pattern harvest

Rule candidate: review-prompt
Pattern: an empty result from a scoped search must name the scope it searched and the one action that can still succeed, not suggest broader keywords.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 12, 2026 06:22
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d5e6f14146da087760061c84737885a2483cd756 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d5e6f14146da087760061c84737885a2483cd756 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check what action='list' shows for an agent whose skill mapping is empty: the new scoped-miss hint then points to a list that is also empty.

[DESIGN-REVIEWED] d5e6f14

Description read

[DESCRIPTION-READ] 603358e9d22339449326d6f7676261dec0da632a8f17015113de22628df41723

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d5e6f14146da087760061c84737885a2483cd756 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d5e6f14

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d5e6f14146da087760061c84737885a2483cd756: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d5e6f14146da087760061c84737885a2483cd756 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

A scoped miss while body indexing is incomplete still gets the indexing message, which does not mention scope. Confirm that order is intended.

What this change ships

Inventory (2 items) — 2 justified

Intent: when an agent's search can only see its mapped skills, a miss should say so, so the agent doesn't conclude the skill isn't installed. This is a FIX (#19294).

  1. When nothing matches in a session whose agent maps skill:// resources, skill_search now says the search covered only that agent's resources and points to action='list'. Unscoped sessions and the CLI still get the old message. — justified
  2. /api/skills search and list responses now carry a scoped field. Grepping "scoped" / get("scoped") finds 1 consumer, mcp_tools/skills.py:220. It isn't the same as the catalog's agent_scoped: with an empty mapping, the search path finds nothing, but the catalog path falls back to the full listing. — justified

[FIRST-PRINCIPLES-REVIEWED] d5e6f14

Description read

[DESCRIPTION-READ] d0f5cd2b9928c4467a749ff48ac9448fdb9ea810c95be360ea9157dff5d3bb01

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d5e6f14146da087760061c84737885a2483cd756 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d5e6f14

Verdict parsed from the review's SHA-scoped output markers for commit d5e6f14146da087760061c84737885a2483cd756.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d5e6f14146da087760061c84737885a2483cd756: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@iamwhatever
iamwhatever force-pushed the fix/skill-search-scoped-no-match-17559 branch from f4085e8 to 6464ebf Compare October 12, 2026 07:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@iamwhatever
iamwhatever force-pushed the fix/skill-search-scoped-no-match-17559 branch from 6464ebf to 2a9e805 Compare October 12, 2026 08:22
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): CHANGES NEEDED

This makes a scoped skill_search miss say that the search covered only the agent's skill:// resources, instead of reading as "not installed". That is the right root cause: the tool did not know the search was scoped. But the shipped count is wrong. src/kiro_crew/dashboard/handlers/prompts.py:2449 sets scope_size = len(only). only comes from session_skill_globs, which returns fnmatch globs, not skills; I checked this in agent_discovery.py. So one supported wildcard entry such as skill://~/.kiro/skills/*/SKILL.md makes src/kiro_crew/mcp_tools/skills.py:~304 print "among the 1 skill(s) mapped to this agent" while dozens are mapped. That is a new false picture inside the message meant to remove one. Design, First Principles and Opus all raise this on head 2a9e805, and no disposition has been posted. The tests only use exact-path mappings, so they cannot catch it. Smallest fix: replace the field with scoped: only is not None (or keep it but drop the number) and remove {scope_size}/"skill(s) mapped" from the miss string. Alternatively, count the resolved catalog entries and add a wildcard-mapping test. — crew-pr-reviewer

@iamwhatever
iamwhatever marked this pull request as draft October 12, 2026 15:47
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Converted to draft by crew-pr-reviewer: the self-review above is CHANGES NEEDED. Mark ready again (gh pr ready 19299) once the points are addressed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 12, 2026
@iamwhatever
iamwhatever force-pushed the fix/skill-search-scoped-no-match-17559 branch from 2a9e805 to 7cd7604 Compare October 12, 2026 16:59
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Glob count shown as a skill count span=653a507c30fd

fixed in 7cd7604: the route now returns "scoped": only is not None instead of len(only), and the miss string no longer states a count ("No skills matched 'x'. Search covers only this agent's skill:// resources ..."). New test_gateway_wildcard_mapping_reports_scope_without_a_count maps one team/*/SKILL.md glob over 3 skills and fails on the old head.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Return scoped: only is not None instead of a count span=9643b595d6d0

adopted in 7cd7604: api_skills returns "scoped": only is not None; the tool reads it as a strict boolean and points the agent at action='list' for the mapped set.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 2 oversized: scope_size counts glob entries, labelled as skills span=6b95af999ea1

fixed in 7cd7604: the route now returns "scoped": only is not None instead of len(only), and the miss string no longer states a count ("No skills matched 'x'. Search covers only this agent's skill:// resources ..."). New test_gateway_wildcard_mapping_reports_scope_without_a_count maps one team/*/SKILL.md glob over 3 skills and fails on the old head. The integer is gone; only the scoped bit ships.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • scope_size counts expanded skill:// globs rather than the skills they match span=0d60fc2849ba

fixed in 7cd7604: the route now returns "scoped": only is not None instead of len(only), and the miss string no longer states a count ("No skills matched 'x'. Search covers only this agent's skill:// resources ..."). New test_gateway_wildcard_mapping_reports_scope_without_a_count maps one team/*/SKILL.md glob over 3 skills and fails on the old head. A stale gateway that still sends scope_size is ignored (test_mapped_scope_miss_ignores_a_stale_count).

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Re the crew-pr-reviewer self-review (comment), point by point, on new head 7cd7604:

  1. scope_size = len(only) counts globs, not skills. Confirmed: session_skill_globs (agent_discovery.py) returns agent_skill_globs(...), a list of fnmatch globs. Fixed with the smallest option: prompts.py now returns "scoped": only is not None; no number ships.
  2. Miss string states a count. {scope_size} and "skill(s) mapped" are gone. The scoped miss now reads: No skills matched 'x'. Search covers only this agent's skill:// resources, so an installed skill outside them is not found; browse action='list' to see the mapped set. skill_search reads scoped as a strict boolean, so a stale gateway's scope_size is ignored.
  3. Tests only used exact-path mappings. Added test_gateway_wildcard_mapping_reports_scope_without_a_count (one team/*/SKILL.md glob over 3 skills) and test_mapped_scope_miss_ignores_a_stale_count. Both fail on 2a9e805 and pass now (262 passed in the two files).
  4. No dispositions posted. Posted one per finding: Design x2, First Principles x1, Opus x1, each keyed by its span on 2a9e805.

Still one commit, Premshay as author with both Co-authored-by trailers. PR body updated to describe scoped; no docs/system-specs page quotes the message.

@iamwhatever
iamwhatever force-pushed the fix/skill-search-scoped-no-match-17559 branch from 7cd7604 to d5b0b71 Compare October 12, 2026 17:18
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 2: scoped duplicates agent_scoped span=bff6ef722f7a

rebutted: the two bits answer different questions, so one name would carry two meanings. agent_scoped (catalog branch, ?agent=) means the listing was filtered; an agent with an empty mapping keeps the unfiltered legacy listing, so it is false there. On the search/list/read branch an empty mapping is not unfiltered: only == [] finds nothing (test_unmapped_custom_agent_still_gets_nothing), so the search was restricted and scoped is true. That is the case the test pins after paths.clear(). The PR description now says why, which is this item's own clear condition.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 3: stale scope_size test rides along span=40dca2af67ec

fixed in d5b0b71: deleted test_mapped_scope_miss_ignores_a_stale_count and the scope_size / "skill(s) mapped" absence assertions in the two other new tests. The wildcard test still asserts scoped is True and the tool test still pins the count-free miss text.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@iamwhatever
iamwhatever force-pushed the fix/skill-search-scoped-no-match-17559 branch from d5b0b71 to 0fc447b Compare October 12, 2026 19:15
A session whose agent maps skill:// resources searches only those, but a
miss read 'No skills matched', which agents took to mean the skill is not
installed. The gateway route now returns scoped=true for a mapped scope,
and the tool says the search covered only this agent's skill:// resources.

The route reports a bit, not a count: the scope is a list of fnmatch
globs, so one wildcard entry would have been reported as one skill.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Premshay <28099628+Premshay@users.noreply.github.com>
@iamwhatever
iamwhatever force-pushed the fix/skill-search-scoped-no-match-17559 branch from 0fc447b to d5e6f14 Compare October 12, 2026 19:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Dispositions for the two advisory notes on head d5e6f14 (no code change):

  1. Design: empty mapping makes action='list' empty too. Accepted as intended. With only == [] the mapped set really is empty, so an empty list is the true answer and confirms the scope, not the keywords, is why nothing matched. The miss text says "to see the mapped set", which stays accurate when that set is empty.
  2. First Principles: an incomplete-index miss gets the indexing message first. Intended order. While indexing is incomplete the miss may only be a coverage gap, and the one action that can still succeed is repeating the query. Once indexing completes, a still-empty scoped search gets the scope message.

Branch is already on origin/main 65df088 (includes #19495). test/test_mcp_core_coverage.py and test/test_skill_mapping_and_lazy_default.py: 219 passed. The crew-pr-reviewer CHANGES NEEDED points were answered in the earlier point-by-point reply; marking ready.

@iamwhatever
iamwhatever marked this pull request as ready for review October 12, 2026 20:34
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 12, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer approval after a full diff read on head d5e6f14146: diff matches the description, no regression to existing behaviour found, CI and readiness green.

@bolichen97
bolichen97 merged commit 2b607d6 into main Oct 12, 2026
104 checks passed
@bolichen97
bolichen97 deleted the fix/skill-search-scoped-no-match-17559 branch October 12, 2026 22:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

skill_search miss in a mapped scope reads as 'skill not installed'

3 participants