Skip to content

fix(knowledge): count knowledge_list_sources items in linear time - #22194

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/21705-knowledge-list-sources-count
Oct 21, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/21705-knowledge-list-sources-count

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: knowledge_list_sources answers quickly on a large knowledge library instead of hanging and taking the shared MCP backend down with it.

On a library with 21 sources and about 76k items, the tool never returned. kirocrew-core is a single-worker MCP backend, so every call queued behind it failed too (including local_knowledge_search), until the gateway recycled the backend after 900 s as "wedged". Retrying did the same again.

Why it matters

The tool is meant to be the cheap way to see what is in the knowledge base and to find a source_id. On a large library it is the most expensive MCP call, and it breaks knowledge search from chat for everyone sharing the backend.

Not a goal

  • Changing what the tool counts or how it prints results.
  • The dashboard source counts (/api/knowledge/sources, /api/knowledge/source-counts); they already use a linear query.
  • The large WAL file or the "vector path unavailable" note mentioned in the issue.

What changed (motivation → approach → change)

Symptom: the per-source count query never finishes on a big library.
Cause: it put OR i.id IN (SELECT ... WHERE sl.source_id = s.id) inside the JOIN. That subquery depends on the outer row, so SQLite runs it again for every (source, item) pair.
Change: in src/kiro_crew/mcp_tools/knowledge.py, the query (now _SOURCE_COUNTS_SQL) builds (source, item) pairs from the two ways an item belongs to a source (it owns it, or it is a location of it), joins them with UNION so an item held both ways counts once, and joins the result to sources one time. The counts and the output text stay the same. This is the same shape the dashboard's /source-counts handler already uses. No other caller in src/ uses the correlated form.

Backwards compatibility

Compatible: same rows, same counts, same output text; only the SQL shape changes.

Removes nothing: the query returns the same columns and rows; no path, key or record is taken away.

Tests

In test/test_mcp_knowledge_search.py, class TestKnowledgeListSourcesCountQuery:

  • test_item_owned_and_located_in_same_source_counts_once: an item owned by a source and also listed as its location counts as 1.
  • test_matches_correlated_query_on_random_libraries (12 seeds): random libraries with deleted items, sourceless items and cross-source locations give the same rows from the new query and from the old correlated one.
  • test_query_plan_has_no_correlated_subquery: EXPLAIN QUERY PLAN of the old shape says CORRELATED (a probe that this SQLite reports it), and the shipped query's plan does not. Swapping the old query back in makes this test fail (checked locally).

Determinism: the 18 ListSources tests passed 3 times in a row with -n0, and once more in a shuffled order (seed 7). The random libraries use a fixed random.Random(seed).

Manual verification

Synthetic library, 21 sources, every item also listed as a location (script run locally, not committed):

items old query new query
5,000 9.30 s 0.01 s
76,000 not run (quadratic) 0.28 s

Both queries gave the same total at 5,000 items.

Related Issues

Closes #21705

Pattern harvest

Rule candidate: semgrep
Pattern: correlated IN (SELECT ... WHERE x = outer.col) inside a JOIN ... ON clause in a SQL string literal.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 19, 2026 15:00
@iamwhatever
iamwhatever requested a review from buluoray October 19, 2026 15:00
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d0a7741ea306dbffec28b08349458a58245d0a5e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d0a7741ea306dbffec28b08349458a58245d0a5e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] d0a7741

Description read

[DESCRIPTION-READ] 04d66aa13b37e0bde11fffe2ca85c345aad071d4dbae04d2749e013248b53dc9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d0a7741ea306dbffec28b08349458a58245d0a5e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (3 items) — 3 justified

Intent: let someone list what is in a large knowledge library from chat without the call hanging and taking knowledge search down for everyone on the shared MCP backend. This is a FIX (Closes #21705).

  1. knowledge_list_sources returns in well under a second on a 76k-item library instead of never returning; the counts and the output text are unchanged — justified
  2. The per-source count query is now a named module constant, _SOURCE_COUNTS_SQL, which the tests import — justified
  3. A test checks the query plan and fails if the correlated per-pair subquery comes back — justified

[FIRST-PRINCIPLES-REVIEWED] d0a7741

Description read

[DESCRIPTION-READ] 5d51d00ccda90e4fb7a75343023e9675719c05d2e5c5cb5c96ed850d0d6399f0

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d0a7741ea306dbffec28b08349458a58245d0a5e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d0a7741

Verdict parsed from the review's SHA-scoped output markers for commit d0a7741ea306dbffec28b08349458a58245d0a5e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d0a7741ea306dbffec28b08349458a58245d0a5e: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d0a7741ea306dbffec28b08349458a58245d0a5e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d0a7741

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d0a7741ea306dbffec28b08349458a58245d0a5e: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever
iamwhatever force-pushed the fix/21705-knowledge-list-sources-count branch from 6e99762 to d0a7741 Compare October 20, 2026 02:23
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main carrying the Harness Parity fix (#22178); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 20, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 21, 2026 08:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: replaces the correlated subquery in the knowledge_list_sources per-source count with a UNION join, so the count runs in linear time; the new tests check the result equals the old query on random libraries and that the query plan has no correlated subquery.

@KiroCrewCaptain
KiroCrewCaptain merged commit 0b89e0c into main Oct 21, 2026
104 of 106 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/21705-knowledge-list-sources-count branch October 21, 2026 08:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

knowledge_list_sources MCP tool is quadratic in library size; on a ~76k-item library it hangs until the gateway recycles the shared core backend

2 participants