fix(knowledge): count knowledge-base list totals with a lateral aggregate by waleedlatif1 · Pull Request #8560 · simstudioai/sim

waleedlatif1 · 2026-10-02T07:37:21Z

Summary

The counted knowledge-base list (GET /api/v2/knowledge, the v1 list, and the internal list with counts) left-joined documents under the reader's ACL predicate and grouped by base. When the planner overestimated a workspace's active bases (soft-deleted bases skew the estimate), it flipped to a hash join fed by a bitmap over doc_acl_gin_idx. Every ordinary document carries the shared ws token, so that bitmap read every tenant's documents to total one base, and the list took ~50 s instead of milliseconds
Totals are now a LATERAL aggregate correlated on the base, so each base is always counted through its own knowledge_base_id index (doc_active_kb_token_count_idx). With sort keys on the base alone, the planner also sorts and limits bases before counting them; at worst it totals every base, as the old query always did
Results are unchanged: same three active-document conditions, same access predicate, same order and limit. An aggregate always yields one row, so a base with no readable documents still lists at zero
The live-source supplement (readLiveSourceDocumentCounts) is untouched: its batches are already restricted to explicit connector IDs
The shared drizzle mock gains innerJoinLateral/leftJoinLateral and a count() that aliases like drizzle's SQL expression

Type of Change

Bug fix

Testing

New lib/knowledge/__integration__/list-totals-plan.integration.ts: runs the v2 list use case on real PostgreSQL, captures the counted-list SQL, and EXPLAINs it with nested loops disabled; every document scan must be correlated on its knowledge base. Red against the previous JOIN + GROUP BY, green with the lateral aggregate
Reproduced the bad plan with EXPLAIN on production statistics: the old query hash-joins a GIN bitmap over the whole document table at an estimate of 247 bases; at 106 it nests. The SQL this change emits, captured from getWorkspaceKnowledgeBases itself, plans a per-base Index Scan using doc_active_kb_token_count_idx (Index Cond: knowledge_base_id = knowledge_base.id) under the Limit for the same estimate, for both createdAt and name sorts
Where the old query already nested, the new one runs the identical per-base index scan with the same buffer count
Compared old vs new results row-for-row across 19 bases in three workspaces: 0 differing rows. Verified a base with no matching documents returns 0/0 rather than dropping out
bun run test:integration for app/api/v1/knowledge, lib/knowledge/__integration__, and app/api/v2/knowledge (real Postgres): 477 passed. These cover ACL-filtered totals on list and detail reads, search-index bases, and the counts flag
lib/knowledge/service.test.ts, lib/knowledge/application/knowledge-bases.test.ts, root bun run test (19/19 tasks), bun run lint, block-registry check, bun run check:audits (57 audits), docs-manifest:check, apps/sim type-check

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…gate The counted knowledge-base list left-joined documents under the reader's ACL predicate and grouped by base. When the planner overestimated how many active bases a workspace has (soft-deleted bases skew the estimate), it switched to a hash join fed by a bitmap over the document ACL GIN index. Every ordinary document carries the shared `ws` token, so that bitmap covered every tenant's documents just to total one base's. Totals are now a LATERAL aggregate correlated on the base, so each base is counted through its own knowledge_base_id index and the page limit stops counting past the page. Results are unchanged: same filters, same order, and an aggregate always yields one row, so empty bases stay at zero. The shared drizzle mock gains innerJoinLateral/leftJoinLateral and a count() that aliases like drizzle's SQL expression.

vercel · 2026-10-02T07:37:27Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 2, 2026 8:09am UTC

greptile-apps · 2026-10-02T07:40:37Z

Confidence Score: 5/5

[High risk] Restructures knowledge-base document counting with lateral joins.

The PR appears safe to merge; no outstanding findings or new actionable issues remain.

Summary

The PR replaces grouped document counts with a per-knowledge-base lateral aggregate and adds a PostgreSQL query-plan regression test.

The changes since the previous review complete fixture teardown.
No new actionable issues were identified.

_{Reviews (4) · Last reviewed commit: "test(knowledge): tear down every fixture..."}

cubic-dev-ai

All reported issues were addressed across 3 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

EXPLAIN the exact counted-list SQL with nested loops disabled and require every document scan to be correlated on its knowledge base, so a set-based join over the ACL index fails the build. Drop the comment's page-limit claim, which is the planner's cost choice rather than a guarantee, and a redundant mock-call assertion.

waleedlatif1 · 2026-10-02T07:59:50Z

@greptile

waleedlatif1 · 2026-10-02T07:59:51Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-02T07:59:57Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 4 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…test Delete the fixture organization and both users alongside the workspace, matching the v1 knowledge route test that seeds the same fixture; permissions cascade from the users.

waleedlatif1 · 2026-10-02T08:09:27Z

@greptile

waleedlatif1 · 2026-10-02T08:09:28Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-02T08:09:33Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 4 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 · 2026-10-02T08:14:31Z

@greptile

waleedlatif1 · 2026-10-02T08:14:33Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-02T08:14:39Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 4 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

icecrasher321 · 2026-10-02T19:18:39Z

should be fixed in #8560

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/service.test.ts Outdated

cubic-dev-ai Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/service.ts

vercel Bot temporarily deployed to Preview October 2, 2026 07:59 Inactive

cubic-dev-ai Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/__integration__/list-totals-plan.integration.ts

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/__integration__/list-totals-plan.integration.ts

test(knowledge): tear down every fixture row in the list totals plan …

4f72a21

…test Delete the fixture organization and both users alongside the workspace, matching the v1 knowledge route test that seeds the same fixture; permissions cascade from the users.

vercel Bot temporarily deployed to Preview October 2, 2026 08:09 Inactive

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/__integration__/list-totals-plan.integration.ts

cubic-dev-ai Bot reviewed Oct 2, 2026

View reviewed changes

icecrasher321 closed this Oct 2, 2026

waleedlatif1 deleted the fix/kb-list-counts-plan branch October 2, 2026 20:23

Conversation

waleedlatif1 commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 2, 2026

Uh oh!

waleedlatif1 commented Oct 2, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 2, 2026

Uh oh!

waleedlatif1 commented Oct 2, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 2, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 2, 2026

Uh oh!

waleedlatif1 commented Oct 2, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

icecrasher321 commented Oct 2, 2026

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

waleedlatif1 commented Oct 2, 2026 •

edited

Loading

vercel Bot commented Oct 2, 2026 •

edited

Loading

greptile-apps Bot commented Oct 2, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading