Skip to content

feat(knowledge): query-time ACL + revocation isolation in the shared KB retrieval chain - #11219

Closed
bolichen97 wants to merge 1 commit into
mainfrom
feat/shared-kb-query-acl
Closed

bolichen97 wants to merge 1 commit into
mainfrom
feat/shared-kb-query-acl

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Query-time ACL and revocation isolation wired into the existing shared Knowledge Library retrieval chain — the query-time half of the Connector Production Stack's shared KB/ACL contracts (KB-01 / ACL-02 / ACL-03 / ACL-05 / ACL-06 / ACL-09 / KB-10). No second KB, no types-only framework: it wires into the existing KnowledgeStore / HybridRetriever and the real query call boundaries.

HybridRetriever.search's source_id/namespace filters are relevance labels ("NOT a security boundary"); the graph leg was unfiltered; the store had no principal/tenant/grant state — so a KB that ingests shared/managed sources could leak one user's content to another at query time.

The gate

Item-scoped, provenance-classified, fail-closed. One KnowledgeStore mixes trusted-local material with managed cloud/structured items, so the gate decides per item, from EVIDENCE, not from the entry point or a source_type name:

  • sources.trust_class (local_admitted | managed) is written once at creation from the creating path's own knowledge (initial_trust_class): the real on-host local creators (local_folder, local_file, obsidian_vault, artifact, agent, doc) stamp local_admitted; every remote connector / unknown / misspelled type defaults to managed. A one-time migration adds the column (default managed) and backfills local_admitted onto the local-creator types.
  • acl.is_managed_trust_class(has_source, trust_class): sourceless ⇒ managed (absence of a source is not proof of no external ACL); local_admitted ⇒ local; managed / unstamped / unknown / dangling (missing source row ⇒ None) ⇒ managed. The query-time gate reads the STAMP only — it never re-guesses trust from a source_type string, so a new/misspelled cloud type, or a managed source whose per-item grant forgot managed=True, cannot be waved through as local.
  • acl.LOCAL_LIBRARY (the local single-user context) sees trusted-local items without a grant but is denied managed items — it carries no verifiable provider-mapped subject. A managed item is always checked against a real current subject/tenant and revalidated, regardless of entry point.

Revalidation / staleness chain (managed items). A static ingest grant + a local acl_version bump is not proof the provider still grants access:

  • acl.RevalidationHook (the provider live-permission interface — a named dependency, not implemented here) + RevalidationOutcome{FRESH,REVOKED,UNVERIFIABLE}. A managed grant is served only when the hook says FRESH OR the stored fresh_as_of is within the staleness window. REVOKED denies even a subject-matching grant; a wired hook returning/raising UNVERIFIABLE is a hard deny (no stamp fall-back); with no hook wired a stale/never-revalidated managed grant is denied. fresh_as_of is meaningful only as a trusted confirmation record (set_item_acl / mark_item_acl_revalidated) — a manual mark / fresh flag is not treated as a completed real-time revocation check, and no other session/binding identity stands in for the current querying subject.
  • store: item_acl gains managed + fresh_as_of; revoke_item_acl resets fresh_as_of + bumps acl_version; mark_item_acl_revalidated records a provider confirmation or observed revocation; grant rows are removed with their item (KB-10).

Every leg gated. search(access_context=...) filters every fused candidate — keyword + vector + the unfiltered graph leg + the protected keyword rescue + citation/location enrichment — before the result window is cut.

Non-search reads + export gated too. The gate is applied to every read that returns item content, not just search: the browse listing (ACL-filtered BEFORE pagination so hidden counts do not leak and visible rows are not pushed off later pages), item detail / content / related / entity-items / per-item export (denied ⇒ 404, so existence does not leak), and the full-library export (bundle items AND the sources/entities/relations/locations reachable only from hidden items are stripped, unconditionally, so an empty visible set cannot leak the rest).

Import trust invariant. An untrusted .knowledge bundle's claim, missing field, or naming an existing source id NEVER produces local trust or a non-managed authorization: an imported source is local only if the same id already exists here as local_admitted; every newly-inserted imported item gets a forced managed grant regardless of the source it names; a bundle-provided item_acl is forced managed (explicit managed=0 ignored) and is restored ONLY for items this import newly inserts (a pre-existing item's grant is never overwritten); malformed entries are skipped, not crashed.

Ingest half (per-row contract). knowledge/rows.py (SourceRow) + IngestionPipeline.ingest_rows + the connector_row_state ledger + BaseConnector.supports_rows/fetch_rows carry each structured row's own identity + content + ACL grant to its item group, writing the grant and the ledger (and the prior group's delete) in one transaction. Its consumers are the vendor connector PRs (Google Drive supports_rows→True; GitHub _source_row_from→SourceRow).

Call boundaries. mcp_tools/knowledge.py + dashboard/handlers/knowledge.py pass LOCAL_LIBRARY (they serve the on-host personal library; managed items mixed in are still gated/denied there). A shared/multi-tenant caller resolves subject+tenant from the authenticated caller — for a managed item, the provider-mapped identity (W01 Binding.subject_ref/tenant_ref from the verifier, NOT a raw session key and NOT L08's operation-scope handle) — and wires a RevalidationHook.

Testing

test/test_knowledge_query_acl.py drives the real store + gate: provenance classification (sourceless / local_admitted / managed / unknown / dangling), a MIXED library (local doc readable, managed cloud doc gated), adversarial cases (a brand-new unknown cloud type is gated; a managed source whose grant omitted managed=True is still gated and revoke-denied), two identities, cross-tenant, graph-leg leakage, and the full revalidation chain (fresh/revoked/error, staleness, mark-revalidated), plus migration backfill. 1004 knowledge tests pass in a clean source-tree-external venv; end-to-end provenance + unknown-type + migration validation from that venv; flake8 clean.

Not in scope / untouched

This PR ships the query-time gate PLUS the per-row ingest contract it governs, so it does edit knowledge/connectors/base.py (adds supports_rows/fetch_rows) and adds knowledge/rows.py + ingest_rows. It does NOT touch connections/control_plane/** or vendor connector dirs, and does not implement a vendor connector. The provider live-revalidation hook and the provider identity mapping (W01 binding) are named, fail-closed dependencies — declared, not faked. Per-provider live cloud verification, and a verifiable one-step local-backup restore (imported bundles are managed until re-admitted locally), remain separate required items.

@bolichen97
bolichen97 requested review from a team and iamwhatever September 16, 2026 06:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 16, 2026
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of f97873ce9f4ced82549a902b6cc8e198637f6d27 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Existing connector-ingested content goes dark on upgrade — the deny ships before any hook, binding, or re-admission path exists to ever show it again.

Watch

  • The migration stamps every non-local-creator source managed, both call boundaries pass LOCAL_LIBRARY, and LOCAL_LIBRARY denies managed items — while the RevalidationHook and W01 binding resolver are "declared, not faked" and the local re-admission/restore path "remain[s] separate required items". So a library holding edition/pluggable-connector content (the base code supports arbitrary sync_scheduler.get_connector(source_type) types, plus legacy quip rows) silently loses it from search, browse, detail, and export on upgrade, with no in-product remedy in this PR. Fail-closed is the right posture for the named multi-human/shared-KB boundary, but shipping the deny N PRs ahead of the first path that can serve a managed item is a sequencing call a human should explicitly accept.
    Clears when: the re-admission/restore item (or the W01 resolver + hook wiring) lands or is committed to in the same milestone, or a maintainer accepts the interim dark window in writing.
  • The cross-PR contract this creates — trust_class provenance, RevalidationHook/RevalidationOutcome, the W01 BindingResolver seam, SourceRow/ingest_rows, and the KB-01/ACL-02..09 IDs — lives only in docstrings and the PR description; docs/system-specs/modules/knowledge.md is untouched and the ACL-* IDs appear nowhere under docs/. The vendor-connector PRs named as consumers will implement against a contract recoverable only from a merged PR body, and AGENTS.md requires the owning spec updated in the same commit.
    Clears when: docs/system-specs/modules/knowledge.md (or a new spec the docs index names) documents the trust-class/grant/revalidation/row contracts in this PR.

[DESIGN-REVIEWED] f97873c

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed f97873ce9f4ced82549a902b6cc8e198637f6d27 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f97873c

Verdict parsed from the review's SHA-scoped output markers for commit f97873ce9f4ced82549a902b6cc8e198637f6d27.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f97873ce9f4ced82549a902b6cc8e198637f6d27: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🔴 BLOCK

Premise-level review of f97873ce9f4ced82549a902b6cc8e198637f6d27 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: BLOCK

Imported libraries become invisible with the restore path "not invented here" and no RFC; plus rows/vendor/resolver plumbing whose only consumers are unlanded PRs.

Not justified as shipped

  • Item 4 — product-shape, no recorded decision: every newly-inserted imported item gets a deny-all managed grant; the diff's own comment: "A one-step local restore … does not exist today and is not invented here."
  • Item 5 — rides along: export drops items the caller cannot see, so export→import→export now loses the imported items.
  • Item 6 — rides along, zero consumers: per-row ingest contract unreachable — no connector returns supports_rows() True.
  • Item 7 — undeclared, zero consumers: registers github_structured/google_drive/salesforce_structured; 0 of 3 modules exist under knowledge/connectors/.
  • Item 8 — zero consumers: _install_binding_resolver imports control_plane.acl_binding_resolver/lifecycle; neither file exists, so it always fails.
  • Item 9 — zero consumers: revoke_item_acl/mark_item_acl_revalidated/RevalidationHook are definitions only in src/ (grepped; callers are tests).
    Subtraction: defer this write API to the PR that wires a revalidation hook.
    Clears when: a counted production caller exists, or the deletion is made.

What this change ships

Inventory (10 items) — 3 justified

Intent: give the shared knowledge library per-user, provider-checked visibility at query time — framed as a FIX for a cross-user leak, but no base path ingests a shared source (knowledge/connectors/ holds only local_folder), so functionally an ADDITION.

  1. Search (every leg, incl. graph) now drops managed items lacking a checked grant — justified
  2. Browse/detail/content/related/export deny hidden items as 404/absent, pre-pagination — justified
  3. New sources.trust_class stamp; migration backfills local-creator types — justified
  4. Imported bundle items become invisible everywhere; no re-admission step exists — product-shape, no recorded decision
  5. Full-library export silently strips unseen items and their closure — rides along
  6. Per-row structured-ingest contract (rows.py, ingest_rows, connector_row_state, sync branch) — rides along, zero consumers
  7. Registration site for three nonexistent vendor connector modules + knowledge_connector_runners key — undeclared, zero consumers
  8. Lazy binding-resolver assembly over modules absent from the tree — zero consumers
  9. Revocation/revalidation write API with no production caller or hook implementation — zero consumers
  10. Touched files fully reformatted (black-baseline removal) — rides along

Blockers

  • product-shape change without accepted RFC — backup/import replaced by import-into-invisibility. Diff: forced managed=1, fresh_as_of=0 grants on all newly-inserted imported items; LOCAL_LIBRARY denies managed; no RevalidationHook exists in the tree. Description concedes the restore step "remain[s] separate required items." The base RFC list names no knowledge/KB/ACL RFC.
    Clears when: an RFC covering the import-trust/restore shape is merged non-draft on base (then push or rebase), or /ai-review override first-principles f97873ce9f4ced82549a902b6cc8e198637f6d27: <decision> from a maintainer.
  • Riders the fix is already complete without (items 6–8). The gate denies managed items with none of these present. Counts: 0 of 3 registered vendor modules exist; no connector returns supports_rows() True (grepped src/); 0 producers for knowledge_connector_runners/knowledge_revalidator/knowledge_identity_resolver/knowledge_query_principal; connections/control_plane/ lacks both imported modules. Zero option: nothing observable changes.
    Subtraction: delete the registration loop and _install_binding_resolver; defer rows.py/ingest_rows/_sync_rows to the vendor/W01 PRs that consume them.
    Clears when: the deletions/deferrals are made, or a consuming module lands on base.

[FIRST-PRINCIPLES-REVIEWED] f97873c

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging f97873ce9f4ced82549a902b6cc8e198637f6d27. 2 of 3 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- test/test_knowledge_acl_w01_integration.py:28 -- Integration suites silently skip when the resolver is absent
pytest.importorskip("kiro_crew.connections.control_plane.acl_binding_resolver")
Missing resolver -> both W01 integration modules skip collection -> ACL integration assertions never execute.
Anchor: a-ratchet-may-only-tighten
Fix: Remove both importorskip calls and provide the tested dependency.

BLOCKING -- src/kiro_crew/knowledge/rows.py:65 -- Blank rows are checkpointed without content
text: str
Blank text -> ingest_rows -> zero chunks -> empty ledger entry persists and checkpoint advances -> row is permanently omitted.
Anchor: residual/crash-data-loss-corruption
Fix: Reject blank or whitespace-only text in SourceRow.__post_init__.

BLOCKING -- src/kiro_crew/dashboard/handlers/knowledge.py:1173 -- Related-item lookup does not authorize its subject
visible = await _acl_visible_ids(request, [it.get("id") for it in items])
Known revoked item ID -> related route derives its entities -> authorized related documents and overlap counts disclose the hidden item’s associations.
Anchor: residual/security
Fix: Return 404 unless _acl_can_see(request, item_id) succeeds before _related_items.

[BLOCK-MERGE] f97873c
[GPT-REVIEWED] f97873c

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

F1 opens at test/test_knowledge_acl_w01_integration.py:28 — a newly added importorskip (a collection-time skip) in a test file the PR adds, anchored to a-ratchet-may-only-tighten, which carries blocking: true in AUTOSDE.yaml:717 and whose FLAG list explicitly includes "a newly added skip/xfail". The rule's flag is authoritative and outranks the weighing.

F2 (fenced): SourceRow accepts blank text; ingest_rows (ingestion.py:1440) hashes it and, on a zero-chunk ingest, can advance the ledger checkpoint (ingestion.py:1613) so the row is treated as persisted and never re-ingested. Blank-text rows (e.g. an issue with an empty body) are ordinary, not an extreme/contradicting condition — the rarity argument required for FLAG cannot be made. UPHOLD-FENCED.

F3 (fenced): get_related_items (knowledge.py:1162) derives entities from item_id via _related_items (knowledge.py:1130) with no _acl_can_see check on the subject before disclosing authorized related documents and overlap counts. The precondition — a caller holding a revoked/hidden item ID — is plausible in ordinary operation (IDs enumerable or previously visible), not extreme; no rarity argument available. UPHOLD-FENCED.

[ADJUDICATION] f97873ce9f4ced82549a902b6cc8e198637f6d27 total=1 uphold=1 downgrade=0
UPHOLD F1 test/test_knowledge_acl_w01_integration.py:28 reason=autosde-blocking-rule
[GPT-ADJUDICATED] f97873ce9f4ced82549a902b6cc8e198637f6d27
[ADJUDICATION-FENCED] f97873ce9f4ced82549a902b6cc8e198637f6d27 fenced=2 flagged=0
UPHOLD-FENCED F2 src/kiro_crew/knowledge/rows.py:65 -- Blank-text rows are ordinary connector output (empty issue body), so the silent-omission condition is common, not an extreme case a human should waive.
UPHOLD-FENCED F3 src/kiro_crew/dashboard/handlers/knowledge.py:1173 -- A caller holding a revoked/enumerable item ID is a plausible real-operation precondition, so the cross-item association disclosure is not a rare-enough residual to pre-clear.
[GPT-ADJUDICATED-FENCED] f97873ce9f4ced82549a902b6cc8e198637f6d27

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f97873ce9f4ced82549a902b6cc8e198637f6d27: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 16, 2026
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision f97873ce9f4ced82549a902b6cc8e198637f6d27 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
bolichen97 added a commit to bolichen97/KiroCrew that referenced this pull request Sep 16, 2026
bolichen97 added a commit to bolichen97/KiroCrew that referenced this pull request Sep 16, 2026
…ntract

Integrated chat408 ACL row API f5bc46a (PR kirodotdev#11219) by normal git merge and
wired the per-row ingest contract. Verified the contract against the f5bc46a
body -- ONE discrepancy from the relayed summary: the SourceRow field is
`resource_ref` (an acl.ProviderResourceRef), not `ref`; managed is a read-only
property fixed True (not a passable field). Everything else matches.

- supports_rows() -> True: Salesforce is structured, so the sync scheduler
  drives fetch_rows -> ingest_rows -> per-row set_item_acl, never the single
  text blob that would collapse many rows' grants into one chunk.
- fetch_rows(source) -> (rows, snapshot, checkpoint): each SourceRow carries its
  OWN key/text/resource_ref/grant. subjects is the EMPTY tuple -- a fail-closed
  deny-all, NEVER acl.PUBLIC_SUBJECT and never a blank default: Salesforce
  per-record sharing is established at query time by the shared revalidation
  hook, not inferred here. tenant is the vendor org id (non-empty).
- Snapshot semantics kept SEPARATE per path: SOQL object path is incremental
  (snapshot=False, since-watermark checkpoint); Report path is a full snapshot
  (snapshot=True, no watermark). Neither reuses the other's refresh strategy.
- Refactored the live read into _collect_rows shared by fetch_rows and the
  legacy fetch (kept for the abstract BaseConnector.fetch).
- props['checkpoint'] is READ as the "ACL landed" signal (shared sync writes it
  only after full persistence); this connector never writes it.

Real SyncScheduler ingest proof (TestSyncSchedulerRealIngest) drives the WHOLE
chain over real TLS: loopback TLS server -> SalesforceProductionRunner -> real
W01 executor -> fetch_rows -> SyncScheduler.sync_source -> ingest_rows ->
store.set_item_acl per row. Reads back store.get_item_grants and asserts each
grant is managed=True, subjects==[] (explicit deny, not public), carries the
row's own salesforce ProviderResourceRef, and that props['checkpoint'] advanced
to the max SystemModstamp only on full success. Both SOQL (2 real TLS pages) and
Report (full snapshot) paths proven. 127 tests pass; fresh make backend + fresh
venv green (57). black/isort/flake8/pyflakes clean. No shared hunk edited.
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 16, 2026
@bolichen97
bolichen97 force-pushed the feat/shared-kb-query-acl branch 4 times, most recently from de20850 to 36bb7bd Compare September 16, 2026 22:42
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 16, 2026
@bolichen97
bolichen97 force-pushed the feat/shared-kb-query-acl branch from 36bb7bd to 38174bc Compare September 16, 2026 22:56
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 16, 2026
…d KB retrieval chain

Adds per-item ACL to the knowledge retrieval gate: a managed cloud/structured
item is served only to a query principal whose per-candidate provider binding
resolves and revalidates; trusted-local items keep serving the on-host personal
library. Provenance-based classification (sourceless and unstamped => managed,
fail-closed), an unforgeable trusted-creator stamp, and provenance that survives
delete, dedup and bundle round-trip.

Per-row structured ingest (SourceRow + connector_row_state + ingest_rows/sync)
carries each row key, text, ProviderResourceRef and grant to its own item, with
the checkpoint advancing only after full persistence. The shared dashboard
handler registers the GitHub/Google/Salesforce connectors import-guarded and
injects each vendor runner factory by the connector own keyword.

Query principal is derived only from the auth middleware established identity
(request app/user/is_dashboard_user), never a raw header; an authenticated app
or the dashboard owner becomes a stable verified principal, and no established
identity denies. The W01 AccessGrant is bridged into an AccessContext, and the
verified-principal marker (QueryPrincipal.verified + principal_is_verified)
supplies the trust contract W01 ControlPlaneBindingResolver requires.
@bolichen97

Copy link
Copy Markdown
Contributor Author

暂停关闭(Connector Production Stack 项目于 2026-09-20 整体暂停)

  • 所属工作流:KB
  • 关闭时状态:需 rebase(dirty)+ 红检查:Backend Tests、Coverage Gate、E2E offline、First Principles Review、GPT 5.6 Review。
  • 分支 feat/shared-kb-query-acl 未删除,head f97873ce9f4c 保留。
  • 恢复方式:gh pr reopen 11219;若 reopen 被拒(base 变动/分支落后),基于同一分支对 main 新开 PR 即可。
  • 交接文档:connector-campaign-20260914/wind-down/HANDOVER-PROMPT-20260920.md

本次关闭不代表该工作被否决,只是项目暂停收口。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant