Repository navigation
feat(connections): W01 L09 control-plane executor, real transport, neutral data channel - #11286
bolichen97 wants to merge 3 commits into
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Half the diff — the #11349 Google Drive/knowledge-ACL squash at HEAD — is outside this PR's described scope, and the body still says HEAD is Watch
[DESIGN-REVIEWED] 1d2300a |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS An undeclared second feature — the Google Drive knowledge source + query-time ACL (head commit Not justified as shipped
What this change shipsInventory (10 items) — 5 justifiedIntent: run connector operations through the governed control plane (four judgments, live-store credential, real HTTPS) and deliver payload data — ADDITION. Note: the workflow-supplied patch conflates ~970 files of already-merged main history; the branch's real delta vs merge-base
[FIRST-PRINCIPLES-REVIEWED] 1d2300a |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/knowledge/sync.py:65 -- Successful structured syncs fail during checkpoint advancement (origin: validation) BLOCKING -- src/kiro_crew/knowledge/connectors/google_drive.py:294 -- Failed content reads become authoritative snapshot omissions BLOCKING -- src/kiro_crew/connections/vendors/google_drive/decode.py:71 -- Malformed collections become authoritative empty snapshots BLOCKING -- src/kiro_crew/dashboard/handlers/knowledge.py:3157 -- Optional connectors are instantiated on the gateway boot path (origin: validation) [BLOCK-MERGE] 1d2300a Adjudication (Opus 4.8) — is blocking on each finding proportionate?F4 is anchored to Fenced notes: F1 — harm: F1/F2/F3 unbounded (crash-data-loss-corruption); F4 governed by blocking rule. False positive or not applicable? A repository writer can comment: |
423528c to
bbb75ee
Compare
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsNo blocking findings; one advisory: an unbounded in-memory issuance registry with no eviction path. FINDING — src/kiro_crew/connections/control_plane/handle.py:400 — [OPUS-REVIEWED] 1d2300a Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
… rows fetched through W01's executor/transport Locate a GitHub operation, dispatch it through W01's real executor and production transport, decode the reply into the neutral payload channel, and page a real structured fetch across two pages — with no naked sender anywhere. Wire the knowledge GitHub structured source's live path onto that transport so a real fetch returns real rows and a real refresh runs. ## What changed Three new leaves under vendors/github/, plus the knowledge connector wiring and docs, all consume-only over the W01 control plane (executor.py + production.py at 423528c; schema versions EXECUTOR=4 / PRODUCTION=3 / RESULT=3 / OPERATION=2, pinned and asserted at compose time): - vendors/github/locator.py: operation_id + params -> one concrete HttpRequest, GitHub's implementation of the transport's injected RequestLocator (mirrors the Graph locator). Endpoint templates come from the existing DESCRIPTORS table; path segments are individually validated and URL-encoded; it holds no credential. - vendors/github/decoder.py: a GitHub 2xx payload -> OperationResult, GitHub's ResultDecode. It puts the fetched records into the ONE neutral data channel — a CollectionPayload for a list reply (JSON array, wrapped collection, or GraphQL nodes/edges), an ObjectPayload for a single object — via result_with_payload, and sets the SINGLE authoritative next_cursor on the envelope (the Link rel="next" URL for REST, the GraphQL endCursor for cursor tools). The collection carries no cursor of its own. - vendors/github/dispatch.py: projects a GithubOperationDescriptor into the executor's OperationDescriptor, composes the production transport with GitHub's locator and decoder, and drives execute() and PageWalk/advance_page. The full W01 gate chain and per-binding custody (BindingSecretSelector) run inside the executor. - knowledge/connectors/github_structured.py: PR-3 delta on top of the PR-2 domain model. GithubStructuredConnector takes an optional GithubTransportProvider; fetch/detect_changes drive a REAL W01 PageWalk for the repo-scoped pull-request and commit entities (invoked, authorized per page, the walk advancing on W01's single next_cursor), read the rows off ExecutionOutcome.payload, convert them with PR-2's converters, fold them through PR-2's diff_rows, and render (text, metadata). detect_changes reports changed iff the since-window returned rows. No payload copy, no second envelope, no out-of-band capture, no cache. Built with no provider, fetch/detect_changes still refuse NotImplementedError (PR-2's parked contract, unchanged). No auth, custody, retry, fencing, error classification, or pagination engine is re-implemented — all W01's. Nothing under vendors/github/ or the connector touches urllib / requests / httpx: the sole sender is W01's build_production_transport. ## Paging: a single cursor, and a question The walk advances on OperationResult.next_cursor alone. GitHub REST list endpoints carry their next-page position only in the Link response header, so the vendor decoder reads Link to POPULATE that single next_cursor; the cursor is then single and authoritative on the envelope and nothing downstream re-parses Link (ExecutionOutcome.metadata is a rate-limit-only allowlist). Because a GitHub REST list literally cannot page without Link, this decoder reading is reported as a question to the conductor rather than worked around. ## Scope: issues and check-runs reported, not written The authorized scope addition (issues list + check-runs list) needs new rows in the shared 45-op descriptors.py (owned by the W02 vendor-data slice), so the exact rows and official endpoints are reported to the conductor for coordination rather than written here. The wired kinds are pull requests and commits, both already in the table with repo-scoped REST list paths. ## Tests - test_connector_github_dispatch.py (22): the load-bearing test drives a real two-page gh_list_pull_requests fetch through the unmodified urllib_http_send over a self-signed HTTPS loopback, custody from the real encrypted SecretVault, asserting two pages, a genuine cursor advance, and the vault-resolved credential on the wire. Plus locator shaping/refusals, both decoder contracts (now payload-carrying), descriptor projection, denied-gate-sends-nothing, wrong-binding refusal. - test_knowledge_github_structured_live.py: fetch returns REAL rows across >=2 pages over TLS (keyed, rendered, credential on wire); detect_changes is True when the since-window returns rows; no-provider refuses NotImplementedError; a provider with no binding returns an empty-but-valid dataset. ## Incorporated dependency PR-2 (feat/knowledge-github-structured, db7011f) supplies the structured-source domain model — typed rows, primary keys, converters, since+key incremental diff, resumable checkpoint, rendering. It is incorporated below this commit by cherry-pick; PR-2 stays independently parked. This PR rebases to drop that commit once PR-2 lands on main. ## Pattern harvest - A vendor RequestLocator that string-concatenates a caller-supplied id into a URL path without a per-segment separator guard is an injection shape; the rule is "a path template filled from caller input encodes each segment with safe='' and rejects a '/' inside one segment". - A vendor decode that returns next_cursor=None for a body it never inspected claims completeness it cannot know; the rule is "next_cursor is None only on a provider signal (no Link rel=next, or hasNextPage false), never as a default". - A cursor carried in two places (a payload copy AND the envelope) is a second thing that can disagree; the rule is "the continuation lives on OperationResult.next_cursor and nowhere else — a payload struct carries rows, not a cursor" (W01 enforces this by removing CollectionPayload.next_cursor). Non-generalizable: the pinned W01 schema-version integers (4/3/3/2) are a coupling assertion, not a rule — dispatch.assert_schema_versions() makes a future bump a loud compose-time failure. Base is feat/connector-control-plane-executor (PR #11286, a non-main base), so CI cannot fire; local gates plus the repo's review profile are the evidence until the dependency lands on main, at which point this same PR retargets to main.
… rows fetched through W01's executor/transport Locate a GitHub operation, dispatch it through W01's real executor and production transport, decode the reply into the neutral payload channel, and page a real structured fetch across two pages — with no naked sender anywhere. Wire the knowledge GitHub structured source's live path onto that transport so a real fetch returns real rows and a real refresh runs. ## What changed Three new leaves under vendors/github/, plus the knowledge connector wiring and docs, all consume-only over the W01 control plane (executor.py + production.py at 423528c; schema versions EXECUTOR=4 / PRODUCTION=3 / RESULT=3 / OPERATION=2, pinned and asserted at compose time): - vendors/github/locator.py: operation_id + params -> one concrete HttpRequest, GitHub's implementation of the transport's injected RequestLocator (mirrors the Graph locator). Endpoint templates come from the existing DESCRIPTORS table; path segments are individually validated and URL-encoded; it holds no credential. - vendors/github/decoder.py: a GitHub 2xx payload -> OperationResult, GitHub's ResultDecode. It puts the fetched records into the ONE neutral data channel — a CollectionPayload for a list reply (JSON array, wrapped collection, or GraphQL nodes/edges), an ObjectPayload for a single object — via result_with_payload, and sets the SINGLE authoritative next_cursor on the envelope (the Link rel="next" URL for REST, the GraphQL endCursor for cursor tools). The collection carries no cursor of its own. - vendors/github/dispatch.py: projects a GithubOperationDescriptor into the executor's OperationDescriptor, composes the production transport with GitHub's locator and decoder, and drives execute() and PageWalk/advance_page. The full W01 gate chain and per-binding custody (BindingSecretSelector) run inside the executor. - knowledge/connectors/github_structured.py: PR-3 delta on top of the PR-2 domain model. GithubStructuredConnector takes an optional GithubTransportProvider; fetch/detect_changes drive a REAL W01 PageWalk for the repo-scoped pull-request and commit entities (invoked, authorized per page, the walk advancing on W01's single next_cursor), read the rows off ExecutionOutcome.payload, convert them with PR-2's converters, fold them through PR-2's diff_rows, and render (text, metadata). detect_changes reports changed iff the since-window returned rows. No payload copy, no second envelope, no out-of-band capture, no cache. Built with no provider, fetch/detect_changes still refuse NotImplementedError (PR-2's parked contract, unchanged). No auth, custody, retry, fencing, error classification, or pagination engine is re-implemented — all W01's. Nothing under vendors/github/ or the connector touches urllib / requests / httpx: the sole sender is W01's build_production_transport. ## Paging: a single cursor, and a question The walk advances on OperationResult.next_cursor alone. GitHub REST list endpoints carry their next-page position only in the Link response header, so the vendor decoder reads Link to POPULATE that single next_cursor; the cursor is then single and authoritative on the envelope and nothing downstream re-parses Link (ExecutionOutcome.metadata is a rate-limit-only allowlist). Because a GitHub REST list literally cannot page without Link, this decoder reading is reported as a question to the conductor rather than worked around. ## Scope: issues and check-runs reported, not written The authorized scope addition (issues list + check-runs list) needs new rows in the shared 45-op descriptors.py (owned by the W02 vendor-data slice), so the exact rows and official endpoints are reported to the conductor for coordination rather than written here. The wired kinds are pull requests and commits, both already in the table with repo-scoped REST list paths. ## Tests - test_connector_github_dispatch.py (22): the load-bearing test drives a real two-page gh_list_pull_requests fetch through the unmodified urllib_http_send over a self-signed HTTPS loopback, custody from the real encrypted SecretVault, asserting two pages, a genuine cursor advance, and the vault-resolved credential on the wire. Plus locator shaping/refusals, both decoder contracts (now payload-carrying), descriptor projection, denied-gate-sends-nothing, wrong-binding refusal. - test_knowledge_github_structured_live.py: fetch returns REAL rows across >=2 pages over TLS (keyed, rendered, credential on wire); detect_changes is True when the since-window returns rows; no-provider refuses NotImplementedError; a provider with no binding returns an empty-but-valid dataset. ## Incorporated dependency PR-2 (feat/knowledge-github-structured, db7011f) supplies the structured-source domain model — typed rows, primary keys, converters, since+key incremental diff, resumable checkpoint, rendering. It is incorporated below this commit by cherry-pick; PR-2 stays independently parked. This PR rebases to drop that commit once PR-2 lands on main. ## Pattern harvest - A vendor RequestLocator that string-concatenates a caller-supplied id into a URL path without a per-segment separator guard is an injection shape; the rule is "a path template filled from caller input encodes each segment with safe='' and rejects a '/' inside one segment". - A vendor decode that returns next_cursor=None for a body it never inspected claims completeness it cannot know; the rule is "next_cursor is None only on a provider signal (no Link rel=next, or hasNextPage false), never as a default". - A cursor carried in two places (a payload copy AND the envelope) is a second thing that can disagree; the rule is "the continuation lives on OperationResult.next_cursor and nowhere else — a payload struct carries rows, not a cursor" (W01 enforces this by removing CollectionPayload.next_cursor). Non-generalizable: the pinned W01 schema-version integers (4/3/3/2) are a coupling assertion, not a rule — dispatch.assert_schema_versions() makes a future bump a loud compose-time failure. Base is feat/connector-control-plane-executor (PR #11286, a non-main base), so CI cannot fire; local gates plus the repo's review profile are the evidence until the dependency lands on main, at which point this same PR retargets to main.
…urrent main Re-based onto current main (6dd9db1) after #11066 squash-merged the L01 control-plane seam + W02 GitHub + W05 Graph into main and deleted the feat/connector-control-plane-seam branch. GitHub auto-retargeted PR #11286 to main, which then read dirty against the stale seam base. How the alignment was decided (content/patch analysis, not commit counts) - merge-base(prev-head, main) = a91a25f; `git cherry -v` marked all four of the old stacked commits `+` because #11066 was a SQUASH merge (different patch-id), so cherry/rebase auto-drop does not apply -- content analysis was required. - Compared this stack's tree against origin/main file-by-file in the connector scope. operation.py / context.py / errors.py are BYTE-IDENTICAL to main (the L01 seam) and untouched here. The W02 (vendors/github) and W05 (vendors/microsoft) trees are byte-identical to main and are NOT reproposed. - The four files that "differ" from main -- control_plane/__init__.py, result.py, test_connections_control_plane.py, connections.md -- were each verified main == seam(2140476), i.e. main carries exactly the seam base this slice's versions were built on, with zero further main-side edits. So this slice's versions are strict supersets and apply as a clean fast-forward, no 3-way conflict. - Everything else the stale base showed as "changed" (warm.py, .github/**, AGENTS.md, docs/**, conftest.py, ...) was main's own 108-commit advance inverted by the stale base -- NOT this slice's work -- and is deliberately left at main's version (e.g. warm.py keeps main's newer PID-reuse hardening). Delivered on top of main (the dependency slices this L09 work imports, none yet in main; each preserved from its own source): - binding.py (L02) - lifecycle.py (L04, VERBATIM from f174887fc; byte-identical, sha256 90b4931a...), plus the connective BindingResolutionError class added to binding.py so lifecycle.py can import it - auth_modes.py (L05) - policy.py (L06) - writes.py (L07) - handle.py (L08) - executor.py, production.py + the neutral data channel additions to result.py / control_plane __init__.py + connections.md (L09, this slice) - test_connections_control_plane*.py suites L09 send path (this slice's core work) - build_production_transport takes store: BindingStore + gate: BindingCustodyGate (was selector: BindingSecretSelector). Per call: (1) gate custody match refuses before locator/store/vault; (2) locator; (3) store.select_secret(binding, reader=vault) FENCES against the LIVE store (revoked / stale-or-forged generation / swapped identity/mode/secret_ref refuse) and takes secret_ref from the STORE's record. binding_secret_ref is neither imported nor called in production.py (AST-checked). A fence refusal / resolution failure emits ZERO calls. PageWalk re-runs the whole chain per page. Evidence (real TLS server + real on-disk SecretVault + real BindingStore) - two bindings -> two different secrets on the wire; B's identity refused with 0 calls; revoke -> next send 0 calls; generation rotation -> 0 calls; no slug fallback (AST + absent-entry refusal); PageWalk revoke between page 1 and 2 -> page 2 emits 0 calls, cursor path never requested. Mutation-verified: the revoked flag and the generation check are redundant load-bearing fences. Invariants (all green): four judgments before dispatch, deny -> 0 calls; routing only via TrustedHandleView; 412 structured, no fabricated precondition; 500/timeout/exception -> write_outcome unknown; redirects per-hop with cross-origin credential strip and no https downgrade; response metadata allowlisted (no credential/session headers); single authoritative cursor OperationResult.next_cursor; three payload shapes, Office bytes byte-equal. Schema: EXECUTOR_SCHEMA_VERSION=4, PRODUCTION_SCHEMA_VERSION=4, RESULT_SCHEMA_VERSION=3, LIFECYCLE_SCHEMA_VERSION=1. Named in-flight boundaries (NOT implemented, not claimed): L03 (auth code/PKCE/state/callback replay protection) is still required and unimplemented. The principal->binding ACL adapter (BindingResolver.resolve -> AccessContext, whose signature differs from lifecycle.BindingStore.resolve) is L04's follow-up, not wired here. MS 412/readback/locator semantics (vendors/microsoft/**) and document/provider-level query ACL remain other owners' work. Gates: 295 passed; black/flake8/isort/mypy clean.
0591b97 to
1648430
Compare
…ctors
The structured vendor connectors (GitHub/Google/Salesforce) are NOT constructed
no-arg: each takes an injected runner/factory (Google operations_factory, SF
call_runner) that composes the W01 control-plane executor with per-source
credential custody + real transport (controlled TLS). Constructed without it the
connector's own validate/fetch refuse fail-closed, so a no-arg registration
would be a dead, misleading entry.
_register_optional_connector now takes runner_factory and constructs the
connector WITH it; setup_knowledge_routes reads a {source_type: runner_factory}
map from app['knowledge_connector_runners'] (installed by the host once the W01
executor, PR #11286, is available) and registers a vendor ONLY when its module
imports AND its runner is installed. Absent module or absent runner => not
registered (fail-closed, never public). No vendor code copied.
Adds an end-to-end test: the handler seam constructs a connector WITH an
injected runner and the SAME instance drives the real SyncScheduler ->
fetch_rows -> ingest_rows -> store -> ACL query chain (scripted runner over the
documented offline path; live W01 TLS is PR #11286's executor, not on main).
4d050bd to
af3f04c
Compare
… rows fetched through W01's executor/transport Locate a GitHub operation, dispatch it through W01's real executor and production transport, decode the reply into the neutral payload channel, and page a real structured fetch across two pages — with no naked sender anywhere. Wire the knowledge GitHub structured source's live path onto that transport so a real fetch returns real rows and a real refresh runs. ## What changed Three new leaves under vendors/github/, plus the knowledge connector wiring and docs, all consume-only over the W01 control plane (executor.py + production.py at 423528c; schema versions EXECUTOR=4 / PRODUCTION=3 / RESULT=3 / OPERATION=2, pinned and asserted at compose time): - vendors/github/locator.py: operation_id + params -> one concrete HttpRequest, GitHub's implementation of the transport's injected RequestLocator (mirrors the Graph locator). Endpoint templates come from the existing DESCRIPTORS table; path segments are individually validated and URL-encoded; it holds no credential. - vendors/github/decoder.py: a GitHub 2xx payload -> OperationResult, GitHub's ResultDecode. It puts the fetched records into the ONE neutral data channel — a CollectionPayload for a list reply (JSON array, wrapped collection, or GraphQL nodes/edges), an ObjectPayload for a single object — via result_with_payload, and sets the SINGLE authoritative next_cursor on the envelope (the Link rel="next" URL for REST, the GraphQL endCursor for cursor tools). The collection carries no cursor of its own. - vendors/github/dispatch.py: projects a GithubOperationDescriptor into the executor's OperationDescriptor, composes the production transport with GitHub's locator and decoder, and drives execute() and PageWalk/advance_page. The full W01 gate chain and per-binding custody (BindingSecretSelector) run inside the executor. - knowledge/connectors/github_structured.py: PR-3 delta on top of the PR-2 domain model. GithubStructuredConnector takes an optional GithubTransportProvider; fetch/detect_changes drive a REAL W01 PageWalk for the repo-scoped pull-request and commit entities (invoked, authorized per page, the walk advancing on W01's single next_cursor), read the rows off ExecutionOutcome.payload, convert them with PR-2's converters, fold them through PR-2's diff_rows, and render (text, metadata). detect_changes reports changed iff the since-window returned rows. No payload copy, no second envelope, no out-of-band capture, no cache. Built with no provider, fetch/detect_changes still refuse NotImplementedError (PR-2's parked contract, unchanged). No auth, custody, retry, fencing, error classification, or pagination engine is re-implemented — all W01's. Nothing under vendors/github/ or the connector touches urllib / requests / httpx: the sole sender is W01's build_production_transport. ## Paging: a single cursor, and a question The walk advances on OperationResult.next_cursor alone. GitHub REST list endpoints carry their next-page position only in the Link response header, so the vendor decoder reads Link to POPULATE that single next_cursor; the cursor is then single and authoritative on the envelope and nothing downstream re-parses Link (ExecutionOutcome.metadata is a rate-limit-only allowlist). Because a GitHub REST list literally cannot page without Link, this decoder reading is reported as a question to the conductor rather than worked around. ## Scope: issues and check-runs reported, not written The authorized scope addition (issues list + check-runs list) needs new rows in the shared 45-op descriptors.py (owned by the W02 vendor-data slice), so the exact rows and official endpoints are reported to the conductor for coordination rather than written here. The wired kinds are pull requests and commits, both already in the table with repo-scoped REST list paths. ## Tests - test_connector_github_dispatch.py (22): the load-bearing test drives a real two-page gh_list_pull_requests fetch through the unmodified urllib_http_send over a self-signed HTTPS loopback, custody from the real encrypted SecretVault, asserting two pages, a genuine cursor advance, and the vault-resolved credential on the wire. Plus locator shaping/refusals, both decoder contracts (now payload-carrying), descriptor projection, denied-gate-sends-nothing, wrong-binding refusal. - test_knowledge_github_structured_live.py: fetch returns REAL rows across >=2 pages over TLS (keyed, rendered, credential on wire); detect_changes is True when the since-window returns rows; no-provider refuses NotImplementedError; a provider with no binding returns an empty-but-valid dataset. ## Incorporated dependency PR-2 (feat/knowledge-github-structured, db7011f) supplies the structured-source domain model — typed rows, primary keys, converters, since+key incremental diff, resumable checkpoint, rendering. It is incorporated below this commit by cherry-pick; PR-2 stays independently parked. This PR rebases to drop that commit once PR-2 lands on main. ## Pattern harvest - A vendor RequestLocator that string-concatenates a caller-supplied id into a URL path without a per-segment separator guard is an injection shape; the rule is "a path template filled from caller input encodes each segment with safe='' and rejects a '/' inside one segment". - A vendor decode that returns next_cursor=None for a body it never inspected claims completeness it cannot know; the rule is "next_cursor is None only on a provider signal (no Link rel=next, or hasNextPage false), never as a default". - A cursor carried in two places (a payload copy AND the envelope) is a second thing that can disagree; the rule is "the continuation lives on OperationResult.next_cursor and nowhere else — a payload struct carries rows, not a cursor" (W01 enforces this by removing CollectionPayload.next_cursor). Non-generalizable: the pinned W01 schema-version integers (4/3/3/2) are a coupling assertion, not a rule — dispatch.assert_schema_versions() makes a future bump a loud compose-time failure. Base is feat/connector-control-plane-executor (PR #11286, a non-main base), so CI cannot fire; local gates plus the repo's review profile are the evidence until the dependency lands on main, at which point this same PR retargets to main.
af3f04c to
573cf0f
Compare
573cf0f to
b3efa41
Compare
b3efa41 to
38b6c03
Compare
38b6c03 to
918ee05
Compare
Security Scope Review -- 🟡 CONCERNSWhich legitimate operations does Verdict basis: the reviewer's own judgement; no confirmed regression. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Candidates written. The four tiers the classifier adjudicates are textually unchanged by this diff — the tightening's actual enforcement points are the file-edit write gate and the OS sandbox mount, neither of which a shell-command verdict can reach. Emitting the review. Scope-Verdict: CONCERNS TIGHTENING: A new path fence around the connections control-plane binding store — the top-level ADJUDICATED: 16 candidates, 0 confirmed newly-refused PLATFORM COVERAGE: posix (10 rows) — home-tilde reads of the store and lock, UNADJUDICATED:
[SCOPE-REVIEWED] 7effc4c |
…urrent main Re-based onto current main (6dd9db1) after #11066 squash-merged the L01 control-plane seam + W02 GitHub + W05 Graph into main and deleted the feat/connector-control-plane-seam branch. GitHub auto-retargeted PR #11286 to main, which then read dirty against the stale seam base. How the alignment was decided (content/patch analysis, not commit counts) - merge-base(prev-head, main) = a91a25f; `git cherry -v` marked all four of the old stacked commits `+` because #11066 was a SQUASH merge (different patch-id), so cherry/rebase auto-drop does not apply -- content analysis was required. - Compared this stack's tree against origin/main file-by-file in the connector scope. operation.py / context.py / errors.py are BYTE-IDENTICAL to main (the L01 seam) and untouched here. The W02 (vendors/github) and W05 (vendors/microsoft) trees are byte-identical to main and are NOT reproposed. - The four files that "differ" from main -- control_plane/__init__.py, result.py, test_connections_control_plane.py, connections.md -- were each verified main == seam(2140476), i.e. main carries exactly the seam base this slice's versions were built on, with zero further main-side edits. So this slice's versions are strict supersets and apply as a clean fast-forward, no 3-way conflict. - Everything else the stale base showed as "changed" (warm.py, .github/**, AGENTS.md, docs/**, conftest.py, ...) was main's own 108-commit advance inverted by the stale base -- NOT this slice's work -- and is deliberately left at main's version (e.g. warm.py keeps main's newer PID-reuse hardening). Delivered on top of main (the dependency slices this L09 work imports, none yet in main; each preserved from its own source): - binding.py (L02) - lifecycle.py (L04, VERBATIM from f174887fc; byte-identical, sha256 90b4931a...), plus the connective BindingResolutionError class added to binding.py so lifecycle.py can import it - auth_modes.py (L05) - policy.py (L06) - writes.py (L07) - handle.py (L08) - executor.py, production.py + the neutral data channel additions to result.py / control_plane __init__.py + connections.md (L09, this slice) - test_connections_control_plane*.py suites L09 send path (this slice's core work) - build_production_transport takes store: BindingStore + gate: BindingCustodyGate (was selector: BindingSecretSelector). Per call: (1) gate custody match refuses before locator/store/vault; (2) locator; (3) store.select_secret(binding, reader=vault) FENCES against the LIVE store (revoked / stale-or-forged generation / swapped identity/mode/secret_ref refuse) and takes secret_ref from the STORE's record. binding_secret_ref is neither imported nor called in production.py (AST-checked). A fence refusal / resolution failure emits ZERO calls. PageWalk re-runs the whole chain per page. Evidence (real TLS server + real on-disk SecretVault + real BindingStore) - two bindings -> two different secrets on the wire; B's identity refused with 0 calls; revoke -> next send 0 calls; generation rotation -> 0 calls; no slug fallback (AST + absent-entry refusal); PageWalk revoke between page 1 and 2 -> page 2 emits 0 calls, cursor path never requested. Mutation-verified: the revoked flag and the generation check are redundant load-bearing fences. Invariants (all green): four judgments before dispatch, deny -> 0 calls; routing only via TrustedHandleView; 412 structured, no fabricated precondition; 500/timeout/exception -> write_outcome unknown; redirects per-hop with cross-origin credential strip and no https downgrade; response metadata allowlisted (no credential/session headers); single authoritative cursor OperationResult.next_cursor; three payload shapes, Office bytes byte-equal. Schema: EXECUTOR_SCHEMA_VERSION=4, PRODUCTION_SCHEMA_VERSION=4, RESULT_SCHEMA_VERSION=3, LIFECYCLE_SCHEMA_VERSION=1. Named in-flight boundaries (NOT implemented, not claimed): L03 (auth code/PKCE/state/callback replay protection) is still required and unimplemented. The principal->binding ACL adapter (BindingResolver.resolve -> AccessContext, whose signature differs from lifecycle.BindingStore.resolve) is L04's follow-up, not wired here. MS 412/readback/locator semantics (vendors/microsoft/**) and document/provider-level query ACL remain other owners' work. Gates: 295 passed; black/flake8/isort/mypy clean.
918ee05 to
0929255
Compare
…re seal + GPT round fixes W01 control-plane, on current main (04727eb). Second commit of the 2-commit stack, folding the a8 L04 integration, L08's F3, the F2 store seal, and this round's GPT review-response fixes. GPT 5.6 round on the prior head raised three findings in my code; all addressed: F1 (production.py response_metadata) — a reflected credential no longer has to be RAW to be caught. The kept-value screen now drops a header value containing the outbound credential in ANY form: raw, base64 (standard or url-safe, with or without padding), or percent-encoded (_credential_echo_forms). Closes the FLAGged encoded-echo escape without relaxing the allowlist. Regression + mutation (raw-only screen fails the encoded-echo test). F2 (production.py decode_json_body) — a NON-EMPTY 2xx body that is not valid UTF-8 JSON, or parses to a non-object, is now REFUSED with the new MalformedResponseBodyError instead of collapsing to {}. A silent {} on a 2xx is indistinguishable from a genuine empty body, so a caller read a reported success carrying no data with no retry signal — silent data loss (truncation, 200-served HTML error pages, gzip faults). An EMPTY body is still a real empty result and yields {}. Regression + mutation (revert-to-{} fails the refuse tests). executor.py top-level-imports — `import logging` and `from kiro_crew.sel import SecurityEvent, sel` moved to module scope (kiro_crew.sel is stdlib-only and same-package, so no import-time dependency risk). Only the SEL USE stays guarded, so a runtime SEL fault still soft-fails a decided call. The SEL audit test now patches the executor module's `sel` name (module-top binding). F2 store-seal real-kernel test (test_sandbox_governance_mask.py) — a NEW test spawns the real namespace launcher (sandbox.namespace_argv) and records a child's ACTUAL write outcome: store/lock/temp writes must be denied by the kernel read-only bind while the store stays readable, and a legitimate pre-sandbox backend write works. It SKIPS with the reason when unprivileged user/mount namespaces are unavailable (a launch that never sandboxed is not a write-denial pass), distinguishing env-launch-failure from an effective seal. macOS (Seatbelt text) and Windows (no launcher seal) remain out of scope and are NOT claimed complete. Preserved: F1 deep-frozen carrier (incl. nested secret_ref), 500 in _AMBIGUOUS_TRANSIT_STATUSES, L08 F3 (HandleTamperedError), and all prior tests.
…e rows + sync + query-time ACL (D1) (#11349) * fix(connections): L09 disconnect-unknown + #11286 CI reds (CodeQL, F1/F2/F4, gate hygiene) on current main Combined fix commit above the aligned base 1648430 (W01 L02/L04/L05/L06/L07/L08/L09 control plane re-based onto current main). Folds the server-committed-then-dropped disconnect fix and this round's CI-red resolutions into one commit; the tree is byte-for-byte identical to the pre-squash HEAD 9d3c2efa0. Disconnect -> unknown (production.py): the ambiguous-outcome `except` tuple also names `ConnectionError` and `http.client.HTTPException`, so a raw `http.client.RemoteDisconnected` (a ConnectionResetError AND a BadStatusLine, but NOT a URLError) records `write_outcome=unknown` and the L07 replay gate refuses a blind re-send. Real-TLS + injected-sender regressions; mutation-verified. CodeQL (3 highs, this slice's tests): TLS test servers pin `minimum_version = TLSVersion.TLSv1_2`; the fencing test stops logging the secret value (references the name, asserts it resolved). F2 (GPT BLOCKING, real): an allowlisted response header whose VALUE echoes the outbound bearer would reach the caller via ExecutionOutcome.metadata. The allowlist gates the NAME; `response_metadata` now takes the outbound `sent_credential` and drops any kept value that contains it. Mutation-verified. F4 (GPT BLOCKING, real, test): the two rotation workers run in their own process group (start_new_session), cwd under tmp_path, and are terminated+waited (then killed) in a finally on every exit path. F1 (integration miss, not new work): adopted L02's binding.py VERBATIM from `e8e11aa23` (byte-identical) — `binding_scoped_secret_ref` / `_secret_scope` / `resolve_binding_for_principal`, and `create_binding` records a per-binding scoped secret_ref (retains `BindingResolutionError` that lifecycle.py imports). Test harness reconciled to the per-binding scheme. F3 (lifecycle.py:347) is L04's. Gate hygiene (added lines): Brand Name `KiroCrew`->`Kiro Crew`; 32 history-narration comment markers reworded to present tense across 11 files. Sibling-slice files edited by authorization, COMMENTS ONLY (AST-invariant): lifecycle.py (code AST == f174887fc), auth_modes.py / writes.py (code AST == e8e11aa23), handle test. Integrated slices carried here (not yet in main): L02 binding (e8e11aa23-derived), L04 lifecycle (lifecycle.py verbatim from f174887fc), L05 auth_modes, L06 policy, L07 writes, L08 handle; L09 executor/production/result/data-channel. Schema: RESULT=3, EXECUTOR=4, PRODUCTION=4, LIFECYCLE=1. Gates: 299 passed; black/flake8/isort/mypy clean (rc non-piped). No vendors/**, platform/governance.py, or registry changes. Named in-flight boundary: L03 (auth code/PKCE/state/callback replay protection) still required and unimplemented; F3 belongs to L04. * feat(knowledge): query-time ACL + revocation isolation in the retrieval chain The shared Knowledge Library retriever had no notion of a querying identity: HybridRetriever.search's source_id/namespace filters are relevance labels (their own docstrings say NOT a security boundary), the graph leg was entirely unfiltered, and the store schema carried no principal/tenant/grant state. A KB that ingests shared sources under an admin/service identity could therefore leak one user's content to another at query time. Add a fail-closed query-time ACL gate wired into the real retrieval chain: - knowledge/acl.py: AccessContext (verified subject+tenant, from the authenticated caller, never self-reported) + AclPolicy. SubjectTenantAclPolicy denies a missing/unreadable/revoked grant, a tenant mismatch (same email in a different tenant is a different identity), and a subject not on the grant list; the only allow-all is the explicit local single-user context. - store.py: item_acl table (subjects/tenant/acl_version), set/revoke/batch-read APIs. revoke keeps the row with an empty subject set and bumps acl_version so the next query denies and any version-keyed cache is invalidated; the row is removed with its item so a grant never outlives the content. - retrieval.py: search() gains access_context and gates every fused candidate (keyword + vector + the unfiltered graph leg + the protected keyword rescue + citation/location enrichment) BEFORE the result window is cut. No context => local single-user bypass, preserving the personal-library behaviour. - mcp_tools/knowledge.py + dashboard/handlers/knowledge.py: pass an explicit AccessContext at the call boundary (local library = single-user context); a shared/multi-tenant caller resolves subject+tenant from the authenticated caller and passes a real context. Tests: test_knowledge_query_acl.py drives the real store+gate over two identities, cross-tenant, same-bare-id-different-source, revoke-then-query, cache-invalidation, unreadable-grant fail-closed, graph-leg leakage, and backward-compat. Existing call-site assertions updated for the new kwarg. * fix(knowledge): item-scoped ACL bypass + managed-item revalidation chain Root re-read the first cut and found two boundary holes; fix both before ship. Root cause 1 -- the bypass was call-surface-scoped, not item-scoped. The local single-user context skipped the grant read and admitted EVERY candidate, but one KnowledgeStore mixes trusted-local material with managed cloud/structured items, so a 'personal dashboard' path could still surface a managed item with no check. - acl.py: bypass is no longer 'see everything'. is_managed_source_type() + an explicit MANAGED_SOURCE_TYPES set classify each item; the retriever re-derives managed from the LIVE source (grant flag OR managed type OR a dangling source row = fail-closed), while a sourceless/local item stays trusted-local. LOCAL_LIBRARY (renamed from ALLOW_ALL, kept as an alias) now sees trusted-local items without a grant but is DENIED managed items -- it carries no verifiable provider-mapped subject. A managed item is always checked against a real current subject regardless of entry point. - retrieval._acl_filter: per-item classify (sourceless/local vs managed vs dangling) then gate; the graph leg, keyword rescue and enrichment all consume the filtered list. - store.get_item_source_types now returns (has_source, source_type) so the gate can tell a sourceless local item from a dangling managed one. Root cause 2 -- a static ingest grant + a locally-bumped acl_version is not proof the provider still grants access. Add a revalidation/staleness chain: - acl.RevalidationHook (the provider live-permission interface -- a named DEPENDENCY, not implemented here) + RevalidationOutcome{FRESH,REVOKED, UNVERIFIABLE}. A managed grant is served only when FRESH from the hook OR its stored fresh_as_of is within the staleness window. REVOKED denies even a subject-matching grant; a wired hook returning/raising UNVERIFIABLE is a hard deny (no fall-back to the stored stamp); with NO hook wired a stale/ never-revalidated managed grant is denied. A static grant can never pose as a live check. - store: item_acl gains managed + fresh_as_of; set_item_acl takes both; revoke resets fresh_as_of; new mark_item_acl_revalidated records a provider confirmation (or observed revocation). Migration backfills legacy rows (managed=0/fresh_as_of=0 => a managed legacy grant is stale until refreshed). Tests: test_knowledge_query_acl rewritten to drive the REAL store+gate over a MIXED library (local doc readable, managed cloud doc gated), the sourceless / dangling-source classification, and the full revalidation chain (hook fresh/revoked/error, staleness, mark-revalidated). 818 knowledge tests pass in a clean source-tree-external venv; flake8 clean. W01 note: a managed item's AccessContext subject/tenant must be the PROVIDER-mapped identity (Binding.subject_ref/tenant_ref from the verifier), not a raw session key; the caller boundary resolves it and the revalidation hook is where the provider probe lives. Neither is faked here. * fix(knowledge): classify managed-vs-local from source provenance, not a type list Root's third read found the second cut still fail-open: MANAGED_SOURCE_TYPES was an enumeration and is_managed_source_type only tested membership, with the doc saying 'unknown type => trusted-local'. A new/misspelled cloud source_type, or a managed source whose per-item grant forgot managed=True, was therefore treated as local -- the same hole, relocated from the whole call surface to unenumerated types. The store type list I cited as an 'admitted-local authoritative set' was actually the orphan-reclaim GC rule, not a trust directory. Redo the classification on PROVENANCE evidence, fail-closed by construction: - store: sources gains a trust_class column (local_admitted | managed), written ONCE at creation from the creating path's own knowledge (initial_trust_class), defaulting to MANAGED for anything not a known LOCAL production creator (local_folder/local_file/obsidian_vault/artifact/agent/doc -- the real on-host ingestion creators, not a runtime type guess). A remote connector source is created managed. Migration adds the column defaulting managed and backfills local_admitted onto exactly those local-creator types -- one-time migration semantics, gated on the column being freshly added. - acl: replace is_managed_source_type with is_managed_trust_class(has_source, trust_class). Sourceless => local; local_admitted stamp => local; managed / unstamped / unknown / dangling (missing source row => trust_class None) => managed. The query-time gate reads the STAMP only; it never re-guesses trust from a source_type string, so a new/misspelled cloud type cannot be waved through. The per-item grant's managed flag is still OR'd in as a backstop. - retrieval: _acl_filter classifies from get_item_trust (has_source, trust_class); get_item_grants also returns trust_class. Adversarial regressions added: a brand-new unknown source_type is stamped managed and gated; a managed source whose grant omitted managed=True is still gated (revoke hook denies it); migration backfill verified. 1004 knowledge tests pass in a clean source-tree-external venv; end-to-end provenance + unknown-type + migration validation from that venv; flake8 clean. The revalidation-chain fix from the prior commit stands: fresh_as_of is only meaningful as a trusted confirmation record (set_item_acl/mark_item_acl_revalidated), and with no provider resolver/revalidator wired a managed grant is denied -- the fresh flag/manual mark is NOT treated as a completed real-time revocation check, and no other session/binding identity stands in for the current querying subject. * docs(knowledge): flag sourceless-item trust axis for central review; model sourced fixtures Per Root's brief, the 'no source does not prove no external ACL' axis needs a CENTRAL independent review, not another list tweak. Rather than silently flip sourceless->managed (which regresses the single-user local library and breaks the retriever's sourceless test idiom) or silently keep it, this documents the axis in is_managed_trust_class (sourceless stays trusted-local for now, with the rationale -- no managed connector produces a sourceless item -- and the conservative alternative -- flip to managed + migrate legacy sourceless rows -- spelled out and held for the review), and keeps the keyword_leg fixtures on a local_folder source (models production: every ingested item has a source), correct under either resolution. No behaviour change vs the pushed head for managed items. 795 knowledge tests pass; flake8 clean. * fix(knowledge): sourceless=managed per decision; trusted stamp issuer; provenance survives bundle round-trip Root decided the reviewed axis: 'no source therefore trusted-local' is NOT a shared-query authorization rule. Implement it fail-closed and harden the stamp against forgery + preserve managed provenance across every path. - acl.is_managed_trust_class: sourceless (has_source False) -> MANAGED. Only a source stamped local_admitted is trusted-local. Absence of provenance evidence is not local; the test habit of omitting a source is not production trust. - store.add_source: the local_admitted stamp is issued from the creator TYPE, not a caller-assertable field. An explicit trust_class kwarg may only NARROW to managed; a local_admitted request for a non-local type is refused down to managed. External API input / a source_type rename / a forged kwarg cannot mint local trust -- the stamp does not prove the issuer, the creator type does. - Managed provenance is durable: re-label/dedup that moves items.source_id never rewrites item_acl.managed (no path does), so a managed item reassigned under a local source stays gated. update_source cannot mutate trust_class (not in _SOURCE_COLUMNS). - export_all/import_bundle now carry trust_class AND item_acl grants, so a bundle round-trip does not silently un-gate a managed item. A bundled trust_class is re-checked by issuer (local only for a local creator type); a bundled managed grant imports with fresh_as_of forced to 0 (stale) so it must be re-revalidated on the importing store, never served off the exporter's old confirmation. Tests: sourceless->managed, forged-stamp-refused, managed-survives-reassignment, bundle-round-trip (local stays visible, managed stays stale-denied) added to test_knowledge_query_acl. Retriever-mechanics tests that used sourceless items migrated to a trusted-local source (models production; every ingested item has a source) -- an isolated-fixture test migration, not a live-library change and not a gate relaxation. 1306 knowledge tests pass; flake8 clean; source-tree-external e2e validated. * feat(knowledge): make the provider-revalidation input contract explicit Per Root: form the per-source service / resource-locator / credential-binding inputs the RevalidationHook needs and their call sites, not just 'wait for W01'. Add two named input structures to acl.py: - ProviderResourceRef(provider, account, resource_id, locator): WHICH provider object a managed item came from -- the resource locator a live-permission probe needs. Documents the minimal locator per provider family (sharepoint/onedrive/google_drive/salesforce/github_structured). This is ingest-side provenance the connector owner must persist alongside the grant. - RevalidationRequest(ctx, item_id, grant, resource, credential_ref): the full input a hook needs. ctx MUST be the provider-mapped subject/tenant (W01 Binding.subject_ref/tenant_ref), not a raw session key and not one binding for every source. credential_ref is an OPAQUE handle (vault key / connection id) the host resolves at probe time -- never a raw secret -- and which binding is correct is W01's trusted association for (provider, account, subject). The retriever still calls revalidate(ctx, item_id, grant) (the keys it holds at query time); the hook docstring now spells out that a real implementation looks up the resource ref + credential by item_id to build the full request, and that those two are the named ingest-side + W01 dependencies -- until they exist the hook cannot answer FRESH and every managed grant stays denied (fail-closed). flake8 clean; ACL suite green. * feat(knowledge): wire query-time ACL identity + revalidator seam into dashboard search Root returned the dashboard/handlers/knowledge.py write slot (GitHub owner's registration import+map is saved separately on their branch; not merged/copied here). Turn the two dashboard search call sites from a hardcoded local-library bypass into a real, fail-closed wiring seam: - _knowledge_access_context(request): resolves the query-time AccessContext. Uses an installed app['knowledge_identity_resolver'] (the W01-backed seam that maps the authenticated caller to a provider subject/tenant) when present; otherwise the on-host personal library runs under acl.LOCAL_LIBRARY (trusted-local visible, managed denied). A resolver that returns None or raises fails closed to LOCAL_LIBRARY -- never an exception out of the handler, never a managed-item leak. It never derives a managed identity from a raw dashboard/ session id; that is the resolver's job and its absence keeps managed denied. - _knowledge_revalidator(request): pulls an optional acl.RevalidationHook off app['knowledge_revalidator']; None until a deployment installs one (managed grants then fall back to their staleness stamp -> stale -> denied). - Both threaded into list_items search, _search_until_exhausted, and search_for_context: HybridRetriever(..., revalidator=...) + search(access_context=...). No call site hardcodes the context any more. So a shared/multi-tenant deployment installs the resolver + hook and the same call sites enforce provider-mapped ACL with live revalidation, with zero further handler changes; the personal library keeps working unchanged today. Tests: test_knowledge_query_acl_wiring covers resolver present/absent/None/raises and revalidator present/absent; existing HybridRetriever stubs updated for the new revalidator kwarg. 1313 knowledge tests pass; flake8 clean; source-tree- external e2e shows the seam feeding the resolved context + revalidator into the gate (fresh->visible, revoked->denied, local-lib->managed denied). * feat(knowledge): per-candidate provider binding + persist ProviderResourceRef Root's decision: one request must NOT resolve a single provider identity for the whole library, and persisting the ProviderResourceRef is this shared store's own responsibility (not an external blocker). GitHub is issue/PR/commit/check-run; Graph carries full container/tenant/endpoint. Store (my responsibility, implemented now): - item_acl gains a resource_ref column; set_item_acl(resource_ref=...) accepts a ProviderResourceRef / dict / json str and persists it; get_item_grants returns it; migration adds the column; export_all/import_bundle carry it. This is the real ingest -> grant linkage a candidate item is revalidated through. ACL model: - acl.ProviderResourceRef gains to_json/from_json and provider-accurate locator docs: github {issue|pull{owner,repo,number}, commit{owner,repo,sha}, check_run{owner,repo,check_run_id}} (NOT a path); every MS Graph provider {tenantId, container ids, endpoint}; salesforce {instanceUrl,sobjectType, recordId}; drive/gmail/zoom/slack/asana object keys. - acl.QueryPrincipal (the whole-query authenticated caller) + LOCAL_PRINCIPAL, and acl.BindingResolver.resolve(principal, provider, account) -> AccessContext | None: maps the principal to the binding it holds on ONE candidate's own (provider, account); None => deny (never reuse another provider's binding). Retriever: - HybridRetriever(binding_resolver=...); search(query_principal=...). _acl_filter resolves the managed context PER CANDIDATE from its resource_ref's (provider, account) via the resolver, then revalidates against THAT context. A managed item with no resource_ref (under a resolver), no binding, a wrong-account/other-provider binding, a resolver error, or a local principal is denied. No resolver => direct-context path (unchanged) for tests/simple hosts. Dashboard: adds _knowledge_query_principal + _knowledge_binding_resolver seams (app['knowledge_query_principal'] / ['knowledge_binding_resolver']) threaded into both search call sites alongside the existing revalidator seam. Tests: test_knowledge_query_acl_binding (matching-provider-only visible, wrong-account no leak, missing resource_ref denied, resolver error denied, local principal denies managed, resource_ref round-trips); wiring tests extended. 1324 knowledge tests pass; flake8 clean; source-tree-external e2e validated. Remaining named deps (fail-closed): W01 BindingResolver + QueryPrincipal resolver implementations; per-provider RevalidationHook; connector ingest calling set_item_acl(managed=True, resource_ref=...). * feat(knowledge): generic per-row ingest contract binding row->item/chunk->ACL Root: the fetch()->(text,meta) path collapsed a structured source's rows into one blob whose per-row refs lived only in source.properties, so set_item_acl and meta['rows'] never became real per-row ACL. Persisting the per-row link is this shared store's responsibility, not an external blocker. Add a minimal generic per-row contract in the EXISTING framework (reusing the per-document group pattern the artifact/agent aggregates use), no second sync/runtime, no vendor change: - knowledge/rows.py: SourceRow{key, text, resource_ref: ProviderResourceRef, subjects, tenant, managed, title} -- one fetched row carrying its own identity + content + ACL grant; a managed row MUST carry a resource_ref (else unqueryable). RowResult / RowsIngestOutcome{fully_persisted, deleted_keys}. - store: connector_row_state(source_id,row_key,content_hash,item_ids,status) PK(source_id,row_key), registered in the doc-state tables so dedup/adoption treat it consistently; get_connector_row_state / set_connector_row_state. - ingestion.ingest_rows(rows, source_id, snapshot): each row keys its own item group; unchanged rows (hash match) skipped; a changed row is chunked+extracted on ITS OWN text and its per-user grant + ProviderResourceRef is written in the SAME off-loop finalize unit as the item write (new ingest_text on_items hook), so item, grant and row-ledger advance together. Different-permission rows never share a chunk or a ref. snapshot=True deletes rows the full fetch dropped (items+grant); snapshot=False (incremental) never deletes an absent row. - connectors/base.py: optional supports_rows()/fetch_rows(source) -> (rows, snapshot, checkpoint). sync.py: sync_source takes the rows path when supports_rows() is True (strict, so a MagicMock connector keeps the fetch path), and advances the source CHECKPOINT (connector resume token in properties) ONLY when outcome.fully_persisted -- a partial/failed/cancelled round leaves the checkpoint and re-attempts next sync; old managed grants are never erased on an incremental round. Plain-document connectors unchanged. Tests: test_knowledge_rows_ingest drives the REAL SyncScheduler -> ingest_rows -> store -> HybridRetriever chain: two rows with different permissions each isolated (binding resolver + revalidation), incremental keeps unchanged rows, full snapshot deletes a dropped row (items+grant gone), partial failure does NOT advance the checkpoint, checkpoint advances only on full success, local principal denies ingested managed rows. embedder=None, isolated tmp store. 1330 knowledge tests pass; flake8 clean. Remaining named deps (fail-closed): W01 BindingResolver + QueryPrincipal implementations; per-provider RevalidationHook; vendor connectors implementing supports_rows/fetch_rows to emit SourceRow (SF, GitHub PR3) -- consumed without pipeline change. * fix(knowledge): SourceRow never invents a grant from a missing field Root: SourceRow.subjects defaulted to (PUBLIC_SUBJECT,), so a connector that omitted the grant produced a PUBLIC row -- inventing authorization from a missing field, which violates 'unknown must not invent authorization'. A blank tenant or managed=False must not let a cloud row escape managed provenance. - subjects is now REQUIRED (no default). An EMPTY tuple is a valid, explicit deny-all; a MISSING grant is a constructor TypeError, never public. PUBLIC is only reachable by the connector passing acl.PUBLIC_SUBJECT on purpose. - tenant is REQUIRED non-empty: a blank tenant is rejected (a cloud row must be tenant-scoped; PUBLIC_TENANT must be given explicitly with a basis). - managed is now a read-only property that is ALWAYS True -- a SourceRow is by definition a managed cloud/structured row, so a connector cannot pass managed=False to bypass managed provenance/revalidation. (Plain-document connectors use the text path, not SourceRow.) - resource_ref stays required. Tests (centralised negatives, same PR): subjects-required-no-public-default, explicit-empty-subjects-is-deny-all-not-public (ingested + queried, nobody incl. matching-tenant subject and local library sees it), blank-tenant-rejected, missing-resource_ref-rejected, managed-always-true-no-bypass. 1335 knowledge tests pass; flake8 clean. * knowledge: register real GitHub/Google/Salesforce connectors at the shared handler seam Wire the three structured vendor connectors into setup_knowledge_routes' connectors map via an import-guarded registration (_register_optional_connector): each is instantiated and keyed by its own source_type() only when its module is importable. A not-yet-landed vendor module (or one whose transport deps are absent) is simply not registered -- add_source rejects the source_type and the retrieval gate never sees it (fail-closed, never public). No vendor implementation is copied here; only the import + instantiation live in the shared handler, so each vendor auto-activates when its own PR lands on main. * knowledge: inject the W01-backed runner when registering vendor connectors The structured vendor connectors (GitHub/Google/Salesforce) are NOT constructed no-arg: each takes an injected runner/factory (Google operations_factory, SF call_runner) that composes the W01 control-plane executor with per-source credential custody + real transport (controlled TLS). Constructed without it the connector's own validate/fetch refuse fail-closed, so a no-arg registration would be a dead, misleading entry. _register_optional_connector now takes runner_factory and constructs the connector WITH it; setup_knowledge_routes reads a {source_type: runner_factory} map from app['knowledge_connector_runners'] (installed by the host once the W01 executor, PR #11286, is available) and registers a vendor ONLY when its module imports AND its runner is installed. Absent module or absent runner => not registered (fail-closed, never public). No vendor code copied. Adds an end-to-end test: the handler seam constructs a connector WITH an injected runner and the SAME instance drives the real SyncScheduler -> fetch_rows -> ingest_rows -> store -> ACL query chain (scripted runner over the documented offline path; live W01 TLS is PR #11286's executor, not on main). * knowledge: register vendor connector on import, inject runner when installed Refine vendor-connector registration so it composes cleanly with the GitHub structured-source PR handler hunk (which registers github unconditionally and must stay edition-overridable). Each connector takes its live-read dependency as an OPTIONAL injected arg (GitHub transport_provider, Google operations_factory, SF call_runner) and self-enforces fail-closed when absent (GitHub refuses at fetch; Google and SF at validate). _register_optional_connector now registers a connector whenever its module imports, injecting the runner factory only when the host installed one; an absent module is still not registered (fail-closed). This keeps GitHub registered and overridable AND the ACL runner injection, with no hard import of a not-yet-landed module. * knowledge: document the connector runner factory as per-source, no stand-in Clarify the shared-handler registration seam per the per-source runner-factory decision: the value installed under knowledge_connector_runners is a PER-SOURCE factory the connector invokes as factory(source) to build that source's own runner (Google operations_factory(source); Salesforce per-source call runner), so one installed factory serves many sources without sharing a credential binding. Also state explicitly that this handler never synthesises a stand-in empty runner to feign activation: an absent factory leaves the connector on its own default (self-refusing live reads, fail-closed); an absent module leaves the source_type unregistered. Comment/docstring only; no behavior change. * knowledge: bridge W01 AccessGrant into a real AccessContext at the ACL seam Root found a real mismatch: W01's ControlPlaneBindingResolver.resolve returns an AccessGrant record (subject/tenant/groups from the trusted store's verified refs) which is NOT an AccessContext -- it has no subject_ids, the exact set the policy subject test reads (grant.subjects & ctx.subject_ids). Feeding it raw would AttributeError. Add acl.bridge_binding_resolver: wraps any resolver returning an AccessGrant and maps it field-for-field into a genuine AccessContext (subject, tenant, groups, bypass_acl=False), None-preserving and fail-closed on a subjectless/tenantless grant; an AccessContext is passed through unchanged. The handler now wraps app[knowledge_binding_resolver] through this bridge, so a host installing W01's resolver gets a working gate with the policy UNCHANGED. Tests drive the real AccessGrant shape (no subject_ids) through the bridge into the real HybridRetriever.search: granted item visible, wrong-account denied, group-from-grant satisfies the subject test, None grant denied; plus the handler seam bridges an installed AccessGrant resolver and passes an AccessContext through. 489 knowledge tests pass, flake8 clean. * test: real W01 ControlPlaneBindingResolver + BindingStore integration through the ACL bridge Adds test_knowledge_acl_w01_integration.py, which drives the ACTUAL W01 control-plane store and resolver rather than an AccessGrant-shaped stand-in: seed a VERIFIED binding via create_binding + BindingStore.insert, resolve it through the real ControlPlaneBindingResolver, bridge the returned AccessGrant to an AccessContext with acl.bridge_binding_resolver, and run the real HybridRetriever ACL gate. Cases: bound principal admitted, wrong-account denied (the store returns no binding for that deployment), revoked binding fenced (store returns None after revoke), unbound principal denied. The W01 control-plane package is not on this branch; it arrives in the merged candidate (this branch + the W01 L04 branch). The module importorskips when it is absent, so the standalone branch stays green (1 skipped) and the merged candidate runs the real integration (4 passed). Only the identity-proof verifier IO is substituted per its Protocol; the store, resolver, revoke fencing and account/deployment keying are the real ones -- an unmodified account genuinely returns no binding, not a faked pass. * knowledge: inject the per-source runner factory by each connector's own keyword The connectors do not share one injection keyword: Google takes operations_factory, GitHub takes transport_provider, and Salesforce takes a POSITIONAL call_runner plus a KEYWORD-ONLY runner_factory. The prior positional inject would have put the per-source factory into Salesforce's call_runner (a single pre-composed runner), mis-typing it. _register_optional_connector now takes inject_kw and injects the factory by that keyword; the registration loop maps each source_type to its connector's own keyword. Verified against the real SalesforceStructuredConnector (SF 74e54): the factory lands in runner_factory and call_runner stays None. Tests cover keyword injection, the SF-shape no-mis-wire, default-when-absent, and absent/broken module fail-closed. * feat(connections): Google Drive knowledge source — real provider + per-file SourceRow + sync/refresh + query-time ACL probe Adds the Google Drive provider stream's knowledge slice (D1), all four segments connected, every outbound call routed through W01's execute/PageWalk (vendor side only assembles: no token, session, HTTP, vault, refresh or revoke). vendors/google_drive/: - descriptors.py W01 OperationDescriptors (service_id=google_drive, effect=read) - locator.py RequestLocator: (descriptor,args) -> HttpRequest (both shared-drive flags) - decode.py ResultDecode: 2xx HttpReply -> OperationResult (nextPageToken->next_cursor) - drive_api.py Drive v3 request builders + mime classification + shortcut/export/media - errors.py Drive failure -> neutral ErrorClass (reason-level classifier, documented) - pagination.py distinct page cursor vs resume/resync token readers - operations.py sequences over an injected OperationRunner (list walk, content, changes+resync) - acl_probe.py GoogleDriveRevalidationHook: real per-query probe AS the subject, revoke-next-query knowledge/connectors/google_drive.py: GoogleDriveConnector — one SourceRow per file (fileId/driveId/mimeType/modifiedTime/version + fingerprint), snapshot vs incremental via changes.list, stale-token resync -> full-snapshot fallback, subjects from REAL permission principals (anyone=public, missing=empty deny-all, never invents a grant). Change-token resync keys on W01's neutral `input` class (Google documents no TTL/status for an expired token; W01 redacts the provider reason) — documented gap in the spec, no fabricated code. Non-input failures still raise. Tests run the REAL W01 execute/PageWalk over a scripted transport that runs the real locator.locate + decode.for_operation (56 tests): multi-page walk, export/media/shortcut, resync, duplicate cursor, removed change, malformed response, missing perms, content-fetch failure, revoke-then-deny. Ceiling code_complete: no live evidence, no real Google request, no business write. Owning spec: docs/system-specs/modules/connector-google-drive.md. Registration (shared knowledge handler is the ACL owner's, not edited here) delivered as a concrete factory/hunk PROPOSAL: connector-google-drive-registration-proposal.md.
|
暂停关闭(Connector Production Stack 项目于 2026-09-20 整体暂停)
本次关闭不代表该工作被否决,只是项目暂停收口。 |
W01 · L09 — the executor that wires the four control-plane judgments, plus the real transport composition, neutral data channel, and the L04 live-store send path
This is the integration point of W01: it turns the upstream primitives from unconsumed helpers into a real executor that runs the four judgments in order before any transport call, resolves the credential per-binding from L04's trusted live store, and delivers the data a consumer needs on the success path.
Base / provenance
Opened against
main. #11066 squash-merged the W01 L01 control-plane seam (+ W02 GitHub, W05 Graph) intomainand deleted thefeat/connector-control-plane-seambranch, so this PR was re-based onto currentmainby content analysis (the L01 seam filesoperation.py/context.py/errors.pyand thevendors/**trees are already inmainand are not re-proposed). The upstream dependency slices this L09 work imports are not yet inmainand are carried here so the executor can import and its suites run; each is preserved from its own source:binding.pylifecycle.py+acl_binding_resolver.py— commit-level integration of L04'sa8e907ee9d3d664a1a8df50d92564b922f9cd162(git cherry-pick), so L04's implementation and its own ~47 tests now execute in this PR's tree (they pass). Conflicts resolved by union;lifecycle.pyis a8 verbatim (code AST == a8, comment-only edits for gate compliance).BindingResolutionErrorlives inbinding.py(L02). The F2 store-seal and L08's F3 handle-shape validation are integrated in this PR (see the commit 2 notes and the ownership section).auth_modes.pypolicy.pywrites.pyhandle.pyexecutor.py,production.py, the neutral-data-channel additions toresult.py/ control-plane__init__.py/connections.md, and thetest_connections_control_plane*.pysuitesCommits (this PR — 2)
44900dcd2— the W01 L02/L04/L05/L06/L07/L08/L09 control plane on currentmain: the executor wiring the four judgments (ensure_usable→permit_operation→decide→replay_decision) before any transport call; the real production composition; the L04 live-store per-binding send path (build_production_transporttakesstore: BindingStore+gate: BindingCustodyGate; per call the gate matches custody, thenstore.select_secret(binding, reader=vault)fences against the LIVE store and takessecret_reffrom the store's record —binding_secret_refis neither imported nor called, AST-checked); the neutral data channel; the single authoritative cursor; the response-metadata allowlist; guarded redirects with cross-origin credential strip; andwrite_outcome=unknownon timeout/ambiguity.09292555c— the review-response commit folded onto the stack (a8 L04 commit-level integration, L08 F3, the F2 store seal, plus the earlier F1/F4/500/CodeQL/Inclusive fixes), rebased onto currentmain(04727ebe0, which carries fix(security): word a resolver stall as unverifiable, not a path match #11221 and fix(dashboard): drop the unused json import left by #9840 #11415):production.pyBindingCustodyGate) — the authorization carrier is frozen at the moment of validation:__post_init__replacesself.bindingwith a deep read-only snapshot (_frozen_binding, incl. the nestedsecret_ref) andtrusted_binding_forhands out a read-only mapping. Root cause: the authorization value travelled as a mutable dict and each consumer decided separately whether to re-read it, so a per-field comparison only moved the seam; removing the mutable carrier makes "validate then mutate the dict" unrepresentable. Regression reproduces the post-construction swap; mutation (freeze disabled) fails it.unknown(production.py) —500is now in_AMBIGUOUS_TRANSIT_STATUSES. A 500 on a non-idempotent write carries no proof of pre/post-commit, and the executor's own contract refuses letting a caller infer "not applied" from a 5xx, so the safe default iswrite_outcome=unknown(L07 thenREPLAY_REFUSEs unless the caller asserts idempotence). A vendor owner that knows its API commits-first still narrows via its ownwrite_outcome. Controlled-TLS test: server commits then 500 →unknown+REPLAY_REFUSE; read-500 staysNone; idempotent override stillALLOWs.test_connections_control_plane.py) — the two spawned refresh workers getcwd=tmp_pathand are terminate/wait-reaped in afinally.py/clear-text-logging— removed an evidenceprintthat interpolated a revealed secret; assertions carry the evidence with no plaintext to a log.sanity-check→confidence-check(comment-only).handle.py) — commit-level integration of L08's94841250a:ensure_usablevalidates the DerivedHandle shape →HandleTamperedErroron a missing/wrong-typed field, andis_expiredgets the_handle_id_is_lookupablepredicate (a malformed handle answersTrue, does not raise). L08's own 10 counter-example tests ride along and pass in this tree.config_dir()/control-plane-bindings/(bindings.json+bindings.lock), out of the agent-writableconnections/dir.lifecycle.py(the two path constants + docstrings only),security/paths.py(one directory entry on_WRITE_PROTECTED_HOME_PATHS), andsandbox.py(the same top-level leaf in_CREW_READONLY_LEAVES+_CREW_PRECREATE_READONLY_DIR_LEAVES+_CREW_NOFOLLOW_READONLY_DIR_LEAVES). Behavior-level tests: file-tool gate refuses store/lock/temp/future-child and keeps reads open; Linux launcherREADONLY_DIRS(config-level) and macOS Seatbelt deny-write/link (text-level). NOT verified at the real-kernel level (no test spawns a sandbox and checks a write's rc), and Windows has no launcher seal (file-tool gate only) — so F2's OS-layer coverage is incomplete and this PR does not claim it complete.Schema versions (pin these)
RESULT_SCHEMA_VERSION = 3EXECUTOR_SCHEMA_VERSION = 4PRODUCTION_SCHEMA_VERSION = 4LIFECYCLE_SCHEMA_VERSION = 1Invariants (all covered by tests)
TrustedHandleView— never the mutable handle.select_secret→assert_live): a revoked binding, a stale/forged generation, or a swapped identity/mode/secret_ref refuses before a socket exists; there is no slug fallback on the send path. APageWalkre-fences every page.write_outcome = unknown(500 with no proof of non-application defaults tounknown; a read never claims one), which the L07 replay gate refuses to blindly replay unless the caller asserts idempotence.Set-Cookie/Authorization/WWW-Authenticateand other credential/session headers can never appear.OperationResult.payloadis one of collection (items) / single object / raw bytes (Office bytes never text-coerced), with the cursor on the envelope, not the payload.Outward API (for the W02 GitHub and W05 MS downstreams)
Verification
297 control-plane tests pass (the guard suites + executor / production / data-channel / metadata / real-TLS e2e suites), including the two-binding-different-secret, revoke→zero-calls, no-slug-fallback, page-2-re-fence, and committed-then-dropped→
unknownregressions on the real TLS path; the disconnect fix is mutation-verified (reverting theexcepttuple fails both new tests).black --target-version py312,flake8,isort,mypyall clean (return codes captured non-piped). Novendors/**,platform/governance.py, or registry old-API changes.Build / fresh-install verified on the current head: real
python -m build→ wheel;zipfileconfirmsexecutor.py/production.py/lifecycle.pyare in the wheel; a fresh venv install; run from a cwd outside the source tree withPYTHONPATHunset resolves the module insite-packages; and a representative call from the installed wheel on a real TLS path exercises both the new live-store fencing (a real secret reaches the wire once; a revoke then fences the next send to zero calls) and the committed-then-dropped→unknown+ replay-refused path. The installed package reportsRESULT=3/EXECUTOR=4/PRODUCTION=4.Named gaps (not claimed as done)
L03 (auth code / PKCE / state / callback replay protection) is still required and unimplemented — the feature surface is known-incomplete. principal→binding resolution is single-binding (refuses others — not a multi-identity runtime); the vault entry name is per-binding (L02's
binding_scoped_secret_ref, adopted here frome8e11aa23); the principal→binding ACL adapter (BindingResolver.resolve→AccessContext, whose signature differs fromlifecycle.BindingStore.resolve) is L04's follow-up and is not wired here; MS's own 412 / readback / baseline / locator semantics live undervendors/microsoft/**(this only preserves the shared signal) and MS owns removing its own disconnectxfail; document/provider-level query ACL is a separately-owned gap.Fenced-finding ownership (as of HEAD
09292555c)control-plane-bindings/directory and is registered on the platform write-protection list (security/paths.py) and the OS-sandbox read-only leaf lists (sandbox.py); the twolifecycle.pypath constants were changed under a one-time cross-owner authorization (their logic otherwise untouched). OS-layer verification is config/text-level only (see the F2 caveat above); not claimed complete.handle.py) — integrated in this PR via L08's commit94841250a(see commit 2's "L08 F3"). No longer "not modified here".09292555craised two further fenced findings (production.py:683encoded-credential echo — Opus adjudication FLAG;production.py:1623malformed-JSON→{}— Opus UPHELD-FENCED) and one non-fencedexecutor.py:641top-level-imports finding; these are open and not yet addressed in this HEAD.