Skip to content

feat(pi): verify a pi member's saved agent spec at spawn - #22911

Merged
iamwhatever merged 2 commits into
mainfrom
feat/pi-saved-spec-verification
Oct 26, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
feat/pi-saved-spec-verification

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Was stacked on #22808 (feat/pi-zero-tool-ban), now merged. Rebased onto main after #22923 (pi steer), #23152 (pi open-gaps doc refresh) and #23979 (pi corpus re-capture): code patches unchanged (git range-diff: context only); the doc/README conflicts were resolved onto main's text (see "Docs" below).

Problem / Motivation

Goal: a crew member with a saved agent spec runs on pi the way it runs on kiro, KAS, claude, codex and deepseek: the spec is verified against what the session consumed, and the member is never run with more than it was saved with.

Today pi is outside ACP_BACKENDS_MEMBER_CAPABILITIES, so every saved-spec member on pi is refused before spawn (capability_harness_unsupported). #22813 kept it out because pi has no mirror: _mirrorless_member_mounts never recorded _session_agent_spec, so a confirmation could only refuse.

Why it matters

pi is a supported member harness (member dispatch and panel already land on it). Without saved-spec verification, any member whose capabilities were saved on the Capabilities page cannot chat on pi at all.

Not a goal

  • A pi mirror, or translating a saved spec's own mcpServers for pi: a declared server still reaches pi only when the shared gateway pools it.
  • Changing what a non-member pi session mounts: its pooled stubs are unfiltered, as before.
  • Per-tool limits on pi (ACP_BACKENDS_MEMBER_TOOL_LIMITS): a spec with disabledTools is still refused before spawn.
  • Any change to acp/harness/pi.py, acp/harness/pi_node.py (fix(pi): run pi under a Node that meets its engine floor #22883) or the pi gate extension.

What changed (motivation → approach → change)

Commit 1 — feat(pi): verify a pi member's saved agent spec at spawn

Commit 2 — fix(pi): mount only the pooled stubs a member's saved spec references
A fresh-context review of commit 1 found pi handed every pooled broker stub to the session, while the claude mirror drops a stub the spec's tools never references. So a saved tools: ["@a"] still mounted pooled b on pi, beside a spec confirmed as consumed: wider than saved.

  • On a member session the mirror-less branch records the spec's tools allowlist (projection.allowlist, the object claude's mirror uses) from the same parse, and _pooled_mcp_servers filters the stubs through it. It fails closed: a member spec that is missing or unreadable grants nothing, because the record is cleared to grant nothing before each read; every session/new / session/load splice evaluates pi's array (which records it) before the pooled stubs, pinned by a test. _pooled_mcp_servers reads only that instance field, so kiro-cli's stub composition gains no branch and no registry lookup (H13; Opus 5.5 finding on c5e678a141, fixed). A pooled server with a tool switched off (in the spec or the global settings) is withheld whole, as the member mounts already are, because pi carries no per-tool deny (GPT 6.1 review finding on de1ae5ef76, fixed). Non-member pi sessions and kiro-cli record none, so their stubs are unchanged; kiro's path gains only a None check (H13).
  • client_class.py declares the field; client_lifecycle._reset_state drops it with the session.

Docs (docs/system-specs/modules/harness-onboarding.md, pi table): row 2 now names pi IN ACP_BACKENDS_MEMBER_CAPABILITIES (and IN ACP_BACKENDS_STEERING_REQUEST, which #22923 made true but the row still called "out of all three steer sets"); the live-spill row counts nine captures (adds steer and saved-spec); the open gaps row drops saved-spec verification and mid-turn steer and keeps only what is still open: per-tool limits (ACP_BACKENDS_MEMBER_TOOL_LIMITS) and the spec's own MCP servers under broker-only. The fixture READMEs keep main's re-captured text and add the saved-spec row.

Not changed: acp/harness/pi.py, acp/harness/pi_node.py (#22883) and the pi gate extension are untouched.

Backwards compatibility

  • Saved-spec members on pi now start instead of being refused. A spec with per-tool disabledTools is still refused before spawn (capability_tool_limits_unsupported; pi is not in ACP_BACKENDS_MEMBER_TOOL_LIMITS).
  • A member session on pi no longer receives pooled stubs its spec's tools does not reference (claude parity). Non-member pi sessions are unchanged.
  • Nothing is removed: no path, key, record, validator or public function.

Tests

  • test/test_pi_saved_spec.py (new, 22 tests), on the real resolver and confirmation:
    • consumed spec is the one parse the mounts came from (no second read); a saved spec is confirmed; a spec edited after its save is refused; an unreadable re-resolve drops the earlier parse and refuses; kiro records nothing;
    • runtime_view: a declared server that did not reach pi keeps the pane unverified (capability_mcp_unreported); no servers reads applied;
    • stubs: a referenced pooled server is present, an unreferenced one is absent, * keeps both, [] and a missing tools key keep none, a switched-off server is dropped, a server with a globally switched-off tool is withheld whole, an unreadable or missing member spec mounts none, stubs asked for before the spec is read are withheld on a member (not on a non-member), a re-resolve takes the edited tools, a non-member pi session keeps the full pool, kiro's stubs are never filtered, the allowlist is dropped on reset;
    • the live capture's frames: the spec's own server called, gated by session/request_permission, server answer in the turn.
  • test/test_member_mount_pi.py: the three exclusion tests from feat(pi): mount the crew member panel on pi member threads #22813 are replaced by the above; the BROKER_ONLY mount fact stays pinned. test/test_harness_parity.py: the membership pin includes pi.
  • Mutation checks (each mutation applied alone, then reverted), all caught: drop the record (3 fail), drop the clear-before-read (1), record from a second parse (1), drop pi from the set (4), drop the stub filter (9), drop the allowlist record (5), record the allowlist for non-members too (1), let a missing member spec grant all (1), let an unrecorded member allowlist fail open (2), keep a narrowed stub (1), drop the reset (1).

Local results

  • Gates: HARNESS_BASE_REF=origin/main scripts/check_harness_parity.py ✓ (x-capture gate: pi's membership cites a replayed live capture) and --test ✓; check_brand_name.py ✓ and check_comment_history.py ✓ (base origin/main); check_file_line_cap.py ✓; check_acp_frame_host_data.py ✓; black ✓; mypy on the changed modules ✓.
  • Focused pytest: 1690 passed (11 skipped) on the rebased head across the pi backend/spawn/session, member mounts, session capabilities, harness parity, frame replay/host-data, client/acp bindings, session MCP and goose/opencode MCP suites. The full suite is left to CI.

Manual verification

test/fixtures/acp_frames/pi/saved-spec-projection-live.jsonl (+ .expected.json), recorded by the in-product recorder while a real AcpClient(acp_backend="pi") ran as a crew member, off pi-acp 0.0.34 over pi 1.3.0 installed pinned into a scratch npm prefix (not the host's global pi), with a scripted model on the loopback interface:

Saved spec Result
matches the saved intent, tools: ["*"], one server confirmed, loaded_capability_template = the member, no gaps; the spec's server was called through the gate
edited after its save refused before the first prompt with capability_runtime_unverified (both spawns); server never called
tools does not reference the server confirmed with gap native_tools (pane unverified); the pooled stub never reached pi, server never spawned
match case with commit 1's record removed refused (capability_runtime_unverified): the record is what makes verification pass

Not verifiable on this host: it refuses user namespaces, so no sandbox backend runs. The capture driver replaced only the routing's sandbox preflight with "no mask", and stood in for gatewayd pooling with the spec's own entry so the call is observable at the server. The gate-extension seal, read-back, envelope nonce, permission path and member confirmation ran as shipped. No sandboxed dashboard-gateway pi turn was observed.

Independent fresh-context reviews: commit 1 — UNSOUND (the pooled-stub width above), fixed in commit 2. Commit 2 — reviewed from the diff and the claude-parity requirement only: MEETS. Its hardening findings are folded in: a member with no recorded allowlist, or a missing spec, now grants no stub (it previously relied on call order / matched claude's grant-all), and the skipped-stub log compares names. Its note that the fixture carries only the first drive's frames is now stated in the corpus README; the third drive's mount set is pinned by the unit tests and the header note.

Related Issues

None. Follows #22808 and #22813.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Sensitive change: none. No file in this diff matches SENSITIVE_GLOBS (checked with fnmatch over both commits), and nothing is inherited.

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners October 21, 2026 21:43
@kyleseaman
kyleseaman requested review from CrysisDeu and removed request for a team October 21, 2026 21:43
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2f674e5578de2ee0597dcc3f94d5bc916ea299f3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 2f674e5578de2ee0597dcc3f94d5bc916ea299f3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The PR declares the three deleted #22813 pins, and each of the three reasons that kept pi out (no recorded spec, an overstated pane, no zero-tool ban) is now covered by the diff or by #22808.

Notes

  • Item 7 — rides along: the pooled stubs are withheld whole for a tool switched off "in the spec or the global settings". The spec half is already refused before spawn: the description says "a spec with disabledTools is still refused before spawn". Subtraction: shrink narrowed_stubs to the global-settings source alone, if projection.disabled_tools holds both.

What this change ships

Inventory (9 items) — 7 justified

Intent: let a crew member with a saved agent spec chat on pi, verified against the spec the session actually used and never with more than was saved. ADDITION.

  1. Saved-spec members on pi now start instead of being refused with capability_harness_unsupported — justified
  2. A pi session records the spec it consumed; a spec edited after its save is refused — justified
  3. A declared server that did not reach pi leaves the Capabilities pane "unverified" — justified
  4. A pi member session no longer mounts pooled stubs that its spec's tools doesn't reference — justified
  5. A pi member whose spec is missing or unreadable mounts no pooled stubs — justified
  6. Non-member pi and kiro stubs are unchanged — justified
  7. A pooled server with any tool switched off is withheld whole on a pi member — rides along, partly redundant with the pre-spawn refusal
  8. A warning is logged for each withheld stub — undeclared
  9. The harness-onboarding table's pi rows are refreshed, including correcting the steer-set row — justified

[FIRST-PRINCIPLES-REVIEWED] 2f674e5

Description read

[DESCRIPTION-READ] c9a1f251aa7f153ea00c513d278c733ebe07d72453b494bc9a7b0acef8efdf5f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2f674e5578de2ee0597dcc3f94d5bc916ea299f3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The stub filter fails closed, the consumed spec comes from the same parse as the mounts, and a declared server that never reaches pi shows as unverified instead of being claimed as running.

Notes

  • The deleted comment in backends_members.py set a condition for pi to join: "Joining needs the spec's servers projected into the handoff…". This PR lists that projection under "Not a goal". In its place, runtime_view leaves the pane unverified / capability_mcp_unreported. So a member whose spec declares its own servers now runs on pi with fewer tools than it was saved with, where before it was refused. The pane flags this, but the PR replaces a recorded joining condition without saying so. Ask the feat(pi): mount the crew member panel on pi member threads #22813 author to sign off, in a PR comment, on "runs narrower and is flagged" instead of "refused".

[DESIGN-REVIEWED] 2f674e5

Description read

[DESCRIPTION-READ] f5c8c392d6e8197478f9d41eecb3c4799f8193b9ad171320a7afb3934341fa0b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2f674e5578de2ee0597dcc3f94d5bc916ea299f3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2f674e5

Verdict parsed from the review's SHA-scoped output markers for commit 2f674e5578de2ee0597dcc3f94d5bc916ea299f3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2f674e5578de2ee0597dcc3f94d5bc916ea299f3 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2f674e5578de2ee0597dcc3f94d5bc916ea299f3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2f674e5

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2f674e5578de2ee0597dcc3f94d5bc916ea299f3 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
Base automatically changed from feat/pi-zero-tool-ban to main October 21, 2026 22:30
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 21, 2026
@kyleseaman
kyleseaman force-pushed the feat/pi-saved-spec-verification branch from ab4f572 to de1ae5e Compare October 21, 2026 22:46
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@kyleseaman
kyleseaman force-pushed the feat/pi-saved-spec-verification branch from de1ae5e to e14dc7c Compare October 21, 2026 23:02
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 21, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 22, 2026
@kyleseaman
kyleseaman force-pushed the feat/pi-saved-spec-verification branch from e14dc7c to 2d2e034 Compare October 22, 2026 14:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 22, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 22, 2026
@kyleseaman
kyleseaman force-pushed the feat/pi-saved-spec-verification branch from 2d2e034 to c5e678a Compare October 22, 2026 15:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 22, 2026
@kyleseaman
kyleseaman force-pushed the feat/pi-saved-spec-verification branch from c5e678a to ca21f75 Compare October 22, 2026 16:19
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 22, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (human review) at ca21f75.

I found no defect. The pi member path records the consumed spec from the same parse as its mounts, clears it before each read, and fails closed: a missing or unreadable member spec grants no pooled stub (client_mcp_array.py). The only other _pooled_mcp_servers caller, the entitlement probe, is kiro-only. All AI lanes pass, the Design note is covered by the rewritten set comment, and every check on this head is green.

It is held because it adds pi to ACP_BACKENDS_MEMBER_CAPABILITIES (backends_members.py). That lets saved-spec members start on pi with native-tool limits reported as a gap rather than enforced, and it changes which pooled MCP stubs a member gets. Changes to the member capability boundary are reserved for a human maintainer.

— crew-pr-reviewer (r121)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 22, 2026
@kyleseaman
kyleseaman force-pushed the feat/pi-saved-spec-verification branch from ca21f75 to 6e2e191 Compare October 26, 2026 13:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 26, 2026
Kyle Seaman added 2 commits October 26, 2026 14:42
pi joins ACP_BACKENDS_MEMBER_CAPABILITIES. pi has no mirror, so
_mirrorless_member_mounts now records _session_agent_spec from the same
session_mcp_projection parse its member mounts and zero-tool answer come
from; confirm_member_projection digests it against the saved intent, and a
spec edited after its save is refused with capability_runtime_unverified.
A re-resolve clears the record before it reads, so an unreadable spec
never leaves an older parse standing.

pi's projection stays BROKER_ONLY: a declared server that is not pooled
does not reach pi, and runtime_view keeps that member unverified rather
than applied. The set's comment says so and cites a live capture off
pi-acp 0.0.34 over pi 1.3.0 (acp_frames/pi/saved-spec-projection-live.jsonl).
A mirror filters the pooled broker stubs through the spec's tools
allowlist; pi has no mirror, so a member on pi received every pooled
stub, and a saved spec with tools ["@A"] still mounted pooled server b
beside a spec confirmed as consumed. On a member session the mirror-less
branch now records the allowlist from the same parse as the consumed
spec, and _pooled_mcp_servers filters the stubs through it. Non-member
pi sessions and kiro-cli record none, so their stubs are unchanged.
@kyleseaman
kyleseaman force-pushed the feat/pi-saved-spec-verification branch from 6e2e191 to 2f674e5 Compare October 26, 2026 14:43
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever merged commit 527ff81 into main Oct 26, 2026
108 checks passed
@iamwhatever
iamwhatever deleted the feat/pi-saved-spec-verification branch October 26, 2026 15:49
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants