Skip to content

fix(chat): relay remote crew approvals - #10638

Closed
ashvinnihalani wants to merge 1 commit into
kirodotdev:mainfrom
ashvinnihalani:fix/remote-crew-approvals
Closed

ashvinnihalani wants to merge 1 commit into
kirodotdev:mainfrom
ashvinnihalani:fix/remote-crew-approvals

Conversation

@ashvinnihalani

Copy link
Copy Markdown
Contributor

Problem / Motivation

A chat can run on a second gateway while its session and approval UI stay on the first gateway. Permission requests reached the first gateway as transcript rows, but they had no approval future there. Approval-mode changes also stayed on the first gateway.

The person saw a request they could not answer, or changed a mode that did not change the gateway running the tool.

Why it matters

Remote crew chats can stop forever at a permission request. They can also run with a different approval mode than the one shown by the session UI.

What changed (motivation → approach → change)

The gateway that owns the UI now mirrors each peer permission as a local approval future. The gateway that runs the tool still owns the final decision.

The UI gateway sends Allow, Reject, Reject once, trust actions, and every mode change to the execution gateway first. It changes local state only after the execution gateway accepts the request. Owner checks, request-ID priority, retry behavior, and teardown cleanup stay in place.

flowchart LR
  subgraph Before
    A1[execution gateway asks]:::ctx --> B1[jump gateway relays text]:::changed --> C1[approval cannot return]:::removed
  end
  subgraph After
    A2[execution gateway asks]:::ctx --> B2[jump gateway mirrors approval]:::added --> C2[person decides]:::ctx --> D2[jump gateway forwards control]:::added --> E2[execution gateway resolves]:::changed
  end
  classDef added fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:2px
  classDef changed fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:2px
  classDef removed fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-dasharray:4 3
  classDef ctx fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E
  linkStyle 0,1 stroke:#DC2626,stroke-dasharray:4 3
  linkStyle 2,3,4,5 stroke:#16A34A,stroke-width:2px
Loading

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

The jump gateway now carries the decision back to the gateway that runs the tool.

Tests

  • Added relay tests for actionable peer permissions and ordinary Allow, Reject, and Reject once actions.
  • Added exact-payload tests for Normal, Reads, Trust, and YOLO mode changes.
  • Added tests for owner checks, ID collisions, retryable peer failures, malformed IDs, teardown cleanup, and completion races.
  • Focused backend validation: 485 passed.
  • Frontend cross-surface validation: 6,975 passed. TypeScript, formatting, imports, lint, types, docs lint, and whitespace checks passed.
  • Full backend validation in this host environment: 103,231 passed and 7 host-specific tests failed. The remaining failures involve local runtime integration and are not in the changed approval paths. GitHub CI remains the clean-environment gate.

Manual verification

Not yet run on three live hosts. The tests exercise both gateway handlers and the network-race boundaries directly.

Related Issues

Fixes #10618

Pattern harvest

Rule candidate: review-prompt
Pattern: A relayed control plane must cover every supported mode and decision, and the execution owner must accept before UI-owner state changes.

Checklist

  • At most two commits, with a Conventional Commits title
  • Existing focused tests pass and new tests cover the fixed behavior
  • Self-review completed; code follows project style guidelines
  • Documentation updated
  • No secrets, credentials, or internal references in the diff

@ashvinnihalani
ashvinnihalani requested review from a team and Zedmor September 13, 2026 23:14
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 13, 2026
@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 70b3d4ddac1e7f3ede6f94874716a7248d0804b2 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

safety_override._commit_activation enforces yolo_policy_permits() fail-closed on every arming path, so the rollback-restore path cannot re-arm YOLO past a tightened ceiling — that candidate finding is dead. The design mirrors the mechanism the old KNOWN GAP comment itself prescribed ("the peer's pending approval mirrored onto this slot's projection and the decision forwarded back"), keeps the execution peer authoritative, and the new peer endpoints ride the existing owner-gated, version-parity-checked _forward_peer_control seam. Remaining findings are CONCERNS-level.

Design-Verdict: CONCERNS

Sound peer-authoritative relay closing a documented gap, but the rollback mechanism restores peer-global state a scoped transition never touched.

Watch

  • Rollback restore is a whole-gateway snapshot applied last-writer-wins: _capture_approval_mode_rollback records "every peer slot and channel mode" even for a single-slot transition, and _restore_approval_mode_rollback rewrites all slots (absent keys forced to "normal") and all channel trust flags. A concurrent legitimate change on the peer inside the 60s window — including a human revoking trust — is silently reverted, re-raising a grant someone just withdrew.
    Clears when: scoped transitions snapshot/restore only the state they changed, or restore refuses on a detected intervening change.
  • The end-to-end path this PR exists to fix (hub → real SSH tunnel proxy_request → peer) has not been run live; every test doubles the tunnel manager, so _forward_peer_control's read-loop behavior against real aiohttp streaming responses is unproven, and the PR says so ("Not yet run on three live hosts").
    Clears when: one manual hub→peer approval and one mode change are verified over a real tunnel before release.

Suggestions

  • The forward → settle-future → mark-row → broadcast → pop-cleanup sequence is written twice (api_approval_resolve, api_chat_slot_approve) with private helpers lazily imported across a module cycle; one settle function in remote_relay (or ApprovalCoordinator) would remove the drift risk between the two copies.

[DESIGN-REVIEWED] 70b3d4d

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 70b3d4ddac1e7f3ede6f94874716a7248d0804b2 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All checks are done: the deleted KNOWN GAP pin in remote_relay.py documented exactly the mechanism this PR builds, SafetyOverride.activate re-checks the YOLO policy internally (so the rollback-restore branch stays governed), the rollback snapshot fields all map to real peer-side state the mode handler mutates, and the one counted gap is the first-match resolve loop still reachable from two Slack call sites. Final review:

First-Principles-Verdict: CONCERNS

The id-keyed first-match resolve this PR fixes on the HTTP route still runs unguarded for its 2 Slack callers — a remote mirror can settle locally with no peer forward.

Not justified as shipped

  • 7 — rides along: the 409 ambiguity refusal also changes purely local collision behavior (base picked first match); same root cause, but a point patch — the loop it guards survives at interaction_coordinator.py:121 with 2 other callers.
  • 8, 10 — rides along: local-path behavior changes (decision on every approval_resolved frame; reject-once now reaches local futures as its own decision) declared only via tests.

What this change ships

Inventory (10 items) — 7 justified

Intent: let a person answer a remote crew's permission request, and make mode changes reach the gateway that runs the tool — a FIX (deleted KNOWN-GAP pin in remote_relay.py + #10618; the pin itself prescribed this mechanism).

  1. A peer's permission request now appears as a clickable approval card on the UI gateway — justified
  2. Allow/Reject/Reject-once travel to the execution peer first; failure leaves the card retryable — justified
  3. All four mode changes apply on the execution peer before local state; unscoped goes to every peer — justified
  4. Card-level YOLO gates on the hub, approves the peer tool, then arms peer YOLO — justified
  5. New rollback-token protocol on /api/chat/mode (capture, 60s one-use restore, 2 error codes) — justified
  6. Retrying an already-decided approval replays idempotently instead of 404 — justified
  7. An approval id matching multiple sessions is refused as ambiguous, local slots included — rides along (see above)
  8. approval_resolved frames carry a decision field on every surface — rides along
  9. Turn teardown retires unresolved mirrored cards; Reject-once same-id successors survive it — justified
  10. Reject-once persists/broadcasts as its own decision on the slot endpoint — rides along

More micro-items exist (_PEER_SINKS widening, _forward_peer_control transport); capped at the 10 a person would notice.

Watch

  • The fix forwards and disambiguates only in the two dashboard handlers; ApprovalCoordinator.resolve's first-match slot loop (interaction_coordinator.py:121) still settles any slot future by bare id — counted callers outside the fixed route: slack/gateway.py:2540, slack/handler.py:5098 (grep resolve_approval(). A state-level id collision with a live mirror re-creates the stall on the messaging surface. Clears when: forwarding/ambiguity moves into resolve(), or mirrors are shown unreachable from channel-carded ids.

[FIRST-PRINCIPLES-REVIEWED] 70b3d4d

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — 🔴 changes requested (blocking)

Reviewed 70b3d4ddac1e7f3ede6f94874716a7248d0804b2 via the fork AI-review pipeline; updated in place on each push.

4 of 4 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/dashboard/remote_relay.py:869 -- A truncated success response leaves peer auto-approval enabled
if capture_rollback and not valid_peer_approval_mode_rollback_token(token):
Peer commits Trust/YOLO -> response drops before token arrives -> hub returns 502 without compensation -> peer remains elevated.
Anchor: residual/security
Fix: Use a hub-generated transaction ID so retries can recover the original rollback token.

BLOCKING -- src/kiro_crew/dashboard/chat_handlers.py:9798 -- Stale rollback can resurrect YOLO
record = state._approval_mode_rollbacks.pop(token, None)
YOLO snapshot captured -> concurrent Normal completes -> later compensation consumes the stale snapshot -> YOLO is re-enabled.
Anchor: residual/security
Fix: Bind snapshots to a mode generation and restore only when the post-transition generation is unchanged.

BLOCKING -- src/kiro_crew/dashboard/chat_handlers.py:9831 -- Rollback bypasses the current duration ceiling
if yolo_snapshot["permanent"]: / result = await asyncio.to_thread(override.activate_declared, source)
Ad-hoc until-shutdown grant captured -> enterprise revokes that duration -> compensation uses declared activation -> non-expiring YOLO returns despite the ceiling.
Anchor: backend-security-controls
Fix: Preserve declared versus ad-hoc identity and re-evaluate the corresponding duration-policy gate during restoration.

BLOCKING -- src/kiro_crew/dashboard/chat_handlers.py:10549 -- Failed peer YOLO extends an existing hub grant
remote_yolo_activation = await asyncio.to_thread(safety_override().activate, "dashboard")
Timed hub YOLO active -> remote-card YOLO renews it -> peer forwarding fails -> rollback is skipped because it was already active -> the grant remains extended after a 502.
Anchor: residual/security
Fix: Restore the exact prior source and expiry when either peer operation fails.

[BLOCK-MERGE] 70b3d4d
[GPT-REVIEWED] 70b3d4d

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

The adjudicable input block is empty; all four findings are FENCED (annotate-only, unbounded-harm class). I verified each against the patch and the base tree.

F1 remote_relay.py:869 — In forward_peer_approval_mode, _forward_peer_control returns {} on a 2xx status with a truncated/unparseable body (patch: reads to empty chunk, json.loads fails → decoded=None, 2xx path returns {}). The token is then empty → RemoteTurnError. The target is appended to accepted_remote_modes only after a successful return, so this target — where the peer already applied Trust/YOLO — is never rolled back by _rollback_remote_modes. Governance-ceiling / auto-approve-all persisting while the hub reports 502. Mid-body TCP truncation over the tunnel is ordinary; conditions do not contradict. Cannot complete a FLAG rarity record.

F2 chat_handlers.py:9798 — _restore_approval_mode_rollback pops a snapshot and re-applies want_yolo from the captured state; a concurrent Normal transition within the 60s TTL is not detected (no generation check), re-enabling YOLO. Interleaving across asyncio.to_thread awaits is ordinary; unbounded governance harm.

F3 chat_handlers.py:9831 — restoration calls override.activate_declared(source) for a captured permanent grant with no re-evaluation of the current duration-policy ceiling; a ceiling revoked inside the TTL window is bypassed. Governance-ceiling bypass, the exact class the fence protects.

F4 chat_handlers.py:10549 — activate extends an already-active timed hub YOLO; on peer-forward failure the compensation only deactivates when not remote_yolo_was_active, so a pre-existing grant stays extended after a 502. All conditions ordinary; auto-approve-all lifetime extended past intent.

For every finding the triggering conditions (network truncation, concurrent mode change, policy revocation in-window, extend-then-fail) are plausible in real operation and non-contradicting, and the harm is YOLO/governance-ceiling persistence — the class where a wrong clearance is unrecoverable. No FLAG record can be completed. When torn: UPHOLD-FENCED.

[ADJUDICATION] 70b3d4ddac1e7f3ede6f94874716a7248d0804b2 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 70b3d4ddac1e7f3ede6f94874716a7248d0804b2

[ADJUDICATION-FENCED] 70b3d4ddac1e7f3ede6f94874716a7248d0804b2 fenced=4 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/dashboard/remote_relay.py:869 -- A 2xx-with-truncated-body leaves the peer in Trust/YOLO while the hub 502s and never rolls that target back; mid-body truncation is ordinary.
UPHOLD-FENCED F2 src/kiro_crew/dashboard/chat_handlers.py:9798 -- Restoration re-applies captured YOLO with no generation check, so a concurrent Normal within the TTL is silently resurrected to YOLO.
UPHOLD-FENCED F3 src/kiro_crew/dashboard/chat_handlers.py:9831 -- Restoring a permanent grant via activate_declared skips the current duration-policy ceiling, re-establishing non-expiring YOLO after the ceiling was revoked.
UPHOLD-FENCED F4 src/kiro_crew/dashboard/chat_handlers.py:10549 -- activate extends an already-active timed grant and the failure path only deactivates when it was not already active, leaving YOLO extended after a 502.
[GPT-ADJUDICATED-FENCED] 70b3d4ddac1e7f3ede6f94874716a7248d0804b2

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 13, 2026
@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 70b3d4ddac1e7f3ede6f94874716a7248d0804b2 via the fork AI-review pipeline; updated in place on each push.

Review details

The sole candidate concerns the remote-slot trust flow in api_chat_slot_approve: the remote block forwards the decision to the peer (adding fut to _remote_approval_forwards) before the local trust/trust_command/trust_base branches run, so a local _deny_trust_pattern after a peer 2xx would leave fut stuck in _remote_approval_forwards and never be retired (ghost card) while the peer is trusted.

Falsifying it: the hub's permission card is a verbatim mirror of the peer's cls (_register_relayed_permission stores the peer's row; _get_pattern_from_pending reads that same mirror). So trust_grantable, full_command, trust_command_key, base_command on the hub equal the peer's. For every reachable input:

  • action == "trust": peer reads its trust_grantable; if != "1" the peer's own api_chat_slot_approve returns _deny_trust_pattern (400) → forward_peer_approval raises RemoteTurnError → the remote block's except discards fut and returns 502. If == "1", the hub reads the same "1" and does not deny.
  • trust_command/trust_base: the browser pattern must equal the peer's full_command/base_consent_pattern(base) for the peer to accept; the hub reads the identical mirrored value, so it accepts too. A missing/wrong pattern makes the peer deny first → 502 with fut discarded.

So the deny-after-peer-accept branch requires the hub and peer to read different grant fields for the same request_id while the peer still returns 2xx. That needs a same-id successor row rewriting _get_pattern_from_pending's newest match between forward and local validation — which the candidate itself could not construct, and which does not occur in practice. (a) — a concrete input that occurs in practice — is not established. Dropped.

No other defect meets the bar in the changed lines.

No findings.

[OPUS-REVIEWED] 70b3d4d

@ashvinnihalani
ashvinnihalani force-pushed the fix/remote-crew-approvals branch from ec9c541 to d7caed1 Compare September 14, 2026 00:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 14, 2026
@ashvinnihalani
ashvinnihalani force-pushed the fix/remote-crew-approvals branch 2 times, most recently from 952e481 to c5a748c Compare September 14, 2026 00:30
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 14, 2026
@ashvinnihalani
ashvinnihalani force-pushed the fix/remote-crew-approvals branch from c5a748c to 6cddcbf Compare September 14, 2026 01:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 14, 2026
Remote-bound chats showed permission requests on the execution peer only, so the gateway that owned the UI could not answer them. Approval-mode changes also stopped at the jump host.\n\nMirror peer permission requests into the owning slot, forward approval decisions and every supported mode to the execution peer, and fence relay teardown against in-flight responses. Preserve owner checks, collision priority, retryability, and one-shot rejection semantics.
@ashvinnihalani
ashvinnihalani force-pushed the fix/remote-crew-approvals branch from 6cddcbf to 70b3d4d Compare September 14, 2026 02:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 14, 2026
@NicholasRBowers NicholasRBowers added needs-pr-triage PR scanner: awaiting automated triage drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 21, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: All 82 non-review checks are green on 70b3d4dd; the only reds are the GPT lane and the readiness rollup it fails. Four GPT [BLOCK-MERGE] findings all land on the approval-mode rollback record this PR introduces, and each carries a concrete named remedy: hub-generated transaction id so a retry can recover the rollback token (remote_relay.py:869), bind the snapshot to a mode generation and restore only when it is unchanged (chat_handlers.py:9798), preserve declared-vs-ad-hoc identity and re-evaluate the duration-policy gate on restore (:9831), and restore the exact prior source and expiry when either peer operation fails (:10549). Design's Watch on the same mechanism ("scoped transitions snapshot/restore only the state they changed") is the same invariant, so one restructure of the rollback record answers all five. The merge conflict is three additive hunks: state.py __slots__ + __init__ are a plain union with main's _approval_stopped, and chat_handlers.py needs main's _app_may_send_to_slot ownership gate kept ahead of this PR's replay/409/remote-YOLO block.

Fix plan: rebase onto main (736 behind), resolve those three hunks as unions, restructure the rollback record per the four remedies above with regression tests, then disposition the advisory items — Design's live hub→peer-over-a-real-tunnel verification and First Principles' ask to move forwarding into ApprovalCoordinator.resolve() are answered with evidence or deferral, not by widening this diff.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@iamwhatever

Copy link
Copy Markdown
Contributor

Hi @ashvinnihalani, thank you for this fix and for the careful write-up on #10618.

This branch had gone conflicting with main and idle, so we took it over and redid it fresh on current main in #16509. Your commit authorship is kept with a Co-authored-by trailer, and the PR description credits this PR.

The new version keeps your core design (mirror the peer's pending approval as a local card, forward the decision and the mode change to the peer first). It leaves out the rollback-token protocol, which is where the GPT lane's four blockers sat: a refused peer call now changes nothing on either side, so there is nothing to roll back.

We will close this PR once #16509 merges. Thanks again!

@iamwhatever

Copy link
Copy Markdown
Contributor

Hi @ashvinnihalani, a quick update on this one.

We redid this fix on current main in #16509, building on your design. While that was in review, the project picked a different model for remote crews: the remote-crew sidebar RFC (#16489). In that model, a crew session is a window onto the peer's own slot, and approvals go straight to the peer (migration wave 3). The approval-mirror approach in both #10638 and #16509 is therefore superseded, and #16509 has been closed.

None of this work is wasted. Your write-up of the gap and the bugs #16509 found (listed in its closing comment) are the checklist wave 3 will use. Thank you for taking this on.

@iamwhatever

Copy link
Copy Markdown
Contributor

Hi @ashvinnihalani, a heads-up: #17662 (stacked on #17642) retires the hub-side turn relay that this PR built on. Under the accepted remote-crew sidebar RFC, a session on a crew is now owned by that crew and opened here through its window, so relay approvals have no path left to land on. Thank you again for this work. It shaped how we thought about approvals across machines, and it helped us get to the window design.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @ashvinnihalani for this PR. Your work on relaying remote crew approvals helped shape the remote-crew-sidebar RFC.

That RFC has now landed, and it makes this PR moot. Approvals now run on the peer's own session through the crew window (#16757), so there is nothing to relay. #17662 deleted the relay path (remote_relay.py, relay_remote_turn, remote_mirror), and #17642 stopped relay-slot births before it.

I am closing this as superseded. The branch is left in place. Thanks again for the work and the ideas it fed into the design.

@iamwhatever iamwhatever closed this Oct 8, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 8, 2026
@mbriones98 mbriones98 removed the drive-to-green PR claimed by drive-to-green pipeline label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote-bound chat does not relay tool approvals or YOLO to the executing peer

4 participants