Repository navigation
test(gui): fake KAS relay reports MCP readiness; fix sidebar scenario - #12238
Conversation
|
Intent: Let the nightly GUI lane get a crew-member reply again (the packaged fake ACP backend speaks the KAS readiness frames the harness now waits for) and make the sidebar smoke scenario judge the layout the product actually draws. |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm the nightly goes green post-merge via [DESIGN-REVIEWED] d5533f3 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fake's KAS frames first execute on the nightly after merge — confirm with Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: make the nightly GUI lane's
Verification run: [FIRST-PRINCIPLES-REVIEWED] d5533f3 |
|
The nightly GUI lane could not get a crew-member reply once the KAS harness began holding a session's first prompt behind managed-MCP readiness: the packaged fake ACP backend never sent `_kiro/mcp/status`, so every member DM (KAS by default) timed out after 90s with "KAS managed MCP readiness timed out". Spawned with the relay's `--agent-engine` argv the fake now reports each declared server `connected` with a client-origin stamp and a one-tool catalog, after session/new and again after session/set_mode, and the barrier releases the prompt. The plain kiro-cli spawn is unchanged. `sidebar-folders-and-older-sessions` now states the seeded layout the way the list view draws it: root folders first, the pinned ungrouped row below them, and the dormant Demos session behind its expander. The pinned row was never above the folder; the earlier passes were lenient reads of the same screen. The GUI lane docs record that a retry runs against the same live gateway, so a scenario must hold on a target its first attempt touched. Refs #11273 Refs #11272
7b86686 to
d5533f3
Compare
|
|
cixuuz
left a comment
There was a problem hiding this comment.
Well-designed fix to the packaged fake ACP backend's KAS-relay readiness reporting. The strongest evidence is test_kas_relay_report_satisfies_the_host_readiness_barrier, which exercises the real production KasMcpReadiness/required_managed_servers against the fake's emitted frames rather than only asserting shape. All four review lanes (Opus, GPT, Design, First Principles) PASS/no-blocking at head d5533f3.
…kirodotdev#12238) The nightly GUI lane could not get a crew-member reply once the KAS harness began holding a session's first prompt behind managed-MCP readiness: the packaged fake ACP backend never sent `_kiro/mcp/status`, so every member DM (KAS by default) timed out after 90s with "KAS managed MCP readiness timed out". Spawned with the relay's `--agent-engine` argv the fake now reports each declared server `connected` with a client-origin stamp and a one-tool catalog, after session/new and again after session/set_mode, and the barrier releases the prompt. The plain kiro-cli spawn is unchanged. `sidebar-folders-and-older-sessions` now states the seeded layout the way the list view draws it: root folders first, the pinned ungrouped row below them, and the dormant Demos session behind its expander. The pinned row was never above the folder; the earlier passes were lenient reads of the same screen. The GUI lane docs record that a retry runs against the same live gateway, so a scenario must hold on a target its first attempt touched. Refs kirodotdev#11273 Refs kirodotdev#11272
Problem / Motivation
The nightly GUI user test (#11273) has been red every night since the
membersscenarios landed. Two different things are red:KAS managed MCP readiness timed out after 90s: kirocrew-cron: unreported, kirocrew-core: unreported, kirocrew-dashboard: unreported.gateway.errin runs 35329960496 and 35434776670 shows the same traceback for everymember-nova-skyslot, fromAcpRuntime._wait_managed_mcp→AcpSessionHandle.wait_mcp_ready.sidebar-folders-and-older-sessionsFAILED on 2026-09-19 on "the pinned Welcome to KiroCrew row is listed above the folder". The screen has never shown that: the 09-16, 09-18 and 09-19 final reports all describe Demos at the top and the pinned row below it. The two earlier PASSes were lenient reads of the same picture.Why it matters
The lane cannot judge any member flow while the fake backend cannot get a member to reply, so the whole
membersfeature reads red for a reason that has nothing to do with the product. A scenario that fails on a wrong expectation costs a retry every night and sends the on-call after a regression that does not exist.What changed (motivation → approach → change)
Member DMs run on the KAS backend (
agent.member_acp_backenddefaults tokas). The KAS harness now holds a session's first prompt until every managed MCP server it declared reportsconnectedover_kiro/mcp/status. The packaged fake (src/kiro_crew/testing/fake_acp_backend.py, the GUI lane's and the E2E harness'sKIROCREW_KIRO_BIN) never sent that frame, so the barrier waited its full 90 seconds and the turn failed. Ordinary chat runs on the kiro-cli harness, which has no such barrier.The fake now knows which spawn it is.
build_kas_argvspawns the relay asacp --agent-engine v3 …; with that flag inargvthe fake reports every server the host declared (the session-levelmcpServersarray and the active_meta.kiro.customAgentsentry's block) asconnected, stamped_meta.kiro.resource.source.origin = clientwith a one-tool catalog, plus the matching_kiro/tools/didChangetags. It reports aftersession/newand again aftersession/set_mode, because the host skips frames queued before itsset_modeas pre-switch. The plainacpspawn sends nothing new. KAS tears a session down with_kiro/session/delete; the fake now treats it like kiro-cli's_kiro.dev/session/terminateand forgets the session, so a prompt after teardown gets the same "Unknown or terminated ACP session" error on both harnesses instead of a false reply.flowchart LR subgraph Before A1[session/new]:::ctx --> B1[set_mode]:::ctx --> C1[wait_mcp_ready]:::ctx --> D1[90s timeout]:::removed end subgraph After A2[session/new]:::ctx --> B2[set_mode]:::ctx --> R2[_kiro/mcp/status connected]:::added --> C2[wait_mcp_ready]:::ctx --> D2[first prompt]:::changed end classDef added fill:#DCFCE7,stroke:#16A34A classDef changed fill:#FEF3C7,stroke:#D97706 classDef removed fill:#FEE2E2,stroke:#DC2626,stroke-dasharray:4 3 classDef ctx fill:#E0F2FE,stroke:#0284C7🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged
The fake, spawned as the KAS relay, now answers the readiness barrier, so a member's first prompt goes out instead of timing out.
The sidebar scenario now describes the seeded layout the way
ChatSidebardraws it: root folders render first, the ungrouped lane (pinned rows leading) sits under them, and the older Demos session sits behind the folder's "Dormant sessions" expander.docs/build/gui-user-test.mdrecords that a retry runs against the same live gateway with nothing re-seeded, so a scenario must hold on a target its own first attempt already touched;docs/ci/e2e-gate.mdnotes the fake's KAS-relay behaviour beside its method list.The
members-private-memory-keeps-threadscenario is not touched here: #12102 already rewrites it around the removed private-memory opt-in. That rewrite still needs this PR's fake change to get a reply.Backwards compatibility
Compatible: on the KAS-relay spawn (
--agent-enginein argv) the fake adds two notifications and forgets a session on_kiro/session/delete; every kiro-cli spawn keeps the exact frames it had. The scenario and doc edits change no code path.Tests
test/test_fake_acp_backend.py:test_relay_flag_is_the_one_build_kas_argv_renderspins the fake's flag tokas_transport.KAS_RELAY_ENGINE_FLAG, so a renamed relay argv breaks the test, not the nightly.test_kas_relay_reports_every_declared_server_connected_on_session_newlocks the status frame: session-level names first, then each agent's block, deduplicated, allconnectedwithorigin: clientand a catalog.test_kas_relay_set_mode_answers_then_reports_the_active_agent_rosterlocks thatset_modestill answers{}first and then reports only the switched-to agent's servers.test_kas_relay_report_satisfies_the_host_readiness_barrierfeeds the emitted frames through the realKasMcpReadinessforrequired_managed_servers(...)and assertspending == ""and no failure — the contract the nightly timed out on.test_kiro_spawn_reports_no_mcp_readinessandtest_main_arms_the_relay_from_argvlock that the plain spawn is unchanged and thatmain()arms the relay from argv.test/gui_user/test_scenarios_and_report.pyloads every shipped YAML, so the sidebar scenario is parsed in CI.Manual verification
The Bedrock GUI lane (
gui-user-test.yml) has no PR trigger and cannot run on a developer box; the scenario and fake changes are validated structurally in CI and first execute on the nightly after merge. A maintainer can rungh workflow run gui-user-test.yml -f scenario=members-dm-hello(or the sidebar scenario) onmainafter merge. Locally the fake's frames were driven throughKasMcpReadinessby hand (the same check the new test automates),features_catalog.py --checkpasses, and the scenario loads through the DSL.Related Issues
Refs #11273
Refs #11272
Refs #12102 (hosts the
members-private-memory-keeps-threadrewrite; both are needed for themembersgroup to go green)Pattern harvest
Rule candidate: review-prompt
Pattern: a harness gains a readiness barrier the packaged fake backend does not speak, so every fake-backed lane on that harness times out — a new required wire frame must land in the fake in the same PR.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)