Skip to content

test(gui): fake KAS relay reports MCP readiness; fix sidebar scenario - #12238

Merged
bolichen97 merged 1 commit into
mainfrom
fix/gui-nightly-11273
Sep 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/gui-nightly-11273

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

The nightly GUI user test (#11273) has been red every night since the members scenarios landed. Two different things are red:

  • A crew member never answers. Since 2026-09-18 every message sent to Nova Sky ends in a red bubble: KAS managed MCP readiness timed out after 90s: kirocrew-cron: unreported, kirocrew-core: unreported, kirocrew-dashboard: unreported. gateway.err in runs 35329960496 and 35434776670 shows the same traceback for every member-nova-sky slot, from AcpRuntime._wait_managed_mcp → AcpSessionHandle.wait_mcp_ready.
  • sidebar-folders-and-older-sessions FAILED on 2026-09-19 on "the pinned Welcome to KiroCrew row is listed above the folder". The screen has never shown that: the 09-16, 09-18 and 09-19 final reports all describe Demos at the top and the pinned row below it. The two earlier PASSes were lenient reads of the same picture.

Why it matters

The lane cannot judge any member flow while the fake backend cannot get a member to reply, so the whole members feature reads red for a reason that has nothing to do with the product. A scenario that fails on a wrong expectation costs a retry every night and sends the on-call after a regression that does not exist.

What changed (motivation → approach → change)

Member DMs run on the KAS backend (agent.member_acp_backend defaults to kas). The KAS harness now holds a session's first prompt until every managed MCP server it declared reports connected over _kiro/mcp/status. The packaged fake (src/kiro_crew/testing/fake_acp_backend.py, the GUI lane's and the E2E harness's KIROCREW_KIRO_BIN) never sent that frame, so the barrier waited its full 90 seconds and the turn failed. Ordinary chat runs on the kiro-cli harness, which has no such barrier.

The fake now knows which spawn it is. build_kas_argv spawns the relay as acp --agent-engine v3 …; with that flag in argv the fake reports every server the host declared (the session-level mcpServers array and the active _meta.kiro.customAgents entry's block) as connected, stamped _meta.kiro.resource.source.origin = client with a one-tool catalog, plus the matching _kiro/tools/didChange tags. It reports after session/new and again after session/set_mode, because the host skips frames queued before its set_mode as pre-switch. The plain acp spawn sends nothing new. KAS tears a session down with _kiro/session/delete; the fake now treats it like kiro-cli's _kiro.dev/session/terminate and forgets the session, so a prompt after teardown gets the same "Unknown or terminated ACP session" error on both harnesses instead of a false reply.

flowchart LR
  subgraph Before
    A1[session/new]:::ctx --> B1[set_mode]:::ctx --> C1[wait_mcp_ready]:::ctx --> D1[90s timeout]:::removed
  end
  subgraph After
    A2[session/new]:::ctx --> B2[set_mode]:::ctx --> R2[_kiro/mcp/status connected]:::added --> C2[wait_mcp_ready]:::ctx --> D2[first prompt]:::changed
  end
  classDef added fill:#DCFCE7,stroke:#16A34A
  classDef changed fill:#FEF3C7,stroke:#D97706
  classDef removed fill:#FEE2E2,stroke:#DC2626,stroke-dasharray:4 3
  classDef ctx fill:#E0F2FE,stroke:#0284C7
Loading

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

The fake, spawned as the KAS relay, now answers the readiness barrier, so a member's first prompt goes out instead of timing out.

The sidebar scenario now describes the seeded layout the way ChatSidebar draws it: root folders render first, the ungrouped lane (pinned rows leading) sits under them, and the older Demos session sits behind the folder's "Dormant sessions" expander. docs/build/gui-user-test.md records that a retry runs against the same live gateway with nothing re-seeded, so a scenario must hold on a target its own first attempt already touched; docs/ci/e2e-gate.md notes the fake's KAS-relay behaviour beside its method list.

The members-private-memory-keeps-thread scenario is not touched here: #12102 already rewrites it around the removed private-memory opt-in. That rewrite still needs this PR's fake change to get a reply.

Backwards compatibility

Compatible: on the KAS-relay spawn (--agent-engine in argv) the fake adds two notifications and forgets a session on _kiro/session/delete; every kiro-cli spawn keeps the exact frames it had. The scenario and doc edits change no code path.

Tests

test/test_fake_acp_backend.py:

  • test_relay_flag_is_the_one_build_kas_argv_renders pins the fake's flag to kas_transport.KAS_RELAY_ENGINE_FLAG, so a renamed relay argv breaks the test, not the nightly.
  • test_kas_relay_reports_every_declared_server_connected_on_session_new locks the status frame: session-level names first, then each agent's block, deduplicated, all connected with origin: client and a catalog.
  • test_kas_relay_set_mode_answers_then_reports_the_active_agent_roster locks that set_mode still answers {} first and then reports only the switched-to agent's servers.
  • test_kas_relay_report_satisfies_the_host_readiness_barrier feeds the emitted frames through the real KasMcpReadiness for required_managed_servers(...) and asserts pending == "" and no failure — the contract the nightly timed out on.
  • test_kiro_spawn_reports_no_mcp_readiness and test_main_arms_the_relay_from_argv lock that the plain spawn is unchanged and that main() arms the relay from argv.

test/gui_user/test_scenarios_and_report.py loads every shipped YAML, so the sidebar scenario is parsed in CI.

Manual verification

The Bedrock GUI lane (gui-user-test.yml) has no PR trigger and cannot run on a developer box; the scenario and fake changes are validated structurally in CI and first execute on the nightly after merge. A maintainer can run gh workflow run gui-user-test.yml -f scenario=members-dm-hello (or the sidebar scenario) on main after merge. Locally the fake's frames were driven through KasMcpReadiness by hand (the same check the new test automates), features_catalog.py --check passes, and the scenario loads through the DSL.

Related Issues

Refs #11273
Refs #11272
Refs #12102 (hosts the members-private-memory-keeps-thread rewrite; both are needed for the members group to go green)

Pattern harvest

Rule candidate: review-prompt
Pattern: a harness gains a readiness barrier the packaged fake backend does not speak, so every fake-backed lane on that harness times out — a new required wire frame must land in the fake in the same PR.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and iamwhatever September 20, 2026 07:51
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Intent: Let the nightly GUI lane get a crew-member reply again (the packaged fake ACP backend speaks the KAS readiness frames the harness now waits for) and make the sidebar smoke scenario judge the layout the product actually draws.
Not a goal: Changing any product behaviour, the KAS readiness barrier itself, or the members-private-memory-keeps-thread scenario (#12102 owns that rewrite).

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed d5533f3af10c29e10b139065cc650f1f69933d6b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d5533f3

Verdict parsed from the review's SHA-scoped output markers for commit d5533f3af10c29e10b139065cc650f1f69933d6b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d5533f3af10c29e10b139065cc650f1f69933d6b: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of d5533f3af10c29e10b139065cc650f1f69933d6b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d5533f3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d5533f3af10c29e10b139065cc650f1f69933d6b: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of d5533f3af10c29e10b139065cc650f1f69933d6b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm the nightly goes green post-merge via gh workflow run gui-user-test.yml -f scenario=members-dm-hello — the real lane has no PR trigger.

[DESIGN-REVIEWED] d5533f3

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of d5533f3af10c29e10b139065cc650f1f69933d6b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The fake's KAS frames first execute on the nightly after merge — confirm with gh workflow run gui-user-test.yml -f scenario=members-dm-hello before trusting green.

Not justified as shipped

  • Item 3 — rides along: _kiro/session/delete handling fixes neither reported defect; it is declared, and its harm is real (the catch-all previously answered {} and a post-teardown prompt drew a false canned reply).
  • Item 6 — rides along: the retry-semantics paragraph in gui-user-test.md is authoring guidance beyond the two defects, declared, harm named (a mutating scenario costs a paid nightly retry).

What this change ships

Inventory (7 items) — 5 justified

Intent: make the nightly GUI lane's members and sidebar scenarios stop failing for reasons that are not product bugs — a FIX (two defects, both with run IDs and Refs #11273/#11272).

  1. Member DMs in fake-backed lanes get a first reply instead of the 90s MCP-readiness timeout — justified
  2. The fake repeats its readiness report after session/set_mode so the switched-to agent's roster clears the barrier — justified
  3. Prompting a torn-down KAS session now errors instead of falsely replying — rides along (declared; derived from the KAS teardown verb, types.py:144)
  4. Sidebar scenario now expects the pinned row below the folder, in the ungrouped lane — justified
  5. Sidebar scenario now accepts a Demos session behind the "Dormant sessions" expander — justified
  6. gui-user-test.md gains a retry-stability authoring rule for scenarios — rides along (declared; beyond the two defects)
  7. Docs and docstring record the fake's KAS-relay behaviour — justified

Verification run: --agent-engine v3 is build_kas_argv's rendered flag (kas_transport.py:66,123); the barrier demands connected + origin: client + catalog (mcp_session_report.py:298-322), exactly what the fake emits and test_kas_relay_report_satisfies_the_host_readiness_barrier proves against the real KasMcpReadiness; the pre-switch frame skip is real (session_handle.py:4373-4424); member_acp_backend defaults to "kas" (config/loader.py:2564); ChatSidebar.tsx:8681-8744 renders root folders first, then the ungrouped pinned lane and dormant expander, so the rewritten expectation matches the code and the old pin was wrong — evidence given, not silence. Grepped _kiro/mcp/status fakes: one packaged fake, no duplicate mechanism.

[FIRST-PRINCIPLES-REVIEWED] d5533f3

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@CrysisDeu

CrysisDeu commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Item 4 — _kiro/session/delete drops fake session state, undeclared (span=b0fc862a643e) — fixed (description)

Legitimate: the diff does make the fake forget a session on KAS's teardown verb, and the body claimed "only two notifications".
The code stays: without it a prompt after _kiro/session/delete would get a canned reply from a session the host already tore down, which is a false success the kiro-cli path (_kiro.dev/session/terminate) never produced.
The PR description now names the teardown handling in "What changed" and "Backwards compatibility" on this same head; no code change, no push.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
The nightly GUI lane could not get a crew-member reply once the KAS
harness began holding a session's first prompt behind managed-MCP
readiness: the packaged fake ACP backend never sent `_kiro/mcp/status`,
so every member DM (KAS by default) timed out after 90s with
"KAS managed MCP readiness timed out". Spawned with the relay's
`--agent-engine` argv the fake now reports each declared server
`connected` with a client-origin stamp and a one-tool catalog, after
session/new and again after session/set_mode, and the barrier releases
the prompt. The plain kiro-cli spawn is unchanged.

`sidebar-folders-and-older-sessions` now states the seeded layout the
way the list view draws it: root folders first, the pinned ungrouped
row below them, and the dormant Demos session behind its expander. The
pinned row was never above the folder; the earlier passes were lenient
reads of the same screen.

The GUI lane docs record that a retry runs against the same live
gateway, so a scenario must hold on a target its first attempt touched.

Refs #11273
Refs #11272
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 20, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/gui-nightly-11273 branch from 7b86686 to d5533f3 Compare September 20, 2026 13:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 20, 2026
@CrysisDeu

CrysisDeu commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Item 3 — _kiro/session/delete handling rides along (span=77b8345b62e4) — rebutted (kept, in scope)

The relay behaviour is one unit: the fake now stands in for KAS, and KAS ends a session with _kiro/session/delete, not kiro-cli's _kiro.dev/session/terminate.
Without the branch a prompt after teardown drew the canned reply from a session the host had already deleted — a false success the readiness report would then make reachable for the first time.
Declared in the description; three lines, no new mechanism.

@CrysisDeu

CrysisDeu commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Item 6 — retry-semantics paragraph in gui-user-test.md rides along (span=a61fe8025394) — rebutted (kept, in scope)

The second attempt of members-private-memory-keeps-thread ended NO_VERDICT for exactly this reason: Harness.run retries against the same live gateway with no reseed, so attempt 1's mutation broke attempt 2's precondition (#11273 triage, attempt 2 in runs 35079909570 and 35329960496).
Recording that in the lane doc is the fix for that half of the reported failure; the scenario file itself is owned by #12102, so the rule is the only place this PR can carry it.
Docs only, no behaviour change.

@cixuuz cixuuz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-designed fix to the packaged fake ACP backend's KAS-relay readiness reporting. The strongest evidence is test_kas_relay_report_satisfies_the_host_readiness_barrier, which exercises the real production KasMcpReadiness/required_managed_servers against the fake's emitted frames rather than only asserting shape. All four review lanes (Opus, GPT, Design, First Principles) PASS/no-blocking at head d5533f3.

@bolichen97
bolichen97 merged commit fa3222c into main Sep 20, 2026
93 checks passed
@bolichen97
bolichen97 deleted the fix/gui-nightly-11273 branch September 20, 2026 20:51
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 20, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…kirodotdev#12238)

The nightly GUI lane could not get a crew-member reply once the KAS
harness began holding a session's first prompt behind managed-MCP
readiness: the packaged fake ACP backend never sent `_kiro/mcp/status`,
so every member DM (KAS by default) timed out after 90s with
"KAS managed MCP readiness timed out". Spawned with the relay's
`--agent-engine` argv the fake now reports each declared server
`connected` with a client-origin stamp and a one-tool catalog, after
session/new and again after session/set_mode, and the barrier releases
the prompt. The plain kiro-cli spawn is unchanged.

`sidebar-folders-and-older-sessions` now states the seeded layout the
way the list view draws it: root folders first, the pinned ungrouped
row below them, and the dormant Demos session behind its expander. The
pinned row was never above the folder; the earlier passes were lenient
reads of the same screen.

The GUI lane docs record that a retry runs against the same live
gateway, so a scenario must hold on a target its first attempt touched.

Refs kirodotdev#11273
Refs kirodotdev#11272
@agenticshieldllc agenticshieldllc mentioned this pull request Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants