Skip to content

fix(gui-test): rewrite members-private-memory-keeps-thread for immutable member memory - #12102

Merged
bolichen97 merged 1 commit into
mainfrom
fix/members-private-memory-scenario-11272
Sep 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/members-private-memory-scenario-11272

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

The nightly GUI user-test scenario members-private-memory-keeps-thread drives the tester through a control the GUI no longer has. Its steps tell the tester to open a member's edit panel, find a field titled "Memory store", click a "Create private memory" button and confirm a dialog. A member's memory store is fixed at creation and its binding is immutable — PUT /api/agents with a changed memory_store returns 409 member_memory_immutable (src/kiro_crew/dashboard/handlers/agents.py). There is nothing on screen for those three steps to click.

The two "Nova Sky" steps also insist on a display name. Nightly artifacts show that member's card rendering as the id nova-sky.

Why it matters

Every night the tester spends a full paid run wandering a member edit panel looking for a field that does not exist, times out against max_steps: 30 / max_seconds: 720, and files a friction issue against a GUI that is behaving. Issue #11273 is that report. A standing red in the nightly lane also costs the signal: a real members regression lands next to a scenario that is always failing and reads as more of the same.

What changed (motivation → approach → change)

The scenario is rewritten in place to exercise the behavior the original friction report (#11272) was about — a member's DM thread must survive the user leaving and coming back — instead of the memory-store switch that used to be one way to leave and return.

File Change
test/gui_user/scenarios/members-private-memory-keeps-thread.yaml Steps chat with the member, click another rail item (Sessions), reopen Crew Members and the same member, and send again. The three memory-control steps are gone. max_steps 30 → 24, max_seconds 720 → 600. Both "Nova Sky" steps hedge the card label, matching members-dm-hello. The expectation that forbade an error bubble drops its Essential source / cannot be a project resource examples, which were symptoms of the removed path.
test/gui_user/features.json The matching record tracks the scenario: title, user_story, entry_path, preconditions.notes, rationale, estimated_steps 12 → 8, and source loses src/kiro_crew/member_essential_context.py, which the flow no longer reaches.
test/gui_user/FEATURES.md Regenerated from features.json by python3 test/gui_user/features_catalog.py --write: the members row picks up the new user story and step count.

Rewriting beats retiring because the name is load-bearing. test/gui_user/test_scenarios_and_report.py pins members-private-memory-keeps-thread in its SHIPPED set and requires the members group to carry two nightly scenarios, so deleting the file fails the suite and drops the group's coverage to one. The name, tier, feature and preconditions are untouched; only what the tester does inside them moves.

The two assertions that made this scenario worth paying a model for are kept: the earlier "remember the blue door" bubble is still on screen after navigating away and back, and "still there?" draws a reply with no error bubble. Those two facts together are what a DOM test cannot judge.

Backwards compatibility

Compatible: the diff is GUI-test scenario data and its generated catalogue. No runtime code, schema, API or config key is touched.

Tests

No new test is added; the change is test data that existing tests constrain.

  • test/gui_user/test_scenarios_and_report.py loads every scenario file and asserts the loaded name set equals SHIPPED, that the nightly tier equals it, and that members holds members-dm-hello plus this scenario. A YAML that stopped parsing, or a renamed or deleted scenario, fails there.
  • test/gui_user/test_features_catalog.py pins FEATURES.md as a faithful render of features.json, so the regenerated table and the edited record must agree.

Manual verification

python3 test/gui_user/features_catalog.py --check is the in-repo check that FEATURES.md matches features.json; the committed file is the --write output of the edited record. grep -rn 'Create private memory\|Memory store' test/gui_user/ finds nothing, so no sibling scenario still points at the removed control.

The live lane that actually walks this scenario needs a display server, a real browser and a model backend, and runs in the nightly workflow rather than on a PR.

Screenshots / video

Why no screenshot: the diff is three files under test/gui_user/; no file under website/ changes, so there is no rendered surface to show.

Related Issues

Closes #11273

The GUI behaviour reported in #11272 is what the rewritten steps cover.

Pattern harvest

Rule candidate: agents-md
Pattern: a GUI-test scenario names a control by its visible label, so removing that control from the frontend leaves a nightly scenario clicking at a UI that is gone — and the failure surfaces as a friction issue blaming the GUI, not the removal. Removing or renaming a user-facing control should sweep test/gui_user/scenarios/ for its label in the same change, the same way a renamed config key sweeps its writers.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and iamwhatever September 19, 2026 21:46
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 19, 2026
@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 5e3b16ead0aee40ec7551ec30ba7c0185ec59174 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5e3b16e

Verdict parsed from the review's SHA-scoped output markers for commit 5e3b16ead0aee40ec7551ec30ba7c0185ec59174.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5e3b16ead0aee40ec7551ec30ba7c0185ec59174: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 5e3b16ead0aee40ec7551ec30ba7c0185ec59174 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the first nightly run passes the rewritten navigate-away-and-return steps before closing the #11272 follow-up.

Suggestions

  • The id members-private-memory-keeps-thread no longer describes what it tests; renaming (with a two-line SHIPPED update) is cheap and the "retiring would break the test" rationale doesn't apply to a rename — worth doing here or as an immediate follow-up.

[DESIGN-REVIEWED] 5e3b16e

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 5e3b16ead0aee40ec7551ec30ba7c0185ec59174 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5e3b16e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5e3b16ead0aee40ec7551ec30ba7c0185ec59174: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 19, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor

Heads-up from the #11273 investigation: the rewritten steps here will still end in a red bubble on the nightly until the fake ACP backend answers the KAS readiness barrier. Since 2026-09-18 every message to nova-sky has failed with KAS managed MCP readiness timed out after 90s: kirocrew-cron: unreported, kirocrew-core: unreported, kirocrew-dashboard: unreported (gateway.err in runs 35329960496 and 35434776670, AcpRuntime._wait_managed_mcp → wait_mcp_ready): member DMs run on kas by default and the packaged fake never sent _kiro/mcp/status. #12238 teaches the fake to report readiness when spawned as the relay (--agent-engine), leaves this scenario file alone, and fixes the sidebar scenario. Both are needed for the members group to go green.

Two notes on the scenario text, since the same run artifacts are in front of me: the tester saw the member card as nova-sky (no display name), so "the member named Nova Sky (its card may read "nova-sky")" avoids a friction entry every night; and the Design lane's rename suggestion is worth taking — the members-dm-hello-style name members-dm-thread-survives-navigation (plus the two SHIPPED references in test_scenarios_and_report.py) describes what the steps now do.

PR #11780 removed the "Create private memory" control: a member now gets
its store at creation and the binding is immutable (PUT /api/agents
changing memory_store returns 409 member_memory_immutable). The nightly
scenario members-private-memory-keeps-thread still drove the tester
through that removed flow, so every run would strand on a control that
no longer exists and re-file a friction issue against #11272.

Rewrite the scenario in place to exercise the behaviour that friction
issue was actually about -- a member's DM thread survives leaving and
returning, keeping its history and still answering -- and drop every
memory-store step. Realign the features.json record (title, user story,
entry path, notes, rationale, estimated_steps 12 -> 8, and the now
irrelevant member_essential_context.py source) and regenerate
FEATURES.md from it.

The scenario keeps its id because test_scenarios_and_report.py pins it
in SHIPPED and requires the members group to hold exactly two nightly
scenarios. Per review of the nightly artifacts, both click steps
now hedge the member's card label the way members-dm-hello already does:
it renders as the id "nova-sky" rather than a display name, and a step
that insists on the display name files a friction entry every night for
a GUI that is behaving.
@bolichen97
bolichen97 force-pushed the fix/members-private-memory-scenario-11272 branch from 6624cc9 to 5e3b16e Compare September 20, 2026 18:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 20, 2026 19:22

@cixuuz cixuuz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the central claim (member_memory_immutable 409 exists in agents.py at this head). Test-data-only scenario rewrite, no runtime code touched. The author's own heads-up about a cross-PR dependency (#12238, also in this run's queue and independently reviewed) is an honest disclosure, not an unresolved defect in this PR's own logic. All bot lanes clean at head 5e3b16e.

@bolichen97
bolichen97 merged commit 0dc290b into main Sep 20, 2026
100 of 101 checks passed
@bolichen97
bolichen97 deleted the fix/members-private-memory-scenario-11272 branch September 20, 2026 19:35
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 20, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…irodotdev#12102)

PR kirodotdev#11780 removed the "Create private memory" control: a member now gets
its store at creation and the binding is immutable (PUT /api/agents
changing memory_store returns 409 member_memory_immutable). The nightly
scenario members-private-memory-keeps-thread still drove the tester
through that removed flow, so every run would strand on a control that
no longer exists and re-file a friction issue against kirodotdev#11272.

Rewrite the scenario in place to exercise the behaviour that friction
issue was actually about -- a member's DM thread survives leaving and
returning, keeping its history and still answering -- and drop every
memory-store step. Realign the features.json record (title, user story,
entry path, notes, rationale, estimated_steps 12 -> 8, and the now
irrelevant member_essential_context.py source) and regenerate
FEATURES.md from it.

The scenario keeps its id because test_scenarios_and_report.py pins it
in SHIPPED and requires the members group to hold exactly two nightly
scenarios. Per review of the nightly artifacts, both click steps
now hedge the member's card label the way members-dm-hello already does:
it renders as the id "nova-sky" rather than a display name, and a step
that insists on the display name files a friction entry every night for
a GUI that is behaving.

Co-authored-by: w <w@k.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nightly GUI user test FAIL: members-private-memory-keeps-thread

3 participants