Repository navigation
fix(gui-test): rewrite members-private-memory-keeps-thread for immutable member memory - #12102
Conversation
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the first nightly run passes the rewritten navigate-away-and-return steps before closing the #11272 follow-up. Suggestions
[DESIGN-REVIEWED] 5e3b16e |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Heads-up from the #11273 investigation: the rewritten steps here will still end in a red bubble on the nightly until the fake ACP backend answers the KAS readiness barrier. Since 2026-09-18 every message to Two notes on the scenario text, since the same run artifacts are in front of me: the tester saw the member card as |
PR #11780 removed the "Create private memory" control: a member now gets its store at creation and the binding is immutable (PUT /api/agents changing memory_store returns 409 member_memory_immutable). The nightly scenario members-private-memory-keeps-thread still drove the tester through that removed flow, so every run would strand on a control that no longer exists and re-file a friction issue against #11272. Rewrite the scenario in place to exercise the behaviour that friction issue was actually about -- a member's DM thread survives leaving and returning, keeping its history and still answering -- and drop every memory-store step. Realign the features.json record (title, user story, entry path, notes, rationale, estimated_steps 12 -> 8, and the now irrelevant member_essential_context.py source) and regenerate FEATURES.md from it. The scenario keeps its id because test_scenarios_and_report.py pins it in SHIPPED and requires the members group to hold exactly two nightly scenarios. Per review of the nightly artifacts, both click steps now hedge the member's card label the way members-dm-hello already does: it renders as the id "nova-sky" rather than a display name, and a step that insists on the display name files a friction entry every night for a GUI that is behaving.
6624cc9 to
5e3b16e
Compare
cixuuz
left a comment
There was a problem hiding this comment.
Verified the central claim (member_memory_immutable 409 exists in agents.py at this head). Test-data-only scenario rewrite, no runtime code touched. The author's own heads-up about a cross-PR dependency (#12238, also in this run's queue and independently reviewed) is an honest disclosure, not an unresolved defect in this PR's own logic. All bot lanes clean at head 5e3b16e.
…irodotdev#12102) PR kirodotdev#11780 removed the "Create private memory" control: a member now gets its store at creation and the binding is immutable (PUT /api/agents changing memory_store returns 409 member_memory_immutable). The nightly scenario members-private-memory-keeps-thread still drove the tester through that removed flow, so every run would strand on a control that no longer exists and re-file a friction issue against kirodotdev#11272. Rewrite the scenario in place to exercise the behaviour that friction issue was actually about -- a member's DM thread survives leaving and returning, keeping its history and still answering -- and drop every memory-store step. Realign the features.json record (title, user story, entry path, notes, rationale, estimated_steps 12 -> 8, and the now irrelevant member_essential_context.py source) and regenerate FEATURES.md from it. The scenario keeps its id because test_scenarios_and_report.py pins it in SHIPPED and requires the members group to hold exactly two nightly scenarios. Per review of the nightly artifacts, both click steps now hedge the member's card label the way members-dm-hello already does: it renders as the id "nova-sky" rather than a display name, and a step that insists on the display name files a friction entry every night for a GUI that is behaving. Co-authored-by: w <w@k.local>
Problem / Motivation
The nightly GUI user-test scenario
members-private-memory-keeps-threaddrives the tester through a control the GUI no longer has. Its steps tell the tester to open a member's edit panel, find a field titled "Memory store", click a "Create private memory" button and confirm a dialog. A member's memory store is fixed at creation and its binding is immutable —PUT /api/agentswith a changedmemory_storereturns 409member_memory_immutable(src/kiro_crew/dashboard/handlers/agents.py). There is nothing on screen for those three steps to click.The two "Nova Sky" steps also insist on a display name. Nightly artifacts show that member's card rendering as the id
nova-sky.Why it matters
Every night the tester spends a full paid run wandering a member edit panel looking for a field that does not exist, times out against
max_steps: 30/max_seconds: 720, and files a friction issue against a GUI that is behaving. Issue #11273 is that report. A standing red in the nightly lane also costs the signal: a real members regression lands next to a scenario that is always failing and reads as more of the same.What changed (motivation → approach → change)
The scenario is rewritten in place to exercise the behavior the original friction report (#11272) was about — a member's DM thread must survive the user leaving and coming back — instead of the memory-store switch that used to be one way to leave and return.
test/gui_user/scenarios/members-private-memory-keeps-thread.yamlmax_steps30 → 24,max_seconds720 → 600. Both "Nova Sky" steps hedge the card label, matchingmembers-dm-hello. The expectation that forbade an error bubble drops itsEssential source/cannot be a project resourceexamples, which were symptoms of the removed path.test/gui_user/features.jsontitle,user_story,entry_path,preconditions.notes,rationale,estimated_steps12 → 8, andsourcelosessrc/kiro_crew/member_essential_context.py, which the flow no longer reaches.test/gui_user/FEATURES.mdfeatures.jsonbypython3 test/gui_user/features_catalog.py --write: themembersrow picks up the new user story and step count.Rewriting beats retiring because the name is load-bearing.
test/gui_user/test_scenarios_and_report.pypinsmembers-private-memory-keeps-threadin itsSHIPPEDset and requires themembersgroup to carry two nightly scenarios, so deleting the file fails the suite and drops the group's coverage to one. The name,tier,featureandpreconditionsare untouched; only what the tester does inside them moves.The two assertions that made this scenario worth paying a model for are kept: the earlier "remember the blue door" bubble is still on screen after navigating away and back, and "still there?" draws a reply with no error bubble. Those two facts together are what a DOM test cannot judge.
Backwards compatibility
Compatible: the diff is GUI-test scenario data and its generated catalogue. No runtime code, schema, API or config key is touched.
Tests
No new test is added; the change is test data that existing tests constrain.
test/gui_user/test_scenarios_and_report.pyloads every scenario file and asserts the loaded name set equalsSHIPPED, that the nightly tier equals it, and thatmembersholdsmembers-dm-helloplus this scenario. A YAML that stopped parsing, or a renamed or deleted scenario, fails there.test/gui_user/test_features_catalog.pypinsFEATURES.mdas a faithful render offeatures.json, so the regenerated table and the edited record must agree.Manual verification
python3 test/gui_user/features_catalog.py --checkis the in-repo check thatFEATURES.mdmatchesfeatures.json; the committed file is the--writeoutput of the edited record.grep -rn 'Create private memory\|Memory store' test/gui_user/finds nothing, so no sibling scenario still points at the removed control.The live lane that actually walks this scenario needs a display server, a real browser and a model backend, and runs in the nightly workflow rather than on a PR.
Screenshots / video
Why no screenshot: the diff is three files under
test/gui_user/; no file underwebsite/changes, so there is no rendered surface to show.Related Issues
Closes #11273
The GUI behaviour reported in #11272 is what the rewritten steps cover.
Pattern harvest
Rule candidate: agents-md
Pattern: a GUI-test scenario names a control by its visible label, so removing that control from the frontend leaves a nightly scenario clicking at a UI that is gone — and the failure surfaces as a friction issue blaming the GUI, not the removal. Removing or renaming a user-facing control should sweep
test/gui_user/scenarios/for its label in the same change, the same way a renamed config key sweeps its writers.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)