Repository navigation
test(gui-user): search the roster before clicking a seeded crewmate's card - #19537
Conversation
85465de to
6e72776
Compare
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Only GUI-test scenario data, one test and one doc change. The scenarios now follow the specified search-only path to an unused crewmate, and the product's listing rule is left alone. Notes
[DESIGN-REVIEWED] 393e655 Description read[DESCRIPTION-READ] bbbecd99aaa994926ddaafcb8c286f3a28880b52c92a90d0c2846b75e03224e4 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
8b0f021 to
aa81d36
Compare
aa81d36 to
80ae802
Compare
|
This branch conflicts with main in test_scenarios_and_report.py. Please rebase it on main. |
80ae802 to
18f0310
Compare
|
Rebased on main (head 18f0310). Main's #20965 renamed the user-facing word crewmate to mate, so the five scenarios now keep main's "Mates" wording and the added step reads the current placeholder "Search mates…" (website/src/i18n/locales/en.json, |
18f0310 to
393e655
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the rebase. The conflict with main is resolved, and the branch now sits on current main.
Reviewed head 393e655.
What I checked:
- Range diff against the previous head 80ae802: the only changes are main's crewmate to mate rename carried into the five scenarios, the doc paragraph and the new test. The added step reads "Search mates…", which matches
search_membersin website/src/i18n/locales/en.json on main. rosterFilter.ts::listedByDefaulton main still hides a row whose DM thread is empty, and a typed search still reaches it, so the fix is still needed.test/gui_user/test_scenarios_and_report.pyon this head: 76 passed. The new guard fails on main's scenarios (crewmate-chat-clean.yaml step 4) and passes here.- The AI lanes report no findings on this head, so there is nothing to reproduce.
One wording issue, inline below. It does not block the flow: typing Nova reaches the card either way.
I am not approving yet: readiness is still checking and several required test jobs were still running when I looked.
| - You are on the Settings page. Open the Developer section and find the Feature Previews list. | ||
| - Turn on the preview whose title starts with "Mates" (older builds may read "Crew Members" or "Crew Members and Crew Mode"; any of those is the right one). Ignore the Webhooks preview. | ||
| - Open the Mates page. After the preview is on, a "Mates" item appears in the left rail, directly under Sessions; click it. | ||
| - Click the search box at the top of the mates list (its placeholder reads "Search mates…") and type Nova. A mate you have not chatted with yet is hidden from the list until a search reaches it, so Nova Sky (its card may read "nova-sky" instead of a display name) is not in the list before you type; the Mate and default cards being there is normal. |
There was a problem hiding this comment.
🟡 "Nova Sky is not in the list before you type" is only true for the first scenario of a run
Nothing resets the target between scenarios (docs/build/gui-user-test.md, "Nothing resets the target between scenarios"). crewmate-chat-clean, members-dm-hello and the others each send a message to nova-sky. Once one of them has run, nova-sky's DM thread holds a message, listedByDefault returns true, and its card is listed before any search. A tester who reads "is not in the list before you type" and then sees the card may report a mismatch with the instructions instead of moving on.
so Nova Sky (its card may read "nova-sky" instead of a display name) is not in the list before you type
Suggestion: hedge it the same way members-private-memory-keeps-thread.yaml already does for the return visit, in all five scenarios and the doc paragraph, for example "Nova Sky may not be in the list before you type; if its card is already listed, that is normal too."
Rule: a scenario step must hold on every run order the lane can produce.
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (7 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: GUI user-test scenarios add a roster search step before clicking a seeded mate's card, with a guard test and a note in the test guide; no product code changes.
Problem / Motivation
Goal: The nightly GUI user test finds the crewmate it was told to click (Nova Sky) on the Crewmates page.
The nightly tester opened the Crewmates page, as
crewmate-chat-cleanandmembers-dm-hellotell it to, and the roster listed onlydefaultandMate. Nova Sky was not there, with no filter on. The scenario stalled at its first click.Why it matters
Every nightly scenario that opens a seeded crewmate's chat (crewmate chat cleanliness, the DM hello, threads, briefing) fails at the same card, so the lane reports a blocker on every run and covers none of those features until the card appears.
Not a goal
docs/system-specs/modules/crew-mode.md.Intended? / Repro
Intended? Yes — the hidden card is by design. Product code is untouched; this PR fixes the GUI-test scenarios that assumed otherwise (conductor ruling on the owner's behalf, option A).
(1) What
seed_home.py/boot.shcreate fornova-skyA crewmate row, not a template.
boot.shrunsseed_home.py --fixture rich --member nova-sky; the seed writesconfig.agents["nova-sky"] = {"kiro_agent": "kirocrew", "workspace": "default", "description": "Nova Sky -- seeded crew member for the GUI user test."}into$KIROCREW_HOME/config.json. Nothing is written under~/.kiro/agents/, so there is no custom agent template named nova-sky. The row binds the sharedkirocrewtemplate, the default folder and Global memory, and its DM thread is empty at boot. Verified by running main's script into an empty home:config.agentskeys['default', 'nova-sky'], no DM thread bound to nova-sky.(2) On main, does the roster search reach it?
Yes.
GET /api/members(handlers/members.py::api_members) emits one row percfg.agentskey, hidden ones included; the hide is client-side inrosterFilter.ts:listedByDefault(m): starred → listed; Mate → listed; elselast_chat_ts > 0. nova-sky has0→ hidden.defaultis listed because the rich fixture's seeded chats fold into it; Mate is exempt.rosterShows(m, query):if (q) return matchesSearch(m, q)— a typed query matchesname,display_nameorkiro_agentas a substring, so "Nova" reachesnova-skyand shows its card.crew-mode.md"Every other row is hidden and appears when the search text matches it ... the search is the one door within the roster list."The tester typed nothing in Search crewmates…. The hint "Crewmates you haven't used yet are hidden. Search to find them." renders only when EVERY row is hidden; with
defaultand Mate listed it does not appear, so nothing on screen told the tester a row was hidden.(3) Is the tester's confusion intended behavior?
Yes.
docs/request-for-change/rfc-session-template-vs-crewmate.md(status: accepted, owner's decision 2026-10-09, newer thanrfc-crewmates-chatted-list.md), rule 3:A seeded crewmate with an empty DM thread is exactly the hidden case. The scenario text ("Click the member card for Nova Sky ... it may be lower in the list") predates the hide rule (#9586, 2026-09-09 vs #17808, 2026-10-07) and assumed a listed card. The confusion is the scenario's stale assumption meeting intended product behavior, so the scenarios change and the product does not. Stamping
crew_recency.jsonor a fake DM message from the seed would pretend the user already chatted with the crewmate, which rule 3 says must not count; starring would seed a user choice the scenarios never make.Repro on
origin/main(standalone script, no test suite)Observed:
config.agents = ['default', 'nova-sky'], no DM thread for nova-sky →last_chat_ts = 0,has_dm_message = false, not starred →listedByDefault()false → card hidden, search-only. Expected by the old scenario text: a card. Expected by the RFC: hidden.Real failure: nightly run 38128758150, job 114438519515, head
f0716252,crewmate-chat-cleanstep 6: "friction (blocker): I could not find a member called Nova Sky or nova-sky anywhere; the roster only ..." → "attempt 1: FAIL in 6 steps".New test on main:
test_scenarios_search_before_clicking_a_seeded_members_cardfails for all five scenarios (crewmate-chat-clean.yaml: step 4 clicks a seeded member's card with no search step before it, and the same for the other four); it passes on this branch.What changed (motivation → approach → change)
The Crewmates page hides a crewmate whose DM thread is empty until the roster search reaches it. That is the accepted design (RFC rule 3 above). The seeded
nova-skyhas an empty thread, so a scenario that says "click its card" asks for a card that is not on the page.The five scenarios that open Nova Sky's chat now do what a person does for an unused crewmate: click Search crewmates…, type
Nova, then click the card. The step says why the card is not there yet, so the tester does not read the short list as a bug. Each scenario'smax_stepsgrows by two for the extra click and typing. The return visit inmembers-private-memory-keeps-threadkeeps a hedge to search again if the card is not listed.max_stepscrewmate-chat-cleanmembers-dm-hellocrewmate-panel-tabscrewmate-reply-threadmembers-private-memory-keeps-threaddocs/build/gui-user-test.mdsays, in its Seeds section, that a seeded member is hidden until searched and that scenarios search first rather than the seed pretending the user chatted. The seed script and the product are untouched.Backwards compatibility
Compatible: only GUI-test scenario text, a test and a doc change; no product code, API or config shape moves.
Removes nothing: the diff adds a step to five scenarios and a test.
Tests
test/gui_user/test_scenarios_and_report.py::TestShippedScenarios::test_scenarios_search_before_clicking_a_seeded_members_card: for every scenario that opens the Crewmates page and clicks a seeded member's card, a step before that click types into Search crewmates… naming the member. Red on main for all five scenarios, green here; a future scenario that clicks a seeded card without searching fails it.Manual verification
N/A — the scenarios are data read by the nightly
gui-user-test.ymllane; the search path they describe is the onerosterFilter.ts::rosterShowsimplements andcrew-mode.mdspecifies. The next nightly run exercises it.Related Issues
Fixes #19081
Pattern harvest
Rule candidate: review-prompt
Pattern: a GUI-test scenario asserts a surface's listing rule as it was when the scenario was written; a product RFC that changes who is listed must sweep the scenarios that click a card.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)