Skip to content

test(gui-user): search the roster before clicking a seeded crewmate's card - #19537

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/members-roster-nova-sky-19081
Oct 23, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/members-roster-nova-sky-19081

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The nightly GUI user test finds the crewmate it was told to click (Nova Sky) on the Crewmates page.

The nightly tester opened the Crewmates page, as crewmate-chat-clean and members-dm-hello tell it to, and the roster listed only default and Mate. Nova Sky was not there, with no filter on. The scenario stalled at its first click.

Why it matters

Every nightly scenario that opens a seeded crewmate's chat (crewmate chat cleanliness, the DM hello, threads, briefing) fails at the same card, so the lane reports a blocker on every run and covers none of those features until the card appears.

Not a goal

  • Changing which crewmates the Crewmates page lists for a real user. The rule (chatted-with, starred, or Mate) stays as specified in docs/system-specs/modules/crew-mode.md.
  • Changing the product UI, the seed script, or any product code: the fix is confined to the GUI-test scenarios (conductor ruling, option A).

Intended? / Repro

Intended? Yes — the hidden card is by design. Product code is untouched; this PR fixes the GUI-test scenarios that assumed otherwise (conductor ruling on the owner's behalf, option A).

(1) What seed_home.py / boot.sh create for nova-sky

A crewmate row, not a template. boot.sh runs seed_home.py --fixture rich --member nova-sky; the seed writes config.agents["nova-sky"] = {"kiro_agent": "kirocrew", "workspace": "default", "description": "Nova Sky -- seeded crew member for the GUI user test."} into $KIROCREW_HOME/config.json. Nothing is written under ~/.kiro/agents/, so there is no custom agent template named nova-sky. The row binds the shared kirocrew template, the default folder and Global memory, and its DM thread is empty at boot. Verified by running main's script into an empty home: config.agents keys ['default', 'nova-sky'], no DM thread bound to nova-sky.

(2) On main, does the roster search reach it?

Yes. GET /api/members (handlers/members.py::api_members) emits one row per cfg.agents key, hidden ones included; the hide is client-side in rosterFilter.ts:

  • listedByDefault(m): starred → listed; Mate → listed; else last_chat_ts > 0. nova-sky has 0 → hidden. default is listed because the rich fixture's seeded chats fold into it; Mate is exempt.
  • rosterShows(m, query): if (q) return matchesSearch(m, q) — a typed query matches name, display_name or kiro_agent as a substring, so "Nova" reaches nova-sky and shows its card.
  • Spec: crew-mode.md "Every other row is hidden and appears when the search text matches it ... the search is the one door within the roster list."

The tester typed nothing in Search crewmates…. The hint "Crewmates you haven't used yet are hidden. Search to find them." renders only when EVERY row is hidden; with default and Mate listed it does not appear, so nothing on screen told the tester a row was hidden.

(3) Is the tester's confusion intended behavior?

Yes. docs/request-for-change/rfc-session-template-vs-crewmate.md (status: accepted, owner's decision 2026-10-09, newer than rfc-crewmates-chatted-list.md), rule 3:

The Crewmates list is a messages list. A crewmate is listed when its DM thread holds a message (has_dm_message), whoever sent it, or when the user starred it; an empty thread is hidden until the search reaches it or it is the open one. ... there is no record of the user's own sends (crew_recency.json is removed) and no "created on the dashboard" or "is the default crew" exemption.

A seeded crewmate with an empty DM thread is exactly the hidden case. The scenario text ("Click the member card for Nova Sky ... it may be lower in the list") predates the hide rule (#9586, 2026-09-09 vs #17808, 2026-10-07) and assumed a listed card. The confusion is the scenario's stale assumption meeting intended product behavior, so the scenarios change and the product does not. Stamping crew_recency.json or a fake DM message from the seed would pretend the user already chatted with the crewmate, which rule 3 says must not count; starring would seed a user choice the scenarios never make.

Repro on origin/main (standalone script, no test suite)

git show refs/remotes/origin/main:scripts/gui-user-test/seed_home.py > /tmp/seed_home_main.py
KIROCREW_HOME=$(mktemp -d) PYTHONPATH=src python /tmp/seed_home_main.py --fixture rich --member nova-sky

Observed: config.agents = ['default', 'nova-sky'], no DM thread for nova-sky → last_chat_ts = 0, has_dm_message = false, not starred → listedByDefault() false → card hidden, search-only. Expected by the old scenario text: a card. Expected by the RFC: hidden.

Real failure: nightly run 38128758150, job 114438519515, head f0716252, crewmate-chat-clean step 6: "friction (blocker): I could not find a member called Nova Sky or nova-sky anywhere; the roster only ..." → "attempt 1: FAIL in 6 steps".

New test on main: test_scenarios_search_before_clicking_a_seeded_members_card fails for all five scenarios (crewmate-chat-clean.yaml: step 4 clicks a seeded member's card with no search step before it, and the same for the other four); it passes on this branch.

What changed (motivation → approach → change)

The Crewmates page hides a crewmate whose DM thread is empty until the roster search reaches it. That is the accepted design (RFC rule 3 above). The seeded nova-sky has an empty thread, so a scenario that says "click its card" asks for a card that is not on the page.

The five scenarios that open Nova Sky's chat now do what a person does for an unused crewmate: click Search crewmates…, type Nova, then click the card. The step says why the card is not there yet, so the tester does not read the short list as a bug. Each scenario's max_steps grows by two for the extra click and typing. The return visit in members-private-memory-keeps-thread keeps a hedge to search again if the card is not listed.

Scenario Search step added before max_steps
crewmate-chat-clean click Nova Sky's card 18 → 20
members-dm-hello find Nova Sky's card 18 → 20
crewmate-panel-tabs click Nova Sky 18 → 20
crewmate-reply-thread click Nova Sky 20 → 22
members-private-memory-keeps-thread click Nova Sky 24 → 26

docs/build/gui-user-test.md says, in its Seeds section, that a seeded member is hidden until searched and that scenarios search first rather than the seed pretending the user chatted. The seed script and the product are untouched.

Backwards compatibility

Compatible: only GUI-test scenario text, a test and a doc change; no product code, API or config shape moves.

Removes nothing: the diff adds a step to five scenarios and a test.

Tests

  • test/gui_user/test_scenarios_and_report.py::TestShippedScenarios::test_scenarios_search_before_clicking_a_seeded_members_card: for every scenario that opens the Crewmates page and clicks a seeded member's card, a step before that click types into Search crewmates… naming the member. Red on main for all five scenarios, green here; a future scenario that clicks a seeded card without searching fails it.

Manual verification

N/A — the scenarios are data read by the nightly gui-user-test.yml lane; the search path they describe is the one rosterFilter.ts::rosterShows implements and crew-mode.md specifies. The next nightly run exercises it.

Related Issues

Fixes #19081

Pattern harvest

Rule candidate: review-prompt
Pattern: a GUI-test scenario asserts a surface's listing rule as it was when the scenario was written; a product RFC that changes who is listed must sweep the scenarios that click a card.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 20:58
@CrysisDeu
CrysisDeu requested a review from bolichen97 October 12, 2026 20:58
@CrysisDeu
CrysisDeu force-pushed the fix/members-roster-nova-sky-19081 branch from 85465de to 6e72776 Compare October 12, 2026 21:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 393e655a952088ac32750af1773b1b8f5425cd46 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 393e655a952088ac32750af1773b1b8f5425cd46 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Only GUI-test scenario data, one test and one doc change. The scenarios now follow the specified search-only path to an unused crewmate, and the product's listing rule is left alone.

Notes

  • "Manual verification: N/A" doesn't hit the trigger. The changed files are scenario data, and CI's own nightly gui-user-test.yml lane runs them. If the next nightly still stalls at the Nova Sky card, this fix is wrong, so check that run.

[DESIGN-REVIEWED] 393e655

Description read

[DESCRIPTION-READ] bbbecd99aaa994926ddaafcb8c286f3a28880b52c92a90d0c2846b75e03224e4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 393e655a952088ac32750af1773b1b8f5425cd46 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 393e655

Verdict parsed from the review's SHA-scoped output markers for commit 393e655a952088ac32750af1773b1b8f5425cd46.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 393e655a952088ac32750af1773b1b8f5425cd46: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 393e655a952088ac32750af1773b1b8f5425cd46 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 393e655a952088ac32750af1773b1b8f5425cd46 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 393e655

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 393e655a952088ac32750af1773b1b8f5425cd46: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Oct 12, 2026
@CrysisDeu CrysisDeu changed the title fix(gui-user-test): list seeded crewmates on the Crewmates page test(gui-user): search the roster before clicking a seeded crewmate's card Oct 12, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/members-roster-nova-sky-19081 branch 2 times, most recently from 8b0f021 to aa81d36 Compare October 12, 2026 23:37
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/gui_user/test_scenarios_and_report.py and test/gui_user/scenarios/ since this head went green (5c59f0d -> 365d87c, #19549 added an unrelated scenario test); CI re-running on 80ae802.

@CrysisDeu
CrysisDeu force-pushed the fix/members-roster-nova-sky-19081 branch from aa81d36 to 80ae802 Compare October 13, 2026 01:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

This branch conflicts with main in test_scenarios_and_report.py. Please rebase it on main.

@iamwhatever
iamwhatever force-pushed the fix/members-roster-nova-sky-19081 branch from 80ae802 to 18f0310 Compare October 23, 2026 01:25
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased on main (head 18f0310). Main's #20965 renamed the user-facing word crewmate to mate, so the five scenarios now keep main's "Mates" wording and the added step reads the current placeholder "Search mates…" (website/src/i18n/locales/en.json, search_members). The test matches "Open the Mates page" / "Search mates", and the doc paragraph uses the same words. Locally: test/gui_user/test_scenarios_and_report.py 76 passed; the new search test fails on main's scenarios and passes here.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever
iamwhatever force-pushed the fix/members-roster-nova-sky-19081 branch from 18f0310 to 393e655 Compare October 23, 2026 01:35
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rebase. The conflict with main is resolved, and the branch now sits on current main.

Reviewed head 393e655.

What I checked:

  • Range diff against the previous head 80ae802: the only changes are main's crewmate to mate rename carried into the five scenarios, the doc paragraph and the new test. The added step reads "Search mates…", which matches search_members in website/src/i18n/locales/en.json on main.
  • rosterFilter.ts::listedByDefault on main still hides a row whose DM thread is empty, and a typed search still reaches it, so the fix is still needed.
  • test/gui_user/test_scenarios_and_report.py on this head: 76 passed. The new guard fails on main's scenarios (crewmate-chat-clean.yaml step 4) and passes here.
  • The AI lanes report no findings on this head, so there is nothing to reproduce.

One wording issue, inline below. It does not block the flow: typing Nova reaches the card either way.

I am not approving yet: readiness is still checking and several required test jobs were still running when I looked.

- You are on the Settings page. Open the Developer section and find the Feature Previews list.
- Turn on the preview whose title starts with "Mates" (older builds may read "Crew Members" or "Crew Members and Crew Mode"; any of those is the right one). Ignore the Webhooks preview.
- Open the Mates page. After the preview is on, a "Mates" item appears in the left rail, directly under Sessions; click it.
- Click the search box at the top of the mates list (its placeholder reads "Search mates…") and type Nova. A mate you have not chatted with yet is hidden from the list until a search reaches it, so Nova Sky (its card may read "nova-sky" instead of a display name) is not in the list before you type; the Mate and default cards being there is normal.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 "Nova Sky is not in the list before you type" is only true for the first scenario of a run

Nothing resets the target between scenarios (docs/build/gui-user-test.md, "Nothing resets the target between scenarios"). crewmate-chat-clean, members-dm-hello and the others each send a message to nova-sky. Once one of them has run, nova-sky's DM thread holds a message, listedByDefault returns true, and its card is listed before any search. A tester who reads "is not in the list before you type" and then sees the card may report a mismatch with the instructions instead of moving on.

so Nova Sky (its card may read "nova-sky" instead of a display name) is not in the list before you type

Suggestion: hedge it the same way members-private-memory-keeps-thread.yaml already does for the return visit, in all five scenarios and the doc paragraph, for example "Nova Sky may not be in the list before you type; if its card is already listed, that is normal too."

Rule: a scenario step must hold on every run order the lane can produce.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever iamwhatever self-assigned this Oct 23, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 23, 2026 03:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (7 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: GUI user-test scenarios add a roster search step before clicking a seeded mate's card, with a guard test and a note in the test guide; no product code changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ux(members): I could not find Nova Sky / nova-sky anywhere; the full roster lists only "default" and "…

4 participants