Skip to content

test(gui-user): match four scenario specs to the shipped Customize, memory and shortcuts UI - #22893

Merged
iamwhatever merged 1 commit into
mainfrom
test/22733-stale-gui-specs
Oct 21, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
test/22733-stale-gui-specs

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Four nightly GUI test scenarios describe the dashboard as it ships today, so they pass when the product works.

In the nightly run 39577973904 (#22733), the tester could not find screens these four scenarios name:

  • capabilities-agents-list-and-open-editor expects an editor with a grouped rail, an Overview diagram and a "Chat with this crewmate" button. The shipped editor has a flat rail that opens on General.
  • customize-tabs expects an "Add crewmate" button, one crewmate card, and "Customize" as the large page title. The Custom agents pane is also supposed to start with its filter row.
  • memory-open-browser-from-overview expects a "Consolidation idle (hours)" field on the first card.
  • settings-shortcuts judges the "Enable shortcuts" switch on the last screenshot. The scenario's own scroll step moves that switch off screen.

Why it matters

These four scenarios fail every night, whatever the product does, and each failure costs two paid attempts. They also hide real regressions on the same surfaces.

Not a goal

What changed (motivation → approach → change)

Each scenario now names what the dashboard draws. The text came from the shipped en catalog and the nightly screenshots, and each stale line traces to the commit that changed that screen:

Scenario Now reads UI changed in
capabilities-agents-list-and-open-editor the shared editor: flat rail (General … Danger Zone), "Edit avatar", BUILT FROM / WORKSPACE fields; roster default + nova-sky; "New crewmate" dea18bc (#21748), c78b138 (#15052)
customize-tabs "Customize" is the heading above the rail and the pane heading names the tab; roster default + nova-sky; "New crewmate"; the "New sessions use" picker sits above the Custom agents filter row c78b138 (#15052), 172afa0 (#18469)
memory-open-browser-from-overview "Memory store" card first, then "Memory Settings" with "Summarize after idle (hours)" 9481302 (#9553), 9d51f95 (#13106)
settings-shortcuts no scroll: the last frame shows the "Enable shortcuts" switch and the "Chat Navigation" rows, so every expectation is judged on it none: the scenario's own steps were wrong

Two unit tests pin the quotes to the en catalog and refuse the retired readings.

Backwards compatibility

Compatible: only scenario text and tests change. No loader, harness or product code changes.

Removes nothing: no scenario, key or check is removed; the four scenarios keep their names, tiers and limits.

Tests

In test/gui_user/test_scenarios_and_report.py:

  • test_customize_and_memory_scenarios_quote_the_shipped_labels reads the button, filter, editor rail, "Edit avatar", "Customize", and memory card and field labels from en.json / en.manual.json. It checks that the scenarios quote them, and that "Add crewmate", "Chat with this crewmate", "Who it is", "Overview" and "Consolidation idle" are gone.
  • test_shortcuts_scenario_judges_everything_on_the_frame_it_lands_on checks that the shortcuts scenario does not scroll and that every expectation names what the first frame shows (the switch and "Chat Navigation"), since the harness judges expectations on the last screenshot.

Both fail on origin/main and pass here. They passed 10 repeats in a row; test/gui_user/: 314 passed.

Manual verification

N/A: the paid GUI suite is not rerun for this PR. The new wording was checked against the 10-21 nightly screenshots (Customize Crewmates list, Custom agents pane, memory browser, Shortcuts pane) and the shipped en catalog. The next nightly run is the end-to-end check.

Related Issues

Part of #22733

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 21, 2026 20:18
@iamwhatever
iamwhatever requested a review from buluoray October 21, 2026 20:18
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 359f7cdffa2b4d8c422f3e6212c0b77347b744c0 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 359f7cdffa2b4d8c422f3e6212c0b77347b744c0 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 359f7cdffa2b4d8c422f3e6212c0b77347b744c0 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 359f7cd

Verdict parsed from the review's SHA-scoped output markers for commit 359f7cdffa2b4d8c422f3e6212c0b77347b744c0.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 359f7cdffa2b4d8c422f3e6212c0b77347b744c0: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 359f7cdffa2b4d8c422f3e6212c0b77347b744c0 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 359f7cd

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 359f7cdffa2b4d8c422f3e6212c0b77347b744c0: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the test/22733-stale-gui-specs branch from 25539f1 to 359f7cd Compare October 21, 2026 20:27
@iamwhatever

iamwhatever commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: earlier-screenshot expectation in settings-shortcuts

  • span=2cbc52c4e6db Earlier-frame expectation conflicts with the harness's evidence contract — fixed in 359f7cd

Fixed: the scenario no longer scrolls. Every expectation (Shortcuts tab highlighted, Enable shortcuts switch on, Chat Navigation rows with key caps, no clipping) is judged on the frame the pane lands on, which is the last screenshot.
No prompt-contract change: the last-screenshot rule stays as is.
test_shortcuts_scenario_judges_everything_on_the_frame_it_lands_on fails if the scenario scrolls or an expectation names an earlier screenshot.

🤖 AI-assisted comment 🤖

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 359f7cdffa2b4d8c422f3e6212c0b77347b744c0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only rewords four scenario specs to match the dashboard as it ships, and it adds tests that check the quoted labels against the en catalog.

Notes

  • The new catalog-quote test only covers the four scenarios edited here. Other scenarios can still quote labels the UI no longer draws. Making it one check that runs over every scenario would fix that, as a follow-up.

[DESIGN-REVIEWED] 359f7cd

Description read

[DESCRIPTION-READ] 3e1cbd341dba1c7c8faa4d376059670b18afadc1e02d069857df1332460bc8fc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at 359f7cd: PR Readiness passed (104 checks, 0 failing, label readiness: passed), mergeable, one commit, and green_age.py reports the green is fresh. The one GPT blocking finding (settings-shortcuts judged an earlier screenshot) is fixed in this head and answered on the PR; Design Review now passes after its earlier tarball-extraction failure. Awaiting maintainer review; Part of #22733.

🤖 AI-assisted comment 🤖

@iamwhatever
iamwhatever enabled auto-merge (squash) October 21, 2026 21:49

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: four GUI-user scenario files updated to the labels the dashboard ships (Customize, memory browser, shortcuts), plus two tests that read those labels from the en locale catalog; test files only.

@iamwhatever
iamwhatever merged commit c1ee575 into main Oct 21, 2026
105 checks passed
@iamwhatever
iamwhatever deleted the test/22733-stale-gui-specs branch October 21, 2026 22:07
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants