Repository navigation
feat(gui-user-test): new-user friction channel beside the verdict - #10224
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- test/gui_user/report.py:239 -- False positive or not applicable? A repository writer can comment: |
273ba2e to
0c95aa2
Compare
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound channel design, but cross-night identity hangs on an LLM reproducing its own wording — the ledger's core dedupe premise is fragile. Watch
Suggestions
[DESIGN-REVIEWED] 9c30877 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All evidence gathered. Here is the review. First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (10 items) — 8 justifiedIntent: let the nightly GUI tester report what confused it as a first-time user, beside PASS/FAIL — an ADDITION, derived from linked issue #10222.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 9c30877 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
0c95aa2 to
a78e525
Compare
a78e525 to
6ce8f0d
Compare
|
CI round 1 →
PR body updated first. |
6ce8f0d to
ccc0ca0
Compare
|
Round 2 →
Body updated first; 30 tests in |
ccc0ca0 to
bacbe72
Compare
|
Round 3 →
Body updated first; 31 tests exercised locally by direct invocation. |
bacbe72 to
c63f8ad
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Maintainer re-review, pinned to d134a36. Requesting changes; the earlier content approval missed a genuine trust-boundary issue, so I am withdrawing that recommendation rather than overriding the new security finding.
In .github/workflows/gui-user-test.yml:294-301, the scheduled workflow selects the first repository-wide artifact matching gui-user-test-friction-ledger and trusts its workflow_run.id. The artifact name alone does not establish that it came from the trusted scheduled workflow/main branch. This persisted input later reaches the issue-writing job. Select an eligible successful scheduled run of this workflow on the trusted branch first, then obtain the ledger artifact only from that run; preserve fail-closed behavior for API/download failures. Add tests for rejecting wrong-workflow, wrong-event and untrusted-branch provenance. I independently inspected the selection/download code; no untrusted artifact was downloaded or executed in this review.
The docs conflict was resolved correctly. Separately, guard snapshot generation when friction.json is absent after a harness failure. Do not weaken the security gate or treat the clean textual rebase as proof of safety. No local runtime verification performed in this delta review.
d134a36 to
16b6a42
Compare
|
Thanks @bolichen97 — agreed on both, and thanks for withdrawing rather than overriding. Pushed Ledger provenance (F1, security). The repository-wide
Fail-closed is preserved: any API error or malformed answer propagates → exit 2 → the merge step fails and the canonical ledger is untouched. Only a complete answer with no eligible run yields Tests ( Missing Docs ( |
16b6a42 to
156042e
Compare
|
Round 9 — pushed Opus — GPT F1 — a re-run excluded itself and read last night's ledger. GPT F2 — GPT finding — Tests: +7 (64 cases total, all pass locally by direct invocation) — collect fold in both orders; re-run reads its own attempt and never lists the nightlies; five untrusted-current-run shapes fall through to the nightlies without listing their artifacts; existing provenance/CLI tests moved to |
|
Round 10 — pushed
Opus cleared this head. 65 test cases pass locally by direct invocation; black / isort / flake8 clean; workflow YAML parses; docs and PR body updated. |
|
Round 11 — pushed
71 test cases pass locally by direct invocation; black / isort / flake8 clean; mypy no new errors; workflow YAML parses; docs and PR body updated. |
|
Round 12 — pushed
72 test cases pass locally by direct invocation; black / flake8 clean; mypy no new errors; workflow YAML parses; PR body updated. |
|
Round 13 — pushed
75 test cases pass locally by direct invocation; black / flake8 clean; mypy no new errors; workflow YAML parses; docs and PR body updated. |
The tester runs as a first-time user (scenarios.PERSONAS, optional per scenario `persona`) and files report_friction entries -- surface, element, what confused me, expected/actual, severity -- whenever it stalls; the harness validates each, stamps feature / scenario / screenshot / step and carries them in summary.json without touching PASS/FAIL. friction.py keys entries across nights, merges them into a ledger artifact, renders the "New-user friction" section for verdict.md, the run summary and the nightly issue, and opens at most five ux(<feature>) issues a night (recurrences comment, cosmetic stays in the summary). Closes #10222
|
Round 14 — pushed
76 test cases pass locally by direct invocation; black / flake8 clean; workflow YAML parses; docs and PR body updated. |
|
Not approving for now: the latest maintainer review is CHANGES_REQUESTED on a trust-boundary finding in |
|
@bolichen97 gentle re-request — your CHANGES_REQUESTED is pinned to What was addressed from your review (the ledger-provenance finding, shared with the GPT review):
If the current head looks right, a fresh review would unblock it — thanks! |
bolichen97
left a comment
There was a problem hiding this comment.
Maintainer review (agent-assisted), pinned to 9c30877.
Re-reviewed all seven changed files. The earlier ledger-provenance blocker is addressed: scheduled default-branch workflow identity, source repository/head repository, and artifact/run binding are checked before accepting the ledger, with failure-closed handling. No blocking findings. Non-blocking: preserve a safely redacted gh error reason for failed issue filing; CalledProcessError alone loses diagnostic stderr.
Verification: reviewed source/tests and observed passing CI and PR Readiness on this exact head; no local test rerun or manual UI reproduction. Approval is for this commit only.
Closes #10222. Part of #9578.
Summary
A second, independent channel on the GUI user-test lane: what confused the tester, reported beside PASS/FAIL and never folded into it. A scenario can pass with five confusions logged, or fail with none.
scenarios.PERSONAS, new optional YAML fieldpersona, defaultnew-user). The tester is a first-time Kiro Crew user -- no docs, ordinary chat-app / editor background -- with a standing instruction to callreport_frictionthe moment it pauses for more than a glance, cannot find a control, clicks the wrong thing, does not understand a label / icon / message, does not know what is happening, or finds the layout hides the main action, then carry on.persona: noneis the bare tester (no tool, unchanged prompt) for expert-path scenarios or to measure the channel's own cost. The block is spliced into the harness system prompt after the non-negotiable RULES and before HOW TO WORK, so the persona cannot outrank the safety rules.report_frictiontool,test/gui_user/friction.py):surface,element,what_confused(first person),expected,actual,severity∈ blocker / slows-down / cosmetic.friction.validate_entryis the only way intosummary.json: fields typed, trimmed and capped at 240 chars, unknown fields refused,featuremust be a registry slug; the harness stampsfeature(from the scenario),scenario, the last screenshot's artifact path and the step. Twelve per attempt; a duplicate, malformed or over-cap call gets a one-line tool result and never fails the task. Friction calls do not count againstmax_steps, and every friction entry in a batched response cites the screenshot and step the model was actually looking at when it decided the batch (not the state after an action that precedes it in the same response).AttemptResult.friction[]carries them;summary["friction_count"]marks a run that actually had the channel (set only for a real run with at least one friction-reporting persona -- a dry run or an all-noneselection never claims "nothing confusing").entry_key(feature, element, what_confused)after case / whitespace / punctuation normalization. The workflow fetches the previousgui-user-test-friction-ledgerartifact from this run's own earlier attempt when it is a re-run of a scheduled night (that attempt already filed issues; the ledger upload overwrites for the same reason), else from the newest completed scheduled run of this workflow on the default branch only (friction.py pick-ledger: run path, event, head branch, head+base repository and conclusion all checked, then the artifact's ownworkflow_runblock against that run -- never a repository-wide name lookup, which a fork PR's artifact could satisfy; none = first night),friction.py mergefolds tonight in (recurrence →count+1,last_seen, the newest sighting's surface / wording / evidence taken whole, worst severity kept; runs are told apart byrun_url, so a second run on the same day refreshes the evidence under its own artifact without counting the day twice, and a re-run of the same run is the same run even after midnight UTC -- no count, no recurrence comment), writesresults/friction.json(tonight's rows with counts and issue numbers) and -- on the scheduled run only -- re-uploads the ledger (90-day retention) after the issue step has recorded what it opened. A dispatch on any ref reads the ledger for its report but never writes it back, so runs on different refs cannot race each other's entries into the canonical ledger (the per-ref concurrency group already serializes one ref). Only a complete API answer with no eligible scheduled run holding a ledger starts a fresh one; a listing or download error fails the merge step, so a transient outage cannot replace the canonical ledger with an empty one. A night the harness did not run (nofriction.json) carries the ledger forward and the issue step files nothing.verdict.md, the run summary and the nightly failure issue end with a "New-user friction" section -- one table per feature in registry order, worst severity first, capped atSECTION_MAX_CHARS= 30k with a counted trailer (issue bodies / comments stop at 64k), where / expected → actual / how many nights / screenshot link into the artifact.report.py --friction results/friction.jsonswaps in the ledger-aware rows (the merge step re-rendersverdict.mdwith them, so the uploaded verdict carries the counts); on the nightly,friction.py snapshotrebuilds that file from the ledger right after the issue step, so the run summary and the nightly report show the issue numbers just opened (the uploaded artifact keeps the pre-filing snapshot). Model text is rendered inert (pipes, backticks, angle brackets, link brackets neutralized,@zero-width-joined, URL schemes andwww.defanged,#<digits>zero-width-split so an on-screen#123never cross-references an issue so a URL the tester read off an injected page never autolinks in a bot-authored issue), same posture as the existing fenced final reports.friction.py issues): each non-cosmetic row without an issue → firstgh issue list --search <key> in:bodyand adopt an OPEN issue whose body carries the exact marker (a number the ledger lost to a cancelled run is never duplicated; a failed lookup skips the row for the night), else →ux(<feature>): <what confused me>with labelsux,channel: gui-user-test,area:mapped from the feature (AREA_LABELS, keyed only by registry slugs -- a test holdsAREA_LABELS ⊆ FEATURES-- defaultarea: dashboard), body marked<!-- gui-user-friction <key> -->; at most five a night, the rest wait in the summary and get their turn on a later night. A recurrence whose row has an issue → one "again on " comment per date (last_commentedmakes a same-day rerun idempotent); every row seen tonight that has an issue is checked, and if a human has since closed that issue, the mapping is retired (kept underclosed_issues) and a fresh issue is opened that links back to it.friction.jsonis scoped to the run's own keys, never to the date, so a same-day dispatch reports its own rows with its own artifact links, not the nightly's. Cosmetic never leaves the summary. The merge step runs underset -euo pipefail, so a failed ledger write afterfriction.jsonfails the step instead of letting a later render paint it green. The issues step exits with theissuescommand's code even though the snapshot runs afterwards, so a partial filing failure stays red. The ledger is written atomically (sibling temp file,fsync,os.replace; a cut-short write leaves the previous document intact) after every successfulghmutation, so a failure partway through a batch keeps the numbers already opened (a rerun files only what failed) and the step exits 2 with the failures listed. Labels are created with--forceif missing (tolerated if not permitted). Workflow gainsactions: readfor the ledger download.Cost
Persona block ≈ 450 input tokens on every model call; each
report_frictioncall is one extra round trip (≈ 7k input with the three kept screenshots, ≈ 150 output). At two to four friction calls per scenario that is ≈ +$0.10 per scenario on Sonnet pricing, ≈ +$1 a night on a twelve-scenario tier -- inside the $10 ceiling, withpersona: noneon low-priority scenarios as the lever if it ever binds. The nightly--budget-usdis unchanged here.What the report looks like
The lane's Bedrock role trusts
mainonly, so a branch dispatch cannot produce a real sample (#9965 recorded the credential failure). The section below is rendered byfriction.pyfrom three hand-written entries fed throughmergetwice (two "nights") -- the exact shape the first nightly after merge will produce, with the tester's own wording in place of mine:The nightly issue for the blocker row would be titled
ux(settings): I could not tell whether the page had already changed theme or was still loading.with labelsux,channel: gui-user-test,area: dashboard.Tests
test/gui_user/test_friction.py(new, 58 tests / 76 cases, written not run under pytest locally -- CI runs them): persona default /none/ unknown; prompt splice order;report_frictionpresent only for a reporting persona in both tool modes and absent from the x11 action vocabulary; entry validation (trim, cap, rejects, unknown feature, non-mapping); key normalization;collectdedupe (a key repeated across attempts folds to the worst severity with the newest evidence, so a cosmetic first sighting cannot hide a blocker retry); harness_record_friction(stores with last screenshot, duplicate / malformed / cap answers never error);AttemptResult.frictionround-trips throughasdict; ledger merge (counts, worst severity, idempotent re-merge,tonightordering, load shapes); issue plan (cosmetic skipped, cap, hold-over, recurrence comment) andfile_issuesagainst a fakegh(labels, title, marker, comment text, label-create failure tolerated); rendering (grouping, ordering, counts, issue numbers, inert cells, empty state,report.render_markdownwith and without the channel,--frictionoverride, console line); atomic_write(no temp file left behind; a failed replace keeps the prior document); reconcile (marker match adopted, digest-in-prose not, failed lookup skips the key); section budget (80 max-length rows → ≤30k, worst rows shown, trailer counts the rest);cancelled/timed_outnightlies accepted as ledger sources,skippednot; CLImerge/issues --dry-run/snapshot(rows carry the issue number, provenance kept) /pick-ledger(writes the pick to--out, echoes nothing from the API, exits 2 on an API failure leaving an earlier answer intact) / missing input → 2; ledger provenance (ledger_source_runsrejects another workflow,pull_request/workflow_dispatch, an untrusted branch, a fork head or foreign base repository, missing blocks, unfinished or cancelled runs, malformed ids and junk; accepts a red nightly;ledger_artifact_idrejects another run's artifact, another branch, another name, expired;pick_ledgerreads a re-run's own prior attempt first, never takes a ledger from an untrusted current run (dispatch / other branch / fork / malformed), walks the nightlies newest first page by page (100 a page, bounded byMAX_RUN_PAGES), skips a run without a ledger, never lists a rejected run's artifacts, returns nulls for a first night and propagates API errors / malformed answers); a fullRunner.attemptagainst a scripted client and a fake display (action +report_frictionin one response → PASS, one step, the entry cites the start screenshot at step 0, text-only tool result, persona in the system prompt). Existing tests are untouched:custom_tools()andSYSTEM_PROMPTare unchanged objects. black / isort / flake8 clean; workflow YAML parses.Not in this PR
A harness step placeholder for the seeded home path (needed by the knowledge P0 scenario) and a seeded-spawn hook (needed to assert the collapsed subagent cards) -- both tracked from #10181. The scenario batches themselves are #10180 / #10182 on top of #10040; this PR is based on
mainand touchestest_scenarios_and_report.pynowhere, so the stack rebases cleanly in either order.