Skip to content

feat(gui-user-test): new-user friction channel beside the verdict - #10224

Merged
bolichen97 merged 1 commit into
mainfrom
feat/gui-user-test-friction
Sep 15, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/gui-user-test-friction

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Closes #10222. Part of #9578.

for our GUI testers, I also want them to take from a new user perspective and report UI elements / workings / layouts that are confusing to them

Summary

A second, independent channel on the GUI user-test lane: what confused the tester, reported beside PASS/FAIL and never folded into it. A scenario can pass with five confusions logged, or fail with none.

  • Persona (scenarios.PERSONAS, new optional YAML field persona, default new-user). The tester is a first-time Kiro Crew user -- no docs, ordinary chat-app / editor background -- with a standing instruction to call report_friction the moment it pauses for more than a glance, cannot find a control, clicks the wrong thing, does not understand a label / icon / message, does not know what is happening, or finds the layout hides the main action, then carry on. persona: none is the bare tester (no tool, unchanged prompt) for expert-path scenarios or to measure the channel's own cost. The block is spliced into the harness system prompt after the non-negotiable RULES and before HOW TO WORK, so the persona cannot outrank the safety rules.
  • Entries (report_friction tool, test/gui_user/friction.py): surface, element, what_confused (first person), expected, actual, severity ∈ blocker / slows-down / cosmetic. friction.validate_entry is the only way into summary.json: fields typed, trimmed and capped at 240 chars, unknown fields refused, feature must be a registry slug; the harness stamps feature (from the scenario), scenario, the last screenshot's artifact path and the step. Twelve per attempt; a duplicate, malformed or over-cap call gets a one-line tool result and never fails the task. Friction calls do not count against max_steps, and every friction entry in a batched response cites the screenshot and step the model was actually looking at when it decided the batch (not the state after an action that precedes it in the same response). AttemptResult.friction[] carries them; summary["friction_count"] marks a run that actually had the channel (set only for a real run with at least one friction-reporting persona -- a dry run or an all-none selection never claims "nothing confusing").
  • Cross-night identity: entry_key(feature, element, what_confused) after case / whitespace / punctuation normalization. The workflow fetches the previous gui-user-test-friction-ledger artifact from this run's own earlier attempt when it is a re-run of a scheduled night (that attempt already filed issues; the ledger upload overwrites for the same reason), else from the newest completed scheduled run of this workflow on the default branch only (friction.py pick-ledger: run path, event, head branch, head+base repository and conclusion all checked, then the artifact's own workflow_run block against that run -- never a repository-wide name lookup, which a fork PR's artifact could satisfy; none = first night), friction.py merge folds tonight in (recurrence → count+1, last_seen, the newest sighting's surface / wording / evidence taken whole, worst severity kept; runs are told apart by run_url, so a second run on the same day refreshes the evidence under its own artifact without counting the day twice, and a re-run of the same run is the same run even after midnight UTC -- no count, no recurrence comment), writes results/friction.json (tonight's rows with counts and issue numbers) and -- on the scheduled run only -- re-uploads the ledger (90-day retention) after the issue step has recorded what it opened. A dispatch on any ref reads the ledger for its report but never writes it back, so runs on different refs cannot race each other's entries into the canonical ledger (the per-ref concurrency group already serializes one ref). Only a complete API answer with no eligible scheduled run holding a ledger starts a fresh one; a listing or download error fails the merge step, so a transient outage cannot replace the canonical ledger with an empty one. A night the harness did not run (no friction.json) carries the ledger forward and the issue step files nothing.
  • Where it shows: verdict.md, the run summary and the nightly failure issue end with a "New-user friction" section -- one table per feature in registry order, worst severity first, capped at SECTION_MAX_CHARS = 30k with a counted trailer (issue bodies / comments stop at 64k), where / expected → actual / how many nights / screenshot link into the artifact. report.py --friction results/friction.json swaps in the ledger-aware rows (the merge step re-renders verdict.md with them, so the uploaded verdict carries the counts); on the nightly, friction.py snapshot rebuilds that file from the ledger right after the issue step, so the run summary and the nightly report show the issue numbers just opened (the uploaded artifact keeps the pre-filing snapshot). Model text is rendered inert (pipes, backticks, angle brackets, link brackets neutralized, @ zero-width-joined, URL schemes and www. defanged, #<digits> zero-width-split so an on-screen #123 never cross-references an issue so a URL the tester read off an injected page never autolinks in a bot-authored issue), same posture as the existing fenced final reports.
  • Issues (nightly only, friction.py issues): each non-cosmetic row without an issue → first gh issue list --search <key> in:body and adopt an OPEN issue whose body carries the exact marker (a number the ledger lost to a cancelled run is never duplicated; a failed lookup skips the row for the night), else → ux(<feature>): <what confused me> with labels ux, channel: gui-user-test, area: mapped from the feature (AREA_LABELS, keyed only by registry slugs -- a test holds AREA_LABELS ⊆ FEATURES -- default area: dashboard), body marked <!-- gui-user-friction <key> -->; at most five a night, the rest wait in the summary and get their turn on a later night. A recurrence whose row has an issue → one "again on " comment per date (last_commented makes a same-day rerun idempotent); every row seen tonight that has an issue is checked, and if a human has since closed that issue, the mapping is retired (kept under closed_issues) and a fresh issue is opened that links back to it. friction.json is scoped to the run's own keys, never to the date, so a same-day dispatch reports its own rows with its own artifact links, not the nightly's. Cosmetic never leaves the summary. The merge step runs under set -euo pipefail, so a failed ledger write after friction.json fails the step instead of letting a later render paint it green. The issues step exits with the issues command's code even though the snapshot runs afterwards, so a partial filing failure stays red. The ledger is written atomically (sibling temp file, fsync, os.replace; a cut-short write leaves the previous document intact) after every successful gh mutation, so a failure partway through a batch keeps the numbers already opened (a rerun files only what failed) and the step exits 2 with the failures listed. Labels are created with --force if missing (tolerated if not permitted). Workflow gains actions: read for the ledger download.

Cost

Persona block ≈ 450 input tokens on every model call; each report_friction call is one extra round trip (≈ 7k input with the three kept screenshots, ≈ 150 output). At two to four friction calls per scenario that is ≈ +$0.10 per scenario on Sonnet pricing, ≈ +$1 a night on a twelve-scenario tier -- inside the $10 ceiling, with persona: none on low-priority scenarios as the lever if it ever binds. The nightly --budget-usd is unchanged here.

What the report looks like

The lane's Bedrock role trusts main only, so a branch dispatch cannot produce a real sample (#9965 recorded the credential failure). The section below is rendered by friction.py from three hand-written entries fed through merge twice (two "nights") -- the exact shape the first nightly after merge will produce, with the tester's own wording in place of mine:

## New-user friction

_2 moment(s) where a first-time user stalled: 1 blocker · 1 slows-down · 0 cosmetic. Reported by the tester persona beside the verdict; a scenario can PASS and still list friction._

### Crew Members (`members`)

| Severity | What confused me | Where | Expected → actual | Seen | Screenshot |
|---|---|---|---|---|---|
| slows-down | I did not expect feature previews to live under Developer. | Settings page → Developer section header | A section named Previews or Labs near the top → It was the last item, under Developer | 2× · last 2026-09-13 | [`members-dm-hello/attempt-1/02-left_click.png`](artifact) |

### Settings (`settings`)

| Severity | What confused me | Where | Expected → actual | Seen | Screenshot |
|---|---|---|---|---|---|
| blocker | I could not tell whether the page had already changed theme or was still loading. | Settings › Display → Theme dropdown | An immediate visible change → Nothing changed for a second, then it did | new | [`settings-theme-toggle/attempt-1/03-left_click.png`](artifact) |

The nightly issue for the blocker row would be titled ux(settings): I could not tell whether the page had already changed theme or was still loading. with labels ux, channel: gui-user-test, area: dashboard.

Tests

test/gui_user/test_friction.py (new, 58 tests / 76 cases, written not run under pytest locally -- CI runs them): persona default / none / unknown; prompt splice order; report_friction present only for a reporting persona in both tool modes and absent from the x11 action vocabulary; entry validation (trim, cap, rejects, unknown feature, non-mapping); key normalization; collect dedupe (a key repeated across attempts folds to the worst severity with the newest evidence, so a cosmetic first sighting cannot hide a blocker retry); harness _record_friction (stores with last screenshot, duplicate / malformed / cap answers never error); AttemptResult.friction round-trips through asdict; ledger merge (counts, worst severity, idempotent re-merge, tonight ordering, load shapes); issue plan (cosmetic skipped, cap, hold-over, recurrence comment) and file_issues against a fake gh (labels, title, marker, comment text, label-create failure tolerated); rendering (grouping, ordering, counts, issue numbers, inert cells, empty state, report.render_markdown with and without the channel, --friction override, console line); atomic _write (no temp file left behind; a failed replace keeps the prior document); reconcile (marker match adopted, digest-in-prose not, failed lookup skips the key); section budget (80 max-length rows → ≤30k, worst rows shown, trailer counts the rest); cancelled / timed_out nightlies accepted as ledger sources, skipped not; CLI merge / issues --dry-run / snapshot (rows carry the issue number, provenance kept) / pick-ledger (writes the pick to --out, echoes nothing from the API, exits 2 on an API failure leaving an earlier answer intact) / missing input → 2; ledger provenance (ledger_source_runs rejects another workflow, pull_request / workflow_dispatch, an untrusted branch, a fork head or foreign base repository, missing blocks, unfinished or cancelled runs, malformed ids and junk; accepts a red nightly; ledger_artifact_id rejects another run's artifact, another branch, another name, expired; pick_ledger reads a re-run's own prior attempt first, never takes a ledger from an untrusted current run (dispatch / other branch / fork / malformed), walks the nightlies newest first page by page (100 a page, bounded by MAX_RUN_PAGES), skips a run without a ledger, never lists a rejected run's artifacts, returns nulls for a first night and propagates API errors / malformed answers); a full Runner.attempt against a scripted client and a fake display (action + report_friction in one response → PASS, one step, the entry cites the start screenshot at step 0, text-only tool result, persona in the system prompt). Existing tests are untouched: custom_tools() and SYSTEM_PROMPT are unchanged objects. black / isort / flake8 clean; workflow YAML parses.

Not in this PR

A harness step placeholder for the seeded home path (needed by the knowledge P0 scenario) and a seeded-spawn hook (needed to assert the collapsed subagent cards) -- both tracked from #10181. The scenario batches themselves are #10180 / #10182 on top of #10040; this PR is based on main and touches test_scenarios_and_report.py nowhere, so the stack rebases cleanly in either order.

@CrysisDeu CrysisDeu added area: ci GitHub Actions, AI review gates, readiness checks channel: insider Reported from an insider prerelease build — candidate release blocker labels Sep 11, 2026
@CrysisDeu
CrysisDeu requested a review from a team September 11, 2026 22:12
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 9c30877cd4be40491cb64adec634187a3a01b53c and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- test/gui_user/report.py:239 -- friction_entries is not None falsely reports no confusion for all-persona: none runs -> Fix: require "friction_count" in summary; use friction_entries only as the row override.
[GPT-REVIEWED] 9c30877

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9c30877cd4be40491cb64adec634187a3a01b53c: <one-sentence reason>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 11, 2026
@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from 273ba2e to 0c95aa2 Compare September 11, 2026 22:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 9c30877cd4be40491cb64adec634187a3a01b53c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Sound channel design, but cross-night identity hangs on an LLM reproducing its own wording — the ledger's core dedupe premise is fragile.

Watch

  • entry_key(feature, element, what_confused) normalizes case/whitespace/punctuation but not paraphrase. The persona will describe the same confusion in different words on different nights → new key → the "one row with a count, not a new row each night" promise (module docstring) fails, recurrence counts fragment, and near-duplicate ux(<feature>) issues accumulate under the 5/night cap, diluting exactly the signal the channel exists to produce. The exact-marker reconcile (ISSUE_MARKER + gh issue list --search <key>) can't adopt a paraphrase.
    Clears when: issue adoption also searches open ux/channel: gui-user-test issues by (feature, element) before opening, or the first few real nightlies show wording is stable enough that duplicates stay rare.
  • Closed means "fixed" with no won't-fix escape: reconcile retires a CLOSED issue's mapping and "a fresh issue is opened that links back to it". A confusion the team closes as by-design recurs every night the persona re-reports it, producing a close → refile → comment → close loop with no in-band suppression.
    Clears when: a closed-as-not-planned issue (gh stateReason, or a wontfix label) permanently suppresses the key instead of refiling.

Suggestions

  • The live gh-mutation path is only exercised by the first real nightly on main (disclosed); consider a one-shot dry-run dispatch on main right after merge to shake out the pick-ledger/merge/upload plumbing before it owns the canonical ledger.

[DESIGN-REVIEWED] 9c30877

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 9c30877cd4be40491cb64adec634187a3a01b53c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All evidence gathered. Here is the review.

First-Principles-Verdict: CONCERNS

PERSONAS is a registry with one real persona; the persona: key is set by zero scenarios — a boolean the diff never needed as an enum.

Not justified as shipped

  • Item 3 — one consumer, generalized: grep persona across test/gui_user/scenarios/*.yaml → 0 of 3 files set it; PERSONAS holds one real variant plus "". The none variant's only support is "for expert-path scenarios or to measure the channel's own cost" — no such scenario exists, so that is so-we-can-later.
  • Item 9 (--dry-run on friction.py issues) — zero consumers: grep dry-run in gui-user-test.yml → 0; only tests pass it.

What this change ships

Inventory (10 items) — 8 justified

Intent: let the nightly GUI tester report what confused it as a first-time user, beside PASS/FAIL — an ADDITION, derived from linked issue #10222.

  1. Every scenario's tester now runs as a first-time-user persona by default (+~$1/night) — justified
  2. The tester gets a report_friction tool; entries land in summary.json — justified
  3. New scenario YAML key persona with a none variant — one consumer, generalized: 0 scenarios set it, one real persona exists
  4. A cross-night friction ledger artifact (90-day retention, fork-provenance-checked) — justified
  5. Verdict, run summary and nightly issue gain a "New-user friction" section — justified
  6. Nightly auto-files up to five ux(<feature>) issues, comments recurrences, retires closed ones — justified
  7. Labels ux, channel: gui-user-test, area:* auto-created — justified
  8. Workflow gains actions: read — justified
  9. New friction.py CLI (pick-ledger / merge / snapshot / issues) — zero consumers for its --dry-run flag only; the rest justified
  10. Console summary gains a "new-user friction: …" line — justified

Watch

  • The persona surface: the description's "persona: none is the bare tester … or to measure the channel's own cost" names no scenario that needs it today (count: 0/3 YAMLs). Splicing the one persona unconditionally removes the key, the registry and the reports_friction property.
    Clears when: a shipped scenario sets persona: none, or the key is deleted.

Subtractions

  • Drop the persona key, PERSONAS, DEFAULT_PERSONA and Scenario.persona_prompt/reports_friction (scenarios.py); splice the one persona block and arm the tool unconditionally — reintroduce an off-switch when a scenario needs one.
  • Delete --dry-run from friction.py issues (0 workflow consumers); tests can call plan_issues() directly, as test_plan_skips_cosmetic… already does.

[FIRST-PRINCIPLES-REVIEWED] 9c30877

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 9c30877cd4be40491cb64adec634187a3a01b53c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9c30877

Verdict parsed from the review's SHA-scoped output markers for commit 9c30877cd4be40491cb64adec634187a3a01b53c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9c30877cd4be40491cb64adec634187a3a01b53c: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from 0c95aa2 to a78e525 Compare September 11, 2026 22:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 11, 2026
@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from a78e525 to 6ce8f0d Compare September 11, 2026 22:51
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

CI round 1 → 6ce8f0d328. All three GPT 5.6 findings on 0c95aa2a96 were real and are fixed, plus the two mechanical reds:

  • BLOCKING friction.py:611 (issue mappings lost when a later gh call fails) -- file_issues now takes a persist callback and the CLI passes the ledger writer; the ledger is written after every successful create or comment, a failed gh call is recorded under failed and the batch continues, and the step exits 2 with the failures listed. Pinned by test_failure_mid_batch_keeps_the_mappings_already_made: the first mapping is on disk before the second create raises, and a rerun plans only the one that failed.
  • FINDING friction.py:435 (tonight excludes capped rows unless they recur) -- candidates are now every un-issued non-cosmetic row in the whole ledger, tonight's sightings first, then held-over rows, each in registry-then-severity order; recurrence comments stay restricted to tonight's rows. Test updated to cover a held-over row not seen tonight.
  • FINDING harness.py:769 (friction_count claimed on dry runs / persona: none) -- friction_channel_ran(scenarios, dry_run) gates the key: only a real run with at least one friction-reporting persona sets it, so the report never says "nothing confusing" for a channel that never ran. Tested.
  • Backend Lint: subprocess.run in _gh pinned to encoding="utf-8" (subprocess-encoding gate).
  • Backend Tests (Windows) shard 1: five test_friction.py failures were fixture mistakes on my side (the two-night helper filed every entry under members, so the registry-order and plan assertions could not hold); fixed, and the module's 30 tests were exercised locally by direct invocation before this push.

PR body updated first.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 11, 2026
@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from 6ce8f0d to ccc0ca0 Compare September 11, 2026 23:18
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 2 → ccc0ca0383, both GPT 5.6 findings on 6ce8f0d328 fixed:

  • BLOCKING gui-user-test.yml:285 (cross-ref runs overwrite the shared ledger) -- the ledger now has one writer: every run still downloads it so its report shows counts, but the upload step is gated on github.event_name == 'schedule' (the issue step already was), so a dispatch on any ref never writes the canonical ledger back and two refs cannot race each other's entries or issue numbers. The per-ref concurrency group serializes runs on one ref. Documented in the workflow comment and the doc.
  • BLOCKING friction.py:448 (same-day rerun posts the recurrence comment twice) -- plan_issues skips rows whose last_commented equals tonight's date (file_issues already stamps it after each successful comment). Pinned by the extended plan test: after the comment is recorded the same-day plan is empty, and a previous-day stamp still comments.

Body updated first; 30 tests in test_friction.py exercised locally by direct invocation.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 11, 2026
@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from ccc0ca0 to bacbe72 Compare September 11, 2026 23:51
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 3 → bacbe72a9e.

  • BLOCKING harness.py:524 (batched action + report_friction records the post-action screen) -- real. The harness now snapshots last_shot / steps once per assistant response, before iterating its tool_use blocks, and every friction entry in that batch cites the snapshot: all blocks in one response were decided from the same screen the model was shown. Pinned by a new end-to-end test that drives Runner.attempt with a scripted client and a fake display (left_click then report_friction in one response → the entry cites 01-start.png at step 0, the action still runs once, the friction tool result is text-only).
  • Backend Tests (Windows) shard 4 on ccc0ca0383 failed in aws_control/.../test_sensitive_source_and_report_identity.py (opencode auth.json classification) -- a file this PR does not touch; treated as main-inherited and left to re-run on this push rather than rebased, since the base is already current.

Body updated first; 31 tests exercised locally by direct invocation.

@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from bacbe72 to c63f8ad Compare September 11, 2026 23:52
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 11, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 12, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer re-review, pinned to d134a36. Requesting changes; the earlier content approval missed a genuine trust-boundary issue, so I am withdrawing that recommendation rather than overriding the new security finding.

In .github/workflows/gui-user-test.yml:294-301, the scheduled workflow selects the first repository-wide artifact matching gui-user-test-friction-ledger and trusts its workflow_run.id. The artifact name alone does not establish that it came from the trusted scheduled workflow/main branch. This persisted input later reaches the issue-writing job. Select an eligible successful scheduled run of this workflow on the trusted branch first, then obtain the ledger artifact only from that run; preserve fail-closed behavior for API/download failures. Add tests for rejecting wrong-workflow, wrong-event and untrusted-branch provenance. I independently inspected the selection/download code; no untrusted artifact was downloaded or executed in this review.

The docs conflict was resolved correctly. Separately, guard snapshot generation when friction.json is absent after a harness failure. Do not weaken the security gate or treat the clean textual rebase as proof of safety. No local runtime verification performed in this delta review.

@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from d134a36 to 16b6a42 Compare September 12, 2026 05:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 12, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Thanks @bolichen97 — agreed on both, and thanks for withdrawing rather than overriding. Pushed 16b6a425a0 (single commit, still on current main).

Ledger provenance (F1, security). The repository-wide actions/artifacts?name=… lookup is gone. friction.py pick-ledger now resolves the source in two verified hops:

  1. GET actions/workflows/gui-user-test.yml/runs?event=schedule&branch=<default>&status=completed → trusted_ledger_runs() keeps a run only if path == .github/workflows/gui-user-test.yml, event == schedule, head_branch == <default branch>, repository.full_name and head_repository.full_name both equal this repository, status == completed with conclusion success or failure (a red nightly still wrote its ledger before the verdict reddened the job; cancelled/skipped are out), and id != $GITHUB_RUN_ID. Newest first.
  2. For each candidate, GET actions/runs/<id>/artifacts?name=gui-user-test-friction-ledger → ledger_artifact_id() accepts an artifact only if its own workflow_run.id is that run and workflow_run.head_branch is the trusted branch, and it is not expired. The first run that holds one wins; a night whose merge step failed (uploaded nothing) is skipped, not treated as a reset.

Fail-closed is preserved: any API error or malformed answer propagates → exit 2 → the merge step fails and the canonical ledger is untouched. Only a complete answer with no eligible run yields {"run_id": null, …} = first night. The trusted branch is github.event.repository.default_branch (the step refuses if it is empty). Download still goes through gh run download <trusted run id> -n ….

Tests (TestLedgerProvenance, 9 tests / 21 cases): rejects another workflow path, pull_request, workflow_dispatch, an untrusted branch, a fork head_repository, a foreign base repository, a missing block, in-progress and cancelled runs, malformed ids and non-dict junk; accepts a red nightly; artifact checks (other run, other branch, other name, expired, missing block, malformed id); pick_ledger walks newest first, never lists a rejected run's artifacts, returns nulls on a first night and raises on API errors / malformed listings; CLI prints the pick and exits 2 on a gh failure.

Missing friction.json (F2). The issue step now exits 0 with a note when results/friction.json is absent (the harness did not run; the merge step only carried the ledger forward). Nothing was tested that night, so no row is "tonight's" and the held-over rows wait for a night with evidence — snapshot is never reached without the file it needs.

Docs (docs/build/gui-user-test.md, new "Where the ledger may come from" bullet) and the PR body updated. Static gates and the 57 friction test cases pass locally; workflow YAML parses.

Comment thread test/gui_user/friction.py Fixed
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 12, 2026
@CrysisDeu
CrysisDeu force-pushed the feat/gui-user-test-friction branch from 16b6a42 to 156042e Compare September 12, 2026 06:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 12, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 9 — pushed 156042e0b5 (single commit) for the three blockers and the finding on 16b6a425a0; all four were real.

Opus — TRUSTED_BRANCH empty on schedule. Confirmed against add-contributor.yml / pr-readiness.yml; now ${{ github.event.repository.default_branch || github.ref_name }} with a comment saying why. Without it the nightly would have failed the merge step every night.

GPT F1 — a re-run excluded itself and read last night's ledger. pick-ledger --self-run $GITHUB_RUN_ID (was --exclude-run) now consults the current run FIRST: it fetches actions/runs/<id> and judges it with the same provenance predicate (trusted_run_id: workflow path, schedule, trusted branch, head + base repository) except that it may be in progress; if that run already holds a ledger artifact — attempt 1 filed issues and uploaded it — that ledger wins, so attempt 2 sees the issue numbers and files nothing twice. An untrusted current run (dispatch, other branch, fork, malformed) never supplies its own ledger and its artifacts are not even listed; the walk then continues to the completed nightlies as before. The ledger upload step gains overwrite: true, since attempt 2 uploads under a name attempt 1 already used in the same run (upload-artifact v7 would otherwise fail and the night's numbers would never persist).

GPT F2 — collect() first-sighting-wins could hide a blocker. A key repeated within a run (retry, or two scenarios on the same control) now folds like merge_ledger folds nights: newest sighting's evidence, worst severity of the two. Docstring states the rule.

GPT finding — verdict.md rendered before the ledger existed. The merge step re-renders it with report.py --friction results/friction.json right after merge, so the uploaded verdict carries recurrence counts (issue numbers remain a nightly-only later step, as documented).

Tests: +7 (64 cases total, all pass locally by direct invocation) — collect fold in both orders; re-run reads its own attempt and never lists the nightlies; five untrusted-current-run shapes fall through to the nightlies without listing their artifacts; existing provenance/CLI tests moved to self_run_id. Docs and PR body updated; black/isort/flake8 clean; workflow YAML parses.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 12, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 10 — pushed 2b500e12b2 (single commit) for the two reds on 156042e0b5:

  • GPT (blocking): ledger written in place. _write now goes through a sibling temp file with fsync and os.replace, and removes the temp file on any failure, so a disk-full or killed runner can never leave a truncated friction_ledger.json for the upload step to publish as the canonical ledger. Test: a failed replace leaves the previous document intact and no .tmp behind.
  • CodeQL (high): py/clear-text-logging-sensitive-data on print(json.dumps(picked)). The pick is API-derived data, and the rule flags echoing it to the job log. pick-ledger now writes its answer to --out (the workflow reads it with jq from $RUNNER_TEMP) and prints only a fixed "previous ledger found" / "no previous ledger" line — nothing that came back from the API reaches stdout. Same fail-closed behaviour; a failed run leaves an earlier answer file untouched (atomic write above).

Opus cleared this head. 65 test cases pass locally by direct invocation; black / isort / flake8 clean; workflow YAML parses; docs and PR body updated.

Comment thread test/gui_user/friction.py Fixed
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 11 — pushed d6cba460d3 (single commit) for the reds on 2b500e12b2:

  • CodeQL py/clear-text-storage-sensitive-data. The SARIF trace shows the source is the function name trusted_run_id (the sensitive-data heuristic classifies it as a secret producer); every value it returns was then "secret" all the way to the file write. Renamed to ledger_source_run. No behaviour change.
  • GPT F1 — friction section could exceed GitHub's 65,536-char body limit. render_section now takes max_chars (SECTION_MAX_CHARS = 30k): rows are already sorted worst-first, so once the budget is spent the rest are counted in one trailer line ("… N more row(s) not shown … in the run artifact's friction.json"). Test: 80 maximum-length rows render ≤ 30k with only blockers shown; a small set has no trailer.
  • GPT F2 — cancellation could duplicate issues. Two halves: (a) LEDGER_CONCLUSIONS now includes cancelled and timed_out — both land after the ledger was written and uploaded, so a validated ledger artifact on such a run is a whole night and refusing it is what would have re-filed its issues (skipped and a None conclusion stay out); (b) before creating an issue for a key, file_issues runs gh issue list --state open --label ux --search "<key> in:body" and adopts an OPEN issue whose body carries the exact <!-- gui-user-friction <key> --> marker (a stray digest in prose is not adopted); a failed lookup skips that key for the night rather than risk a duplicate. Result gains adopted.
  • GPT finding — issues exit code overwritten by snapshot. The step captures the issues code, still runs the snapshot (the ledger already holds every number that was filed), then exits with the captured code.

71 test cases pass locally by direct invocation; black / isort / flake8 clean; mypy no new errors; workflow YAML parses; docs and PR body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 12 — pushed df502b6766 (single commit):

  • CodeQL py/clear-text-storage-sensitive-data moved to the next function: the SARIF trace now names trusted_ledger_runs() as the source — the heuristic classifies any trusted* callee as a secret producer. Dropped the word from every identifier (ledger_source_runs, LEDGER_EVENT, workflow env LEDGER_BRANCH); no behaviour change. Comments still say "trusted" where that is what is meant.
  • GPT (blocking): #123 in model text would cross-reference an issue. _cell now zero-width-splits # before a digit (#\u200b123, also owner/repo#45), alongside the existing @, URL and bracket defangs; #hashtag is untouched. Test added.

72 test cases pass locally by direct invocation; black / flake8 clean; mypy no new errors; workflow YAML parses; PR body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 13 — pushed b15bf52026 (single commit). The GPT job on df502b6766 flagged two blockers and a finding but its PR comment did not update (stale-verdict notice); read from the job log:

  • Ledger lookup was a single page of 30 runs. pick_ledger now walks the completed nightlies 100 a page, newest first, until one holds a ledger or a short page ends the listing, bounded by MAX_RUN_PAGES (5 → ~500 nights, far past the 90-day artifact retention). A long streak of merge-step failures can no longer turn into a false "first night". Tests: 100 ledgerless nights in front of the one with a ledger → page 2 is read and wins; a short page ends the walk; the walk is bounded.
  • --artifact-url was stamped on every row with last_seen == today. A re-run that did not observe an earlier attempts row would have pointed that rows screenshot at an artifact without it. issues now takes --friction results/friction.json and stamps only its run_keys; without it the stamp is refused (exit 2), never guessed. Workflow passes the file (the step already guards on its presence).
  • Finding: closed-issue check skipped count < 2 rows. Every row seen tonight that has an issue is checked now, so a same-day re-run cannot leave a fresh sighting mapped to a thread a human closed in between.

75 test cases pass locally by direct invocation; black / flake8 clean; mypy no new errors; workflow YAML parses; docs and PR body updated.

The tester runs as a first-time user (scenarios.PERSONAS, optional per
scenario `persona`) and files report_friction entries -- surface,
element, what confused me, expected/actual, severity -- whenever it
stalls; the harness validates each, stamps feature / scenario /
screenshot / step and carries them in summary.json without touching
PASS/FAIL. friction.py keys entries across nights, merges them into a
ledger artifact, renders the "New-user friction" section for verdict.md,
the run summary and the nightly issue, and opens at most five
ux(<feature>) issues a night (recurrences comment, cosmetic stays in
the summary).

Closes #10222
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Round 14 — pushed 9c30877cd4 (single commit) for the two GPT blockers on b15bf52026:

  • A re-run that crossed midnight UTC counted as a new night. merge_ledger now checks run_url first: an identical run URL is the same run whatever the date — no count, no last_seen move, no evidence refresh (attempt 1 already filed and uploaded), so tonight() for the new date is empty and the re-run files and comments nothing. A different run the next day is still a recurrence. Test covers all three cases.
  • Merge step ran under set -uo pipefail. Now set -euo pipefail; every command in that step resolves, downloads, merges or re-renders the ledger, so any failure (the ledger write after friction.json, the verdict re-render) fails the step and nothing downstream uploads or files against a night that was not persisted. Checked each line for -e compatibility (|| { …; exit 1; }, if !, if/elif only).

76 test cases pass locally by direct invocation; black / flake8 clean; workflow YAML parses; docs and PR body updated.

@iamwhatever

Copy link
Copy Markdown
Contributor

Not approving for now: the latest maintainer review is CHANGES_REQUESTED on a trust-boundary finding in .github/workflows/gui-user-test.yml (the scheduled run selects the first repository-wide artifact matching the friction-ledger name, which a PR run can produce). A workflow that files issues off downloaded artifact content needs that provenance check tightened before approval.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

@bolichen97 gentle re-request — your CHANGES_REQUESTED is pinned to d134a3622c; the PR has gone through several fix rounds since and current head 9c30877cd4 is green (65 checks incl. CodeQL and PR Readiness) and MERGEABLE on main.

What was addressed from your review (the ledger-provenance finding, shared with the GPT review):

  • Ledger source is no longer "any recent artifact". pick-ledger reads the current run's own earlier attempt first (--self-run), then only completed scheduled runs of this workflow on the default branch (LEDGER_BRANCH = github.event.repository.default_branch || github.ref_name, LEDGER_EVENT), paginated with a page cap, and verifies the artifact's own workflow_run block before adopting it.
  • Result is written to an --out file rather than stdout (CodeQL clear-text finding), and identifiers avoid the trusted* prefix CodeQL treats as a sensitive-data source.
  • Follow-on hardening from later rounds: atomic ledger write, duplicate-key folding (worst severity, newest evidence), per-section size cap, adopt-existing-open-issue before create, closed-issue check for every tonight row, set -euo pipefail and exit-code propagation in the workflow steps.

If the current head looks right, a fresh review would unblock it — thanks!

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer review (agent-assisted), pinned to 9c30877.

Re-reviewed all seven changed files. The earlier ledger-provenance blocker is addressed: scheduled default-branch workflow identity, source repository/head repository, and artifact/run binding are checked before accepting the ledger, with failure-closed handling. No blocking findings. Non-blocking: preserve a safely redacted gh error reason for failed issue filing; CalledProcessError alone loses diagnostic stderr.

Verification: reviewed source/tests and observed passing CI and PR Readiness on this exact head; no local test rerun or manual UI reproduction. Approval is for this commit only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci GitHub Actions, AI review gates, readiness checks channel: insider Reported from an insider prerelease build — candidate release blocker

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gui-user-test: new-user friction channel -- tester persona reports what confused it, beside the verdict

5 participants