Skip to content

fix: move removing harness seeds into the stub's localStorageEntries - #36136

Merged
bolichen97 merged 1 commit into
mainfrom
fix/31778-harness-seed-remove-clear
Nov 13, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/31778-harness-seed-remove-clear

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Capture harnesses that seed a stubbed page once, and also removed or cleared keys in that seed, get their seed on every run whatever order Playwright runs init scripts in.

Thirty capture harnesses seed a page that goes through stubDashboardApi from their own addInitScript, and that script also calls localStorage.removeItem or localStorage.clear. localStorageEntries (the fix for #4760) cannot express a removal, so these seeds still ride on Chromium running the stub's clear before them. Playwright does not define that order.

Why it matters

These harnesses take the screenshots that PRs attach as evidence. If the order flips, the stub's clear wipes the seed and the capture comes up with the wrong theme, slot, locale or panel layout, without any error, so a wrong screenshot reads as proof.

Not a goal

What changed (motivation → approach → change)

Every harness below now seeds its page inside the stub's own init script, so the seed lands on every run, not only when Chromium happens to run the stub's clear first. That works because each of these pages registers its seed once and the stub clears storage at the top of its script: every removeItem or clear in the harness's own script removed a key that clear had already removed. Without them only setItem writes are left, which is what localStorageEntries holds, and the stub writes those after its clear.

Harness seed Change
16 seeds: setItem writes plus a redundant removeItem/clear writes move into the stub's localStorageEntries (values computed in Node); the removal goes
capture-crew-board.mjs its seed only repeated the stub's own mc-theme and mc-onboarded, so it is dropped
capture-composer-collapse.mjs (preserveStorage: true) its first-boot removal of mc-composer-collapsed acted on a new context that never has the key, so it is dropped

capture-blocked-link-chip-host-only.mjs and capture-redacted-reveal.mjs passed SLOT as a second argument to addInitScript, which takes one, so they wrote the string undefined as the active slot; the entry now carries SLOT. Comments that explained a removal now say the stub's clear leaves that key absent.

harnessStorageSeeds.test.ts gains a second check: an init script on a stubbed page that calls removeItem or clear is reported, and the reported files must be exactly the ten re-seeding harnesses, so a new one fails the test.

Backwards compatibility

Compatible: only capture harnesses under website/scripts/ and one test change; no product code, API or stored data is touched.

Removes nothing: the dropped removals act on keys the stub's own clear already removed, and no shipped code reads the harness scripts.

Tests

website/src/test/harnessStorageSeeds.test.ts: the new case fails on the parent's harness files (reverted in a throwaway worktree: 1 failed, 1 passed) and passes on this head; the existing flat-setItem case stays at zero. Run 10 times and shuffled, the same result each time.

Proof: not provable -- frontend test (prove.py runs Python only)

Manual verification

node --check on every changed harness.

Screenshots / video

Why no screenshot: only capture harness scripts and a source-shape test change; no product surface renders differently.

Related Issues

Part of #31778

Sibling sweep

Swept: git grep -lE "localStorage.(removeItem|clear)(" -- website/scripts/, kept where the file also calls stubDashboardApi( or stubSplitPanes( (then the test's AST walk over each addInitScript on a stubbed page)
Sibling: website/scripts/capture-continue-turn.mjs -- tracked in #31778
Sibling: website/scripts/capture-diff-split-preference.mjs -- tracked in #31778
Sibling: website/scripts/capture-file-changes-older-turn.mjs -- tracked in #31778
Sibling: website/scripts/capture-files-tree-empty-folder.mjs -- tracked in #31778
Sibling: website/scripts/capture-pierre-chat-diffs.mjs -- tracked in #31778
Sibling: website/scripts/capture-pierre-files-tab.mjs -- tracked in #31778
Sibling: website/scripts/capture-pierre-misc-surfaces.mjs -- tracked in #31778
Sibling: website/scripts/capture-pulls-first-paint.mjs -- tracked in #31778
Sibling: website/scripts/capture-stale-collapse.mjs -- tracked in #31778
Sibling: website/scripts/capture-turn-snapshot-budget.mjs -- tracked in #31778
Sibling: website/scripts/lib/stub-dashboard-api.mjs -- not affected -- its clear is the one the harnesses now rely on

Pattern harvest

Rule candidate: review-prompt
Pattern: a test harness writes browser storage from its own init script on a page another init script clears; the two scripts' order is undefined

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in website/src/pages/chat/AssistantMessage.tsx since this head's CI ran (2f7e9f2 -> b2aac45); CI re-running.

The last run was built on a main whose reply menu used GitFork without importing it, which failed the frontend build and every job behind it (Frontend Lint & Type Check, Build Wheel, Build Desktop, Coverage Gate). #36124 fixed that on main; this branch now sits on top of it. The review lanes and the readiness signal on that run stopped on an API rate limit, not on a finding. The diff itself is unchanged.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 8207ff6ec0561b76401f5e5134253c22f77c9421 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 8207ff6ec0561b76401f5e5134253c22f77c9421 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The PR only changes capture harness scripts under website/scripts/ and one source-shape test. No user-visible control, string or state is added or changed, so the missing screenshots leave no evidence gap.

[UX-REVIEWED] 8207ff6

Description read

[DESCRIPTION-READ] 72cb09732cc78d4d656edc0b742036dcc7c0882226195797ea486249a9d1d44a

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8207ff6ec0561b76401f5e5134253c22f77c9421 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Each moved seed now runs after the clear at the top of the stub's own init script, so the order no longer depends on Chromium. Every removal that was dropped targeted a key that clear had already wiped, or a key a brand-new context never has.

Notes

[DESIGN-REVIEWED] 8207ff6

Description read

[DESCRIPTION-READ] 7e833f836a2a1094d3dd40cb81ca3604f196497caeadcdbfbc731935df30e8f6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8207ff6ec0561b76401f5e5134253c22f77c9421 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8207ff6

Verdict parsed from the review's SHA-scoped output markers for commit 8207ff6ec0561b76401f5e5134253c22f77c9421.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8207ff6ec0561b76401f5e5134253c22f77c9421 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8207ff6ec0561b76401f5e5134253c22f77c9421 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8207ff6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8207ff6ec0561b76401f5e5134253c22f77c9421 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Moves harness seeds that also removed or cleared keys into the stub's localStorageEntries. In stub-dashboard-api.mjs:266-271, one init script clears storage, writes mc-theme and mc-onboarded, then writes the entries, so every dropped removal acted on a key that clear had already removed. composer-collapse uses preserveStorage but a fresh context, so its dropped first-boot removal changes nothing. crew-board's dropped seed repeated the theme already passed to the stub. Every value now built in Node (SLOT, PROJECT, fileTab, bucket, SPLIT, tabs, s.attr, rail.collapsed) is declared before the stub call that uses it. The SLOT-as-second-argument bug fix in two harnesses is correct. The new test expects exactly the ten RESEEDED_PAGES files. I checked about 90 harness files with init scripts that call clear/removeItem: apart from those ten, none calls stubDashboardApi or stubSplitPanes, so the expectation should hold. The failing CI checks are unrelated: a Python backend test and a CodeQL rate limit.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 0c8dada23d · 373 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
Eighteen capture harnesses seeded a stubbed page from their own
addInitScript that also called localStorage.removeItem or clear, so the
seed still raced the clear inside stubDashboardApi's init script, whose
order against a caller's script Playwright does not define. Each page
registers its seed once, so the removal is already done by the stub's
clear: the removal goes and the writes move into localStorageEntries.
A seed that only repeated the stub's own theme and onboarding keys is
dropped, and capture-composer-collapse loses a first-boot removal that
a new context never needed.

The guard test now also reports a removeItem or clear seed on a stubbed
page. The ten harnesses that re-seed one page per scenario stay pinned
by name until they move to a fresh page per scenario.

Part of #31778
@bolichen97
bolichen97 force-pushed the fix/31778-harness-seed-remove-clear branch from 0c8dada to 8207ff6 Compare November 13, 2026 05:06
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_reply_format_once.py since this head's CI ran (b2aac45 -> b677e0d); CI re-running.

The last run merged this branch with a main where the new mimocode harness was missing from the reply-format harness table, so test/test_reply_format_once.py::TestEveryMessageWhereTheBlockCannotBeTakenOut::test_the_table_names_every_selectable_harness failed on Backend Tests (3.12, 8) and Backend Tests (Windows) (8), and Coverage Gate failed closed behind them. #36290 fixed that on main; this branch now sits on top of it. CodeQL's JavaScript analysis stopped on an API rate limit while uploading its results, not on a finding. The diff itself is unchanged.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on head 8207ff6. Each dropped removeItem or clear targets a key that the stub's own init script already clears, because no changed harness passes preserveStorage except composer-collapse, which uses a fresh context. Every moved entry reads values that are defined before the stub call. node --check is clean on all 19 harnesses. The new harnessStorageSeeds case passes against the head's scripts and fails against the parent's, listing exactly the 19 fixed harnesses.
Done well: the seed writes now run after the clear by construction. The change also fixes a real bug in blocked-link-chip-host-only and redacted-reveal, where the extra SLOT argument to addInitScript made those harnesses store the string 'undefined' as the active slot.
Optional: the body's counts are off by one. The parent has 30 init-script sites in 29 harnesses, and the first table row covers 17 seeds, not 16.
Optional: RESEEDED_PAGES pins file names, not sites. A second removing seed in a file already on the list would still pass. A per-file site count, or a line in the doc comment, would close that.
Optional: five moved seeds still pass mc-theme 'dark' and mc-onboarded '1', which the stub already writes. Main has the same pattern elsewhere.

@bolichen97
bolichen97 merged commit 51349e1 into main Nov 13, 2026
98 of 107 checks passed
@bolichen97
bolichen97 deleted the fix/31778-harness-seed-remove-clear branch November 13, 2026 09:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants