Repository navigation
test: require the Playwright host-platform guard on every workflow install - #37703
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test and a docs section that protect the #37653 stopgap, and both are removed together when Playwright is bumped. Notes
[DESIGN-REVIEWED] 3b12a0a Description read[DESCRIPTION-READ] 5f1307075e6cb462b2915ba5073bc9a1df86d661dbd56f680f0b459f7ccd754c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. The PR adds only a static test and a docs section. The three Install Playwright browser (Chromium) steps in ci.yml each set and export PLAYWRIGHT_HOST_PLATFORM_OVERRIDE before the install and write it to $GITHUB_ENV, which is the shape _guarded() accepts, so the test passes on head. Backend Tests and New-test stability are green on both platforms. The tmp_path tests cover the cases that should go red: no guard, a version-pinned install, a guard placed after the install, an export without the $GITHUB_ENV write, and a $GITHUB_ENV write only. The test reads only checked-in files and tmp_path, so it is deterministic, and it changes no workflow or product code. The coverage gaps (composite actions, An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head 3b12a0a. It adds only a static workflow test and one docs section; no workflow or product code changes. The new test file passes (10 tests). It is not vacuous: when I removed the host-platform guard from one ci.yml install step, the main check failed and named that exact step, and with the guard restored it passed. The fixture cases cover the orderings that should fail: a guard after the install, an export without the GITHUB_ENV write, and a GITHUB_ENV write alone. The test reads only checked-in files and its tmp_path, so it is deterministic, and nothing blocks.
Problem / Motivation
Goal: A new
playwright installstep in a workflow cannot go red on an ubuntu 26.04 runner again without a test catching it first.When
kirocrew-linux-4-coremoved to ubuntu 26.04, Playwright 1.58.2 refusednpx playwright install chromium(Playwright does not support chromium on ubuntu26.04-x64). Render Benchmark went red on about 30 PRs and twice onmainin one morning (#37645). #37653 fixed the three install steps inci.yml, but nothing stops the next new install step from missing that guard, and the rule was written nowhere.Why it matters
The next runner-image move, or a new Playwright job, would turn every PR red again with a failure that says nothing about the PR's code.
Not a goal
What changed (motivation → approach → change)
test/test_workflow_playwright_host_guard.py.github/workflows/and fails on anyplaywright installstep that lacks the host-platform guarddocs/ci/ci-and-reviews.mdplaywright install": the rule, the guard, and the real fixThe guard counts when the step's shell sets
PLAYWRIGHT_HOST_PLATFORM_OVERRIDEbefore the install AND writes it to$GITHUB_ENV, or when the job or workflowenvsets it. A step that only writes$GITHUB_ENV, or sets it after the install, is still flagged, because the install in that same step would not see it.The test asserts a list of offending steps is empty, and that
ci.ymlis still seen as a file with installs. It holds no total count, so PRs that add or remove install steps can merge in any order without conflicting on it.The docs name the real fix: bump Playwright from 1.58.2 to a release with a native ubuntu26.04 host entry (1.65.1 has one), then delete the guard and this test together.
Backwards compatibility
Compatible: the test passes on today's
main; it only reads workflow files.Removes nothing: it adds a test and a docs section.
Tests
test/test_workflow_playwright_host_guard.py: 10 passed. Two tests read the real workflows; eight use small workflow files intmp_path(no guard, aplaywright@<version>install, guard after the install, only exported, only written to$GITHUB_ENV, inline guard, jobenv, commented-out install).if ... figuard from the Render Benchmark step inci.ymlmakestest_every_playwright_install_carries_the_host_platform_guardfail with('ci.yml', 'bench-render', 'Install Playwright browser (Chromium)'). Restoring it makes it pass.-n 4, passed. The test reads only checked-in files andtmp_path.test_flake_pattern_ratchet.py: 68 passed.scripts/local-gate.py: backend 371 passed, frontend 534 files passed. isort, flake8, black, semgrep,docs_lint.py: clean.Proof: not provable -- test-only change
Manual verification
Evidence that the #37653 guard works on ubuntu 26.04, which this test now protects:
mainafter ci: install Playwright's ubuntu24.04 Chromium on the ubuntu-26.04 fork runner #37653: Render Benchmark job 132941715543 ran onImage: ubuntu-26.04, installed chromium v1208 asubuntu24.04-x64, and passed every budget.mainpushes in about 6 hours. Examples: 43755316797, 43757029509, 43757825652.ci.ymlinstalls Playwright today:git grep -nE "playwright install" -- .githubfinds its three steps.gui-user-test.ymluses the image's own Chrome.Related Issues
Closes #37645
Checklist