Skip to content

test: require the Playwright host-platform guard on every workflow install - #37703

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/playwright-install-guard-test
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/playwright-install-guard-test

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A new playwright install step in a workflow cannot go red on an ubuntu 26.04 runner again without a test catching it first.

When kirocrew-linux-4-core moved to ubuntu 26.04, Playwright 1.58.2 refused npx playwright install chromium (Playwright does not support chromium on ubuntu26.04-x64). Render Benchmark went red on about 30 PRs and twice on main in one morning (#37645). #37653 fixed the three install steps in ci.yml, but nothing stops the next new install step from missing that guard, and the rule was written nowhere.

Why it matters

The next runner-image move, or a new Playwright job, would turn every PR red again with a failure that says nothing about the PR's code.

Not a goal

What changed (motivation → approach → change)

Part Change
test/test_workflow_playwright_host_guard.py New static test. It reads every file in .github/workflows/ and fails on any playwright install step that lacks the host-platform guard
docs/ci/ci-and-reviews.md New section "A runner image move and playwright install": the rule, the guard, and the real fix

The guard counts when the step's shell sets PLAYWRIGHT_HOST_PLATFORM_OVERRIDE before the install AND writes it to $GITHUB_ENV, or when the job or workflow env sets it. A step that only writes $GITHUB_ENV, or sets it after the install, is still flagged, because the install in that same step would not see it.

The test asserts a list of offending steps is empty, and that ci.yml is still seen as a file with installs. It holds no total count, so PRs that add or remove install steps can merge in any order without conflicting on it.

The docs name the real fix: bump Playwright from 1.58.2 to a release with a native ubuntu26.04 host entry (1.65.1 has one), then delete the guard and this test together.

Backwards compatibility

Compatible: the test passes on today's main; it only reads workflow files.

Removes nothing: it adds a test and a docs section.

Tests

  • test/test_workflow_playwright_host_guard.py: 10 passed. Two tests read the real workflows; eight use small workflow files in tmp_path (no guard, a playwright@<version> install, guard after the install, only exported, only written to $GITHUB_ENV, inline guard, job env, commented-out install).
  • Red without the guard: deleting the if ... fi guard from the Render Benchmark step in ci.yml makes test_every_playwright_install_carries_the_host_platform_guard fail with ('ci.yml', 'bench-render', 'Install Playwright browser (Chromium)'). Restoring it makes it pass.
  • Determinism: 10 serial runs in shuffled order, 10/10 passed; one run under -n 4, passed. The test reads only checked-in files and tmp_path.
  • test_flake_pattern_ratchet.py: 68 passed. scripts/local-gate.py: backend 371 passed, frontend 534 files passed. isort, flake8, black, semgrep, docs_lint.py: clean.

Proof: not provable -- test-only change

Manual verification

Evidence that the #37653 guard works on ubuntu 26.04, which this test now protects:

Related Issues

Closes #37645

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 21:23
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 3b12a0aacb0cf6ee854853fd1e679146732ff2ce ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3b12a0aacb0cf6ee854853fd1e679146732ff2ce touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3b12a0aacb0cf6ee854853fd1e679146732ff2ce — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test and a docs section that protect the #37653 stopgap, and both are removed together when Playwright is bumped.

Notes

  • The scan only reads .github/workflows/*.yml. A playwright install placed in a composite action under .github/actions/, or in a script that a step calls, gets past it unflagged. None exists today. If one is ever added, widen the glob instead of adding a second check.

[DESIGN-REVIEWED] 3b12a0a

Description read

[DESCRIPTION-READ] 5f1307075e6cb462b2915ba5073bc9a1df86d661dbd56f680f0b459f7ccd754c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3b12a0aacb0cf6ee854853fd1e679146732ff2ce — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3b12a0a

Verdict parsed from the review's SHA-scoped output markers for commit 3b12a0aacb0cf6ee854853fd1e679146732ff2ce.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3b12a0aacb0cf6ee854853fd1e679146732ff2ce false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 3b12a0aacb0cf6ee854853fd1e679146732ff2ce and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3b12a0a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3b12a0aacb0cf6ee854853fd1e679146732ff2ce false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

The PR adds only a static test and a docs section. The three Install Playwright browser (Chromium) steps in ci.yml each set and export PLAYWRIGHT_HOST_PLATFORM_OVERRIDE before the install and write it to $GITHUB_ENV, which is the shape _guarded() accepts, so the test passes on head. Backend Tests and New-test stability are green on both platforms. The tmp_path tests cover the cases that should go red: no guard, a version-pinned install, a guard placed after the install, an export without the $GITHUB_ENV write, and a $GITHUB_ENV write only. The test reads only checked-in files and tmp_path, so it is deterministic, and it changes no workflow or product code. The coverage gaps (composite actions, uses: steps, only the first install line per step) are non-blocking because none of those shapes exists today.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 3b12a0aacb · 175 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head 3b12a0a. It adds only a static workflow test and one docs section; no workflow or product code changes. The new test file passes (10 tests). It is not vacuous: when I removed the host-platform guard from one ci.yml install step, the main check failed and named that exact step, and with the guard restored it passed. The fixture cases cover the orderings that should fail: a guard after the install, an export without the GITHUB_ENV write, and a GITHUB_ENV write alone. The test reads only checked-in files and its tmp_path, so it is deterministic, and nothing blocks.

@KiroCrewCaptain
KiroCrewCaptain merged commit 6ee86c9 into main Nov 15, 2026
96 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/playwright-install-guard-test branch November 15, 2026 22:50
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
buluoray pushed a commit that referenced this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: Render Benchmark (counter budgets) fails on main: Playwright cannot install chromium on the ubuntu-26.04 runner image

2 participants