-
Notifications
You must be signed in to change notification settings - Fork 2
adding playwright VRT integration #86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rigoucr
wants to merge
8
commits into
3.x
Choose a base branch
from
feature/playwright-vrt
base: 3.x
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
38a911b
Updating Fire commands - adding playwright VRT integration
rigoucr d2dac42
Debuging and reshaping the the vrt init command
rigoucr 0c4c209
multiple improvements
rigoucr 4388bc3
Final details for Playwright + fire integration
rigoucr 5672e26
fix: playwright-vrt simplify atk scaffold, add readme.md, small clean…
laura-johnson c0e80e0
fix: playwright-vrt make vrt:init or vinit aliases on VrtPlaywrightIn…
laura-johnson b0bfa7f
fix: playwright-vrt update to view report instruction
laura-johnson 3ade80a
fix: playwright-vrt do not enable atk and qa_accounts
laura-johnson File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # AGENTS | ||
|
|
||
| ## Entrypoint | ||
| - CLI entrypoint is `bin/fire` (executes `bin/fire.php`); it requires Composer autoload to exist, so run `composer install` before local runs. | ||
|
|
||
| ## Config + discovery | ||
| - `fire.yml` and optional `fire.local.yml` are read from the project root (four directories above `vendor/fourkitchens/fire`); if missing, only `InitCommand.php` is registered and the CLI prompts to run `fire init`. | ||
| - Local env auto-detection sets `local_environment` based on `.lando.yml` or `.ddev/config.yaml` in the project root. | ||
|
|
||
| ## Command sources | ||
| - Core commands live in `src/Robo/Plugin/Commands/*Command.php` and are discovered via Robo `CommandFileDiscovery`. | ||
| - Project-level custom commands are loaded from `fire/src/Commands` in the project root (path is computed by stripping `vendor/fourkitchens/` from the package path). | ||
|
|
||
| ## Config template | ||
| - `fire init` uses the template at `assets/templates/fire.yml`; update it if you change required config fields. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 24.15.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| # Playwright VRT | ||
|
|
||
| Visual Regression Testing for this project, set up with [Fire](https://github.com/fourkitchens/fire) and powered by [`@fkbender/playwright-vrt-scripts`](https://github.com/fourkitchens/playwright-vrt-scripts). | ||
|
|
||
| VRT works by capturing screenshots of your site from two environments — a **baseline** (typically the live Pantheon environment) and a **candidate** (your local environment) — and comparing them pixel by pixel. Failures mean something visually changed. | ||
|
|
||
| --- | ||
|
|
||
| ## Drupal modules | ||
|
|
||
| `drupal/automated_testing_kit` and `drupal/qa_accounts` are required in your `composer.json` but **not enabled**. Enable them only after a config split is in place to ensure they cannot reach production: | ||
|
|
||
| ```bash | ||
| ddev drush en automated_testing_kit qa_accounts -y | ||
| ``` | ||
|
|
||
| These modules are not required for VRT. They are needed for ATK-based E2E tests (Phase 2). | ||
|
|
||
| --- | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Node (version pinned in `.nvmrc`) | ||
| - Playwright browsers installed (`npx playwright install --with-deps` — done automatically by `fire vrt:playwright:init`) | ||
| - A populated `.env` file (see below) | ||
|
|
||
| ### `.env` | ||
|
|
||
| This file is gitignored and must exist locally. It is generated automatically by `fire vrt:playwright:init`, but can be created manually if needed: | ||
|
|
||
| ``` | ||
| BASELINE_URL="https://live-yoursite.pantheonsite.io" | ||
| CANDIDATE_URL="https://yoursite.ddev.site" | ||
| BASELINE_TERMINUS_ENV=live | ||
| BASELINE_TERMINUS_SITE=yoursite | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Running VRT Locally | ||
|
|
||
| ```bash | ||
| # Full two-pass comparison: captures baseline from BASELINE_URL, candidate from CANDIDATE_URL | ||
| npm run vrt:local | ||
|
|
||
| # CI-style run (used in automated pipelines) | ||
| npm run vrt:ci | ||
| ``` | ||
|
|
||
| Results open automatically in your browser as an HTML report. Failing tests show a diff of baseline vs. candidate. | ||
|
|
||
| --- | ||
|
|
||
| ## Adding Pages to Test | ||
|
|
||
| Open `tests/data/vrtCommonPages.json` and add an entry: | ||
|
|
||
| ```json | ||
| [ | ||
| { | ||
| "name": "Home", | ||
| "path": "/", | ||
| "screenshotName": "home-page" | ||
| }, | ||
| { | ||
| "name": "Login", | ||
| "path": "/user/login", | ||
| "screenshotName": "user-login" | ||
| }, | ||
| { | ||
| "name": "News listing", | ||
| "path": "/news", | ||
| "screenshotName": "news-listing" | ||
| } | ||
| ] | ||
| ``` | ||
|
|
||
| Each entry requires three fields: | ||
|
|
||
| | Field | Purpose | | ||
| |---|---| | ||
| | `name` | Human-readable label shown in the test report | | ||
| | `path` | Site-relative URL to visit (must start with `/`) | | ||
| | `screenshotName` | Base filename for saved screenshots (no extension, no spaces) | | ||
|
|
||
| The spec automatically generates tests for every page × every device profile (Desktop, iPad, iPhone 12), so adding one entry produces three new tests. | ||
|
|
||
| --- | ||
|
|
||
| ## Device Profiles | ||
|
|
||
| Tests run against three device profiles, all using Chromium: | ||
|
|
||
| | Profile | Viewport | | ||
| |---|---| | ||
| | Desktop Chrome | 1280 × 720 | | ||
| | iPad (gen 7) | 810 × 1080 | | ||
| | iPhone 12 | 390 × 844 | | ||
|
|
||
| Device profiles are defined in `tests/support/4k_utilities.js`. Firefox and WebKit are available but commented out in `playwright.config.js` — remove the comments to enable them. | ||
|
|
||
| --- | ||
|
|
||
| ## Running a Single Test or Page | ||
|
|
||
| ```bash | ||
| # Run only tests matching a page name | ||
| npx playwright test --grep "Home" | ||
|
|
||
| # Run only desktop tests | ||
| npx playwright test --grep "desktop" | ||
|
|
||
| # Run all VRT tests (tagged @vrt) | ||
| npx playwright test --grep "@vrt" | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Viewing the Report | ||
|
|
||
| After a run, open the HTML report from inside `tests/playwright/`: | ||
|
|
||
| ```bash | ||
| npm run report | ||
| ``` | ||
|
|
||
| Or from the project root: | ||
|
|
||
| ```bash | ||
| npx playwright show-report tests/playwright/playwright-report | ||
| ``` | ||
|
|
||
| On CI, reports are uploaded as artifacts. Each failing test shows the baseline screenshot, the candidate screenshot, and a pixel-diff image. | ||
|
|
||
| --- | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| **Tests fail with image differences I didn't expect** | ||
|
|
||
| - Check that `stage_file_proxy` is enabled so images load correctly in your local environment. | ||
| - Make sure your local database is synced from a recent pull of the live environment. | ||
| - Confirm there are no pending CSS or JS builds that differ from what's on live. | ||
|
|
||
| **`networkidle` timeout errors** | ||
|
|
||
| The test waits for the page to become network-idle before screenshotting. If a page has long-polling or persistent background requests, this will time out. Add a custom wait or increase the timeout for that specific test. | ||
|
|
||
| **Snapshots are stale after a design change** | ||
|
|
||
| Once a visual change is intentional and approved, update the baseline by re-running the VRT capture against the new live environment. | ||
|
|
||
| **`.env` file not found** | ||
|
|
||
| Run `fire vrt:playwright:init` to regenerate it, or copy the values from a teammate or the project's password manager entry. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| #toolbar-administration { | ||
| visibility: hidden; | ||
| } | ||
|
|
||
| .gin-secondary-toolbar__layout-container { | ||
| visibility: hidden; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| [ | ||
| { | ||
| "name": "Home", | ||
| "path": "/", | ||
| "screenshotName": "home-page" | ||
| }, | ||
| { | ||
| "name": "Login", | ||
| "path": "/user/login", | ||
| "screenshotName": "user-login" | ||
| } | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| { | ||
| "name": "playwright-vrt", | ||
| "version": "1.0.0", | ||
| "description": "Playwright VRT and E2E testing", | ||
| "main": "playwright.atk.config.js", | ||
| "scripts": { | ||
| "vrt": "playwright-vrt", | ||
| "vrt:local": "playwright-vrt-local", | ||
| "vrt:ci": "playwright-vrt-ci" | ||
| }, | ||
| "keywords": [], | ||
| "dependencies": { | ||
| "@eslint/eslintrc": "3.3.5", | ||
| "@eslint/js": "8.57.1", | ||
| "@fkbender/playwright-vrt-scripts": "1.0.0-alpha.2", | ||
| "@playwright/test": "1.59.1", | ||
| "axios": "1.16.0", | ||
| "cspell": "8.19.4", | ||
| "dotenv": "16.6.1", | ||
| "fast-xml-parser": "4.5.6", | ||
| "junit-report-merger": "9.0.3", | ||
| "winston": "3.19.0", | ||
| "yaml": "2.9.0" | ||
| }, | ||
| "devDependencies": { | ||
| "@eslint/eslintrc": "3.3.5", | ||
| "@eslint/js": "8.57.1", | ||
| "eslint": "8.57.1", | ||
| "eslint-config-airbnb-base": "15.0.0", | ||
| "eslint-config-standard-with-typescript": "43.0.1", | ||
| "eslint-plugin-chai-friendly": "1.2.0", | ||
| "eslint-plugin-cypress": "3.6.0", | ||
| "eslint-plugin-import": "2.32.0", | ||
| "globals": "15.15.0", | ||
| "prettier": "3.8.3" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| /* | ||
| * Automated Testing Kit configuration. | ||
| */ | ||
| var currentEnv = process.env.TERMINUS_ENV || 'local'; | ||
| var runOnPantheon = false; | ||
| if (currentEnv != 'local') { | ||
| runOnPantheon = true; | ||
| } | ||
|
|
||
| module.exports = { | ||
| operatingMode: "native", | ||
| drushCmd: "__DRUSH_CMD__", | ||
| articleAddUrl: 'node/add/article', | ||
| contactUsUrl: "form/contact", | ||
| logInUrl: "user/login", | ||
| logOutUrl: "user/logout", | ||
| imageAddUrl: 'media/add/image', | ||
| mediaDeleteUrl: 'media/{mid}/delete', | ||
| mediaEditUrl: 'media/{mid}/edit', | ||
| mediaList: 'admin/content/media', | ||
| menuAddUrl: 'admin/structure/menu/manage/main/add', | ||
| menuDeleteUrl: 'admin/structure/menu/item/{mid}/delete', | ||
| menuEditUrl: 'admin/structure/menu/item/{mid}/edit', | ||
| menuListUrl: 'admin/structure/menu/manage/main', | ||
| nodeDeleteUrl: 'node/{nid}/delete', | ||
| nodeEditUrl: 'node/{nid}/edit', | ||
| pageAddUrl: 'node/add/page', | ||
| registerUrl: "user/register", | ||
| resetPasswordUrl: "user/password", | ||
| termAddUrl: 'admin/structure/taxonomy/manage/terms/add', | ||
| termEditUrl: 'taxonomy/term/{tid}/edit', | ||
| termDeleteUrl: 'taxonomy/term/{tid}/delete', | ||
| termListUrl: 'admin/structure/taxonomy/manage/terms/overview', | ||
| termViewUrl: 'taxonomy/term/{tid}', | ||
| xmlSitemapUrl: 'admin/config/search/simplesitemap', | ||
| authDir: "tests/support", | ||
| dataDir: "tests/data", | ||
| supportDir: "tests/support", | ||
| testDir: "tests", | ||
| pantheon : { | ||
| isTarget: runOnPantheon, | ||
| site: "__PANTHEON_SITE__", | ||
| environment: currentEnv | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to add a part here where we describe how to edit the common pages.