Repository navigation
Add screenshot regression test #286
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
40a1853
Add playwright-based screenshot tests
panglesd c4cc995
Wait for slipshow to be ready before taking screenshots
panglesd 927bc23
Port some manually-checked cram tests to screenshot tests
panglesd ff4cb81
Store screenshots in a screenshots/ subfolder of each test folder
panglesd b5f7fd0
Accept the blurry rendering of BOBKonf step 5 on Chromium
panglesd acc6ca2
See CI font rendering system
panglesd de963ae
Compile the screenshot test entries with dune rules
panglesd 43c42c3
Run the screenshot tests in Playwright's Docker image
panglesd 37fba73
Update the Firefox screenshots of slides to the Docker rendering
panglesd da74551
CI: less permissions
panglesd 20bb152
More consistent timeout
panglesd bf16341
Screenshot tests: Be sure to wait for readiness
panglesd 324fb59
Only set will-change: transform during zooms
panglesd c31fff1
Move between steps with State messages instead of reloading
panglesd bdb01c4
Update the Chromium screenshots after the will-change fix
panglesd cd8fcf1
Update changelog
panglesd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| name: Playwright Tests | ||
| on: | ||
| push: | ||
| branches: [ main ] | ||
| pull_request: | ||
| branches: [ main ] | ||
| permissions: | ||
| contents: read | ||
| jobs: | ||
| test: | ||
| timeout-minutes: 60 | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| working-directory: test/screenshot-based | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Use OCaml 4.14.x | ||
| uses: ocaml/setup-ocaml@v3 | ||
| with: | ||
| ocaml-compiler: 4.14.x | ||
|
|
||
| - name: Install OCaml dependencies | ||
| working-directory: . | ||
| run: opam install --deps-only -y . | ||
|
|
||
| - name: Generate the HTML files | ||
| working-directory: . | ||
| run: opam exec -- dune build --profile release | ||
|
|
||
| # In Playwright's Docker image, so that the rendering is the same as when | ||
| # the screenshots were generated | ||
| - name: Run Playwright tests | ||
| run: ./playwright-docker.sh | ||
|
|
||
| - uses: actions/upload-artifact@v4 | ||
| if: ${{ !cancelled() }} | ||
| with: | ||
| name: playwright-report | ||
| path: test/screenshot-based/playwright-report/ | ||
| retention-days: 30 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Tests | ||
|
|
||
| There are two kinds of tests | ||
|
|
||
| ## CLI tests | ||
|
|
||
| CLI tests take the form of [cram tests](https://dune.readthedocs.io/en/stable/tests.html#cram-tests) | ||
|
|
||
| Run them with: | ||
|
|
||
| ```shell | ||
| dune test | ||
| ``` | ||
|
|
||
| ## Browser tests | ||
|
|
||
| Browser tests take the form of screenshot tests, implemented using [playwright visual tests](https://playwright.dev/docs/test-snapshots). | ||
|
|
||
| Running the tests is done in two steps, from the `screenshot-based` folder. | ||
|
|
||
| First, generate the HTML files of all tests with dune: | ||
|
|
||
| ```shell | ||
| dune build --profile release | ||
| ``` | ||
|
|
||
| Each test folder has a `dune` file that compiles its `.slp` and `.md` entries | ||
| to HTML files next to them, with the same base name. When adding an entry to a | ||
| `run.json`, add the corresponding rule. | ||
|
|
||
| Then, run the tests in [Playwright's Docker image](https://playwright.dev/docs/docker). | ||
| Browser rendering depends on the fonts and libraries of the host, so using the | ||
| same image everywhere ensures that the screenshots are the same locally and in | ||
| the CI. This only requires Docker, not OCaml nor the Playwright browsers: | ||
|
|
||
| ```shell | ||
| ./playwright-docker.sh | ||
| ``` | ||
|
|
||
| The arguments of `playwright-docker.sh` are passed to `playwright test`. For | ||
| instance, to run only some tests, use `-g` with a regular expression matching | ||
| the test titles, which have the form `<folder>/<entry> › step <N>`, and | ||
| optionally `--project` to select a browser: | ||
|
|
||
| ```shell | ||
| # One test folder | ||
| ./playwright-docker.sh -g 'math/' | ||
|
|
||
| # One step of one entry, in one browser | ||
| ./playwright-docker.sh -g 'bobkonf.html.*step 5$' --project=chromium | ||
| ``` | ||
|
|
||
| Use `.*` rather than the `›` separator, and end with `$` so that `step 5` does | ||
| not also match `step 50`. Add `--list` to see the selected tests without | ||
| running them, and `--update-snapshots` to accept the current rendering of the | ||
| selected tests. Screenshots generated outside of the Docker image may differ | ||
| slightly from the ones of the CI. | ||
|
|
||
| To see the screenshots of failed tests, including the ones of the CI (download | ||
| the `playwright-report` artifact of the workflow run), use: | ||
|
|
||
| ```shell | ||
| npx playwright show-report <report folder> | ||
| ``` |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.