test(e2e): add Playwright integration-test suite#1553
Open
dawsontoth wants to merge 22 commits into
Open
Conversation
Standalone Playwright harness under e2e/ (its own pnpm root, deliberately kept out of the app workspace) covering the highest-value auth flows against the deployed dev app: - sign-in page render + client-side validation - email-verification screens (/#/verifying, /#/verify-email) - full email round-trip via Mailosaur: signup -> verification link -> verified login - authenticated shell + org users list The LLM authors/maintains a deterministic suite; it is not in the loop at run time. anon specs need no credentials; setup/authed/round-trip skip cleanly until a test account and Mailosaur creds land in .env.e2e. Also exclude e2e/ from the app's vitest config: its *.spec.ts import '@playwright/test' and crash vitest's collector when run from the worktree (or once e2e/ merges to the repo root). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Scheduled monitor for the trusted lane (daily + merge-to-stage), built to keep the LLM off the pass-path: - run-trusted.sh: deterministic Playwright run -> results/results.json. No LLM. - on-schedule.sh: scheduler entry; runs the suite and, only on failure, invokes `claude -p` with a default-deny allowlist (Read + gh issue ops). - triage.md: classify each failure (regression/flake/test-drift) and file or update an `e2e-failure` issue on HarperFast/studio. - automation/README.md: flow, launchd example, laptop setup, churn notes. Adds a JSON reporter, gitignores results/, and tags the signup round-trip @roundtrip so daily runs can exclude it to limit dev account churn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
poll-stage.sh runs on a frequent schedule and invokes on-schedule.sh only when the stage branch advances (its SHA, read via gh, compared against the last-tested SHA stored outside the repo in $STUDIO_E2E_STATE). Documents the two-schedule setup (daily unconditional + 15-min poll) and the dev deploy-lag caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makes the suite easy to run against a locally-running studio during development, so regressions surface while enhancing the app: - test:local / test:local:ui / test:local:headed target http://localhost:5173 and reuse the running dev server (or start one). UI mode is the tight feedback loop. - README leads with the local-dev workflow and notes the local dev server talks to the stage backend (per .env.local), so authed/round-trip specs need a stage-valid account; anon specs need none. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The standalone e2e project doesn't inherit the app workspace's pnpm policies, and it runs on the automation laptop with real secrets present, so a compromised dependency is a real threat. Add defense in depth: - onlyBuiltDependencies: [] — make "no dependency runs install/build scripts" explicit (pnpm 11 blocks them by default; this keeps enabling one a reviewed change). - minimumReleaseAge 10080 (7d) — don't adopt just-published versions on future updates; inert while installing the committed lockfile with --frozen-lockfile. - run-trusted.sh installs with --ignore-scripts (the browser is fetched by the explicit `playwright install`, not a postinstall). Surfaced by the adversarial review of the automation plan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolveOrgId now polls for the org signal — a single-org redirect OR the async-loaded org-card link on the picker — instead of reading the DOM once, removing the redirect/load race flagged earlier. The org-users spec skips cleanly (with a clear reason) when the account has no accessible org or the org route 403s, rather than failing on a missing table. Surfaced while running against stage: a dev-specific PLAYWRIGHT_ORG_ID pinned in .env.e2e is invalid on stage, which now yields a clean skip instead of a red failure. With it unset, auto-resolution goes green on stage (8/8) — org id is a per-environment value, like the base URL and credentials. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…le.sh launchd runs scripts with a bare PATH (/usr/bin:/bin:...) — no nvm, no Homebrew, no ~/.local/bin — so the triage step's `claude` (~/.local/bin) and `gh` (Homebrew) were unreachable. Prepend those dirs and source nvm up front. run-trusted.sh already self-heals node/pnpm via nvm (verified under `env -i`), so the run path was fine; this covers the triage tools. Surfaced during dedicated-machine setup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A broken triage (claude not logged in, gh token unreadable, network) previously left the script exiting 0, so a scheduled run would look successful while silently reporting nothing. Capture the claude exit code, log loudly, and exit non-zero. Surfaced during dedicated-machine setup.
`claude -p` exits 0 when the agent finishes even if a gh call inside it failed (a 401 still 'completed'), so the run reported success while filing nothing. triage.md now must end with TRIAGE_RESULT: filed|updated|noop|error, and on-schedule.sh requires a success marker or exits 3. Also documents that gh's token must be file-based (~/.config/gh/hosts.yml), not keychain, since the keychain token is unreachable inside Claude Code's Bash sandbox / launchd. Surfaced during dedicated-machine setup.
install-launchd.sh generates and (re)loads the daily + 15-min poll LaunchAgents with derived paths — replaces fragile manual plist pasting (interactive zsh mangles heredocs/comments). env.sh centralizes the launchd PATH/nvm setup; on-schedule.sh and poll-stage.sh source it. Fixes poll-stage.sh, which called gh without the PATH fix, so under launchd gh wasn't found, the SHA read was empty, and the poller silently never triggered. Surfaced during dedicated-machine setup.
run-trusted.sh runs the suite with CI=1 so the config's retries (2) + bounded workers apply. A slow-deployment page-load timeout now self-heals on retry instead of filing a spurious flake issue; a genuine failure still fails all retries and reports non-zero. Surfaced by a real run where a sluggish dev (1.1m vs ~13s) timed out two page loads while the same field rendered fine in a sibling test.
on-schedule.sh tees the triage output to ~/Library/Logs/studio-e2e/triage.last.log (override via STUDIO_E2E_TRIAGE_LOG) and runs claude -p with --verbose, so an in-flight triage is watchable with 'tail -f' instead of buffered in a command substitution until claude exits. Still captures the output to verify the TRIAGE_RESULT marker (PIPESTATUS[0] for claude's status).
The round-trip creates a real account on the target each run; it now self-deletes via the logged-in session (GET /User/current -> DELETE /User/{id}) so the environment doesn't accumulate test users. Runs in a finally and is best-effort, so a cleanup failure never fails the test. This keeps full round-trip coverage on every scheduled run without churn — the alternative to gating @roundtrip to a weekly cadence.
…k the scheduler macOS has no timeout(1), so on-schedule.sh runs claude -p in the background with a watchdog (TERM, then KILL after a grace). STUDIO_E2E_TRIAGE_TIMEOUT (default 300s). A timed-out triage exits non-zero with no success marker, so the existing TRIAGE_FAILED guard fires instead of the run hanging forever. Motivated by the wedged triage seen during setup.
… app is pre-served The PR-lane sandbox builds the PR frontend and serves it via 'vite preview', then points Playwright at localhost. Without this, targeting localhost makes the config try to spin up 'pnpm dev' on :5173 (and under CI reuseExistingServer is off), fighting the sandbox's own server. The sandbox sets PLAYWRIGHT_NO_WEBSERVER=1.
…ontainer When the run happens in a --cap-drop=ALL container (PR-lane sandbox), Chromium's own sandbox can't initialize; the container is the sandbox, so launch Chromium with --no-sandbox when PLAYWRIGHT_NO_SANDBOX is set. Never set for local/CI.
… lock) In the egress-locked PR-lane sandbox the browser must reach the backend through the same allowlist proxy as everything else; localhost (the served build) bypasses it. Uses Playwright's use.proxy when PLAYWRIGHT_PROXY is set.
The scheduler + triage prompt are control plane and now live in the studio-e2e-harness repo (trusted/), out of reach of studio PRs — the relocated lane is verified running from there. The specs stay here, co-located for local dev; README now points at the harness for the automation + sandbox. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Coverage Report
File CoverageNo changed files found. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a standalone Playwright end-to-end integration test harness under the e2e/ directory, featuring Docker configuration, package files, and test specs for authentication and organization user management. Feedback on these changes highlights critical issues with non-existent, hallucinated versions of Node.js, TypeScript, and Node types in the Dockerfile, package.json, and lockfile that will break the build. Additionally, improvements are suggested to make the throwaway account cleanup logic more robust against early test failures and to replace a manual polling loop with Playwright's native expect.poll helper.
cb1kenobi
reviewed
Jul 24, 2026
… skip signup: wrap the whole flow in try/finally so the throwaway account is cleaned up on any post-signup failure (self-delete needs a session, so a pre-login failure is a no-op — noted). org-users: replace the manual waitForTimeout poll in resolveOrgId with expect.poll (capturing the value in the polled closure); skip only on a real 403 during load and fail red if the error boundary shows with no 403, so a genuine break can't hide as a permission skip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dawsontoth
commented
Jul 24, 2026
Replace the static SuperSecret123! with a per-run randomPassword() (randomUUID-based, always upper+lower+digit+symbol) so no static credential lives in the repo. Addresses review on signup-verification.anon.spec.ts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dawsontoth
commented
Jul 24, 2026
Add test:e2e:docker + test:e2e:snapshots at the repo root so the canonical Linux-container runs (esp. screenshot baselines) are discoverable without cd-ing into e2e/ or remembering the compose invocation. Document them in e2e/README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Playwright end-to-end integration-test suite for studio (
e2e/), plus a one-linevitest.config.tsexclude so the app's unit tests don't collect these specs.What this is
e2e/— its ownpackage.json/ lockfile /pnpm-workspace.yaml,deliberately NOT part of the app's pnpm workspace, so its deps + browser downloads stay out of the
app build. Holds only narrow, disposable test credentials, never
.env.local.email-verification screens, an org-users list, and a full signup → email round-trip (Mailosaur)
→ verify → login flow.
*.anon.spec.ts(no session) vs*.authed.spec.ts(logged in, via asaved cookie storageState from
auth.setup.ts).cd e2e && pnpm test:local:ui) or headless against deployeddev (
pnpm test). Screenshot baselines come from the Linux container (docker compose run).Why it's shaped this way
The autonomous control plane that runs these tests — daily, on merge-to-
stage, and on PRs(triaging failures into issues, sandboxing untrusted PR code) — lives in a separate repo,
studio-e2e-harness, deliberately kept out of this
repo so a studio PR can't alter what decides whether or how its own code is run. The specs live
here, co-located with the app for local dev; both lanes exercise these same files.
Merging this to
stageis what lets the trusted lane (and, later, the PR lane) run against a basebranch that actually contains the suite.
Notes
e2e/README.mddocuments running it + the conventions worth knowing (hash routing, cookie auth,Sonner toasts, link/token email verification, the Mailosaur round-trip, screenshot baselines).
.env.e2e(real creds) is gitignored;.env.e2e.exampleshows the shape. Signup isdomain-allowlisted server-side (
ALLOWLIST_EMAIL_DOMAINS), so the round-trip spec skips itselfcleanly if the Mailosaur domain isn't allowlisted on the target env.
🤖 Generated with Claude Code