Skip to content

e2e job always reports success, so the required ci-success check passes even when e2e is broken #2358

Description

@kirilly

The Tests workflow has an e2e job that can't currently pass, but it shows green on every run, and ci-success (the check our main/dev ruleset requires) depends on it. So a broken e2e never blocks a merge, and nobody notices it's broken.

Evidence

Run https://github.com/peanutprotocol/peanut-ui/actions/runs/27425406519/job/81061724408. The "Run E2E tests" step logs TEST_HARNESS_SECRET is not set and exits 1, but the step and the job both report success.

Why it's green

  • In .github/workflows/tests.yml the step has continue-on-error: true, so the job passes no matter what happens.
  • The step never passes TEST_HARNESS_SECRET (or API_BASE_URL / UI_BASE_URL) into the env, and nothing starts the API or UI, so e2e/global-setup.ts throws at getHarnessSecret() before any test runs. It cannot pass as written.
  • e2e is listed in ci-success.needs, and the ruleset "Require ci-success on api + ui (main, dev)" makes ci-success the required check. So the required gate is satisfied by a job that is guaranteed green.
  • The report job lists e2e in needs but only parses the unit JUnit, so its "all green" summary never reflects e2e anyway.

Options

  • Wire the secret and stand up the API/UI, then remove continue-on-error so real failures block.
  • Or, until it's wired, skip the job cleanly with an if: guard and take it out of ci-success.needs (the way eslint is already handled), so we stop claiming a passing check that never actually ran.

Either way, stop counting a can't-fail job as a required check.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions