Skip to content

[CI] e2e matrix (e2e / e2e-case *-sync-compose / e2e-storage) is flaky and blocks unrelated approved PRs #7768

Open

Description

@Aias00

Summary

The e2e aggregator and its e2e-case / e2e-storage children fail across many unrelated PRs. Because e2e is a required check, otherwise-approved PRs show red (BLOCKED / UNSTABLE) and cannot be merged.

Evidence

Failing checks aggregated across currently-approved open PRs:

check occurrences
e2e-case (...) (e.g. e2e-*-sync-compose, e2e-logging-*-compose) 89
e2e (aggregator) 21
e2e-storage (...) (h2/mysql/opengauss/postgres) 14

Root cause 1 — artifact cascade from build-docker-images

e2e-case fails with (e.g. #7091):

##[error]Unable to download artifact(s): Artifact not found for name: shenyu-images

because the separate build-docker-images job failed (see the Maven-Central issue).

Root cause 2 — config-sync timing race

e2e-storage (#7668) creates a divide selector and immediately calls /anything, but the gateway has not yet synced the selector:

ERROR org.apache.shenyu.plugin.api.utils.WebFluxResultUtils - can not match selector data: divide , path is /anything

The selector is created and then deleted while assertions are still in flight.

Root cause 3 — external service / MQ timing

Affected open PRs (sample)

#7038, #7080, #7088, #7091, #7183, #7188, #7189, #7191, #7192, #7198, #7236, #7387, #7513, #7666, #7668, #7699.

Proposed mitigations

  • Decouple e2e from build-docker-images (build images inline or gate explicitly) so a Maven flake does not cascade.
  • Replace the external httpbin.org dependency with the in-repo local httpbin fixture for all e2e suites (as introduced in [ISSUE #6461] Follow advertised WebSocket master addresses #7661).
  • Ensure a unique RocketMQ/RabbitMQ consumer group per run + seekToBegin.
  • Make the e2e aggregator non-required (the individual cases already carry signal) or auto-retry flaky cases.

Relates to #6281 (flaky-test tracking).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions