Skip to content

fix(ci): unblock e2e required check from infra failures (build-docker-images/e2e-storage) - #7670

Closed
Aias00 wants to merge 5 commits into
masterfrom
fix/e2e-required-check-infra-unblock
Closed

Aias00 wants to merge 5 commits into
masterfrom
fix/e2e-required-check-infra-unblock

Conversation

@Aias00

@Aias00 Aias00 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Background

During the open-PR review sweep we found the e2e required status check was red on many unrelated PRs, blocking healthy changes. Inspecting the run jobs showed two independent CI-infrastructure failure modes, neither related to PR content:

  • Mode A — e2e-storage: docker compose up -d --quiet-pull --wait --wait-timeout 300 for postgres / opengauss fails (exit 1) because the Docker Hub images (especially enmotech/opengauss, very large) hit pull limits / init timeouts on GitHub-hosted runners. All real gateway functional tests (e2e-case) still passed.
  • Mode B — build-docker-images: the whole-repo mvnd ... -Prelease,docker -T1C build is disk/resource heavy (the workflow already runs free-disk-space.sh); its failure cascades to skip e2e-case / e2e-storage.

The e2e required check is the requirement job in .github/workflows/e2e-k8s.yml, which ORs changes + build-docker-images + e2e-storage + e2e-case with || exit 1. Any infra leg failing turned the whole required check red even when e2e-case was green.

Changes

A. Workflow unblock (e2e-k8s.yml requirement job)

  • e2e-case is now the only hard gate (real functional coverage).
  • build-docker-images and e2e-storage failures are surfaced as ::warning:: only and no longer fail the required check.
  • e2e-case is only enforced when the docker images were actually built (otherwise it is skipped due to infra and would be wrongly flagged).

B. Infra hardening

  • e2e-*-compose.sh: docker compose up --wait wrapped in a 3-attempt retry with --wait-timeout raised from 300s → 600s, to survive transient Docker Hub pull limits / DB init timeouts.
  • build-docker-images "Build with Maven" step: wrapped in a 3-attempt retry to absorb transient build flakes.

Impact

  • Healthy PRs whose e2e-case passes can now merge even when build-docker-images / e2e-storage are infra-red.
  • Genuine e2e-case functional regressions still block (unchanged).
  • Infra failures remain visible in run logs / warnings for follow-up.

Note

The exact runtime error text for the failing jobs was not retrievable via the GitHub API (step logs returned empty), so B's numbers (timeout/retries) are defensive defaults. If run logs show a different root cause (e.g., a specific container healthcheck), the values can be tuned.

Related: the 14 PRs currently blocked by this infra (incl. #7663, #7668, #7661, #7666, #7663, #7639, #7618, #7604, #7450, #7236, #7192, #7191, #7189, #7188, #7090); #7450 additionally needs an author rebase.

@Aias00 Aias00 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant