Cut the e2e pipeline cost: environments by trigger, weekly bootstrap sweep, less setup per leg

[sscarduzio]

Makes the e2e pipeline cheaper to run. No assertion is weakened, and no ELK version stops being tested. The e2e legs run on the environments each trigger needs, the bootstrap sweep moves to its own scheduled workflow, and every leg carries less fixed setup.

What each trigger runs

The scheduling model @coutoPL asked for in review.

all-e2e-tests.yml

trigger	images	environments	legs
daily cron, manual dispatch	prod and dev (built from develop)	docker, eck-2.16.1, eck-3.5.0	12 + 12
push to master	prod	docker, eck-3.5.0	8
push to develop	dev	docker, eck-3.5.0	8
pull request to master	prod	docker, eck-3.5.0	8
pull request to develop	dev	docker, eck-3.5.0	8
draft pull request	as the ready one	docker	4

ready_for_review is a trigger, so a draft that becomes a pull request gets its ECK leg. Bootstrap tests do not run on pushes or pull requests.

bootstrap-tests.yml (new)

trigger	images	environment	legs
Sunday cron	prod	docker	34
Wednesday cron	dev (built from develop)	docker	34
manual dispatch	both	docker	68

Before this PR: 80 legs on every PR to master, every push to master and every night; 13 on a PR to develop.

Coverage: what still runs

Change	Coverage after
ECK legs by trigger	The daily cron runs every ELK version on all three environments, against both image sets. Pushes and pull requests run docker and eck-3.5.0; a draft runs docker.
Bootstrap sweep on a schedule, in its own workflow	All 34 versions still swept, on docker, against prod and dev images. It no longer gates on prod-e2e-tests, so a flaky suite leg cannot skip it (it ran on 14 of the last 30 nightlies).
One describe.skip spec deleted	Nothing. Kibana-config has never executed an assertion in this repository and cannot run in either environment here; the ROR KBN copy is untouched.
Tenancy tests moved out of runTests()	The same assertions run. Two of them read neither callback, so passes 2 and 3 were byte-identical copies. The third (share link) keeps its assertions; the first test in runTests() already covers the variants it dropped.
Malformed-settings step removed from Settings.cy.ts	Nothing real. Its only live assertion was expect(statusCode).to.eq(200) on saving a broken config.
cy.wait(200) between ESC keydown and keyup	Nothing. Same events, same order, no gap. Now KibanaNavigation.closeOpenOverlays().
Disk cleanup, yarn/Cypress caches, paths-ignore	Nothing. Runner setup only.

Net effect on the suite: same specs, same assertions, 13 test bodies instead of 17 in Tenancy.cy.ts.

The changes

Pick the e2e environments by trigger — a plan job decides the images (prod, dev) and the environments for the trigger, and the test jobs only ask it. ELK_VERSIONS, ECK_3X and ECK_2X are workflow env vars at the top, one line each to bump. The earlier ECK "pairing" and the --flip trick are gone: the daily cron covers every (version, environment) cell, so a PR does not have to.

Move the bootstrap sweep to its own workflow — --run bootstrap runs no tests; the only assertion is that the stack came up. Its input is the plugin images, which a PR to this repo cannot change. Yield over 75 nightlies: 7 failures in 3,317 executed legs, 6 of them the 7.10.0 already removed from the list. The dev sweep dispatches the plugin pre-build workflows for all 34 versions from develop; the wait timeouts are 4 hours per plugin, which is not measured, because this run has never existed.

Move the yarn and Cypress caches into their own action — .github/cypress-cache, used by run-e2e-tests. Measured over 10 legs of nightly 34547123638: 30s for the install and 5-22s for the Cypress binary on every leg.

Skip the disk cleanup when the runner already has room — deleting the preinstalled toolchains costs about 80s. A bootstrap leg asks for 8 GB, a suite leg for 25. Each rm -rf runs in the background and is waited on by pid, so a failed deletion fails the step.

Delete the spec that runs in no environment — Kibana-config.cy.ts is describe.skip on the whole file: two replicas disagree about the active config on docker, and kibana.yml is read-only on ECK. Four unreferenced kibana.yml fixtures go with it. ror-config.cy.ts stays: #133 kept it with the reason on the skip (the suite lifts it when #116 merges), together with the two fixtures it reads.

Drop an assertion that could only fail if ROR got it right — Settings.cy.ts saved readonlyrest: and asserted a 200.

Run Tenancy's two callback-free tests once, not three times — and the share-link test once, because the tenancy-survives-the-variant property is already asserted by the first test in runTests().

Name the retried tests on the run summary — a retried test that passed leaves no trace in the run output. The after:spec handler lists them on the GitHub run summary. Cypress keeps one handler per event name, so the report shares the handler with the video discard.

Numbers

Measured before the review rework and still true: the per-spec timings, the bootstrap yield, the cache savings, and the 80 legs per PR baseline. Not re-measured: the "about 60%" saving of the first version. The legs per trigger above are simulated against the if: expressions; the daily cron costs more than before (24 suite legs plus two plugin builds, against 12), pushes and pull requests cost less.

Checked before pushing

• yarn tsCheck clean and yarn lint at 0 errors.
• actionlint clean on both workflows.
• Every YAML parsed with a duplicate-key-rejecting loader.
• The plan step run locally for the seven trigger shapes (cron, dispatch, push master/develop, PR to master/develop, draft, fork) gives the expected images and matrix.
• Every if: expression simulated for the nine trigger shapes, with needs propagation: the leg tables above.

Deliberately left alone

• Test-settings.cy.ts and Settings.cy.ts test 1 (~53s). Their page objects assert real responses, so removing them drops the "the button is wired to the endpoint" coverage.
• RorMenu.ts:49's 300ms settle. cy.get().should() retries the assertion but never re-runs the click.
• Sharding. The account pool peaks at 20 concurrent jobs; sharding adds runner-minutes per leg and lengthens every queue.

Related pull requests

PR	Repo	What it does
sscarduzio/readonlyrest_kbn#1039	KBN plugin	Mirrors the spec changes here into automatic-tests/.
#130	this repo	Gives ro_strict real coverage: roStrictSettings.yaml is a whitespace-only copy of roSettings.yaml.
#131	this repo	Moves two API-only specs out of Cypress into a plain HTTP suite.

🤖 Generated with Claude Code

Summary by CodeRabbit

• CI Improvements

  • Improved end-to-end test workflow selection, cancellation, retries, caching, and disk-space handling.
  • Added reporting for tests that pass after retries.
  • Reduced duplicate tenancy test executions.
  • Updated scheduled bootstrap testing across supported ELK versions.

• Test Maintenance

  • Removed obsolete disabled Kibana configuration coverage and unused fixtures.
  • Simplified outdated test scenarios and navigation handling.

• Documentation

  • Clarified that master and develop test suites run on daily schedules.

[Dzuming]

What do we do with this reload Kibana config logic? Currently, it's blocked because:

• It's not working in the case of multiple Kibana instances
• It's not working in the case of ECK

So, it should be diagnosed and fixed on the ReadonlyREST Kbn plugin side. We need a follow-up task to decide if we will fix it or drop it because of the cost of changes in this logic, which is only for testing purposes.

[sscarduzio]

The spec is deleted here because it cannot run in either environment of this repo: two replicas behind the proxy on docker, read-only kibana.yml on ECK. The decision to fix the reload logic on the plugin side or drop it belongs in a follow-up ticket. Can you open it, or shall I?

[Dzuming]

Please, create a Jira task for it

[coutoPL]

LGTM

[coutoPL]

LGTM