Repository navigation
Generalize rancher-upgrade pipeline to a TEST_MODULE selector - #1027
Open
floatingman wants to merge 21 commits into
Open
floatingman wants to merge 21 commits into
floatingman wants to merge 21 commits into
Conversation
Implements the Longhorn vertical of the Rancher server upgrade validation (rancher#967, parent PRD rancher#946). New validation/rancherupgrade package with a LonghornUpgradeTestSuite whose phase methods run as separate go test invocations selected via -run, one per pipeline stage: - TestLonghornPreUpgrade: create project, install the Longhorn chart at the cluster-scoped catalog latest, create a checksum workload (PVC + nginx deployment with deterministic names) on a Longhorn volume. - TestLonghornPostUpgrade: gate on the management server-version setting reaching rancherUpgradeInput.targetVersion, assert the Longhorn chart survived the server upgrade, verify the checksum workload data. - TestLonghornChartUpgrade: gate on server version, resolve latest from the cluster-scoped catalog, upgrade longhorn + longhorn-crd, assert the installed chart version, verify the checksum workload again. Supporting actions: - actions/clusters: WaitRancherVersion polls the server-version setting until it semver-equals the target, retrying transient read errors. - actions/storage: CreateChecksumWorkload/VerifyChecksumWorkload build and verify a deterministically named PVC/deployment pair whose sha256-verified data file must survive across processes; no session cleanup is registered by design. - actions/charts: UpgradeLonghornChart upgrades longhorn and longhorn-crd with the deletingConfirmationFlag value preserved and OpenAPI validation disabled, waits for Deployed, then gates on deployments and daemonsets. TearDownSuite deliberately skips session.Cleanup and the suite is admin-only: upgrade-subject resources must survive into later phases and the pipeline environment teardown is the cleanup. Both deviations are documented in the package README, along with the token-rotation and catalog-follower-restart notes for the pipeline. Includes schemas/pit_schemas.yaml for the three phase methods. Verified live against a 2.14.3 HA env with a node-driver downstream RKE2 cluster: pre-upgrade PASS (Longhorn 109.3.1+up1.11.2, checksum seeded), server upgraded to 2.15.2 via qa-infra-automation, post-upgrade PASS, chart upgrade PASS (109.3.1+up1.11.2 -> 110.1.0+up1.12.1, checksum intact).
…estrator Dedicated pipeline for the Rancher server upgrade validation (issue rancher#968, PRD rancher#946), community leg: deploy 2.14-latest env, pre-upgrade Longhorn phase, in-place server upgrade via make rancher-upgrade with rotated-token re-injection, post-upgrade and chart-upgrade phases against one shared Qase run, destroy-always with failure-only diagnostics. Versions resolve at runtime via make rancher-resolve-version when the params are empty. Structurally cloned from validation/Jenkinsfile.e2e and the airgap Jenkinsfile call shapes; runs go-pit-validation-job per phase with -runs TestLonghornUpgradeTestSuite/<phase> selectors.
The per-phase executor job can land on a different Jenkins agent than the orchestrator (observed in the first rancher-upgrade run: default-n367q vs default-xj00w). The LB ingress on the tofu-created ephemeral SG only allows the orchestrator's detected public IP, so a phase on another agent could not reach the Rancher API and rancher.NewClient timed out after 22 minutes of retries. awsNodeTemplate carries the three keys the executor's Authorize Security Group stage reads (securityGroup, vpcId, region) so it authorizes its own runner IP on tf-<prefix>-sg before dispatching to the API, matching the shape Jenkinsfile.e2e ships for the same reason.
A declarative pipeline re-applies its parameters{} block to the job
definition after each run, which silently overwrote the JJB-configured
staging values (fork repos + slice branches) with the hard-coded main
defaults on the mower job: the first run deployed from the fork, the
next one fell back to upstream main and lost make
rancher-resolve-version ('No rule to make target'). Defaulting these
five params to their current value when set keeps the per-controller
JJB configuration authoritative instead of racing it.
The one-minute bind wait raced a freshly installed Longhorn: the first volume's provisioning (engine-image pull, instance-manager startup, CSI registration) can exceed a minute on a cold cluster, which failed the pre-upgrade phase with 'context deadline exceeded' in the mower run (the same step passed when the volume bound faster). Ten minutes matches the bound used for the rest of the storage waits.
Jenkins evaluates declarative post conditions in a fixed order with always before failure, so the failure-block diagnostics ran after the always block had already run infra-down and docker rmi: kubectl/helm had no rancher-infra-tools image left to run in (pull access denied, exit 125) and the cluster was already gone regardless. Diagnostics now run first inside always, gated on a non-SUCCESS result, while the images and the live cluster still exist; the failure block keeps only the DESTROY_ON_FAILURE fallback destroy.
The post-upgrade phase failed with an instant 401 from rancher.NewClient even though the rotated token had been extracted (length 66) and yq-injected into the dispatched config. Two fail-fast checks now pinpoint which side is broken: the injected token must get HTTP 200 from the upgraded Rancher API before any phase is dispatched, and the config about to be dispatched must byte-match the token in generated.tfvars. Both run under set +x so the token value stays out of the xtrace (the previous run echoed it into the build log).
The smoke-check curl died with exit 60 (TLS verification) against pitupgrade.qa.rancher.space before it ever reached auth — the same endpoint the tests already call with insecure: true. Pass -k so the check actually adjudicates the token.
token=$(awk ...) creates an unexported shell variable, but yq's strenv() reads the process environment, so the injection silently wrote an empty string into rancher.adminToken — the post-upgrade phase then authenticated with an empty bearer token (instant 401). curl in the same step still worked because its header uses shell expansion, which is why the smoke-check passed while the config check failed (expected length 66, got length 0). export the variable so the child yq process can see it.
- logrus milestone logs in each upgrade phase (version resolution, chart install/upgrade, workload seed, checksum verification) so the job log shows where each phase is without reading the assertions backwards. - rancher-infra-tools/rancher-go-test now carry a build-scoped tag (pitupgrade-<BUILD_NUMBER>) used by every direct docker invocation in this pipeline, so this job no longer depends on whatever another job last built or removed under the shared :latest tag on the same agent; the :latest alias is still applied for qa-jenkins-library's make.runTarget image resolution.
Jenkins evaluates post conditions in a fixed order (always before
failure, cleanup last), so the failure{} destroy block could never
work: always had already shredded workspace_name.txt and tfvars and
removed the infra-tools image teardown depends on, and with
DESTROY_AFTER_TESTS=false + DESTROY_ON_FAILURE=true a failed run
skipped teardown entirely, leaking the AWS environment.
Fold the destroy decision into always (after diagnostics and Qase
completion), gated on DESTROY_AFTER_TESTS or failure+DESTROY_ON_
FAILURE; move the docker rmi and sensitive-artifact shredding into a
cleanup block so they run last.
…ard, flavor plumbing - actions/storage: reject checksum content outside [A-Za-z0-9-] before sh -c interpolation - rancherupgrade: fail fast when resolved latest equals the installed chart version (stale cluster-scoped catalog follower) - Jenkinsfile.rancher-upgrade.e2e: document timeout budget, plumb RANCHER_CHART_REPO_FLAVOR into make args, resolve kubeconfig via INFRA_DIR, select deploy repo by flavor - README + pit_schemas: correct downstream-cluster wording to the Rancher local cluster - actions/clusters: debug logging on server-version read/parse failures
…ersion scope - pvc.go: createNginxDeploymentWithPVC takes an explicit deploymentName, so CreatePVCWorkload keeps random names while CreateChecksumWorkload passes its deterministic one; shared construction replaces ~25 duplicated lines - checksum.go: reuse the shared helper; drop now-unused charts/wloads imports - README: previous-release-line-to-latest wording (pipeline default, not a suite constraint) and a deliberate-deviation bullet for the out-of-band server upgrade
…r#968) Replace the three hardcoded Longhorn -run regexes with an UPGRADE_TEST_MODULES map (longhorn, neuvector, fleet, observability) selected by a new TEST_MODULE parameter (default longhorn, preserving existing behavior). The checkout stage fails fast on an unknown module before any infra spend and logs the selected regexes per stage. Stages renamed to module-agnostic names; the chart-upgrade stage is skipped for fleet (no standalone chart), which closes the shared Qase run in the post-upgrade stage instead. Discharges the rancher#986 review discussion about generic post-upgrade stages as modules land.
The fleet suite serves its GitRepo over SSH from a downstream node, but downstream executors mount no .ssh, so the suite cannot reach a key by itself. For TEST_MODULE=fleet the config-creation stage injects the orchestrator's node PEM and the tfvars ssh user into the sshNode section of the cattle-config; other modules are unaffected.
The observability suite requires the stackstateConfigs operator inputs (license, url, cluster API key, admin password); the pipeline injects them for TEST_MODULE=observability: the license from the STACKSTATE_LICENSE folder credential (failing fast before infra spend when absent), a STACKSTATE_URL parameter defaulting to the in-cluster router service the co-located agent can always reach, and a self-consistent cluster API key the suite installs the server with.
Build 24 showed the license check firing only after the pre-upgrade environment was already deployed, burning the deploy on a missing credential. The version-resolution stage now verifies STACKSTATE_LICENSE exists and is non-empty before any infra spend; the post-deploy config injection reuses it.
floatingman
requested review from
hamistao,
khushboo-rancher and
lscalabrini01
October 26, 2026 14:12
Add an all entry to UPGRADE_TEST_MODULES whose broad per-phase regex picks up every module suite in the package (and future ones); thread the fleet ssh key and the observability license when TEST_MODULE=all, and pre-deploy-guard the license for it too. Fleet has no chart subtest, so the chart phase runs its Setup/TearDown with zero matched methods.
Take the Jenkinsfile from the integration branch verbatim: in all-mode dispatch the observability chart upgrade before longhorn's (Longhorn's engine live-upgrades stall a concurrent observability rollout's volume reattach; build 52).
Comment on lines
+66
to
+79
| neuvector: [ | ||
| pre: '-run "TestNeuVectorUpgradeTestSuite/TestNeuVectorPreUpgrade"', | ||
| post: '-run "TestNeuVectorUpgradeTestSuite/TestNeuVectorPostUpgrade"', | ||
| chart: '-run "TestNeuVectorUpgradeTestSuite/TestNeuVectorChartUpgrade"' | ||
| ], | ||
| fleet: [ | ||
| pre: '-run "TestFleetUpgradeTestSuite/TestFleetPreUpgrade"', | ||
| post: '-run "TestFleetUpgradeTestSuite/TestFleetPostUpgrade"' | ||
| ], | ||
| observability: [ | ||
| pre: '-run "TestObservabilityUpgradeTestSuite/TestObservabilityPreUpgrade"', | ||
| post: '-run "TestObservabilityUpgradeTestSuite/TestObservabilityPostUpgrade"', | ||
| chart: '-run "TestObservabilityUpgradeTestSuite/TestObservabilityChartUpgrade"' | ||
| ], |
Contributor
There was a problem hiding this comment.
We do not have these tests on this branch
Contributor
Author
There was a problem hiding this comment.
We will once all related PRs are merged.
lscalabrini01
approved these changes
Oct 28, 2026
lscalabrini01
left a comment
lscalabrini01
left a comment
Contributor
There was a problem hiding this comment.
Looks good, but we should wait for the related PRs be merged
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scoped to #968 (pipeline slice; parent PRD #946). Generalizes the six-stage rancher-upgrade pipeline from Longhorn-only dispatch to a per-module selector, so the same pipeline runs any
validation/rancherupgradesuite end to end. This also discharges the #986 review discussion asking for generic post-upgrade stages as modules land (khushboo-rancher's comment onJenkinsfile.rancher-upgrade.e2e:929).Change
TEST_MODULEparameter (longhorn | neuvector | fleet | observability, defaultlonghorn, preserving current behavior exactly).-runregexes become anUPGRADE_TEST_MODULESmap of per-phase selectors consumed by the pre/post/chart-upgrade stage dispatches.Pre-Upgrade Tests,Post-Upgrade Tests,Chart-Upgrade Tests); build description now carries the module.fleetships inside Rancher with no standalone chart, so its map has no chart phase: the chart-upgrade stage is skipped via awhenguard, and the post-upgrade stage completes the shared Qase run for such modules (thepost{always}fallback still covers failure paths for every module).Verification
validation/Jenkinsfile.rancher-upgrade.e2e; withTEST_MODULE=longhornthe three stage regexes are byte-identical to the previous constants, and build add basic CI for go build & golangci-lint #9/Introducing codeowners #10/testing, DNM #11's proven parameter set is unchanged apart from the new parameter's default.-list-verified method names from Add NeuVector pre/post/chart-upgrade validation to rancherupgrade package #1024/Add Fleet GitRepo survival and lockstep validation to rancherupgrade package #1025/Add SUSE Observability pre/post/chart-upgrade validation to rancherupgrade package #1026.Merge order
#986 → this PR → #1024 (neuvector) → #1025 (fleet) → #1026 (observability). Once #986 merges, this PR retargets cleanly since its only delta is the pipeline file.