Repository navigation
Add Fleet GitRepo survival and lockstep validation to rancherupgrade package - #1025
Open
floatingman wants to merge 26 commits into
Open
floatingman wants to merge 26 commits into
floatingman wants to merge 26 commits into
Conversation
Implements the Longhorn vertical of the Rancher server upgrade validation (rancher#967, parent PRD rancher#946). New validation/rancherupgrade package with a LonghornUpgradeTestSuite whose phase methods run as separate go test invocations selected via -run, one per pipeline stage: - TestLonghornPreUpgrade: create project, install the Longhorn chart at the cluster-scoped catalog latest, create a checksum workload (PVC + nginx deployment with deterministic names) on a Longhorn volume. - TestLonghornPostUpgrade: gate on the management server-version setting reaching rancherUpgradeInput.targetVersion, assert the Longhorn chart survived the server upgrade, verify the checksum workload data. - TestLonghornChartUpgrade: gate on server version, resolve latest from the cluster-scoped catalog, upgrade longhorn + longhorn-crd, assert the installed chart version, verify the checksum workload again. Supporting actions: - actions/clusters: WaitRancherVersion polls the server-version setting until it semver-equals the target, retrying transient read errors. - actions/storage: CreateChecksumWorkload/VerifyChecksumWorkload build and verify a deterministically named PVC/deployment pair whose sha256-verified data file must survive across processes; no session cleanup is registered by design. - actions/charts: UpgradeLonghornChart upgrades longhorn and longhorn-crd with the deletingConfirmationFlag value preserved and OpenAPI validation disabled, waits for Deployed, then gates on deployments and daemonsets. TearDownSuite deliberately skips session.Cleanup and the suite is admin-only: upgrade-subject resources must survive into later phases and the pipeline environment teardown is the cleanup. Both deviations are documented in the package README, along with the token-rotation and catalog-follower-restart notes for the pipeline. Includes schemas/pit_schemas.yaml for the three phase methods. Verified live against a 2.14.3 HA env with a node-driver downstream RKE2 cluster: pre-upgrade PASS (Longhorn 109.3.1+up1.11.2, checksum seeded), server upgraded to 2.15.2 via qa-infra-automation, post-upgrade PASS, chart upgrade PASS (109.3.1+up1.11.2 -> 110.1.0+up1.12.1, checksum intact).
…estrator Dedicated pipeline for the Rancher server upgrade validation (issue rancher#968, PRD rancher#946), community leg: deploy 2.14-latest env, pre-upgrade Longhorn phase, in-place server upgrade via make rancher-upgrade with rotated-token re-injection, post-upgrade and chart-upgrade phases against one shared Qase run, destroy-always with failure-only diagnostics. Versions resolve at runtime via make rancher-resolve-version when the params are empty. Structurally cloned from validation/Jenkinsfile.e2e and the airgap Jenkinsfile call shapes; runs go-pit-validation-job per phase with -runs TestLonghornUpgradeTestSuite/<phase> selectors.
The per-phase executor job can land on a different Jenkins agent than the orchestrator (observed in the first rancher-upgrade run: default-n367q vs default-xj00w). The LB ingress on the tofu-created ephemeral SG only allows the orchestrator's detected public IP, so a phase on another agent could not reach the Rancher API and rancher.NewClient timed out after 22 minutes of retries. awsNodeTemplate carries the three keys the executor's Authorize Security Group stage reads (securityGroup, vpcId, region) so it authorizes its own runner IP on tf-<prefix>-sg before dispatching to the API, matching the shape Jenkinsfile.e2e ships for the same reason.
A declarative pipeline re-applies its parameters{} block to the job
definition after each run, which silently overwrote the JJB-configured
staging values (fork repos + slice branches) with the hard-coded main
defaults on the mower job: the first run deployed from the fork, the
next one fell back to upstream main and lost make
rancher-resolve-version ('No rule to make target'). Defaulting these
five params to their current value when set keeps the per-controller
JJB configuration authoritative instead of racing it.
The one-minute bind wait raced a freshly installed Longhorn: the first volume's provisioning (engine-image pull, instance-manager startup, CSI registration) can exceed a minute on a cold cluster, which failed the pre-upgrade phase with 'context deadline exceeded' in the mower run (the same step passed when the volume bound faster). Ten minutes matches the bound used for the rest of the storage waits.
Jenkins evaluates declarative post conditions in a fixed order with always before failure, so the failure-block diagnostics ran after the always block had already run infra-down and docker rmi: kubectl/helm had no rancher-infra-tools image left to run in (pull access denied, exit 125) and the cluster was already gone regardless. Diagnostics now run first inside always, gated on a non-SUCCESS result, while the images and the live cluster still exist; the failure block keeps only the DESTROY_ON_FAILURE fallback destroy.
The post-upgrade phase failed with an instant 401 from rancher.NewClient even though the rotated token had been extracted (length 66) and yq-injected into the dispatched config. Two fail-fast checks now pinpoint which side is broken: the injected token must get HTTP 200 from the upgraded Rancher API before any phase is dispatched, and the config about to be dispatched must byte-match the token in generated.tfvars. Both run under set +x so the token value stays out of the xtrace (the previous run echoed it into the build log).
The smoke-check curl died with exit 60 (TLS verification) against pitupgrade.qa.rancher.space before it ever reached auth — the same endpoint the tests already call with insecure: true. Pass -k so the check actually adjudicates the token.
token=$(awk ...) creates an unexported shell variable, but yq's strenv() reads the process environment, so the injection silently wrote an empty string into rancher.adminToken — the post-upgrade phase then authenticated with an empty bearer token (instant 401). curl in the same step still worked because its header uses shell expansion, which is why the smoke-check passed while the config check failed (expected length 66, got length 0). export the variable so the child yq process can see it.
- logrus milestone logs in each upgrade phase (version resolution, chart install/upgrade, workload seed, checksum verification) so the job log shows where each phase is without reading the assertions backwards. - rancher-infra-tools/rancher-go-test now carry a build-scoped tag (pitupgrade-<BUILD_NUMBER>) used by every direct docker invocation in this pipeline, so this job no longer depends on whatever another job last built or removed under the shared :latest tag on the same agent; the :latest alias is still applied for qa-jenkins-library's make.runTarget image resolution.
Jenkins evaluates post conditions in a fixed order (always before
failure, cleanup last), so the failure{} destroy block could never
work: always had already shredded workspace_name.txt and tfvars and
removed the infra-tools image teardown depends on, and with
DESTROY_AFTER_TESTS=false + DESTROY_ON_FAILURE=true a failed run
skipped teardown entirely, leaking the AWS environment.
Fold the destroy decision into always (after diagnostics and Qase
completion), gated on DESTROY_AFTER_TESTS or failure+DESTROY_ON_
FAILURE; move the docker rmi and sensitive-artifact shredding into a
cleanup block so they run last.
…ard, flavor plumbing - actions/storage: reject checksum content outside [A-Za-z0-9-] before sh -c interpolation - rancherupgrade: fail fast when resolved latest equals the installed chart version (stale cluster-scoped catalog follower) - Jenkinsfile.rancher-upgrade.e2e: document timeout budget, plumb RANCHER_CHART_REPO_FLAVOR into make args, resolve kubeconfig via INFRA_DIR, select deploy repo by flavor - README + pit_schemas: correct downstream-cluster wording to the Rancher local cluster - actions/clusters: debug logging on server-version read/parse failures
…ersion scope - pvc.go: createNginxDeploymentWithPVC takes an explicit deploymentName, so CreatePVCWorkload keeps random names while CreateChecksumWorkload passes its deterministic one; shared construction replaces ~25 duplicated lines - checksum.go: reuse the shared helper; drop now-unused charts/wloads imports - README: previous-release-line-to-latest wording (pipeline default, not a suite constraint) and a deliberate-deviation bullet for the out-of-band server upgrade
- Extract the fleet git-repo helpers from validation/fleet/upgrade into the actions module as exported, error-returning, godoc-commented actions (GitPushCommit, CreateLocalFleetGitRepo, CreateFleetSSHSecret, VerifyNewGitCommit, VerifyRepoUpdate, VerifyRepoPause) and refactor the donor suite onto them; the initial-sync verification moves to the callers because the actions module cannot depend on interoperability. - Add FleetUpgradeTestSuite to validation/rancherupgrade with exactly two phases: TestFleetPreUpgrade seeds the SSH secret, the node-local git remote and the deterministic GitRepo on the shared downstream cluster, then records the pre-upgrade fleet-controller/fleet-agent image versions and the synced commit SHA in a labeled state ConfigMap in fleet-default; TestFleetPostUpgrade waits for the upgraded server version first, verifies the GitRepo is still in sync, pushes a new commit through the full reconcile loop, and asserts controller and agent upgraded in lockstep. No chart-upgrade phase: Fleet has no standalone chart in the Rancher catalog. - Append PIT schema entries for both phases to validation/rancherupgrade/schemas/pit_schemas.yaml.
The ansible-provisioned downstream cluster's nodes carry no Rancher machine sshkeys link, so ssh.CreateSSHNode resolves an empty key URL against them (seen live: Get "": unsupported protocol scheme ""). Downstream executors also mount no .ssh, so the key cannot come from disk. The rancher-upgrade pipeline now threads the node PEM and ssh user through the sshNode section of the test config for the fleet module, and fleetSSHNode builds the node from the downstream worker list plus that material, picking the first node by name so both phases land on the same host.
The post-upgrade phase pushes the reconcile-loop commit through the same node-local remote, so it needs the same pipeline-injected node material as the pre-upgrade phase.
The ansible-provisioned instances get public IPs from terraform (associate_public_ip_address), but kubelet does not report them as node addresses, so the first fleet attempt dialed the internal VPC IP and timed out (dial tcp 172.31.13.186:22). When the node reports no external address, map the internal one to the instance's public IP with the aws CLI using the awsNodeTemplate region.
lscalabrini01
previously approved these changes
Oct 22, 2026
lscalabrini01
left a comment
lscalabrini01
left a comment
Contributor
There was a problem hiding this comment.
LGTM! just a nit, the gomod check is failing
The JMESPath projection Reservations[].Instances[].PublicIpAddress yields a flat list of strings (with nulls for instances that lack one), not a nested list.
VerifyGitRepo's own exponential backoff tops out near a minute, which is too short for the fleet controller's first-ever git clone job on a cold cluster (gitjob scheduling, image pulls) and for the controller restart the server upgrade causes before the post-upgrade phase (seen live: the pre-upgrade phase failed 56s after GitRepo creation with 'timed out waiting for the condition' while the network path was open). Wrap it in a ten-minute retry that surfaces the last verification error.
The fleet-default controller only processes clusters registered in the fleet-default workspace, while the local cluster is registered in fleet-local: a fleet-default GitRepo targeting cluster local matched no cluster, so no bundles were ever created and the GitRepo sat unready without a display error for the whole verification window (seen live on build rancher#20). CreateFleetSSHSecret and CreateLocalFleetGitRepo now take the workspace explicitly (donor callers keep fleet-default), and the rancherupgrade suite resolves it from the target: fleet-local for the local cluster, fleet-default otherwise.
The fleet cluster registration for the local cluster lives in fleet-local, so the verification's registration lookup followed the GitRepo into the target's workspace instead of assuming fleet-default (seen live: clusters.fleet.cattle.io "local" not found from /v1/fleet.cattle.io.clusters/fleet-default/local).
floatingman
requested review from
hamistao and
khushboo-rancher
and removed request for
a team
October 26, 2026 14:13
actions/fleet/fleet.go imports github.com/rancher/fleet/pkg/apis directly, so the module must list it as a direct require. Also merges upstream/main (fleet v0.17.0-alpha.1, go 1.27.3) so the tidied module matches what verify-gomod checks on the merge ref.
Consistent module-upgrade file naming across the rancherupgrade suites (review follow-up, khushboo-rancher); the suite is already named TestFleetUpgradeTestSuite.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #970 (parent PRD #946). Adds the Fleet vertical to the
validation/rancherupgradepackage from #986: GitRepo survival, new-commit reconcile, and fleet controller/agent version lockstep across the server upgrade.Helper extraction (move, not copy)
The commit-push and SSH-repo machinery was generalized out of
validation/fleet/upgrade/upgrade.go(now deleted) into exported, error-returning, godoc-commented functions inactions/fleet/fleet.go(git detects the rename):GitPushCommit,CreateLocalFleetGitRepo,CreateFleetSSHSecret,VerifyNewGitCommit,VerifyRepoUpdate,VerifyRepoPause.namegeneratorrandoms stay at the donor call sites) and droppingclientfromGitPushCommit(unused).VerifyGitRepocall moved to callers: the actions module cannot importinteroperability(module direction), and the donor tests now callinteroperability/fleet.VerifyGitRepoexplicitly, preserving the original create-time verification semantics for all three donor tests.Suite
TestFleetPreUpgradeactions/ssh.CreateSSHNode(first-worker selection keeps the node stable across phases), SSH secret infleet-default, fleet-examples clone doubling as the node-local git remote, GitReporancher-upgrade-fleet-gitreposynced viaVerifyGitRepo, then records controller/agent deployment versions plus the synced SHA into ConfigMapfleet-default/rancher-upgrade-fleet-stateTestFleetPostUpgradeWaitRancherVersion; GitRepo still in sync; a new empty commit is pushed and the full reconcile loop is verified (new SHA fetched, bundle redeploys on the target cluster, not frozen state); lockstep assertion:fleet-controller(local) andfleet-agent(downstream, with thecattle-fleet-local-systemnamespace split handled) versions differ from the recordings and equal each otherNo chart-upgrade method: Fleet ships inside Rancher with no standalone chart (per the PRD).
Verification
go vet -tags validationclean forvalidation/rancherupgrade,validation/fleet/...(donor package still whole), and the actions module; golangci-lint reports no findings on any touched file.-listshowsTestFleetUpgradeTestSuite;-run TestFleetUpgradeTestSuite/TestFleetPreUpgradeand.../TestFleetPostUpgradeeach select exactly one method; skip guard verified to fire before any client creation.schemas/pit_schemas.yamlextended with theRancher Upgrade/Fleetsuite (append-only; custom field 15 = method names) and parses clean.Pending
-runregex mechanism.