Repository navigation
Add rancherupgrade package: Longhorn pre/post/chart-upgrade validation - #986
floatingman wants to merge 22 commits into
Conversation
Implements the Longhorn vertical of the Rancher server upgrade validation (rancher#967, parent PRD rancher#946). New validation/rancherupgrade package with a LonghornUpgradeTestSuite whose phase methods run as separate go test invocations selected via -run, one per pipeline stage: - TestLonghornPreUpgrade: create project, install the Longhorn chart at the cluster-scoped catalog latest, create a checksum workload (PVC + nginx deployment with deterministic names) on a Longhorn volume. - TestLonghornPostUpgrade: gate on the management server-version setting reaching rancherUpgradeInput.targetVersion, assert the Longhorn chart survived the server upgrade, verify the checksum workload data. - TestLonghornChartUpgrade: gate on server version, resolve latest from the cluster-scoped catalog, upgrade longhorn + longhorn-crd, assert the installed chart version, verify the checksum workload again. Supporting actions: - actions/clusters: WaitRancherVersion polls the server-version setting until it semver-equals the target, retrying transient read errors. - actions/storage: CreateChecksumWorkload/VerifyChecksumWorkload build and verify a deterministically named PVC/deployment pair whose sha256-verified data file must survive across processes; no session cleanup is registered by design. - actions/charts: UpgradeLonghornChart upgrades longhorn and longhorn-crd with the deletingConfirmationFlag value preserved and OpenAPI validation disabled, waits for Deployed, then gates on deployments and daemonsets. TearDownSuite deliberately skips session.Cleanup and the suite is admin-only: upgrade-subject resources must survive into later phases and the pipeline environment teardown is the cleanup. Both deviations are documented in the package README, along with the token-rotation and catalog-follower-restart notes for the pipeline. Includes schemas/pit_schemas.yaml for the three phase methods. Verified live against a 2.14.3 HA env with a node-driver downstream RKE2 cluster: pre-upgrade PASS (Longhorn 109.3.1+up1.11.2, checksum seeded), server upgraded to 2.15.2 via qa-infra-automation, post-upgrade PASS, chart upgrade PASS (109.3.1+up1.11.2 -> 110.1.0+up1.12.1, checksum intact).
|
LonghornUpgradeTestSuite SetupSuite TestSuites above were modified. TestSuites below use modified code from this PR. TestLonghornUpgradeTestSuite |
…estrator Dedicated pipeline for the Rancher server upgrade validation (issue rancher#968, PRD rancher#946), community leg: deploy 2.14-latest env, pre-upgrade Longhorn phase, in-place server upgrade via make rancher-upgrade with rotated-token re-injection, post-upgrade and chart-upgrade phases against one shared Qase run, destroy-always with failure-only diagnostics. Versions resolve at runtime via make rancher-resolve-version when the params are empty. Structurally cloned from validation/Jenkinsfile.e2e and the airgap Jenkinsfile call shapes; runs go-pit-validation-job per phase with -runs TestLonghornUpgradeTestSuite/<phase> selectors.
|
LonghornUpgradeTestSuite SetupSuite TestSuites above were modified. TestSuites below use modified code from this PR. TestLonghornUpgradeTestSuite |
The per-phase executor job can land on a different Jenkins agent than the orchestrator (observed in the first rancher-upgrade run: default-n367q vs default-xj00w). The LB ingress on the tofu-created ephemeral SG only allows the orchestrator's detected public IP, so a phase on another agent could not reach the Rancher API and rancher.NewClient timed out after 22 minutes of retries. awsNodeTemplate carries the three keys the executor's Authorize Security Group stage reads (securityGroup, vpcId, region) so it authorizes its own runner IP on tf-<prefix>-sg before dispatching to the API, matching the shape Jenkinsfile.e2e ships for the same reason.
A declarative pipeline re-applies its parameters{} block to the job
definition after each run, which silently overwrote the JJB-configured
staging values (fork repos + slice branches) with the hard-coded main
defaults on the mower job: the first run deployed from the fork, the
next one fell back to upstream main and lost make
rancher-resolve-version ('No rule to make target'). Defaulting these
five params to their current value when set keeps the per-controller
JJB configuration authoritative instead of racing it.
lscalabrini01
left a comment
lscalabrini01
left a comment
There was a problem hiding this comment.
Looks good, just some comments
The one-minute bind wait raced a freshly installed Longhorn: the first volume's provisioning (engine-image pull, instance-manager startup, CSI registration) can exceed a minute on a cold cluster, which failed the pre-upgrade phase with 'context deadline exceeded' in the mower run (the same step passed when the volume bound faster). Ten minutes matches the bound used for the rest of the storage waits.
Jenkins evaluates declarative post conditions in a fixed order with always before failure, so the failure-block diagnostics ran after the always block had already run infra-down and docker rmi: kubectl/helm had no rancher-infra-tools image left to run in (pull access denied, exit 125) and the cluster was already gone regardless. Diagnostics now run first inside always, gated on a non-SUCCESS result, while the images and the live cluster still exist; the failure block keeps only the DESTROY_ON_FAILURE fallback destroy.
The post-upgrade phase failed with an instant 401 from rancher.NewClient even though the rotated token had been extracted (length 66) and yq-injected into the dispatched config. Two fail-fast checks now pinpoint which side is broken: the injected token must get HTTP 200 from the upgraded Rancher API before any phase is dispatched, and the config about to be dispatched must byte-match the token in generated.tfvars. Both run under set +x so the token value stays out of the xtrace (the previous run echoed it into the build log).
The smoke-check curl died with exit 60 (TLS verification) against pitupgrade.qa.rancher.space before it ever reached auth — the same endpoint the tests already call with insecure: true. Pass -k so the check actually adjudicates the token.
token=$(awk ...) creates an unexported shell variable, but yq's strenv() reads the process environment, so the injection silently wrote an empty string into rancher.adminToken — the post-upgrade phase then authenticated with an empty bearer token (instant 401). curl in the same step still worked because its header uses shell expansion, which is why the smoke-check passed while the config check failed (expected length 66, got length 0). export the variable so the child yq process can see it.
- logrus milestone logs in each upgrade phase (version resolution, chart install/upgrade, workload seed, checksum verification) so the job log shows where each phase is without reading the assertions backwards. - rancher-infra-tools/rancher-go-test now carry a build-scoped tag (pitupgrade-<BUILD_NUMBER>) used by every direct docker invocation in this pipeline, so this job no longer depends on whatever another job last built or removed under the shared :latest tag on the same agent; the :latest alias is still applied for qa-jenkins-library's make.runTarget image resolution.
| @@ -0,0 +1,61 @@ | |||
| - projects: [RANCHERINT] | |||
| suite: Rancher Upgrade/Longhorn | |||
There was a problem hiding this comment.
Could we consider reusing the test cases from rancher-tests/validation/longhorn/ instead of creating a new suite?
Maintaining duplicate test suites can create technical debt over time. Running our primary test suite against pre-upgraded Rancher environments would allow us to validate upgrades using our existing coverage, keeping maintenance light and consistent
There was a problem hiding this comment.
@khushboo-rancher The new package is deliberately not a parallel Longhorn suite; it carries no functional coverage of its own. Its three tests only encode the upgrade contract (seed -> survive external upgrade -> still upgradable), and every step delegates to the same shared actions the primary suite uses (actions/charts install/upgrade, actions/storage workload seed/verify), so nothing is duplicated at the action layer.
Reusing validation/longhorn directly doesn't fit that contract mechanically: its TearDownSuite runs session.Cleanup() and its resources use random names, i.e. it proves install-time functionality in one process and then removes everything. The upgrade flow needs the opposite: deterministic names, no teardown, and state that survives across separate go test processes straddling an out-of-band server upgrade; if the primary suite ran pre-upgrade, its cleanup would erase exactly the state the post-upgrade phase exists to assert survived.
That said, running the primary suite against the pre-upgraded env is a great complementary regression signal (broad pre-state coverage), and it could be added later as an additional pre-upgrade phase before the seed, with its own -run selection. I'd keep the thin contract suite as the always-on survival check either way, since it's ~150 lines calling shared actions.
There was a problem hiding this comment.
We should do the functional coverage. Can we skip the teardown in the existing test cases and reuse them?
There was a problem hiding this comment.
@khushboo-rancher Done in 3c70d58. The pipeline now runs the primary validation/longhorn suite as pre-upgrade functional coverage before the contract seed (LONGHORN_FUNCTIONAL_COVERAGE, default on), and the suite skips its own teardown when the dispatched config carries the rancherUpgradePipeline marker: its projects, PVCs, and workloads survive the server upgrade, so the post-upgrade phases assert against an environment carrying real Longhorn load. The default -run selector excludes TestVolumeEncryption, which needs cloud credentials the pipeline's default config does not carry (LONGHORN_FUNCTIONAL_TESTCASE overrides it when the config provides them). The deterministic contract suite stays the always-on survival oracle, since this suite's random names cannot be re-found cross-process after the upgrade.
There was a problem hiding this comment.
Sounds good. I'm not sure, why a cloud credential is needed for the volumeencryption test. It should be a key/token. For now, it is ok to skip the encryption test but need to revisit it later.
There was a problem hiding this comment.
@khushboo-rancher The encryption itself is key-based: the test creates a secret with CRYPTO_KEY_VALUE/CRYPTO_KEY_PROVIDER=secret (Longhorn's Kubernetes-secret crypto provider, per the Longhorn docs example referenced in the test). The cloud credential is only consumed by the encrypted-backup verification leg: it creates a real S3 bucket, takes a Longhorn backup into it, downloads the backup object and asserts the plaintext marker is absent from the encrypted bytes, then restores. That leg needs AWS access keys the pipeline's default config does not carry, hence the default exclusion; the Jenkinsfile and README now spell this out (15d8f21). Revisiting the credential shape (e.g. splitting the S3 leg behind its own config key or carrying S3 creds in the pipeline config) is queued as follow-up.
| // ── Phase 4: Longhorn chart upgrade through the upgraded catalog ── | ||
| // Completes the shared Qase run after this final phase; do not rely on | ||
| // the downstream job's own QASE_TEST_RUN_COMPLETE for the shared run. | ||
| stage('Longhorn Chart Upgrade Tests') { |
There was a problem hiding this comment.
Do we want specific stage per integration point? I would suggest having a general stage like post check. Imagine having tests for 10 or more modules, we can't have those many stages rather than they should be steps.
There was a problem hiding this comment.
@khushboo-rancher Agreed on the end state. With one interop module today, the three-phase stages are load-bearing rather than decorative: each has distinct preconditions (post-upgrade waits for the server version, chart-upgrade then mutates the chart), and per-phase Jenkins stage status is how the green-run acceptance for #968 is read: stage-level pass/fail and timing per phase, plus the per-phase -run regex in the log.
When the later modules (NeuVector, Observability, ...) land, the right shape is exactly what you describe: one general post-upgrade stage looping module -> regex as steps, since each dispatch is already a single build job call; folding them into a loop is mechanical. Happy to restructure then or now, if you'd rather land the generic shape before module 2 arrives, since it also keeps the stage list constant as modules are added.
There was a problem hiding this comment.
I would prefer structuring it right way now itself.
There was a problem hiding this comment.
@khushboo-rancher Done in 3c70d58: the pipeline is now three stages (Setup, Deploy Rancher Pre-Upgrade Environment, Rancher Upgrade Validation). All four test dispatches are steps in the validation stage and share one dispatchUpgradePhase helper, so the stage list stays constant as modules are added; the module-selector PR slots its per-module regexes into the same helper.
lscalabrini01
left a comment
lscalabrini01
left a comment
There was a problem hiding this comment.
Looks good, just a comment
| latest, err := catalogClient.GetLatestChartVersion(charts.LonghornChartName, catalog.RancherChartRepo) | ||
| require.NoError(s.T(), err) | ||
| logrus.Infof("chart-upgrade phase: upgrading longhorn to %s from the upgraded catalog", latest) | ||
|
|
||
| installOptions := &charts.InstallOptions{ | ||
| Cluster: s.cluster, | ||
| Version: latest, | ||
| } | ||
| err = charts.RetryOnWatchError(charts.DefaultWatchRetries, func() error { | ||
| return charts.UpgradeLonghornChart(s.client, installOptions) | ||
| }) | ||
| require.NoError(s.T(), err) | ||
|
|
||
| chart, err := shepherdCharts.GetChartStatus(s.client, s.cluster.ID, charts.LonghornNamespace, charts.LonghornChartName) | ||
| require.NoError(s.T(), err) | ||
| require.Equal(s.T(), latest, chart.ChartDetails.Spec.Chart.Metadata.Version) |
There was a problem hiding this comment.
just confirming; the test can pass without actually upgrading here. is that intended? i.e. if the cluster this is ran against is already on latest.
There was a problem hiding this comment.
@slickwarren Yes, intended at the suite level: this package encodes the upgrade contract (seed -> survive external upgrade -> still upgradable) rather than performing the Rancher upgrade itself. The pipeline's make rancher-upgrade stage performs it between phases, and the post phases only assert outcomes (server at TargetVersion, chart still installed, checksum intact). Against a cluster already at the target version, the assertions are vacuously true, but the pipeline can't reach that state because it deploys RANCHER_VERSION_FROM first; that state would only occur when running the suite standalone, which isn't a supported entry point.
For line 148 specifically: the pre phase installs catalog-latest under the pre-upgrade Rancher, the chart phase re-resolves catalog-latest under the upgraded Rancher, so a true version transition happens whenever the two catalogs disagree (the expected case, e.g., 2.14 -> 2.15 catalog). When they serve the same Longhorn version, this phase degrades to an idempotent same-version upgrade assertion, and the real signal is the checksum survival. If you want that case made explicit, I can assert pre-installed != post-upgrade latest at the top of the chart phase.
| post { | ||
| failure { | ||
| script { | ||
| if (params.DESTROY_ON_FAILURE) { | ||
| destroyInfrastructure('DESTROY_ON_FAILURE') | ||
| } | ||
| } | ||
| } | ||
|
|
||
| always { | ||
| script { | ||
| // Failure diagnostics run here, not in the failure block: | ||
| // Jenkins evaluates post conditions in a fixed order with | ||
| // always first, so anything in failure{} would already have | ||
| // lost the docker images (rmi below) and the live cluster | ||
| // (destroy below). Collect while both still exist. | ||
| if (currentBuild.currentResult != 'SUCCESS') { | ||
| collectUpgradeDiagnostics() | ||
| } | ||
|
|
||
| // Complete the shared Qase run when a phase failed before the | ||
| // final stage could close it (no-op once already completed). | ||
| if (env.QASE_RUN_COMPLETED != 'true' && fileExists('testrun') && fileExists('testrunconfig.yaml')) { | ||
| catchError(buildResult: 'SUCCESS', stageResult: 'SUCCESS') { | ||
| property.useWithProperties(['QASE_AUTOMATION_TOKEN']) { | ||
| sh './testrun --startRun=false' | ||
| } | ||
| env.QASE_RUN_COMPLETED = 'true' | ||
| } | ||
| } | ||
|
|
||
| if (params.DESTROY_AFTER_TESTS) { | ||
| destroyInfrastructure('DESTROY_AFTER_TESTS') | ||
| } | ||
|
|
||
| // Docker cleanup | ||
| sh "docker rmi -f ${GO_TEST_IMAGE} ${INFRA_TOOLS_IMAGE} rancher-go-test:latest rancher-infra-tools:latest || true" | ||
|
|
||
| // Sensitive artifact cleanup — shred or remove credentials from workspace | ||
| sh ''' | ||
| set +e | ||
|
|
||
| cleanup_file() { | ||
| if [ -f "$1" ]; then | ||
| if command -v shred >/dev/null 2>&1; then | ||
| shred -u "$1" || rm -f "$1" | ||
| else | ||
| rm -f "$1" | ||
| fi | ||
| fi | ||
| } | ||
|
|
||
| cleanup_dir() { | ||
| if [ -d "$1" ]; then | ||
| find "$1" -type f -exec sh -c ' | ||
| for file do | ||
| if command -v shred >/dev/null 2>&1; then | ||
| shred -u "$file" || rm -f "$file" | ||
| else | ||
| rm -f "$file" | ||
| fi | ||
| done | ||
| ' sh {} + | ||
| rm -rf "$1" | ||
| fi | ||
| } | ||
|
|
||
| cleanup_file ".go_test_env" | ||
| cleanup_file "rancher-token.json" | ||
| cleanup_file "rancher-token-metadata.json" | ||
| cleanup_file "workspace_name.txt" |
There was a problem hiding this comment.
Jenkins runs the always block before the failure block. By the time failure runs, always has already deleted workspace_name.txt, the tfvars and the infra-tools image, which teardown needs. This causes two problems:
- DESTROY_AFTER_TESTS=false, DESTROY_ON_FAILURE=true: when a run fails, teardown can't find the workspace and skips. The AWS environment is left running.
- DESTROY_ON_FAILURE=false: when a run fails, the environment is still destroyed, because always tears it down whenever DESTROY_AFTER_TESTS is true. So you can't keep a failed environment around to debug it.
With the defaults (both flags true), everything works.
There was a problem hiding this comment.
@slickwarren You're right about the ordering: Jenkins evaluates post conditions in a fixed order (always before failure, cleanup last), so the failure{} block was dead; as written, it always had already shredded workspace_name.txt and the tfvars and removed the infra-tools image, so with DESTROY_AFTER_TESTS=false + DESTROY_ON_FAILURE=true, a failed run skipped teardown entirely and leaked the AWS environment.
Addressed in b14003d: the destroy decision now lives entirely in always (after diagnostics and Qase completion), gated on DESTROY_AFTER_TESTS || (failure && DESTROY_ON_FAILURE); the docker rmi and sensitive-artifact shredding moved into a cleanup block, which runs last, so teardown still has the files and image it needs. DESTROY_ON_FAILURE=false on a failed run now actually keeps the environment around for debugging.
Jenkins evaluates post conditions in a fixed order (always before
failure, cleanup last), so the failure{} destroy block could never
work: always had already shredded workspace_name.txt and tfvars and
removed the infra-tools image teardown depends on, and with
DESTROY_AFTER_TESTS=false + DESTROY_ON_FAILURE=true a failed run
skipped teardown entirely, leaking the AWS environment.
Fold the destroy decision into always (after diagnostics and Qase
completion), gated on DESTROY_AFTER_TESTS or failure+DESTROY_ON_
FAILURE; move the docker rmi and sensitive-artifact shredding into a
cleanup block so they run last.
| // rancher-infra-tools — used by make.groovy / infra.* for all make | ||
| // targets (library resolves the :latest alias); direct invocations | ||
| // in this file use the build-pinned tag. | ||
| sh "docker build --no-cache --platform linux/amd64 -t ${INFRA_TOOLS_IMAGE} -t rancher-infra-tools:latest -f ${env.TESTS_DIR}/validation/pipeline/Dockerfile.infra ." |
There was a problem hiding this comment.
Can we combine few stages together. The official Jenkins Pipeline Syntax Guide recommends using stage blocks for high-level phases of the delivery process (like Build, Test, and Deploy) and using steps within those stages for individual tasks.
Please consider this comment for all the stages.
There was a problem hiding this comment.
@khushboo-rancher Done in 3c70d58, applied to all stages: Checkout + Build Docker Images + Setup Tools + Detect Public IP + Resolve Rancher Versions are now one Setup stage; the nine nested stage() calls inside the deploy phase became plain steps with echo markers; and the four test stages merged into the single Rancher Upgrade Validation stage with per-phase dispatch steps. The timeout budget moved 300 -> 360 minutes to cover the added functional-coverage dispatch.
… pre-upgrade - validation/Jenkinsfile.rancher-upgrade.e2e: 11 top-level stages -> 3 high-level phases (Setup, Deploy, Upgrade Validation) per the Jenkins Pipeline Syntax Guide; per-phase work (deploy tasks, test dispatches) becomes steps; the four test dispatches share one dispatchUpgradePhase helper so the stage list stays constant as modules are added - new pre-upgrade functional-coverage step reuses the primary validation/longhorn suite (LONGHORN_FUNCTIONAL_COVERAGE, default on) with a -run selector that excludes TestVolumeEncryption (needs cloud credentials the default config lacks); timeout budget 300 -> 360 minutes - validation/longhorn: TearDownSuite skips session.Cleanup when the config carries the rancherUpgradePipeline marker, so the suite's resources survive the server upgrade; normal runs still clean up - actions/clusters: fold the version-equality check into the poll return - actions/storage: fmt.Sprintf instead of string concatenation
The .golangci.yaml lens builds with tags validation,2.13, which the suite files exclude via !2.13; the unexported helpers moved into rancherupgrade.go read as unused in that view. Carrying the suite tag keeps the package's files in one build unit.
khushboo-rancher
left a comment
khushboo-rancher
left a comment
There was a problem hiding this comment.
Thanks for consolidating the stages! Does below stages make sense, this will give a better understanding what is actually failing.
- Setup
- Deploy Rancher
- Pre upgrade tasks
- Upgrade Rancher
- Post upgrade tasks
- Post always destroy the environment
| @@ -0,0 +1,221 @@ | |||
| package storage | |||
There was a problem hiding this comment.
Are we using this file actions/storage/checksum.go anymore as we are reusing the existing tests from longhorn_test.go?
There was a problem hiding this comment.
@khushboo-rancher Yes, it is still load-bearing: the upgrade-contract tests use it. TestLonghornPreUpgrade seeds with storage.CreateChecksumWorkload (validation/rancherupgrade/longhorn_test.go:91) and TestLonghornPostUpgrade/TestLonghornChartUpgrade verify with storage.VerifyChecksumWorkload; those assertions are what prove data survives the server upgrade and the chart upgrade. The functional-coverage dispatch reuses the primary suite in addition, not instead.
| const rancherUpgradePipelineConfigKey = "rancherUpgradePipeline" | ||
|
|
||
| // upgradePipelineConfig mirrors validation/rancherupgrade's PipelineInput marker | ||
| // section; declared locally so this suite does not depend on another validation | ||
| // package for a single bool. | ||
| type upgradePipelineConfig struct { | ||
| Enabled bool `json:"enabled" yaml:"enabled"` | ||
| } | ||
|
|
||
| // runningInUpgradePipeline reports whether the loaded test config carries the | ||
| // rancherUpgradePipeline marker section. | ||
| func runningInUpgradePipeline() bool { | ||
| pipeline := new(upgradePipelineConfig) | ||
| operations.LoadObjectFromMap(rancherUpgradePipelineConfigKey, config.LoadConfigFromFile(os.Getenv(config.ConfigEnvironmentKey)), pipeline) | ||
|
|
||
| return pipeline.Enabled | ||
| } | ||
|
|
||
| func (l *LonghornTestSuite) TearDownSuite() { | ||
| if runningInUpgradePipeline() { | ||
| // The rancher-upgrade pipeline runs this suite as pre-upgrade functional | ||
| // coverage: its workloads must survive the in-place server upgrade so the | ||
| // post-upgrade phases assert against an environment carrying real load. | ||
| // The pipeline environment teardown is the cleanup. | ||
| l.T().Log("rancher-upgrade pipeline detected: skipping teardown so the suite's resources survive the server upgrade") | ||
| return | ||
| } |
There was a problem hiding this comment.
Can this be handled by simple bool like skipTearDown which is by default No and set to yes when running via pipeline? It could be placed in a constant.go file too where we can handle all the constants in future. This logic looks complicated and not scalable and increases maintenance in future.
There was a problem hiding this comment.
@khushboo-rancher Done in 15d8f21: the suite now reads a plain skipTearDown bool from the config (unset/false = full teardown, the default), defined in a new validation/longhorn/constants.go that also took over the suite's other package constants. The mirror struct and marker fingerprint helper are gone, and the pipeline config sets skipTearDown: true. The rancherUpgradePipeline marker remains only where the upgrade suite needs it (the pipeline-bug guard).
| @@ -0,0 +1,164 @@ | |||
| //go:build (validation || pit.weekly) && !2.8 && !2.9 && !2.10 && !2.11 && !2.12 && !2.13 | |||
There was a problem hiding this comment.
Do we want to rename this file to longhorn_upgrade_test.go and move to longhorn directory? If needed we should be able to run these upgrade tests (at least Longhorn chart upgrade) on a daily pipeline too.
There was a problem hiding this comment.
@khushboo-rancher Kept it in validation/rancherupgrade for now: the sibling module suites in #1024/#1025/#1026 follow the same <module>_test.go naming in this package (neuvector_test.go, fleet_test.go, observability_test.go), so renaming or moving only this file would diverge at merge. It also shares the cross-phase contract in rancherupgrade.go (UpgradeInput, loadUpgradeInput, the checksum constants) and its own build tags; moving it under validation/longhorn would duplicate those or point the primary suite's package at upgrade plumbing. For the daily goal, #1027's TEST_MODULE selector can already dispatch ./validation/rancherupgrade with a Longhorn regex on a daily leg (the chart-upgrade phase is self-contained against a fresh environment). Happy to rename all four module files together if you prefer that naming.
| * 1. Setup: checks out tests + qa-infra-automation, builds the pinned | ||
| * images, installs yq, detects the runner IP, and resolves the upgrade | ||
| * window | ||
| * 2. Deploy Rancher Pre-Upgrade Environment: deploys a 2.14-latest |
There was a problem hiding this comment.
Deploys (n-1) Rancher, right?
There was a problem hiding this comment.
@khushboo-rancher Yes: stage 2 deploys the previous release line (n-1). With RANCHER_VERSION_FROM empty, the pipeline resolves the latest 2.14.x final at runtime (SOURCE_LINE=2.14 via make rancher-resolve-version); a pin overrides. The header comment now says "previous release line (n-1, currently 2.14-latest)" (15d8f21).
| defaultValue: DEFAULT_RANCHER_PASSWORD, | ||
| description: 'Rancher bootstrap password for initial setup' | ||
| ) | ||
| password( |
There was a problem hiding this comment.
Do we need this configurable in Jenkins parameter?
There was a problem hiding this comment.
@khushboo-rancher Kept it parameterized: Jenkinsfile.e2e exposes the same pair (RANCHER_BOOTSTRAP_PASSWORD + RANCHER_ADMIN_PASSWORD, both defaulting to DEFAULT_RANCHER_PASSWORD) and both feed the deploy envVars through the qa-jenkins-library rancher target, so keeping the pair identical keeps the two pipelines' runbooks and rebuild flows interchangeable. Happy to fold them into constants if you prefer a minimal surface.
| string( | ||
| name: 'S3_BUCKET_NAME', | ||
| defaultValue: 'jenkins-terraform-state-storage', | ||
| description: 'S3 bucket name where Terraform state is stored' | ||
| ) | ||
| string( | ||
| name: 'S3_BUCKET_REGION', | ||
| defaultValue: 'us-east-2', | ||
| description: 'AWS region where the S3 bucket is located' | ||
| ) | ||
| string( | ||
| name: 'S3_KEY_PREFIX', | ||
| defaultValue: 'terraform.tfstate', | ||
| description: 'S3 key prefix for the Terraform state files' | ||
| ) |
There was a problem hiding this comment.
Same as above, are we adding value in having these configurable in pipeline?
There was a problem hiding this comment.
@khushboo-rancher Same rationale: S3_BUCKET_NAME/S3_BUCKET_REGION/S3_KEY_PREFIX mirror Jenkinsfile.e2e (lines 191-205), Jenkinsfile.multibranch.e2e (107-121) and the airgap pipelines, and are consumed verbatim as the terraform backend makeArgs (BUCKET/KEY/REGION), letting a run isolate its state in another bucket without a code change. Happy to hardcode them here if you would rather trim the surface.
|
Pushed 21309f0: stabilize |
|
Root cause of the prime Fixed in 85656a7: the test's ServiceMonitor endpoint now sets |
Longhorn 1.12's manager gathers volume metrics live per /metrics request, so the endpoint takes 30s or more and every Prometheus scrape times out even with a 60s scrapeTimeout (live-verified across three fresh pipeline environments, runs 56-59). Its series never appear, so the monitoring assertions cannot pass in this pipeline until the exporter latency is fixed upstream. Re-enable via LONGHORN_FUNCTIONAL_TESTCASE.
|
Follow-up to the scrape-timeout fix (85656a7): run 59 (pitupgradelh30) still failed with zero |
Summary
Implements #967 (parent PRD #946): a new
validation/rancherupgradepackage validating the Rancher server upgrade path (2.14.x -> latest) with interoperability products installed. Longhorn is the first vertical.The suite runs as three pipeline stages, each a separate
go testinvocation selected via-run; all cross-phase state lives in-cluster under deterministic names:TestLonghornPreUpgradeTestLonghornPostUpgradeserver-versionreachingrancherUpgradeInput.targetVersion, assert Longhorn chart survived the server upgrade, verify checksum dataTestLonghornChartUpgradeSupporting actions:
actions/clusters.WaitRancherVersion: polls the managementserver-versionsetting until it semver-equals the target; transient read errors during the upgrade rollout are retried.actions/storage.CreateChecksumWorkload/VerifyChecksumWorkload: deterministically named PVC/deployment pair whose data file must survive across processes.actions/charts.UpgradeLonghornChart: CRD-first chart upgrade preservingdeletingConfirmationFlag, OpenAPI validation disabled for the large CRDs, double watch to Deployed, then deployment + daemonset readiness gates.Deliberate deviations (do not "fix")
TearDownSuitedoes not callsession.Cleanup()and the helpers register no cleanup funcs: pre-phase resources must survive into later phases; the pipeline environment teardown is the cleanup.GetChartStatus/install action validation mismatch). Documented in the package README.Config
Verification
go vet -tags validationandgolangci-lint run --build-tags validationclean on the touched packages (the only lint findings inactions/are pre-existing in untouched files).-run "TestLonghornUpgradeTestSuite/<Phase>"selects exactly its phase method.109.3.1+up1.11.2installed, checksum seeded (independent verify: PVC Bound,sha256sum /auto-mnt/checksum-data=b12ea220...f34).make rancher-upgrade(Slice 1, feat(rancher): make rancher-upgrade target with runtime version resolution qa-infra-automation#243).109.3.1+up1.11.2 -> 110.1.0+up1.12.1, version assertion + checksum green.Pipeline notes (Slice 3)
make rancher-upgraderotates the admin token and writes it toansible/rancher/<env>-ha/generated.tfvars; the pipeline must refreshrancher.adminTokenbetween phases (pre-upgrade token is revoked).cattle-cluster-agentre-syncs; if the chart phase reports "failed to find chart longhorn from the chart repo", restart the cluster-agent and retry (~4 min observed).