Repository navigation
Add NeuVector pre/post/chart-upgrade validation to rancherupgrade package - #1024
Open
floatingman wants to merge 19 commits into
Open
floatingman wants to merge 19 commits into
floatingman wants to merge 19 commits into
Conversation
Implements the Longhorn vertical of the Rancher server upgrade validation (rancher#967, parent PRD rancher#946). New validation/rancherupgrade package with a LonghornUpgradeTestSuite whose phase methods run as separate go test invocations selected via -run, one per pipeline stage: - TestLonghornPreUpgrade: create project, install the Longhorn chart at the cluster-scoped catalog latest, create a checksum workload (PVC + nginx deployment with deterministic names) on a Longhorn volume. - TestLonghornPostUpgrade: gate on the management server-version setting reaching rancherUpgradeInput.targetVersion, assert the Longhorn chart survived the server upgrade, verify the checksum workload data. - TestLonghornChartUpgrade: gate on server version, resolve latest from the cluster-scoped catalog, upgrade longhorn + longhorn-crd, assert the installed chart version, verify the checksum workload again. Supporting actions: - actions/clusters: WaitRancherVersion polls the server-version setting until it semver-equals the target, retrying transient read errors. - actions/storage: CreateChecksumWorkload/VerifyChecksumWorkload build and verify a deterministically named PVC/deployment pair whose sha256-verified data file must survive across processes; no session cleanup is registered by design. - actions/charts: UpgradeLonghornChart upgrades longhorn and longhorn-crd with the deletingConfirmationFlag value preserved and OpenAPI validation disabled, waits for Deployed, then gates on deployments and daemonsets. TearDownSuite deliberately skips session.Cleanup and the suite is admin-only: upgrade-subject resources must survive into later phases and the pipeline environment teardown is the cleanup. Both deviations are documented in the package README, along with the token-rotation and catalog-follower-restart notes for the pipeline. Includes schemas/pit_schemas.yaml for the three phase methods. Verified live against a 2.14.3 HA env with a node-driver downstream RKE2 cluster: pre-upgrade PASS (Longhorn 109.3.1+up1.11.2, checksum seeded), server upgraded to 2.15.2 via qa-infra-automation, post-upgrade PASS, chart upgrade PASS (109.3.1+up1.11.2 -> 110.1.0+up1.12.1, checksum intact).
…estrator Dedicated pipeline for the Rancher server upgrade validation (issue rancher#968, PRD rancher#946), community leg: deploy 2.14-latest env, pre-upgrade Longhorn phase, in-place server upgrade via make rancher-upgrade with rotated-token re-injection, post-upgrade and chart-upgrade phases against one shared Qase run, destroy-always with failure-only diagnostics. Versions resolve at runtime via make rancher-resolve-version when the params are empty. Structurally cloned from validation/Jenkinsfile.e2e and the airgap Jenkinsfile call shapes; runs go-pit-validation-job per phase with -runs TestLonghornUpgradeTestSuite/<phase> selectors.
The per-phase executor job can land on a different Jenkins agent than the orchestrator (observed in the first rancher-upgrade run: default-n367q vs default-xj00w). The LB ingress on the tofu-created ephemeral SG only allows the orchestrator's detected public IP, so a phase on another agent could not reach the Rancher API and rancher.NewClient timed out after 22 minutes of retries. awsNodeTemplate carries the three keys the executor's Authorize Security Group stage reads (securityGroup, vpcId, region) so it authorizes its own runner IP on tf-<prefix>-sg before dispatching to the API, matching the shape Jenkinsfile.e2e ships for the same reason.
A declarative pipeline re-applies its parameters{} block to the job
definition after each run, which silently overwrote the JJB-configured
staging values (fork repos + slice branches) with the hard-coded main
defaults on the mower job: the first run deployed from the fork, the
next one fell back to upstream main and lost make
rancher-resolve-version ('No rule to make target'). Defaulting these
five params to their current value when set keeps the per-controller
JJB configuration authoritative instead of racing it.
The one-minute bind wait raced a freshly installed Longhorn: the first volume's provisioning (engine-image pull, instance-manager startup, CSI registration) can exceed a minute on a cold cluster, which failed the pre-upgrade phase with 'context deadline exceeded' in the mower run (the same step passed when the volume bound faster). Ten minutes matches the bound used for the rest of the storage waits.
Jenkins evaluates declarative post conditions in a fixed order with always before failure, so the failure-block diagnostics ran after the always block had already run infra-down and docker rmi: kubectl/helm had no rancher-infra-tools image left to run in (pull access denied, exit 125) and the cluster was already gone regardless. Diagnostics now run first inside always, gated on a non-SUCCESS result, while the images and the live cluster still exist; the failure block keeps only the DESTROY_ON_FAILURE fallback destroy.
The post-upgrade phase failed with an instant 401 from rancher.NewClient even though the rotated token had been extracted (length 66) and yq-injected into the dispatched config. Two fail-fast checks now pinpoint which side is broken: the injected token must get HTTP 200 from the upgraded Rancher API before any phase is dispatched, and the config about to be dispatched must byte-match the token in generated.tfvars. Both run under set +x so the token value stays out of the xtrace (the previous run echoed it into the build log).
The smoke-check curl died with exit 60 (TLS verification) against pitupgrade.qa.rancher.space before it ever reached auth — the same endpoint the tests already call with insecure: true. Pass -k so the check actually adjudicates the token.
token=$(awk ...) creates an unexported shell variable, but yq's strenv() reads the process environment, so the injection silently wrote an empty string into rancher.adminToken — the post-upgrade phase then authenticated with an empty bearer token (instant 401). curl in the same step still worked because its header uses shell expansion, which is why the smoke-check passed while the config check failed (expected length 66, got length 0). export the variable so the child yq process can see it.
- logrus milestone logs in each upgrade phase (version resolution, chart install/upgrade, workload seed, checksum verification) so the job log shows where each phase is without reading the assertions backwards. - rancher-infra-tools/rancher-go-test now carry a build-scoped tag (pitupgrade-<BUILD_NUMBER>) used by every direct docker invocation in this pipeline, so this job no longer depends on whatever another job last built or removed under the shared :latest tag on the same agent; the :latest alias is still applied for qa-jenkins-library's make.runTarget image resolution.
Jenkins evaluates post conditions in a fixed order (always before
failure, cleanup last), so the failure{} destroy block could never
work: always had already shredded workspace_name.txt and tfvars and
removed the infra-tools image teardown depends on, and with
DESTROY_AFTER_TESTS=false + DESTROY_ON_FAILURE=true a failed run
skipped teardown entirely, leaking the AWS environment.
Fold the destroy decision into always (after diagnostics and Qase
completion), gated on DESTROY_AFTER_TESTS or failure+DESTROY_ON_
FAILURE; move the docker rmi and sensitive-artifact shredding into a
cleanup block so they run last.
…ard, flavor plumbing - actions/storage: reject checksum content outside [A-Za-z0-9-] before sh -c interpolation - rancherupgrade: fail fast when resolved latest equals the installed chart version (stale cluster-scoped catalog follower) - Jenkinsfile.rancher-upgrade.e2e: document timeout budget, plumb RANCHER_CHART_REPO_FLAVOR into make args, resolve kubeconfig via INFRA_DIR, select deploy repo by flavor - README + pit_schemas: correct downstream-cluster wording to the Rancher local cluster - actions/clusters: debug logging on server-version read/parse failures
…ersion scope - pvc.go: createNginxDeploymentWithPVC takes an explicit deploymentName, so CreatePVCWorkload keeps random names while CreateChecksumWorkload passes its deterministic one; shared construction replaces ~25 duplicated lines - checksum.go: reuse the shared helper; drop now-unused charts/wloads imports - README: previous-release-line-to-latest wording (pipeline default, not a suite constraint) and a deliberate-deviation bullet for the out-of-band server upgrade
Add the NeuVector vertical to the Rancher server upgrade validation: - actions/neuvector/admission.go: reusable, error-returning admission enforcement helpers. EnableAdmissionControlDenyNamespace applies the NvAdmissionControlSecurityRule CR (documented name "local") in protect mode with a namespace-criterion deny rule (containsAny op), creates the deny namespace and waits for the neuvector-validating-admission-webhook ValidatingWebhookConfiguration. VerifyAdmissionDenial attempts a known-bad pod create and proves the API server rejects it with an admission-webhook denial, tolerating controller rule reloads within a bounded window. CRD shape grounded in the neuvector-crd chart schema and the NeuVector console-exported manifest docs. - actions/charts: UpgradeNeuVectorChart upgrades neuvector + neuvector-crd with stale-catalog-safe action retry, PendingUpgrade->Deployed watches and workload readiness, carrying over the installed release values (helm --reuse-values semantics) so registry, hardening and k3s settings survive instead of a blind version bump. - validation/charts: export VerifyNeuVectorServicesActive and VerifyNeuVectorWebUIAccessible for cross-package reuse (move-not-copy; in-package callers updated). - validation/rancherupgrade: NeuVectorUpgradeTestSuite with the three -run-selectable phases (pre-upgrade install + denial proof, post-upgrade server-version gate + services/web UI/enforcement survival, chart upgrade to the cluster-scoped latest + health/enforcement re-verification), following the Longhorn suite contract. - schemas: PIT schema entries for the three new test methods.
The Steve client returns errors like "Bad response statusCode [404]. Status [404 Not Found]..." that k8sErrors.IsNotFound does not match, so on a fresh cluster the deny-namespace lookup, the admission CR lookup and the webhook poll all surfaced their 404s instead of taking the create/wait paths (seen live: TestNeuVectorPreUpgrade failed with namespaces "rancher-upgrade-neuvector-deny" not found). Route all four checks through isNotFound, which also matches the Steve error text.
floatingman
requested review from
hamistao,
khushboo-rancher and
lscalabrini01
and removed request for
a team
October 26, 2026 14:14
Consistent module-upgrade file naming across the rancherupgrade suites (review follow-up, khushboo-rancher); the file moves with its suite, which was already named TestNeuVectorUpgradeTestSuite.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #969 (parent PRD #946). Adds the NeuVector vertical to the
validation/rancherupgradepackage from #986, following the same three-phase contract: phases are separatego testinvocations selected via-runregex, cross-phase state lives in-cluster under deterministic names, and the suite intentionally skips withoutrancherUpgradeInput.targetVersion(failing instead when the pipeline marker is set but the target is missing).Suite
TestNeuVectorPreUpgradeneuvector+neuvector-crd, cluster-scoped catalog latest) incattle-neuvector-system, enables anNvAdmissionControlSecurityRulein protect mode denying workload creates in the deterministic namespacerancher-upgrade-neuvector-deny, and proves the validating webhook denies a known-bad createTestNeuVectorPostUpgradeserver-version == targetVersionfirst; chart survived; expected services active and manager web UI reachable through the Rancher service proxy; enforcement still deniesTestNeuVectorChartUpgradeneuvector+neuvector-crdto the cluster-scoped latest preserving the installed release values (PRD Further Notes on values migration); health and enforcement re-verifiedSupporting changes
actions/charts:UpgradeNeuVectorChartmodeled onUpgradeLonghornChart(retry on stale catalog after the server upgrade, watch to Deployed, workload readiness gates, values reuse).actions/neuvector(new package): reusable admission-enforcement helpers, grounded in the shippedneuvector-crdschema (NvAdmissionControlSecurityRule) and the NeuVector admission-controls documentation; the deny rule targets only the dedicated namespace, so system namespaces keep NeuVector's default allow rules.validation/charts:VerifyNeuVectorServicesActive/VerifyNeuVectorWebUIAccessibleexported (rename only, callers updated) for cross-package reuse, consistent with the existing chart-test verification helpers.Verification
go vet -tags validationacross the validation tree and the actions module; golangci-lint reports no findings on any touched file (pre-existing baseline findings unchanged).go test -tags validation -run NoMatchcompiles the package;-listshowsTestNeuVectorUpgradeTestSuite, and-run TestNeuVectorUpgradeTestSuite/TestNeuVectorPostUpgradeselects exactly one method.schemas/pit_schemas.yamlextended with theRancher Upgrade/NeuVectorsuite (custom field 15 = Go method names) and parses clean.Pending