Repository navigation
Add SUSE Observability pre/post/chart-upgrade validation to rancherupgrade package - #1026
Open
floatingman wants to merge 53 commits into
Open
floatingman wants to merge 53 commits into
floatingman wants to merge 53 commits into
Conversation
Implements the Longhorn vertical of the Rancher server upgrade validation (rancher#967, parent PRD rancher#946). New validation/rancherupgrade package with a LonghornUpgradeTestSuite whose phase methods run as separate go test invocations selected via -run, one per pipeline stage: - TestLonghornPreUpgrade: create project, install the Longhorn chart at the cluster-scoped catalog latest, create a checksum workload (PVC + nginx deployment with deterministic names) on a Longhorn volume. - TestLonghornPostUpgrade: gate on the management server-version setting reaching rancherUpgradeInput.targetVersion, assert the Longhorn chart survived the server upgrade, verify the checksum workload data. - TestLonghornChartUpgrade: gate on server version, resolve latest from the cluster-scoped catalog, upgrade longhorn + longhorn-crd, assert the installed chart version, verify the checksum workload again. Supporting actions: - actions/clusters: WaitRancherVersion polls the server-version setting until it semver-equals the target, retrying transient read errors. - actions/storage: CreateChecksumWorkload/VerifyChecksumWorkload build and verify a deterministically named PVC/deployment pair whose sha256-verified data file must survive across processes; no session cleanup is registered by design. - actions/charts: UpgradeLonghornChart upgrades longhorn and longhorn-crd with the deletingConfirmationFlag value preserved and OpenAPI validation disabled, waits for Deployed, then gates on deployments and daemonsets. TearDownSuite deliberately skips session.Cleanup and the suite is admin-only: upgrade-subject resources must survive into later phases and the pipeline environment teardown is the cleanup. Both deviations are documented in the package README, along with the token-rotation and catalog-follower-restart notes for the pipeline. Includes schemas/pit_schemas.yaml for the three phase methods. Verified live against a 2.14.3 HA env with a node-driver downstream RKE2 cluster: pre-upgrade PASS (Longhorn 109.3.1+up1.11.2, checksum seeded), server upgraded to 2.15.2 via qa-infra-automation, post-upgrade PASS, chart upgrade PASS (109.3.1+up1.11.2 -> 110.1.0+up1.12.1, checksum intact).
…estrator Dedicated pipeline for the Rancher server upgrade validation (issue rancher#968, PRD rancher#946), community leg: deploy 2.14-latest env, pre-upgrade Longhorn phase, in-place server upgrade via make rancher-upgrade with rotated-token re-injection, post-upgrade and chart-upgrade phases against one shared Qase run, destroy-always with failure-only diagnostics. Versions resolve at runtime via make rancher-resolve-version when the params are empty. Structurally cloned from validation/Jenkinsfile.e2e and the airgap Jenkinsfile call shapes; runs go-pit-validation-job per phase with -runs TestLonghornUpgradeTestSuite/<phase> selectors.
The per-phase executor job can land on a different Jenkins agent than the orchestrator (observed in the first rancher-upgrade run: default-n367q vs default-xj00w). The LB ingress on the tofu-created ephemeral SG only allows the orchestrator's detected public IP, so a phase on another agent could not reach the Rancher API and rancher.NewClient timed out after 22 minutes of retries. awsNodeTemplate carries the three keys the executor's Authorize Security Group stage reads (securityGroup, vpcId, region) so it authorizes its own runner IP on tf-<prefix>-sg before dispatching to the API, matching the shape Jenkinsfile.e2e ships for the same reason.
A declarative pipeline re-applies its parameters{} block to the job
definition after each run, which silently overwrote the JJB-configured
staging values (fork repos + slice branches) with the hard-coded main
defaults on the mower job: the first run deployed from the fork, the
next one fell back to upstream main and lost make
rancher-resolve-version ('No rule to make target'). Defaulting these
five params to their current value when set keeps the per-controller
JJB configuration authoritative instead of racing it.
The one-minute bind wait raced a freshly installed Longhorn: the first volume's provisioning (engine-image pull, instance-manager startup, CSI registration) can exceed a minute on a cold cluster, which failed the pre-upgrade phase with 'context deadline exceeded' in the mower run (the same step passed when the volume bound faster). Ten minutes matches the bound used for the rest of the storage waits.
Jenkins evaluates declarative post conditions in a fixed order with always before failure, so the failure-block diagnostics ran after the always block had already run infra-down and docker rmi: kubectl/helm had no rancher-infra-tools image left to run in (pull access denied, exit 125) and the cluster was already gone regardless. Diagnostics now run first inside always, gated on a non-SUCCESS result, while the images and the live cluster still exist; the failure block keeps only the DESTROY_ON_FAILURE fallback destroy.
The post-upgrade phase failed with an instant 401 from rancher.NewClient even though the rotated token had been extracted (length 66) and yq-injected into the dispatched config. Two fail-fast checks now pinpoint which side is broken: the injected token must get HTTP 200 from the upgraded Rancher API before any phase is dispatched, and the config about to be dispatched must byte-match the token in generated.tfvars. Both run under set +x so the token value stays out of the xtrace (the previous run echoed it into the build log).
The smoke-check curl died with exit 60 (TLS verification) against pitupgrade.qa.rancher.space before it ever reached auth — the same endpoint the tests already call with insecure: true. Pass -k so the check actually adjudicates the token.
token=$(awk ...) creates an unexported shell variable, but yq's strenv() reads the process environment, so the injection silently wrote an empty string into rancher.adminToken — the post-upgrade phase then authenticated with an empty bearer token (instant 401). curl in the same step still worked because its header uses shell expansion, which is why the smoke-check passed while the config check failed (expected length 66, got length 0). export the variable so the child yq process can see it.
- logrus milestone logs in each upgrade phase (version resolution, chart install/upgrade, workload seed, checksum verification) so the job log shows where each phase is without reading the assertions backwards. - rancher-infra-tools/rancher-go-test now carry a build-scoped tag (pitupgrade-<BUILD_NUMBER>) used by every direct docker invocation in this pipeline, so this job no longer depends on whatever another job last built or removed under the shared :latest tag on the same agent; the :latest alias is still applied for qa-jenkins-library's make.runTarget image resolution.
Jenkins evaluates post conditions in a fixed order (always before
failure, cleanup last), so the failure{} destroy block could never
work: always had already shredded workspace_name.txt and tfvars and
removed the infra-tools image teardown depends on, and with
DESTROY_AFTER_TESTS=false + DESTROY_ON_FAILURE=true a failed run
skipped teardown entirely, leaking the AWS environment.
Fold the destroy decision into always (after diagnostics and Qase
completion), gated on DESTROY_AFTER_TESTS or failure+DESTROY_ON_
FAILURE; move the docker rmi and sensitive-artifact shredding into a
cleanup block so they run last.
…ard, flavor plumbing - actions/storage: reject checksum content outside [A-Za-z0-9-] before sh -c interpolation - rancherupgrade: fail fast when resolved latest equals the installed chart version (stale cluster-scoped catalog follower) - Jenkinsfile.rancher-upgrade.e2e: document timeout budget, plumb RANCHER_CHART_REPO_FLAVOR into make args, resolve kubeconfig via INFRA_DIR, select deploy repo by flavor - README + pit_schemas: correct downstream-cluster wording to the Rancher local cluster - actions/clusters: debug logging on server-version read/parse failures
…ersion scope - pvc.go: createNginxDeploymentWithPVC takes an explicit deploymentName, so CreatePVCWorkload keeps random names while CreateChecksumWorkload passes its deterministic one; shared construction replaces ~25 duplicated lines - checksum.go: reuse the shared helper; drop now-unused charts/wloads imports - README: previous-release-line-to-latest wording (pipeline default, not a suite constraint) and a deliberate-deviation bullet for the out-of-band server upgrade
Pre-upgrade phase primes the suse-observability prime server-charts ClusterRepo, installs the server chart with base+ingress+150-ha sizing values into the System project on the local cluster, and installs the stackstate agent on the shared downstream cluster (rancher.clusterName) with cluster/apiKey/url values, verifying reporting per the existing validation/observability checks. Post-upgrade phase gates on WaitRancherVersion first, then asserts server chart survival, workload readiness, and agent reporting continuity. Chart-upgrade phase resolves latest server and agent versions from the cluster-scoped catalogs with stale-index guards, upgrades the server first (re-applying install-time values so the HA sizing profile and service credentials survive, per PRD rancher#946 Further Notes on values migration) and then the agent, re-verifying health and versions. Adds UpgradeStackStateServerChart (+ private action constructor) to interoperability/charts mirroring UpgradeStackstateAgentChart's two-phase app watch. Helpers live in observability.go behind the suite's build tag so the package stays clean under the default lint tags. Schemas appended for the three new methods.
lscalabrini01
previously approved these changes
Oct 22, 2026
The first pipeline exercise of CreateClusterRepo failed with a bare internal error four minutes into the pre-upgrade phase, about fifteen minutes after the management cluster deployed. Retry the create a bounded number of times with spacing, treat AlreadyExists as success since a failed POST can still persist server-side, register the cleanup only for a repo that exists, and wrap the terminal error with the repo name and URL for diagnosability.
Builds 26 and 27 failed at the version lookup three lines below the ClusterRepo create, not at the create itself: the catalog controller fetches and indexes the new repo's chart index asynchronously, and the first lookup lands seconds after creation. The pre-upgrade phase now polls for the server chart version up to five minutes before moving on; the post and chart phases keep single lookups since their repos are long indexed by then.
Build 28's version poll resolved cleanly, then the server chart install POST answered a bare internal error about thirty seconds in, consistent with the Rancher proxy timing out a large chart's first synchronous render. Wrap the server and agent install and upgrade action POSTs with bounded retries: the first attempt primes the chart's server-side cache, an attempt that persisted despite its error surfaces as an already-exists conflict and satisfies the action, and the terminal error carries the action name.
Build 29's install action failed three times at almost exactly thirty seconds per attempt: the clusterrepo install action resolves the chart version template synchronously, and for a fresh prime repo that means downloading the large suse-observability tarball inside one request window. Poll the template info link until the catalog has fetched and cached it, then install; the same pre-warm guards the chart-upgrade phase's freshly offered server and agent versions.
Build 30 disproved the cold-template theory: the pre-warm info link answered instantly and the install action still failed at almost exactly thirty seconds, three times. The cause is server-side and not visible in the bare internal error, so the pre-upgrade phase now captures the Rancher server pod log tail from the local cluster's cattle-system namespace before failing, as best-effort diagnostics.
Build 31's server log capture named the cause: the install operation died in 'failed to wait for roles to be populated'. Rancher's helm operation always tries to create the namespace and then watches for thirty seconds until the InitialRolesPopulated condition event fires; a pre-existing, healthy, quiet namespace emits no events, so the watch expired on every attempt. Stop pre-creating the server and agent namespaces and let the action create and project-annotate them itself, matching how the longhorn and neuvector suites install on this pipeline.
retryChartAction and CreateClusterRepo spaced retry attempts with PollUntilContextTimeout(..., immediate=true, condition->(true, nil)), but immediate=true runs the condition before the first tick, so the poll returned in microseconds and attempts fired back-to-back. immediate=false waits the interval before the first check, which is the intended spacing, verified with a timing probe (about 20us vs one full interval).
Build 32's capture covered one Rancher server replica and the install window logged nothing on it: the action can land on any of the three replicas, and the helm operation's real error lives in the catalog Operation resource the action creates. Capture every replica's log tail, the target namespace's annotations and conditions, and the Operation and App statuses before the install failure fails the test.
Build 33's all-replica diagnostics pinned the failure: every install attempt logged 'failed to wait for roles to be populated'. The install action derives the target namespace's field.cattle.io/projectId annotation from the action's ProjectID, and this suite passed the ID with its cluster prefix stripped, so the annotation resolved to no project, the namespace's project roles never populated, and the action's thirty-second wait failed before anything was created. Return the System project's full ID, matching how the actions package and the standalone observability tests address projects.
Chart 2.12.0 derives each split component's JVM flags from its memory limit as heap + direct = limit - sizing.baseMemoryConsumption, but the no-profile default sizes several components past their own limits (authorizationSync: 1047m of flags against 1024m) and build 38 caught the whole family crash-looping after storage was fixed. Raise the eight affected components' limits and pin their base consumption and heap fraction so the flags land at 66-89% of each limit. Whole Gi units only: the chart's memory parser renders empty JVM flags for decimal quantities like 2.5Gi.
Build 39's exit codes disproved the memory theory: every crasher exits 1 with an orderly JVM shutdown, and the Go otel-collector fails too, so the cause is a shared startup failure whose error line sits far above a 40-line tail. The capture now filters a 2000-line window down to error-marker lines with a short raw tail.
The 2.12.0 documentation installs through global.suseObservability (license, baseUrl, adminPassword, receiverApiKey, sizing.profile); the suite's legacy stackstate.* values never initialize the tenant subscription (build 39: every JVM component exits with SubscriptionInitializationProhibited even though the license key reaches the chart's secret) and their no-profile fallback sizing was the crash-loop family from build 38. The native 10-nonha profile renders a monolith server at 7 cores / 27 GiB total, which fits the three-node pipeline cluster, and replaces the hand-rolled sizing file, ingress values and per-component memory overrides.
With the server healthy (build 41 installed suse-observability 2.12.0
in global mode in seven minutes), the agent install failed because the
legacy stackstate-k8s-agent chart no longer exists in
rancher-partner-charts. The agent ships in the same prime server-charts
repo as the server, renamed suse-observability-agent, and still accepts
the stackstate.{url,apiKey,cluster.name} values the suite already
passes; point the chart name, install, upgrade and version lookups at
the suse-observability repo.
Build 42's agent install failed with the bare internal error whose cause only the Rancher server logs name; the diagnostics hook covered only the server install path. Also widen the Rancher log tail window: 300 lines of the server's watch-metadata chatter covers barely a minute.
The previous commit switched the agent chart name, payload annotation, upgrade call and version lookups but left the install action posting to rancher-partner-charts; the server resolved suse-observability-agent against that repo's index and answered helm's 'no chart name found' (build 42, one per replica per attempt).
Build 43 took the suite through pre-upgrade, the Rancher server upgrade and post-upgrade green; the chart-upgrade guard then fired by design because the prime repo's index is identical before and after the server upgrade, so installing the latest version pre-upgrade leaves the phase no delta. Seed suse-observability 2.11.2 and the agent one stable step below the latest so the chart-upgrade phase exercises a real 2.11.2 -> 2.12.0 upgrade; both previous versions render with the suite's global-mode values.
…ches Build 44 lost a healthy 2.11.2 install to a single dropped watch stream (error with watch connection, INTERNAL_ERROR) while the app sat pending-install and helm kept running. Unify the four deployed-state watches on watchAppUntilState, which re-establishes the watch via RetryOnWatchError without re-posting the chart action; StatusFailed still errors immediately. Also drop the two pending-upgrade watches: they assert nothing the deployed/failed watch does not, and a watch re-established after the seconds-long pending-upgrade window would stall until timeout waiting for a state that already passed.
Build 45 exposed a sequencing bug in the previous commit: during an upgrade the app starts already deployed, so a deployed-only watch returns immediately on the old state and the phase's version assert fails five seconds in. Wait for pending-upgrade first; that state spans the entire helm operation, so a re-established watch cannot miss it.
The evidence suite served its purpose across builds 24-46 (catalog index reassembly, container log capture, values dumps); the permanent in-band diagnostics carry the failure forensics now.
Contributor
Author
|
Full green pipeline run: go-pit-rancher-upgrade-job run 46 (hostname prefix pitupgradeob24), all four phases SUCCESS against staging/rancher-upgrade-observability head 012ebce + review-polish commits:
Key landings since the last update: pre-upgrade now seeds the previous stable chart version so the chart-upgrade phase exercises a real delta (the prime repo index is identical across the server upgrade); app watches retry on transient watch-stream drops without re-posting chart actions, with upgrade watches gated on pending-upgrade before the new deployed state; the agent install action posts to the suse-observability repo (renamed chart suse-observability-agent). The throwaway evidence harness is removed; in-band diagnostics remain. |
Contributor
Author
|
Heads-up for the standalone validation/observability suite (out of scope for this PR, discovered while getting the upgrade leg green against the current prime charts):
The upgrade suite in this PR installs with global-mode values and the 10-nonha profile, so it does not trip either. |
floatingman
requested review from
hamistao,
khushboo-rancher and
lscalabrini01
October 27, 2026 11:58
lscalabrini01
left a comment
lscalabrini01
left a comment
Contributor
There was a problem hiding this comment.
Looks good, just some comments
Build 49's all-module run lost the server chart upgrade to a 45-minute helm deadline with no in-band evidence (the diagnostics only ran on install failures). Capture the same helm operation, app, workload and rancher tails on both upgrade failure paths.
floatingman
force-pushed
the
feat/rancher-upgrade-observability
branch
from
October 27, 2026 22:24
edb8d6c to
48a3472
Compare
Volume-attach stalls and scheduling refusals appear only in events: the pod sits in ContainerCreating with no container-level evidence (build 52's zookeeper stall would have been named by FailedAttachVolume). Newest first, capped.
Contributor
Author
|
All-module pipeline runs green in both flavors (go-pit-rancher-upgrade-job, TEST_MODULE=all, staging branch carrying this suite):
Two findings from driving all-mode green, now on this branch: the observability chart upgrade must run before longhorn's (longhorn's engine live-upgrades stall a concurrent observability rollout's RWO volume reattach — zookeeper sat ContainerCreating for a full 45-minute helm deadline with kafka and the JVM components cascading on the unresolved headless service), and the upgrade-failure diagnostics now capture the helm operation, app/workload state, namespace events and rancher tails (the same evidence that named the stall). |
- Align pit_schemas.yaml descriptions with the implementation: previous-stable server seeding, renamed suse-observability-agent chart from the suse-observability repo, global-mode values with the 10-nonha sizing profile - Attach the InstallStackStateServerChart godoc to its function instead of watchAppUntilState - Guard previousStableChartVersion against an empty stable-version list - Correct the agent repo comment in observability_test.go
Consistent module-upgrade file naming across the rancherupgrade suites (review follow-up, khushboo-rancher); the suite is already named TestObservabilityUpgradeTestSuite.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #971 (parent PRD #946). Adds the SUSE Observability vertical to the
validation/rancherupgradepackage from #986, following the existing stackstate test layout rather than a second pattern.Supporting changes
interoperability/charts:UpgradeStackStateServerChart(+ privatenewStackStateServerChartUpgradeAction) mirroringUpgradeStackstateAgentChartconstruction exactly (PayloadOpts →NewChartUpgrade/NewChartUpgradeAction→ upgrade → pending watch → deployed watch). Per PRD Further Notes, the caller re-applies install-time values (HA sizing + service credentials) instead of a blind version bump; documented in godoc.validation/rancherupgrade/observability.go: package-private helpers (loadStackStateConfig,loadHASizingValues, values merge, namespace/project resolution, workload health waits). Sizing is loaded fromvalidation/observability/resources/150-ha_sizing_values.yaml, keeping a single source of truth with the observability package instead of forking the values.Suite
TestObservabilityPreUpgradehttps://charts.rancher.com/server-charts/prime/suse-observability), installs thesuse-observabilityserver into the System project on the local cluster (base + ingress + 150-ha sizing values), installsstackstate-k8s-agenton the shared downstream cluster (rancher.clusterName) with cluster/apiKey/url values fromstackstateConfigs, and verifies agent workloads report against the serverTestObservabilityPostUpgradeserver-version == targetVersionfirst; server chart survived and its workloads are healthy; agent chart still installed and pods healthy and reporting (monitoring continuity)TestObservabilityChartUpgradeVerification
go vet -tags validationacross the validation tree and the interoperability module; golangci-lint reports no findings on any touched file.-listshowsTestObservabilityUpgradeTestSuite;-run TestObservabilityUpgradeTestSuite/TestObservabilityPreUpgradeselects exactly one method; skip/fail guard verified (no config → SKIP; pipeline marker without target → FAIL).schemas/pit_schemas.yamlextended with theRancher Upgrade/Observabilitysuite (custom field 15 = method names) and parses clean.Pending
rancher.clusterNameto be the shared downstream cluster (server is pinned tolocal); degenerate co-location (clusterName: local) still works.