fix(k8s): gate deploy health on a serving CNPG cluster instead of a fully-ready one - #2985
Draft
devantler wants to merge 3 commits into
Draft
fix(k8s): gate deploy health on a serving CNPG cluster instead of a fully-ready one#2985devantler wants to merge 3 commits into
devantler wants to merge 3 commits into
Conversation
…ully-ready one kstatus marks a CNPG Cluster NotReady for the whole of any rolling update, and CNPG rolls every instance whenever the operator, the barman-cloud plugin sidecar, or the PostgreSQL image changes. Those are routine Renovate bumps: plugin v0.13.0 -> v0.14.0 (#2971) restarted every instance of all four clusters on 2026-08-05. A database that is merely rolling, or serving on 2 of 3 instances, therefore failed the apps/infrastructure health check, timed out the deploy, and evicted every PR from the merge queue - wedding-db 2026-06-19, coroot-db 2026-07-14, umami-db twice on 2026-08-05. Gate on 'the database is serving' instead: a primary is elected and at least one instance is ready. A cluster with no ready instance matches no current expression, stays InProgress, and still fails the deploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
✅
|
The three resources this PR edits are individually pinned in the EKS CI authorization contract, so adding `healthCheckExprs` to them moved their fingerprints and the aggregate surface hash, failing `Validate EKS Authorization` and its `CI - Required Checks` rollup. Re-approved after running the measurement the constant's own note requires, against main 72fe791: 520 rendered documents on both sides with membership identical (zero added, removed or renamed by apiVersion|kind|namespace|name set difference), and exactly three entries moved -- precisely the three re-approved here. The complete rendered delta is 19 lines, all ADDED and all part of a `healthCheckExprs` entry; none removed or modified. All 67 grant-bearing documents (Role/ClusterRole/RoleBinding/ClusterRoleBinding/ ServiceAccount) are byte-identical, as are all 116 `aws`-bearing lines, so nothing granted to the aws/aws service account is touched. Gate verified still enforcing: widening one verb on the pinned `kro-tenant-rgd` ClusterRole makes the validator report that ClusterRole and the aggregate surface as unapproved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
Validation (run locally — GitHub Actions is in a major outage)GitHub Actions has been in a RED → GREEN. Before: The 35 Membership measurement (the discipline the constant's own note requires), branch vs
Gate proven still enforcing (ablation). Widening one verb on the pinned
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A single database losing one replica has now evicted every PR from the merge queue four times:
wedding-db 2026-06-19, coroot-db 2026-07-14, and umami-db twice on 2026-08-05. Each time the
database was serving perfectly well — primary healthy, backups green — and each time the merge lane
was blocked for hours.
The reason it keeps recurring is that the trigger is routine, not exceptional. CloudNativePG
restarts every instance whenever the operator, the barman-cloud sidecar, or the PostgreSQL image
changes, and Renovate ships those bumps continuously. #2971 (barman-cloud plugin v0.13.0 → v0.14.0)
merged earlier today and rolled every instance of all four clusters; three sat at 2 of 3 ready for
about eight minutes. Any
merge_grouprun during that window fails its deploy and its PR is evicted.Every mitigation so far aimed at preventing a replica from degrading (synchronous replication,
force/prunedisabled). Those are worth having, but they cannot fix this: umami-db already carriedthe synchronous-replication mitigation and was stranded anyway, and no mitigation stops a rolling
update from happening at all. The gate itself is what needs to change.
What
Health-check a CNPG
Clusteron "the database is serving" — a primary is elected and at least oneinstance is ready — instead of demanding every instance be ready. Applied to the three Kustomizations
that own a
Cluster:apps(umami-db, backstage-db),infrastructure(coroot-db), and the tenantRGD's two Kustomization templates, which covers wedding-db and every future tenant.
A cluster with no ready instance, or no elected primary, matches no
currentexpression, staysInProgress, and still fails the deploy — so a genuinely broken database is still caught.This deliberately moves "one replica is down" from blocking to paging, which is direction #1 on
#2639. The paging half is #2979, whose
cnpg-degraded-alertCronJob is by its own description"deliberately independent of any Flux health gate, so relaxing that gate cannot silently remove this
coverage" — the two are designed to compose. Worth landing #2979 alongside or before this, so the
alerting path is live when the blocking path stops being the de-facto detector.
Verification
ksail workload validateandksail --config ksail.prod.yaml workload validate— both clean, 554files each; all three changed files explicitly validated rather than skipped.
The CEL expression was compiled and evaluated against every cluster state observed on prod
today, alongside a control expression reproducing current kstatus behaviour. The two diverge in
exactly the degraded cases — confirming the change is not a no-op — and agree on the failure cases:
Field references checked against all four live clusters:
status.readyInstancesandstatus.currentPrimaryare present on every one.One honest edge: if
statusis absent entirely the expression errors rather than returningfalse. Flux's own documented examples don't guard against this either, and the docs frame the
consequence as waiting until timeout — the same outcome kstatus already gives an uninitialised
resource. The realistic fresh-apply state (
status: {}) returns false cleanly.Fixes #2639
Important
Merge-order gate: #2985 and #3004 both re-approve the same
expectedRenderedSurfaceSHAfrom the same base value. Whichever merges second will conflict on that line and must re-measure and re-approve against the newmain— the second PR's hash is not valid once the first lands.