Gated Cilium rollout has stepped zero nodes in 10 days, and its gate has suppressed every Talos config sync and all ksail reconciliation since

🤖 Generated by the Agentic Engineer

Evidence

The gated Cilium homogeneous-device rollout has not stepped a single node, and the gate it
installed has been suppressing unrelated work for over a week.

Read live from prod on 2026-08-05 (read-only):

daemonset/cilium in kube-system: updateStrategy=OnDelete, desired=9, ready=9,
updatedNumberScheduled absent (0), generation=21, observedGeneration=21.
All 9 of 9 agent pods carry controller-revision-hash of ControllerRevision 19
(created 2026-07-25T22:59:29Z). Every pod's start time is 2026-07-28 and every pod has
0 restarts.
The DaemonSet's newest ControllerRevision is 24 (created 2026-08-03T08:09:07Z); 20
(2026-07-28T18:19:31Z) and 23 (2026-08-01T13:44:46Z) also went by.

So since revision 20 was cut on 2026-07-28 the fleet has been 0/9 on the current spec, and it
stays that way by construction: OnDelete only replaces an agent when something deletes its pod,
and nothing does. New revisions accumulate; none is ever adopted.

Why it matters beyond Cilium

.github/actions/deploy-prod/action.yml gates the 🔄 Update cluster (Talos machine config) step
on the rollout gate:

if: >-
  steps.cilium_rollout_gate.outputs.active != 'true' &&
  steps.wait_flux_revision.outcome == 'success'

and scripts/guard-cilium-homogeneous-device-rollout.sh keeps active=true for as long as the
homogeneous-devices component is referenced and its kustomization still says type: OnDelete.
Both remain true, so that step is skipped on every prod deploy. Measured on the most recent prod
deploy (merge-group, 2026-08-04T23:41:46Z, job green):

##[end-action id=__self.cilium_rollout_gate;outcome=success;...]     Cilium ... gate active
##[end-action id=__self.cluster_update;outcome=skipped;conclusion=skipped;duration_ms=0]

That step is the pipeline's only Talos machine-config sync and its only ksail cluster update.
Consequences, all currently live:

Talos machine config has not been synced to prod since the gate went active (2026-07-26).
No ksail cluster reconciliation reaches prod at all — the component reconcilers for CNI,
CSI, load balancer, cert-manager, policy engine, autoscaler, registry credentials and the GitOps
sync resource are all reached only from that command.
This is the concrete reason Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922 / Root OCIRepository verification does not survive onto the live cluster #2938 cannot pass: ensureVerifyIfConfigured runs inside
SetupInstance, which on the update path is reached only from the cluster.workload.tag and
cluster.workload.flux.distributionVersion reconcilers — none of which execute while the command
itself never runs. The live root ocirepository/flux-system still has spec.verify absent.
Cluster Autoscaler is held suspended by the gate for the same duration.

#2951/#2962 made this suppression visible (a job-summary warning) and bounded the reporting. They
did not address the rollout being stalled, which is what keeps the gate active.

Expected behaviour

The gated rollout either completes or is rolled back, within the window it was designed for, and the
deploy pipeline does not silently lose its only machine-config sync while it waits.

Proposed direction

Step the 9 agents onto the current revision, one node at a time with verification between steps —
the procedure OnDelete was chosen to allow — then remove the gate by taking the component off
OnDelete. require_cilium_fleet_current already refuses removal until
observed==generation, scheduled==desired, updated==desired, pods==desired, ready==desired,
so gate removal stays fail-closed and needs no change.
Or, if the widened device set is no longer wanted, roll the component back and remove the gate.
Add a forcing function so an open-ended gate cannot recur: the gate should fail the deploy — not
only warn — once it has been active beyond its intended window, so a stall surfaces as breakage
instead of a green deploy that quietly skips its own config sync.

Acceptance criteria

daemonset/cilium reports updatedNumberScheduled == desiredNumberScheduled == numberReady,
verified by reading it back from the live cluster.
No agent pod remains on a superseded ControllerRevision.
A subsequent prod deploy shows cluster_update with outcome=success, not skipped.
Cluster Autoscaler is restored to its pre-gate replica count by the gate's own restore path.
A gate that stays active past its intended window fails a deploy rather than warning inside a
green one.

Size

Medium — the stepping itself is an operator action against prod networking (deliberately not
automated here); the forcing function is a small workflow change.

Blocks #2922 and #2938. Related: #2610, #2792 (activation), #2951, #2962.

🔗 Related PRs

#2633 - fix(deploy): verify incoming image on Talos nodes [merged]
#2793 - feat(cilium): activate homogeneous device selection [merged]
#2927 - fix(deploy): await pre-handoff kustomize-controller pod termination [merged]
#2930 - feat(ci): verify published evidence before promoting the prod digest [merged]
#2947 - fix(ci): refuse to promote a prod digest whose evidence cannot be verified [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Still active today — re-verified live, not inherited from an earlier note

Checked against current state rather than trusting the existing "gated" record, since a closed blocker is not a released one.

The suppression is unchanged. On the most recent production deploy (the merge_group CI run of 2026-08-04 23:41Z, job 🚀 Deploy to Prod, conclusion success), the step outcomes are:

end-action id=__self.cilium_rollout_gate;outcome=success
end-action id=__self.cluster_update;outcome=skipped

The job is green and its most important step did not run — so job conclusion is not a usable signal here; the per-step outcome is.

The rollout has still stepped nothing. The DaemonSet remains on OnDelete with its pods several revisions behind the current spec, so no deploy will ever advance it — only a deliberate operator step or a rollback will.

What has changed since this issue was filed — the risk argument has flipped

The reason for not simply enabling root artifact verification was that verification against an unsigned artifact would stop the root source reconciling. That is no longer the state. In the same deploy, the whole publication chain is green and in the right order:

push_staging → resolve_staging → cosign_sign → generate_sbom
→ attest_provenance → attest_sbom → verify_evidence → promote_latest

verify_evidence succeeds before promote_latest, which is exactly what #2859 and #2849 set out to guarantee. #2627's publication half is therefore done — 9 of its 10 children are closed, and the only open one is #2922.

So the standing blocker is no longer a supply-chain gap. It is this gate: cluster_update is the step that would apply spec.verify to the live root OCIRepository, and it has not run. Live prod still reads spec.verify: null.

Why this is not something to fix in the workflow

The gate is deliberate and correct — it exists because KSail owns Cluster Autoscaler and could reconcile it during the command, opening a scale-up race before the first canary is verified. Removing or weakening it to unblock the security control would trade one control for another, so I have not proposed that.

The unblock is the operator action this issue already names: step the rollout, or roll the component back. That is a judgement call on a live cluster and a mutation outside my authority, so it stays with the maintainer.

Net effect

One operator action now gates a security control that is otherwise ready to land. Blocking chain: this issue → #2922 → #2938, and #2627 closes behind them.

Detailed cluster-side evidence is kept in the private operator notes rather than here.

fix(ci): escalate the Cilium gate suppression from warning to failure

🤖 Generated by the Agentic Engineer

Re-measured on 2026-08-06 (read-only). The issue still holds, and three things have changed since it was written that make the remediation riskier than it was — worth recording before anyone releases the gate.

1. The stalled fleet is running a release candidate, not the declared GA build.

running pods (all 9):  quay.io/cilium/cilium:v1.20.0-rc.0@sha256:ab193b23...
DaemonSet template:    quay.io/cilium/cilium:v1.20.0@sha256:383968cd...
Git (helm-release.yaml): version: 1.20.0

Every agent pod still has a 2026-07-28 start time. So the practical consequence of "0/9 on the current spec" is concrete: production has been serving on v1.20.0-rc.0 while Git has declared GA v1.20.0 since 2026-08-05T17:32 (#2981) — a bump that merged after this issue was filed. generation/observedGeneration are now 24/24 (they were 21/21 here).

2. The HelmRelease reports success, which makes this invisible from the Flux surface.

helmrelease/cilium: lastAttemptedRevision=1.20.0  lastApplied=1.20.0  Ready=UpgradeSucceeded

Helm genuinely did succeed — it rewrote the DaemonSet template. OnDelete then declined to act on it. So the one surface an operator would check to answer "did the CNI upgrade land?" says yes while no node moved. Same shape as the green deploy whose cluster_update is skipped duration_ms=0: the status reports the step that ran, not the outcome that didn't.

3. This raises the cost of the obvious fix. Releasing the gate means dropping type: OnDelete from cilium/components/homogeneous-devices/, which is what the guard keys on. Doing that now would let the DaemonSet replace all 9 agents at once, onto a build that has never been canaried on a single node — the exact outcome the gate was installed to prevent. The longer it holds, the larger the pending step: rc.0 → GA today, plus whatever lands next.

So "finish or roll back the rollout" is no longer symmetric with "flip the strategy back". Concluding it deliberately — step one node on v1.20.0, verify the datapath, then proceed — looks like the only safe direction, and the alternative is an explicit rollback to a build matching what is actually running.

Gate bookkeeping is intact, for whoever does conclude it: deployment/cluster-autoscaler-hetzner-cluster-autoscaler carries cilium-device-rollout-previous-replicas=1 and cilium-device-rollout-approved-template-sha=94910ca6..., and currently sits at 0 replicas. The release path can restore it to 1.

I could not check whether that approved SHA still matches the current rendered template — the guard derives the candidate from kubectl kustomize plus the three Flux substitution sources, one of which is a Secret I do not have access to. Given the v1.20.0 bump landed after the approval, assume it does not match until that is computed.

🤖 Generated by the Agentic Engineer

New consequence: this now breaks the merge queue, not only Talos sync

Day 11. The gate's cost is no longer confined to suppressed machine-config sync — it is now
evicting pull requests from the merge queue, and the error it surfaces names the wrong subsystem.

CI run 31110692734 (merge
group for #3001) failed, and so did its 🩹 Heal Prod job, which re-deploys main. Both failed at
the same step — 🔑 Stage Flux and consumer GHCR pull credential — with:

The Flux image-verification policy owner did not quiesce before the image-verification policy handoff.

Nothing about that message points here. The actual chain, read live from prod (read-only, today):

daemonset/cilium: updateStrategy=OnDelete, desired=9, ready=9,
updatedNumberScheduled absent (0), generation=24/observedGeneration=24.
Template image is v1.20.0; all 9 running pods are v1.20.0-rc.0.
A Flux Kustomization health check on a DaemonSet requires updatedNumberScheduled to reach
desiredNumberScheduled. Under OnDelete with no node stepped, that value cannot ever rise,
so the check cannot converge — it can only burn its timeout.
kustomization/infrastructure-controllers is therefore stuck
Reconciling=True / Progressing — "Running health checks … with a timeout of 25m0s".
kustomization/infrastructure is consequently Ready=False,
DependencyNotReady: dependency 'flux-system/infrastructure-controllers' is not ready, and itself
Reconciling=True (generation=721, observedGeneration=719).
refresh-flux-ghcr-auth.sh's pause_flux_policy_handoff() requires that infrastructure
(IMAGE_VERIFICATION_FLUX_KUSTOMIZATION) be quiescent — no Reconciling=True — before it will
take the handoff. It retries 60 × 2s and then emits the message above.

So the deploy bridge is reporting a symptom two layers downstream of the un-stepped CNI.

Why it looks intermittent

The 11 merge-group deploys before this one succeeded today. The health-check windows are bounded
(25m for infrastructure-controllers, 20m for infrastructure) and are followed by a brief
failure-and-retry gap, so a deploy that lands inside a window fails and one that lands in a gap
passes. That makes this a recurring systemic failure with a random-looking arrival, not a
transient — re-queuing does not fix it, it just re-rolls the dice. Per the merge-queue rule in
AGENTS.md, #3001 has not been re-queued.

Impact

Any PR can be evicted from the merge queue at any time, with a misattributed error.
🩹 Heal Prod fails the same way, so the automatic restore-main path is also unavailable
while a window is open. In this run publish was skipped, so no speculative artifact reached the
mutable tag and prod stayed on its previous revision — the safe direction, but by luck of
ordering rather than by design.

Ask

Finishing or rolling back the rollout is a cluster mutation (stepping the 9 agent pods), which is
outside this engineer's authority — it needs a maintainer. Until then, expect further evictions.

Two follow-ups are worth separating from that decision, and both are in-authority engineering:

Diagnosability: the handoff timeout should name the blocking dependency and its failing health
check, instead of reporting only the image-verification handoff. Tracing this took a live
four-resource walk that the log could have short-circuited.
Fail the deploy when the Cilium rollout gate outruns its intended window #2972 already proposes failing the deploy when the gate outruns its window; this is evidence that
the window is being outrun in a way that damages unrelated delivery.

🤖 Generated by the Agentic Engineer

The un-adopted delta is no longer just this component's fields — rollback is no longer free

Re-verified live on 2026-08-06 (read-only). Two things changed since this issue was filed, and
together they invalidate the rollback shortcut the component's own runbook offers.

1. Drift is widening, not static

The issue recorded the newest ControllerRevision as 24 (2026-08-03). It is now 27
(2026-08-05T20:45:20Z). The fleet is still entirely on revision 19 (2026-07-25), so a new
revision has been cut since this issue was opened and adopted by nothing.

2. The delta now contains more than the device-set change

Diffing the running template (rev 19) against the desired one (rev 27) shows the un-adopted change
is not limited to this component's devices / nodePort / updateStrategy patches:

Container image: v1.20.0-rc.0 → v1.20.0. Verified against the running pods, not just the
revision object: all 9/9 agents report the -rc.0 image, while daemonset/cilium's desired
spec carries the GA digest. So the fleet is held on a pre-release build of the CNI purely
because OnDelete prevents any pod from adopting the newer template.
A SPIRE agent socket hostPath volume and its container mount were removed in the desired
template and are still present on every running agent.

Why this matters for the decision

The component's ROLLBACK note says re-commenting the reference rolls nothing, because no node has
been stepped. That is no longer true. Removing the component also restores RollingUpdate
alongside rollOutCiliumPods: true, and the template the fleet would then roll to is rev-27-derived
— so the "rollback" would perform a fleet-wide CNI image upgrade and a mutual-auth volume change
as a side effect, on all nine nodes, unattended.

This is exactly the hazard step 3 of the runbook anticipates ("If ANY desired Cilium pod template
value changes while this gate is active … re-step and re-verify every stale node"). It has now
happened, twice.

Consequence

Both available paths are now operator-stepped work in a low-traffic window, not a one-line change:

Proceed — execute the activation runbook (step 2 onward). Each stepped node now adopts the
device-set change and the image upgrade and the volume removal simultaneously, so the per-node
verification must cover all three, and a canary result from before rev 20 approves nothing.
Roll back — un-reference the component, but treat it as a deliberate fleet-wide CNI upgrade
and sequence it accordingly, rather than as the no-op the note describes.

I have not taken either path: both require cluster mutation, which is maintainer-only.

Blast radius while it stays open: the gate continues to skip the pipeline's only Talos
machine-config sync and its only ksail cluster update, so #2922 / #2938 / #2627 remain
structurally unpassable, and platform#3001 (pentad-clear, promoted) cannot clear the merge queue —
its merge_group CI fails on the resulting Flux non-convergence, which is #3002.

🤖 Generated by the Agentic Engineer

The stall has changed character since this was filed — it is now swallowing the CNI GA upgrade

Re-read live from prod today (read-only, 2026-08-06 ~17:2xZ). The mechanism is unchanged, but what
is queued behind it is not.

Running fleet — unchanged since 2026-07-28:

All 9 of 9 agent pods carry controller-revision-hash 7f7c864994 = ControllerRevision 19
(created 2026-07-25T22:59:29Z, image quay.io/cilium/cilium:v1.20.0-rc.0).
Every pod's start time is 2026-07-28; 0 restarts.
daemonset/cilium: updateStrategy=OnDelete, desired=9, ready=9, available=9,
updatedNumberScheduled absent (0), generation=24, observedGeneration=24.

Desired state — moved three more times, and crossed a release boundary:

ControllerRevision	Created	Image
19	2026-07-25T22:59:29Z	`v1.20.0-rc.0` ← what all 9 nodes actually run
20	2026-07-28T18:19:31Z	`v1.20.0-rc.0`
23	2026-08-01T13:44:46Z	`v1.20.0-rc.1`
26	2026-08-03T08:09:07Z	`v1.20.0-rc.1`
27	2026-08-05T20:45:20Z	`v1.20.0` (GA) ← current desired

The DaemonSet's pod template today names v1.20.0 GA on the agent container and all five init
containers. Revision 27 landed ~16 hours after this issue was filed, so the framing here — a
stalled device-selection rollout — now understates it:

production has run a release-candidate CNI (rc.0) for 12 days, and rc.1, rc.1 and the
v1.20.0 GA release have each queued unadopted behind the same OnDelete gate.

Why this changes the priority, not just the numbers

It is no longer a config-shape rollout that can wait. The queued revisions now include the
GA release of the CNI. Every fix that shipped between rc.0 and GA is declared-but-not-running.
It is self-reinforcing. updatedNumberScheduled is 0/9 by construction under OnDelete, so
each new upstream release widens the gap rather than closing it. Four revisions have accumulated;
nothing in the system will ever adopt one.
It wedges the whole GitOps chain, live right now. flux-system/infrastructure-controllers is
Ready=Unknown reason=Progressing — "Running health checks … with a timeout of 25m0s" — with
wait: true and healthChecks: null, so Flux waits on kstatus for every applied resource, and
the DaemonSet can never satisfy updated == desired. That cascades:
infrastructure → DependencyNotReady, apps → DependencyNotReady.
lastAppliedRevision and lastAttemptedRevision have diverged.

So the consequences listed in the original body are not just still live — the set of things
declared-but-not-running now includes the CNI version itself.

Unchanged: the stepping is an operator action

Nothing here proposes automating it. Stepping 9 agents through a CNI upgrade against prod networking
is deliberately a human action, and this run holds read-only cluster access by design. The two halves
remain:

Operator half (this issue): step the fleet onto revision 27, verifying between nodes, then take
the component off OnDelete. require_cilium_fleet_current already fail-closes gate removal.
Forcing-function half (Fail the deploy when the Cilium rollout gate outruns its intended window #2972): make an over-window gate fail the deploy instead of warning
inside a green one — so the next stall surfaces in hours, not twelve days.

Correction to my own earlier evidence

The original body recorded the newest ControllerRevision as 24 created 2026-08-03T08:09:07Z.
Reading the same object today, that timestamp belongs to revision 26. The timestamps and the
conclusion are unaffected; only that ordinal was wrong.

🤖 Generated by the Agentic Engineer

Closing: the rollout completed and the gate is gone

Live read from prod, 2026-09-19 (read-only):

daemonset/cilium in kube-system: updateStrategy=RollingUpdate (no longer OnDelete), desired=8, updated=8, ready=8.
All 8 agent pods run the DaemonSet's current image, quay.io/cilium/cilium:v1.20.1@sha256:ae9ea21f…. That's past the v1.20.0 GA this issue found stuck behind the gate.

Both halves of the problem are resolved. The stalled rollout finished, and the "gate suppresses work without limit" risk now has a guard: the rollout-gate check in deploy-prod reads as active only while the homogeneous-devices component is referenced with type: OnDelete. scripts/report-cilium-rollout-gate-suppression.sh then warns after 7 days and fails the deploy after 14, measured from the gate's activation marker. So a future gated rollout can't silently stall for weeks.

Navigation Menu

Repository navigation

Gated Cilium rollout has stepped zero nodes in 10 days, and its gate has suppressed every Talos config sync and all ksail reconciliation since #2963

Evidence

Why it matters beyond Cilium

Expected behaviour

Proposed direction

Acceptance criteria

Size

Still active today — re-verified live, not inherited from an earlier note

What has changed since this issue was filed — the risk argument has flipped

Why this is not something to fix in the workflow

Net effect

New consequence: this now breaks the merge queue, not only Talos sync

Why it looks intermittent

Impact

Ask

The un-adopted delta is no longer just this component's fields — rollback is no longer free

1. Drift is widening, not static

2. The delta now contains more than the device-set change

Why this matters for the decision

Consequence

The stall has changed character since this was filed — it is now swallowing the CNI GA upgrade

Why this changes the priority, not just the numbers

Unchanged: the stepping is an operator action

Correction to my own earlier evidence

Closing: the rollout completed and the gate is gone

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Gated Cilium rollout has stepped zero nodes in 10 days, and its gate has suppressed every Talos config sync and all ksail reconciliation since #2963

Description

Evidence

Why it matters beyond Cilium

Expected behaviour

Proposed direction

Acceptance criteria

Size

Activity

coderabbitai commented on Aug 5, 2026

devantler commented on Aug 5, 2026

Still active today — re-verified live, not inherited from an earlier note

What has changed since this issue was filed — the risk argument has flipped

Why this is not something to fix in the workflow

Net effect

devantler commented on Aug 6, 2026

devantler commented on Aug 6, 2026

New consequence: this now breaks the merge queue, not only Talos sync

Why it looks intermittent

Impact

Ask

devantler commented on Aug 6, 2026

The un-adopted delta is no longer just this component's fields — rollback is no longer free

1. Drift is widening, not static

2. The delta now contains more than the device-set change

Why this matters for the decision

Consequence

devantler commented on Aug 6, 2026

The stall has changed character since this was filed — it is now swallowing the CNI GA upgrade

Why this changes the priority, not just the numbers

Unchanged: the stepping is an operator action

Correction to my own earlier evidence

devantler commented on Sep 19, 2026

Closing: the rollout completed and the gate is gone

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions