Repository navigation
Cilium rollout gate silently suppresses Talos machine-config sync indefinitely #2951
Description
🤖 Generated by the Agentic Engineer
Evidence
The 🔄 Update cluster (Talos machine config) step in .github/actions/deploy-prod/action.yml — the platform's only Talos machine-config sync — has been skipped on every prod deploy for at least four days, while each deploy reported success.
| Deploy | __self.cluster_update |
|---|---|
| 2026-08-04T16:09Z (30927795575) | outcome=skipped duration_ms=0 |
| 2026-08-04T15:33Z (30924710728) | outcome=skipped |
| 2026-08-03T07:58Z (30795581205) | outcome=skipped |
| 2026-08-01T11:04Z (30696960482) | outcome=skipped |
The cause is the first arm of its condition:
if: >-
steps.cilium_rollout_gate.outputs.active != 'true' &&
steps.wait_flux_revision.outcome == 'success'In the 16:09Z run wait_flux_revision was success, and the gate step printed Cilium homogeneous-device rollout gate active: Cluster Autoscaler is suspended. The gate is still active: kube-system/cluster-autoscaler-hetzner-cluster-autoscaler reads 0/0 on prod.
The problem
The gate is deliberate and its reasoning is sound — KSail owns Cluster Autoscaler and cluster update could reconcile it back to one replica mid-rollout, racing a scale-up before the first Cilium canary is verified. The step's own comment describes it as temporary, released by "a reviewed rollout-completion or rollback artifact".
The defect is not the coupling, it is that the coupling is unbounded and silent:
- A safety gate for one component indefinitely disables an unrelated reconcile loop.
- Nothing reports it. The deploy is green, the summary says success, and the skipped step reads as normal conditional behaviour.
- The longer it holds, the larger the un-reconciled machine-config delta becomes — and the first
cluster updateafter release applies all of it at once, against nodes that have drifted.
It also silently blocks unrelated work: #2938 and #2922 both depend on cluster update applying the root OCIRepository cosign verify block. KSail shipped that fix in v7.178.13 and this repo already pins v7.178.14, so both issues look like KSail defects and are in fact blocked here.
Affected audience and impact
Operators of the prod cluster. Machine config in Git is not the machine config on the nodes, with no signal that they diverge, and root artifact verification cannot be applied while it holds.
Expected behaviour
A temporary gate should be visibly temporary. Either the deploy surfaces that it is suppressing machine-config sync, or the two are decoupled so a Cilium rollout does not stop Talos reconciliation.
Acceptance criteria
- A prod deploy that skips
cluster updatebecause the rollout gate is active says so in its job summary — not only as a skipped step. - The suppression is bounded: after a defined threshold (deploy count or elapsed time) the deploy warns loudly, and the threshold is asserted by a test.
- A regression test fails if the gate can suppress
cluster updatewith no surfaced signal. - The decision is recorded: keep the coupling with a bound, or decouple autoscaler suspension from machine-config sync.
- Verified against a real deploy log, not only against the workflow source — the whole point here is that a correct-looking source produced a silent skip.
Out of scope
Releasing the current Cilium rollout gate — that is an operational judgement about the rollout, not a repo change. The verify-block outcomes tracked in #2938 and #2922.
Rough size: small — a summary line plus a bounded threshold and its test. The decision on decoupling may be larger and can be a follow-up.
Part of #2627.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
coderabbitai commentedon Aug 4, 2026
🔗 Related PRs
#2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
#2633 - fix(deploy): verify incoming image on Talos nodes [merged]
#2793 - feat(cilium): activate homogeneous device selection [merged]
#2927 - fix(deploy): await pre-handoff kustomize-controller pod termination [merged]
#2930 - feat(ci): verify published evidence before promoting the prod digest [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
docs(ksail): name the real reason the block does not reach prod
devantler commentedon Aug 4, 2026
Still active — two more deploys tonight, and the blast radius is wider than this issue records
Confirming against the two prod deploys produced by tonight's merges:
__self.cluster_update30954732610(22:01Z)outcome=skipped conclusion=skipped duration_ms=030956176478(22:22Z)outcome=skipped conclusion=skipped duration_ms=0The gate condition still holds right now:
cluster-autoscaler-hetzner-cluster-autoscalerreads 0replicas on prod.
KSAIL_VERSIONis pinned 7.178.14. Both deploys reported success.That extends the documented run to at least six days of prod deploys in which the platform's only
Talos machine-config sync has not executed, with a green summary each time.
One addition to the impact section
This issue notes that it silently blocks #2938 and #2922. Working #2922 today I can confirm that
from the other direction, and it cost real effort: I published a root-cause analysis attributing the
missing cosign
verifyto a KSail code path, because the deploy was green and the step's absenceread as ordinary conditional behaviour. It was wrong — the command simply never ran. I have since
corrected that analysis and linked this issue as a dependency of both #2922 and #2960.
That is the defect doing exactly what the issue predicts, one level up: it does not just suppress
reconciliation, it actively misleads diagnosis of anything downstream of it, because a skipped
step inside a green deploy is indistinguishable from a step that ran and did nothing. Whatever
remediation is chosen, the "make it visible" half is worth as much as the decoupling half.
Unclaimed as of this comment.
fix(deploy): make rollout-gate suppression of machine-config sync vis…
devantler commentedon Aug 4, 2026
Decision (acceptance criterion 4): keep the coupling, bound it
Keep it. The gate's reasoning is sound and I found nothing that weakens it: KSail owns Cluster
Autoscaler,
cluster updatereconciles it, and doing that mid-rollout could restore a replica andrace a scale-up onto a node that boots the widened Cilium device set before any canary is verified.
Decoupling would mean either teaching
cluster updateto leave the autoscaler alone — a KSailchange, outside this repo — or splitting machine-config sync out of it. Both are larger than the
defect being fixed, and neither is justified by evidence I have.
What was actually wrong is the silence, so that is what #2962 fixes. The gate may still suppress
machine-config sync; it may no longer do so invisibly or indefinitely.
Against the acceptance criteria
📣 Report suppressed machine-config syncstep writes a summary block naming the skipped sync, the start date, elapsed days, and what else is blocked behind itwarn_after_days=7; the test asserts the constant exists and that it is a real bound in(0,30], so it cannot be neutered by setting it to 999cluster update, must run underalways(), and must take the gate state from the gate's own output. Each of those was ablated and each firesWhy seven days
The component's own runbook is "step nine nodes one at a time, verifying each, then soak". A week is
generous for that. Past it the rollout is not proceeding as designed, and the un-synced machine-config
delta becomes the larger risk. Run against the current repo state it fires immediately — 9 days
elapsed against a 7-day bound — which is the honest reading.
On criterion 5, precisely
The input is verified against real deploy logs, not the workflow source:
__self.cluster_updateisoutcome=skipped duration_ms=0in runs30954732610and30956176478, and the gate's activationdate comes from the commit that activated it (
dc51cf64, 2026-07-26), not from a guess.The output — the summary block appearing in a real deploy — can only be observed after this merges
and the next merge-queue deploy runs. I will read that job summary back and record it here rather
than close on the source alone; that is the whole point of this criterion.
Deliberately not done
Releasing the gate. That is the operational judgement this issue puts out of scope, and it is
genuinely yours: it needs someone to decide whether the Cilium rollout should be finished node by
node or rolled back. This fix makes that decision visible and dated; it does not make it. Worth
knowing that #2922 and #2938 stay blocked until it happens.
devantler commentedon Aug 4, 2026
Closing — criterion 5 verified against a real deploy log
#2962 merged at
2026-08-04T23:48:56Z. Its own merge-queue deploy exercised the new step, which isthe verification this criterion insisted on — the whole point being that a correct-looking source had
already produced a silent skip once.
From the
🚀 Deploy to Prodjob log of run30960849127:
And in that same run, confirming it is reporting a real suppression rather than firing
speculatively:
So the deploy that skipped machine-config sync now says so, loudly, in the same run — which it did
not do for the nine days before this.
One precise limit on that evidence: what I read back is the
::warning::annotation and thestep's
outcome=success. The job-summary block itself is written to$GITHUB_STEP_SUMMARY, aseparate file that is not exposed through the API — the log correctly contains zero occurrences
of its text, which is the designed behaviour, not a gap. The block's content is pinned by the
contract test instead.
All criteria
📣 Report suppressed machine-config syncstep,outcome=success, livewarn_after_days=7; the test asserts the constant and that it is a real bound in(0,30], so it cannot be neutered to 999cluster update, run underalways(), and take gate state from the gate's own output — each ablated, each firesWhat review caught that I did not
CodeRabbit found three 🟠 Major defects my own two self-review passes had missed, and all three were
real — most notably that a future activation date within one day slipped past the very guard I had
added during self-review, because bash truncates integer division toward zero. My test only covered
+30d, which yields-30and passed, so the guard looked proven while the boundary walked through it.Investigating it turned up a portability trap worth recording: BSD
date -j -f '%Y-%m-%d'fillsunspecified fields from the current time while GNU
date -dgives midnight. Measured at23:29:56Z, BSD parsed
2026-08-05as2026-08-05 23:29:56Z. That means the defect is live on theLinux runner and cannot be reproduced on macOS, where a local ablation reads as proof the guard
works. Both sides now resolve through one explicit-midnight parser.
Still open, and it is not mine
Releasing the gate itself — out of scope here by design, and a real operational decision: finish
the Cilium rollout node-by-node, or roll it back. This change makes the cost visible and dated; it
does not make that call. #2922 and #2938 remain blocked behind it, and the deploy will now say so
on every run until it is resolved.