fix(cilium): roll back the homogeneous-device gate so prod deploys resume - #3030
fix(cilium): roll back the homogeneous-device gate so prod deploys resume#3030devantler wants to merge 4 commits into
Conversation
The control asserted the reporter exits 0 for a single in-bound marker, but hard-coded 2026-07-26 as that marker's date. On 2026-08-09 that reached fail_after_days=14, so the reporter correctly failed and the control failed with it -- taking every platform pull request down. Use the existing in-bound relative date, as every other date assertion in this file already does. The duplicate-marker case keeps its absolute dates because it asserts failure and fails closed on duplication before age is evaluated.
…sume The gate reached its 14-day hard-fail bound, which stopped the deploy's Talos machine-config sync and, through the merge queue, every merge in the repository. No agent was ever stepped, so the widened device set never ran anywhere and the rollback restores the device set the fleet is already on. Re-comments the component reference and removes the activation marker in the same diff, which is what the reference's own comment requires. The pre-publish guard has an explicit path for this and publishes without the fleet-current check that gate REMOVAL requires. Both rollout tests asserted the activated state only, and one built its fixture by copying the live overlay, so they encoded the rollout as permanent. They now construct each state and assert both.
✅
|
The rollback moves eight non-RBAC lines in one HelmRelease, which moves the aggregate surface fingerprint the validator pins. Measured across all five roots: 519 documents both sides, membership identical, four roots byte-identical. All 67 RBAC documents and all 123 aws-bearing lines compared directly and unchanged, with a planted-change control proving the comparison is not blind. Fingerprint taken from the required job's own output.
Requesting review at Worth your attention in this diff: it re-approves the pinned authorization-surface fingerprint in @coderabbitai review |
|
✅ Action performedReview finished.
|
📝 WalkthroughWalkthroughThe Cilium homogeneous-device component is now disabled by default in the Hetzner controllers configuration. Activation tests cover both enabled and rollback states. Autoscaler-gate tests create an explicit active fixture, and suppression-signal tests use a relative date. The EKS role-policy validator records the updated rendered Cilium fields and fingerprint. Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Readiness at
|
Closing — this is the wrong directionThe maintainer's intent for #3028 is to complete the homogeneous-device rollout, not reverse it. Not merged. Dequeued and cancelled mid-deploy; the consequences that did reach the cluster, and the What was worth keeping is re-filed as #3031 — the autoscaler-gate test builds its fixture by Not carried over, deliberately:
Still open and unrelated to direction: #3029's one-line fixture fix, which reds every PR in this |

Why
The Cilium rollout gate hit its 14-day hard-fail bound, and the blast radius turned out to be the
whole repository: the deploy stopped syncing Talos machine config two weeks ago, and because every
merge goes through the queue and the queue deploys prod, nothing could merge either — including the
fix for the gate.
You asked me to get prod healthy. Rolling the component back is the option that does that and can
actually merge: it removes the gate from its own merge-queue run, so this PR is not blocked by the
thing it is fixing.
Nothing is lost by rolling back. Not one agent was ever stepped — all nine are still on the
pre-activation template — so the widened device set has never run on this cluster, and the rollback
restores the device set the fleet is already using.
What
Re-comments the component reference and removes the activation marker in the same diff, which is what
that reference's own comment requires. The staged work is preserved and default-off, ready to activate
again through its runbook in a low-traffic window.
Operational note — this rolls the Cilium agents. Removing the component restores the chart's
RollingUpdate strategy, so the nine agents step to the current template. That is the desired
direction: they are currently pinned on a release candidate, and the current template is the
1.20.0 GA release. Restoring the sync is also what stops the machine-config drift growing.
Also fixes the two rollout tests, which asserted the activated state as if it were permanent — one of
them built its fixture by copying the live overlay, so it broke the moment the component was rolled
back. Each now constructs the state it tests and both states are asserted.
Part of #3028— deliberately not a closing keyword. The remaining acceptance criteria are onlyobservable after this deploys, so I will verify the fleet and the restored sync on the cluster and
close the issue on evidence rather than on merge.
Includes #3029 (that PR's fixture fix is in this branch); it can close once this lands.