Skip to content

Fail the deploy when the Cilium rollout gate outruns its intended window #2972

Closed
Security
Closed
Fail the deploy when the Cilium rollout gate outruns its intended window#2972
Security
Assignees
devantler

Description

@devantler
devantler
opened on Aug 5, 2026

🤖 Generated by the Agentic Engineer

Evidence

The gated Cilium rollout has now been active for 10 days and has stepped zero nodes, and the
whole time every prod deploy has been green while silently skipping its only Talos machine-config
sync and its only ksail cluster update. Re-verified live today (read-only, admin@prod):

daemonset/cilium  updateStrategy=OnDelete  desired=9  ready=9  updatedNumberScheduled=0
all 9 agent pods on controller-revision-hash 7f7c864994  (revision 19)

The suppression is real and ongoing — see the parent #2963 for the full consequence list — but the
reason it survived 10 days is narrower and separable: nothing makes an over-running gate visible as
failure. A green deploy that quietly skipped its config sync looks exactly like a green deploy that
did not need one.

Scope of this child

Only the parent's fifth acceptance criterion:

A gate that stays active past its intended window fails a deploy rather than warning inside a green
one.

Deliberately not in scope: stepping the 9 agents, or rolling the component back. Those are
operator actions against prod networking that the parent keeps unautomated on purpose, and no
forcing function can perform them.

Expected behaviour

scripts/guard-cilium-homogeneous-device-rollout.sh records when the gate first went active and
fails the deploy once the gate has been active longer than its intended window.

  • Activation time is stored as epoch seconds in a companion annotation on the same Cluster
    Autoscaler deployment that already carries the guard's ownership markers, so the check is integer
    arithmetic with no date parsing. (date -j -f on BSD and date -d on GNU disagree about what an
    unspecified time-of-day means, so a parsed-date comparison is platform-dependent and cannot be
    ablated locally against the Linux runner.)
  • The gate suspends the autoscaler first and checks the window second, so a stalled rollout stays
    fenced rather than trading one safety property for the new one.
  • The window is exceeded ⇒ the guard fails, naming the elapsed time and the two legitimate
    resolutions (complete the rollout, or roll the component back). Raising the window is explicitly
    not one of them.
  • The gate already active with no recorded activation time (exactly today's prod state) ⇒ the
    clock is backfilled to now and the deploy stays green. This is the honest bound: the guard cannot
    know retroactively when a gate it never timestamped went active, and reddening every prod deploy
    the moment this merges would freeze the merge queue on a change whose whole point is to make a
    stall visible in an orderly way. The backfill is announced in the deploy log, and the window then
    applies normally.
  • Releasing the gate clears the annotation alongside the existing ownership markers, so a later
    rollout starts a fresh clock.

Acceptance criteria

  • The guard records gate activation time and clears it on release.
  • An active gate inside its window succeeds; an active gate past its window fails the deploy.
  • Both states are covered by scripts/tests/test-cilium-homogeneous-devices-autoscaler-gate.sh,
    including the missing-annotation backfill path.
  • The window default is stated in the script and overridable for tests.

Size

Small — one script, its test, no manifest change.

Part of #2963.

Activity

  1. github-project-automation
    added this to 🌊 Project Boardon Aug 5, 2026
  2. devantler
    added the
    Security
    issue type on Aug 5, 2026
  3. devantler
    self-assigned this
    on Aug 5, 2026
  4. devantler
    added a parent issue on Aug 5, 2026

  5. coderabbitai commented on Aug 5, 2026

    @coderabbitai
    coderabbitaibot
    on Aug 5, 2026 – with coderabbitai
    Contributor
    🔗 Related PRs

    #2793 - feat(cilium): activate homogeneous device selection [merged]
    #2927 - fix(deploy): await pre-handoff kustomize-controller pod termination [merged]
    #2930 - feat(ci): verify published evidence before promoting the prod digest [merged]
    #2947 - fix(ci): refuse to promote a prod digest whose evidence cannot be verified [merged]
    #2962 - fix(deploy): make rollout-gate suppression of machine-config sync visible and bounded [merged]

    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan
    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
  auto_enrich:
    enabled: false

    💬 Have feedback or questions? Drop into our discord!

  6. devantler
    moved this to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Aug 5, 2026
  7. devantler
    assigned
    devantler
    and unassigned
    devantler
    on Aug 21, 2026
  8. devantler
    added a commit that references this issue on Aug 21, 2026

    chore: agent-claim #2972 nonce=615f16112bb938e74a8b20fc05fc45d4

    a782666

  9. devantler commented on Aug 21, 2026

    @devantler
    devantler
    on Aug 21, 2026
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Closing as already delivered. This shipped on the day the issue was filed, and the issue was
    simply never closed behind it.

    fix(ci): fail the deploy when the Cilium rollout gate outruns its window (#2973, 97d41324,
    2026-08-05) — the same title as this issue — added the hard bound to
    scripts/report-cilium-rollout-gate-suppression.sh, whose own reporting half had landed the day
    before in #2962 (a312e62b).

    Acceptance criteria against main (a0537f33)

    • Records gate activation time and clears it on release. Activation is the
      platform.devantler.tech/rollout-gate-activated: <YYYY-MM-DD> marker declared beside the
      component reference in k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml;
      releasing the gate removes the component reference and its marker in the same diff. An active
      gate with no marker, a duplicated marker, an unparseable date, or a future date each fail
      closed      .
    • Inside its window succeeds; past its window fails the deploy. warn_after_days=7 emits a
      ::warning::; fail_after_days=14 calls fail, which emits ::error:: and exits 1. The
      step is wired at .github/actions/deploy-prod/action.yml:330 under
      always() && steps.cilium_rollout_gate.outputs.active == 'true' with no continue-on-error,
      so the non-zero exit reddens the deploy rather than warning inside a green one.
    • Both states covered, in scripts/tests/test-cilium-rollout-gate-suppression-signal.sh —
      including the exact >= boundary on both thresholds, the future-date and one-day-ahead cases
      (where integer division would otherwise hide the skew as 0 days), the duplicated-marker
      case, and the override.
    • Window default stated and overridable.
      readonly fail_after_days="${CILIUM_ROLLOUT_FAIL_AFTER_DAYS_OVERRIDE:-14}".

    One deviation from the issue's design, and it is an improvement

    This issue specified an epoch-seconds annotation on the Cluster Autoscaler Deployment, guarded in
    scripts/guard-cilium-homogeneous-device-rollout.sh and covered in
    test-cilium-homogeneous-devices-autoscaler-gate.sh. What shipped instead dates the gate from a
    reviewed marker in Git. That is the stronger choice on the axis that matters here: a cluster
    annotation is operator-writable, so the same kubectl annotate command a refusal hands out is also
    the way to silence it — a far-future value would make the gate immortal, and a zero-padded value
    would abort in shell arithmetic before the guard's own message. The Git marker cannot be moved
    without a reviewed diff, so completing or rolling back the rollout is the only way to clear it. The
    coverage living in the suppression-signal test rather than the autoscaler-gate test follows from
    that design, and is not a gap.

    The issue's own framing still holds exactly as written — "a green deploy that quietly skipped its
    config sync looks exactly like a green deploy that did not need one"    . That is now false in the
    deploy, which is what this asked for.

  10. devantler
    closed this as completedon Aug 21, 2026
  11. github-project-automation
    moved this from 🏃🏻‍♂️ In Progress to ✅ Done in 🌊 Project Boardon Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Security

Fields

No fields configured for Security.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions