node-rotation-controller

A Kubernetes controller that proactively rotates Karpenter-managed nodes within a defined maintenance window, using make-before-break (surge) semantics, before Karpenter's forceful expireAfter triggers.

Designed for EKS Auto Mode and any Karpenter v1+ environment where node expiration is forceful and disruption budgets do not apply.

Status

v0.6.1 — v1 surge MVP, released (pre-1.0). The v1 make-before-break rotation state machine (spec §5.2), ageThreshold/candidate derivation (§3.2), surge placeholder (§3.3), metrics and Warning Events (§4.2), the Helm chart, and the Karpenter v1 startup preflight (§5.1) are implemented, with unit tests and an envtest smoke test in CI. So is the opt-in, window-bounded surge-less forceful fallback (surge.forcefulFallback, default off; ADR-0001), which deletes a candidate's NodeClaim inside the window — still via Karpenter's voluntary, PDB-respecting path — when a graceful surge cannot finish before that node's own expireAfter deadline, rather than leaving it to be force-expired at an uncontrolled time. Candidates are ordered by earliest deadline, and a node carrying an operator-owned karpenter.sh/do-not-disrupt annotation is excluded from selection.

v0.6.0 is about the throughput forecast — the check that tells you, before a single node moves, whether your window is wide enough to rotate the fleet. It now models a rotation from what a healthy rotation actually costs: surge.provisioningEstimate (candidate created → new node Ready) plus surge.drainEstimate (a PDB-respecting drain) plus cooldownAfter, instead of deriving it from the force-kill deadline (ADR-0003). Its inputs are exported as metrics, so the forecast is observable rather than a one-off startup log. surge.failurePause splits the pause after a failed attempt off cooldownAfter (ADR-0004), so the settle pause between successful rotations can go all the way to 0 — leaving PDBs to serialize drains — without also unbounding candidate cycling under a systematic failure. And the documentation site now hosts a browser policy simulator: the controller's own schedule and selection code, compiled to WebAssembly and guarded in CI against drift, run against a policy you edit in the page.

v0.6.1 is a chart-only follow-up: the controller Deployment can now set an optional top-level priorityClassName, so on a shared pool the component that rotates the pool need not run at the default priority. It is the controller's own priority, unrelated to the surge placeholder's negative-priority class (spec §3.3). No controller behavior, CRD schema, annotation or metric changed.

The core surge path is validated on EKS Auto Mode across a full rotation regression suite, including a trick-free forceful-fallback run on a synchronized batch and a 12-hour tight-race soak under a sub-daily window (Scenario P). A genuine same-AZ capacity shortage (ICE) driving rollback remains the open item before v1.0 (see the roadmap).

Still pre-1.0 — the configuration surface may change between minor releases. docs/specification/ remains the source of truth for the design; see Compatibility for the Karpenter contract.

日本語版: README.ja.md / docs/ja/specification/

Why

Karpenter classifies node disruption into two categories:

Category	Examples	NodePool Disruption Budgets	Pre-provisioned replacement
Graceful	Drift, Consolidation	Applied	Yes (make-before-break)
Forceful	Expiration, Spot Interruption	Not applied	No

Expiration is intentionally forceful (see the upstream forceful-expiration design) so that AMI patches and security updates cannot be indefinitely delayed by misconfigured budgets. The upstream design explicitly lists "operators implement their own graceful rotation" as one acceptable path. EKS Auto Mode further enforces a 21-day hard cap on node lifetime that cannot be lifted.

The practical consequence: nodes will be force-drained at some point within 21 days — the drain honors PDBs only up to terminationGracePeriod, not indefinitely — and Karpenter will only provision a replacement after the drain begins. This can land in peak business hours.

This controller closes that gap by:

1. Watching NodeClaim resources approaching expiration
2. Restricting rotation to a configurable maintenance window (e.g., Saturday 02:00–06:00)
3. Inducing a NodePool-owned replacement node first via a low-priority placeholder Pod (never a standalone NodeClaim — see the spec, §3.3), waiting until the reserved capacity is ready, then deleting the old NodeClaim (surge)
4. Letting Karpenter's standard termination controller graceful-drain the old node, where PDBs do apply

What it is not

• Not a replacement for Karpenter Consolidation, Drift, or Disruption Budgets — it composes with them
• Not a Spot interruption handler (use AWS Node Termination Handler)
• Not an OS-patch reboot tool (use kured)
• Not a pod descheduler (use descheduler)
• Not a replacement for application-side warm-up (readinessProbe, readinessGate, slow_start) — surge places nodes, applications must place themselves

Compatibility

The compatibility contract is the stable karpenter.sh/v1 CRD surface — not a specific Karpenter controller minor. This matters for EKS Auto Mode, which does not expose the managed Karpenter version to users: the controller runs against any cluster serving a compatible karpenter.sh/v1 NodePool/NodeClaim API.

• Runtime target: EKS Auto Mode and any Karpenter v1+ cluster exposing a karpenter.sh/v1-compatible NodePool/NodeClaim API.
• Build/test baseline: the repository compiles and tests against the sigs.k8s.io/karpenter version pinned in go.mod. That pins the typed Go API it is built against — it is not a requirement that the cluster run that exact Karpenter minor.
• No internals, no cloud APIs: the controller interacts only through Kubernetes API objects (NodeClaim/NodePool CRDs, core Node/Pod); it never calls Karpenter controller internals or a cloud-provider API. Unknown Auto Mode internals are fine as long as the public karpenter.sh/v1 surface is compatible.
• A startup preflight fails fast if karpenter.sh/v1 (nodeclaims/nodepools) is not served or not readable.

See the compatibility policy for the full list of required CRD fields, labels, and annotations.

Project layout

.
├── docs/
│   ├── specification/         Full design specification (English, per chapter)
│   ├── runbook.md             Production runbook (English)
│   ├── ja/specification/      Japanese translation
│   ├── ja/runbook.md          Production runbook (Japanese)
│   └── reference/             ADRs and performance notes
├── charts/                    Helm chart (node-rotation-controller)
├── examples/                  Ready-to-adapt RotationPolicy manifests
├── cmd/                       Controller entry point (manager bootstrap + startup preflight)
└── internal/                  Reconciler and supporting packages: rotation state machine
                               (controller), schedule/selection, surge placeholder, window,
                               policy, metrics, preflight

Installation

Requires Karpenter v1+ already installed. This chart does not install Karpenter or its CRDs — it only operates the NodeClaim/NodePool resources Karpenter owns.

Install the published chart from the GitHub Container Registry (OCI):

helm install node-rotation-controller \
  oci://ghcr.io/akashisn/charts/node-rotation-controller \
  --namespace node-rotation-system --create-namespace \
  --set-json 'rotationPolicies=[{"spec":{"nodePoolSelector":{"matchLabels":{"workload":"api"}},"maintenanceWindows":[{"timezone":"Asia/Tokyo","days":["Wed","Sat"],"start":"02:00","end":"06:00"}]}}]'

Or install from a local checkout of this repository:

helm install node-rotation-controller charts/node-rotation-controller \
  --namespace node-rotation-system --create-namespace \
  --set-json 'rotationPolicies=[{"spec":{"nodePoolSelector":{"matchLabels":{"workload":"api"}},"maintenanceWindows":[{"timezone":"Asia/Tokyo","days":["Wed","Sat"],"start":"02:00","end":"06:00"}]}}]'

• What the chart installs. The controller (replicas=2 with leader election), its RBAC, the cluster-scoped RotationPolicy CRD (from the chart's crds/ directory) plus a sample RotationPolicy object, and the dedicated negative-priority PriorityClass for the surge placeholder Pod (spec §3.3, §4.3, §5.1).
• Configuring rotation. List policies under rotationPolicies (the spec §5.4 schema) — the chart renders one RotationPolicy per entry, so you can give each NodePool a different window / ageThreshold / surge. The default values ship a single-entry sample, which the quickstart above points at your NodePools. See charts/node-rotation-controller/values.yaml.
• Bring your own policies. Set rotationPolicies: [] to author your own RotationPolicy objects out-of-band instead (one per divergent policy); a NodePool matched by none is simply not rotated. See examples/ for ready-to-adapt policies — a single catch-all, divergent per-NodePool policies, specificity resolution, and maintenance-window composition.

Maintainer note (first release only): the ghcr.io image and chart packages may be created private on first publish. Make node-rotation-controller and charts/node-rotation-controller public in the GitHub Packages settings so unauthenticated helm install / image pulls work, then verify with a logged-out client — e.g. helm pull oci://ghcr.io/akashisn/charts/node-rotation-controller --version <X.Y.Z> (the chart version has no leading v — the release guard strips it), or fetch the image manifest anonymously and expect HTTP 200. (Querying or changing package visibility via the GitHub API needs a token with read:packages / write:packages; the Packages settings UI needs no token.) Releases are cut by pushing a vX.Y.Z tag (see the Release workflow).

Getting involved

This project is pre-1.0 and under active development; the v1 scope is the surge MVP described in the specification. Design feedback and implementation contributions are both welcome via GitHub Issues and PRs.

See CONTRIBUTING.md for the development workflow and CODE_OF_CONDUCT.md for community standards.

Development

Requires aqua and make; Docker is needed only for make docker-build. The Go toolchain and all CLI tooling (golangci-lint, gopls, setup-envtest, kind, ko, kustomize, helm, kubectl, terraform, awscli) are version-pinned in aqua.yaml — install aqua, and the make targets provision and use the pinned versions automatically (aqua lazily installs each on first use; a make run links them onto $PATH for you). The Go version in aqua.yaml is kept in sync with the go directive in go.mod.

Command	Purpose
make build	Compile the manager binary into bin/manager
make test	Run unit tests and the envtest-based smoke test
make lint	Run golangci-lint
make helm-lint	Lint and render the Helm chart
make docker-build	Build the container image

make test downloads the envtest control-plane binaries on first run.

See CONTRIBUTING.md for the workflow (issues, branches, PRs).

License

Apache 2.0 — see LICENSE.