Skip to content

Add a startup probe to the calico-node DaemonSet - #5163

Open
caseydavenport wants to merge 4 commits into
tigera:masterfrom
caseydavenport:casey-node-startup-probe
Open

Add a startup probe to the calico-node DaemonSet#5163
caseydavenport wants to merge 4 commits into
tigera:masterfrom
caseydavenport:casey-node-startup-probe

Conversation

@caseydavenport

@caseydavenport caseydavenport commented Aug 12, 2026

Copy link
Copy Markdown
Member

Description

Type of change: enhancement

calico-node typically becomes ready within a couple of seconds of container start, but nothing notices until the next readiness poll. During a rolling update that latency is paid once per node, so on a large cluster it dominates how long the DaemonSet takes to roll.

This adds a startup probe to calico-node running the same health check readiness already runs, polling every 2s with a 300s total budget. Two effects:

  • Pods are marked ready within ~2s of actually being ready instead of on the next readiness poll.
  • Kubernetes suspends the liveness probe until the startup probe passes, so a slow start on a large cluster can no longer be restarted part way through. Previously the liveness probe was live from container start and would restart the container after ~3 minutes.

The tradeoff is that a container hung at startup now takes up to 300s to be restarted rather than ~3 minutes. That felt like the right direction, since the restart is unlikely to help and the 300s budget is only ever paid when startup is actually failing.

Readiness and liveness are unchanged. Windows nodes are not changed.

Note that the operator's probe defaulting only fills in liveness and readiness probes, so every startup probe field is set explicitly here - the Kubernetes defaults would give a 30s budget, which would restart calico-node on any large cluster.

The startup probe's timing is overridable through the calico-node DaemonSet overrides on the Installation resource, alongside the existing readiness and liveness overrides. The probe handler stays operator-owned.

TigeraStatus also accounts for the new probe: the window it waits before reporting a pod as running but not ready now includes the startup budget, since Kubernetes holds off readiness until the startup probe passes. Without that, every node would be reported as failing part way through a slow start.

Affected components: calico-node DaemonSet, TigeraStatus reporting.

Release Note

Reduces the time calico-node takes to be marked ready after starting, which shortens calico-node rolling updates on large clusters.

For PR author

  • Tests for change.
  • If changing pkg/apis/, run make gen-files
  • If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

  • Milestone set according to targeted release.
  • Appropriate labels:
    • kind/bug if this is a bugfix.
    • kind/enhancement if this is a a new feature.
    • enterprise if this PR applies to Calico Enterprise only.

The startup probe polls every 2s with a 300s budget and suspends the
liveness probe until Felix is up.
@caseydavenport
caseydavenport force-pushed the casey-node-startup-probe branch from edd9a38 to c7e6643 Compare August 12, 2026 18:56
@caseydavenport
caseydavenport requested a review from fasaxc August 12, 2026 19:40
The status grace period now includes the startup budget, so a slow start is not flagged as failing.
Copilot AI lite review requested due to automatic review settings August 13, 2026 15:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves calico-node rollout behavior by introducing a startup probe for the calico-node DaemonSet and wiring that probe through the operator’s override and status-reporting plumbing.

Changes:

  • Add a StartupProbe to the rendered calico-node container, using the same exec health check as readiness, with a tighter polling cadence.
  • Extend DaemonSet/container override support (and override annotation/status messaging) to include startup probe timing overrides.
  • Update API docs, CRD schema, and tests (rendering, overrides, enterprise modifier, and status grace-period logic) to account for the new startup probe.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
pkg/render/node.go Renders a startup probe for calico-node and refactors probe construction.
pkg/render/node_test.go Asserts startup probe rendering alongside readiness/liveness.
pkg/render/common/components/components.go Adds startup probe support to container override extraction/merge and override annotation tracking.
pkg/render/common/components/components_test.go Tests startup probe override application and annotation behavior.
pkg/imports/crds/operator/operator.tigera.io_installations.yaml Updates generated CRD schema to expose startupProbe timing overrides.
pkg/enterprise/installation/node.go Ensures enterprise modifier appends the BGP metrics check to startup probe command when applicable.
pkg/enterprise/installation/node_test.go Unit tests for enterprise modifier behavior on readiness + startup probes.
pkg/enterprise/installation/node_enterprise_test.go Integration test assertion includes startup probe command modification.
pkg/controller/status/status.go Includes startup probe in readiness grace period derivation and status messaging for overrides.
pkg/controller/status/status_test.go Tests longer readiness grace period behavior when startup probe is present.
docs/api_design.md Documents that StartupProbe is exposed when a container renders one.
api/v1/zz_generated.deepcopy.go Generated deepcopy updated for the new StartupProbe field.
api/v1/calico_node_types.go Adds StartupProbe *ProbeOverride to the CalicoNodeDaemonSet container override API.
Files not reviewed (1)
  • api/v1/zz_generated.deepcopy.go: Generated file

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pkg/render/node.go Outdated
Copilot AI review requested due to automatic review settings August 13, 2026 15:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Files not reviewed (1)
  • api/v1/zz_generated.deepcopy.go: Generated file
Suppressed comments (2)

pkg/render/node.go:1730

  • The comment says the startup probe has "~300s of startup", but with PeriodSeconds=2, TimeoutSeconds=5, and FailureThreshold=150 the worst-case window is much larger (150*(2s+5s)=1050s). This is also reflected in status_test.go, so the comment is misleading for readers trying to reason about timing.
	// Poll every 2s so readiness lands fast; 150 failures gives ~300s of startup, more if checks time out.

pkg/controller/status/status.go:960

  • readinessGracePeriod now adds startup+readiness settling time using (period+timeout) per attempt. With calico-node's startup probe settings (2s period, 5s timeout, 150 failures) this yields a ~1050s startup window and ~18min total grace before reporting NotReady, which appears inconsistent with the PR description's "300s total budget" and may delay degraded reporting significantly.
	for _, c := range p.Spec.Containers {
		// Kubelet holds off readiness until the startup probe passes, so the budgets add.
		d := probeSettlingTime(c.StartupProbe) + probeSettlingTime(c.ReadinessProbe)
		if d > grace {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants