Add a startup probe to the calico-node DaemonSet - #5163
Open
caseydavenport wants to merge 4 commits into
Open
Conversation
5 tasks
The startup probe polls every 2s with a 300s budget and suspends the liveness probe until Felix is up.
caseydavenport
force-pushed
the
casey-node-startup-probe
branch
from
August 12, 2026 18:56
edd9a38 to
c7e6643
Compare
The status grace period now includes the startup budget, so a slow start is not flagged as failing.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves calico-node rollout behavior by introducing a startup probe for the calico-node DaemonSet and wiring that probe through the operator’s override and status-reporting plumbing.
Changes:
- Add a
StartupProbeto the rendered calico-node container, using the same exec health check as readiness, with a tighter polling cadence. - Extend DaemonSet/container override support (and override annotation/status messaging) to include startup probe timing overrides.
- Update API docs, CRD schema, and tests (rendering, overrides, enterprise modifier, and status grace-period logic) to account for the new startup probe.
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/render/node.go | Renders a startup probe for calico-node and refactors probe construction. |
| pkg/render/node_test.go | Asserts startup probe rendering alongside readiness/liveness. |
| pkg/render/common/components/components.go | Adds startup probe support to container override extraction/merge and override annotation tracking. |
| pkg/render/common/components/components_test.go | Tests startup probe override application and annotation behavior. |
| pkg/imports/crds/operator/operator.tigera.io_installations.yaml | Updates generated CRD schema to expose startupProbe timing overrides. |
| pkg/enterprise/installation/node.go | Ensures enterprise modifier appends the BGP metrics check to startup probe command when applicable. |
| pkg/enterprise/installation/node_test.go | Unit tests for enterprise modifier behavior on readiness + startup probes. |
| pkg/enterprise/installation/node_enterprise_test.go | Integration test assertion includes startup probe command modification. |
| pkg/controller/status/status.go | Includes startup probe in readiness grace period derivation and status messaging for overrides. |
| pkg/controller/status/status_test.go | Tests longer readiness grace period behavior when startup probe is present. |
| docs/api_design.md | Documents that StartupProbe is exposed when a container renders one. |
| api/v1/zz_generated.deepcopy.go | Generated deepcopy updated for the new StartupProbe field. |
| api/v1/calico_node_types.go | Adds StartupProbe *ProbeOverride to the CalicoNodeDaemonSet container override API. |
Files not reviewed (1)
- api/v1/zz_generated.deepcopy.go: Generated file
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.
Files not reviewed (1)
- api/v1/zz_generated.deepcopy.go: Generated file
Suppressed comments (2)
pkg/render/node.go:1730
- The comment says the startup probe has "~300s of startup", but with PeriodSeconds=2, TimeoutSeconds=5, and FailureThreshold=150 the worst-case window is much larger (150*(2s+5s)=1050s). This is also reflected in status_test.go, so the comment is misleading for readers trying to reason about timing.
// Poll every 2s so readiness lands fast; 150 failures gives ~300s of startup, more if checks time out.
pkg/controller/status/status.go:960
- readinessGracePeriod now adds startup+readiness settling time using (period+timeout) per attempt. With calico-node's startup probe settings (2s period, 5s timeout, 150 failures) this yields a ~1050s startup window and ~18min total grace before reporting NotReady, which appears inconsistent with the PR description's "300s total budget" and may delay degraded reporting significantly.
for _, c := range p.Spec.Containers {
// Kubelet holds off readiness until the startup probe passes, so the budgets add.
d := probeSettlingTime(c.StartupProbe) + probeSettlingTime(c.ReadinessProbe)
if d > grace {
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Type of change: enhancement
calico-node typically becomes ready within a couple of seconds of container start, but nothing notices until the next readiness poll. During a rolling update that latency is paid once per node, so on a large cluster it dominates how long the DaemonSet takes to roll.
This adds a startup probe to calico-node running the same health check readiness already runs, polling every 2s with a 300s total budget. Two effects:
The tradeoff is that a container hung at startup now takes up to 300s to be restarted rather than ~3 minutes. That felt like the right direction, since the restart is unlikely to help and the 300s budget is only ever paid when startup is actually failing.
Readiness and liveness are unchanged. Windows nodes are not changed.
Note that the operator's probe defaulting only fills in liveness and readiness probes, so every startup probe field is set explicitly here - the Kubernetes defaults would give a 30s budget, which would restart calico-node on any large cluster.
The startup probe's timing is overridable through the calico-node DaemonSet overrides on the Installation resource, alongside the existing readiness and liveness overrides. The probe handler stays operator-owned.
TigeraStatus also accounts for the new probe: the window it waits before reporting a pod as running but not ready now includes the startup budget, since Kubernetes holds off readiness until the startup probe passes. Without that, every node would be reported as failing part way through a slow start.
Affected components: calico-node DaemonSet, TigeraStatus reporting.
Release Note
For PR author
make gen-filesmake gen-versionsFor PR reviewers
A note for code reviewers - all pull requests must have the following:
kind/bugif this is a bugfix.kind/enhancementif this is a a new feature.enterpriseif this PR applies to Calico Enterprise only.