feat(validator): add optional startupProbe - #334
Merged
Merged
Conversation
The chart renders only livenessProbe and readinessProbe, so a slow-starting validator can only be accommodated by raising liveness initialDelaySeconds, which also delays steady-state failure detection. A startupProbe separates the two: while it runs, liveness and readiness are suspended, and once it passes they behave normally. This is not hypothetical. On 2026-08-18 a prod ghost pod took 182s to serve /healthz against the chart default initialDelaySeconds of 60, collected six Unhealthy events and was killed 21s after it came up. The version it replaced took exactly 60s, i.e. it had been clearing the probe with no margin at all. Disabled by default, so existing releases render unchanged (verified: helm template with default values emits no startupProbe). Applies to both the ghost and ghost-vao deployments. Default budget when enabled is failureThreshold 30 x periodSeconds 10 = 5 minutes.
paulohpigatto
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an optional
startupProbeto thevalidatorchart, for both theghostandghost-vaodeployments. Disabled by default. Chart version 0.6.11 -> 0.7.0 (minor, new feature, no behaviour change for existing releases).Why
The chart renders only
livenessProbeandreadinessProbe. That means the only way to accommodate a validator that starts slowly is to raise livenessinitialDelaySeconds, which also delays steady-state failure detection: the two concerns are forced to share one number.This bit prod. On 2026-08-18 a prod
ghostpod took 182s to serve/healthzagainst the chart defaultinitialDelaySeconds: 60, accumulated sixUnhealthyevents, and was killed 21 seconds after it finally came up (app logged normal startup, no errors; image pull took 1s). The version it replaced took exactly 60s — it had been clearing the probe with no margin at all.A
startupProbeis the mechanism designed for this: while it runs, liveness and readiness are suspended, so startup budget and failure detection are tuned independently.Backwards compatibility
global.startup.enableddefaults tofalse, so existing releases render byte-identically. Verified:helm lintpasses. README regenerated withhelm-docs.Usage
The template guards with
and .Values.global.startup .Values.global.startup.enabled, so consumers that have not merged the new values key are safe.Related
chronicleprotocol/app-of-apps#1922 currently works around this by raising liveness
initialDelaySecondsto 300 for the prod validator. Once this chart version is adopted there, that override should be replaced withstartup.enabled: true, which is the better shape.