A HelmRelease whose post-renderer fails at reconcile still reports Ready #4376

Activity

1. github-project-automation

   added this to 🌊 Project Boardon Oct 2, 2026
2. devantler

   added the

   Bug

   issue type on Oct 2, 2026
3. devantler

   added a parent issue on Oct 2, 2026

   • A HelmRelease postRenderer that cannot apply merges green and silently blocks the release from upgrading #3581
4. devantler commented on Oct 3, 2026

   

   devantler

   on Oct 3, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   Investigation, 2026-10-03 — no reliable durable signal exists at the pinned controller version, so nothing was built

   Read from the released source of the helm-controller version prod runs (v1.5.5; the same code is in v1.6.5, the newest release):

   • A failed upgrade marks the release Released=False / UpgradeFailed, emits a Warning UpgradeFailed event and increments status.failures.
   • The status summary runs after every action, including a failed one, and unconditionally records the new spec's post-renderer digest as observed. A code comment says this happens only after a successful action; the code does not do that.
   • On the next reconcile the chart and values still match the stored release and the digest now matches, so the release is judged in-sync and the condition is rewritten to Released=True / UpgradeSucceeded for the old release. No event marks that flip.

   So the stale Ready this issue describes is an upstream defect, and it applies to any failed upgrade triggered only by a post-renderer or common-metadata change.

   What already holds. The failure itself does raise an alert: the platform's Flux reconciliation alert covers every HelmRelease namespace at error severity, and it delivered this exact post-render failure on 2026-09-04 with the full error text. The alert and its delivery path are live today. And since #4377, a pull request whose post-renderer cannot apply fails CI before it can merge, which closes the cause this issue was split from.

   What is missing is durability. The alert is one message; seconds later every health surface reads healthy and nothing designed records the failure.

   Why no detector was shipped. A derived signal exists (Ready=True with status.failures > 0 and a Released transition much later than the last deploy; 44 of 44 releases are clean against it today), but it rests on side effects of the defect rather than a designed signal: it clears on any unrelated spec edit while the stale state persists, misses a failure shortly after a deploy, and could not be observed from a real controller without cluster writes. Building it would cost about what the existing database-degraded alerter does.

   Recommended path. Treat the first acceptance criterion as met by the existing alert plus #4377, report the defect upstream (the status summary should advance the observed post-renderer and common-metadata digests only when the release action succeeded or none was needed; #3577 is a reproduction), and keep this issue open on that upstream fix. The alternative is the derived-signal alerter with the limits above. Reporting to the upstream project is the maintainer's call.
5. devantler commented on Oct 7, 2026

   

   devantler

   on Oct 7, 2026

   Last edited by devantler

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer (interactive Claude Code session)

   Re-verified 2026-10-07: still an upstream defect, still unreported; report text prepared below

   • Production runs helm-controller v1.5.5. In that release, in v1.6.5 (the newest, 2026-10-01) and on upstream main, the status summary sets the observed post-renderer and common-metadata digests to the current spec every time it runs, whether or not the release action succeeded. The code that decides "in sync" carries a comment saying those digests advance after each successful action, and it only compares them while the object's generation is still unobserved. Both halves were read from the released source today.
   • No upstream report exists. A search of the upstream tracker for the digest field returns five items, none about a failed upgrade reading as ready (the closest, feat(homepage): fix logo and enhance with widgets and bookmarks #1412, fixed a post-renderer change arriving during an upgrade).
   • Nothing changed locally. The Flux reconciliation alert still delivers the failure once, and ci(helm): execute HelmRelease post-renderers against the rendered chart #4377 still stops a pull request whose post-renderer cannot apply. Neither makes the failure last.

   What is left. The third acceptance criterion needs the upstream report linked here, and the local mitigation written down in the repository. Filing is a write to a third-party project; this session was told not to make any, so it is not filed. Under the contribution rule recorded on 2026-10-05 a later run may file it once it has read the upstream project's contribution and AI rules.

   Prepared report (built from the released source and the #3577 failure, nothing private):

   Title: A failed upgrade caused only by a postRenderers change reads Ready=True on the next reconcile

   Versions: observed on a 1.5.x controller; the code described below is identical in v1.5.5, v1.6.5 and on main.

   What happens: A HelmRelease is installed and ready. Its .spec.postRenderers is changed to a patch that cannot be applied to the chart's rendered output; nothing else changes. The upgrade fails in the post-render step: Released=False with reason UpgradeFailed, a Warning event, and status.failures goes up. On the next reconcile the object reports Ready=True and Released=True / UpgradeSucceeded, quoting the previous release, with observedGeneration equal to generation, and the log says the release is in sync. The cluster still runs the output of the old post-renderers. The failure is visible only in the one event and in the controller log.

   Why: summarize in internal/reconcile/release.go sets status.observedPostRenderersDigest (and observedCommonMetadataDigest) from the current spec unconditionally. DetermineReleaseState in internal/reconcile/state.go treats a release as out of sync when the spec digest differs from the observed one, and its comment says the observed digests are updated after each successful release action. After the failed attempt the digests already match, the chart and values match the stored release, so the next reconcile finds nothing to do. The same applies to a failed upgrade caused only by a commonMetadata change.

   Expected: the release stays not ready (or keeps retrying under its remediation settings) until an upgrade with the new post-renderers has succeeded.

   Before filing: reduce the #3577 case to a minimal chart and patch so the steps can be run by someone else, confirm which controller version ran on 2026-09-04 (I did not), and say plainly that the newer versions were read, not run.

   ---

   Added the same day: the local mitigation is written down in #4636 (in the merge queue): a runbook with the signals, a read-only check and its limits, and the repair. Its review corrected two things worth knowing here. A failed upgrade that stores no release raises status.failures only, and a count above zero under Ready=True does not by itself mean the failure was hidden, because a retry or a rollback leaves the same count; the runbook says how to tell them apart and where it cannot. What keeps this issue open after #4636 merges is the upstream report.
6. devantler

   self-assigned this

   on Oct 7, 2026
7. devantler

   added 3 commits that reference this issue on Oct 7, 2026

   chore: agent-claim #4376 nonce=3d408b95382ed2c81b837ab934496279

   9c6d8cf

   docs(operations): correct what the failure count proves and how to re…

   ...

   12e3578

   docs(operations): stop the runbook calling a rollback healthy, and fo…

   ...

   61ce983