Repository navigation
Generated secrets are regenerated on every GitOps render; ignoreDifferences does not prevent it #9
Description
Summary
The chart's generated secrets cannot survive a GitOps render, and ArgoCD's
ignoreDifferences does not protect them. This blocks managing PatchMon with
ArgoCD at all — verified empirically on k8s-sto2, twice.
The mechanism
templates/secret.yaml seeds JWT_SECRET, SESSION_SECRET and AI_ENCRYPTION_KEY
with randAlphaNum, keeping them stable across upgrades via a lookup on the existing
Secret. lookup only resolves under helm upgrade/install. ArgoCD renders with
helm template, where it always returns empty — so every render mints three new values:
$ helm template patchmon patchmon/patchmon --version 0.3.6 -f values.yml | grep JWT_SECRET
JWT_SECRET: "fAWUcEVwQRqDHzWPsBZikBELm..." # render 1
JWT_SECRET: "GkWMNDFpxZ0nwVvA3HUPRq7HQ..." # render 2, same inputs
@BryanR77 documented this in #7 and added patchmon.session.secret /
patchmon.encryption.aiKey so the values can be pinned — but pinning them means
putting three secrets in git in plaintext, which is not a fix for GitOps users.
ignoreDifferences does not save you
The obvious mitigation is ignoreDifferences on the Secret's /data plus
RespectIgnoreDifferences=true. It does not work. Tested by syncing only that one
resource and hashing before/after:
| Sync options | Result |
|---|---|
ServerSideApply=true + RespectIgnoreDifferences=true |
regenerated |
RespectIgnoreDifferences=true (client-side apply) |
regenerated |
Both times all three keys were replaced. Restored from a pre-made copy of the Secret;
no pod restarted, so nothing was lost — but on an unattended selfHeal: true sync this
would be silent and permanent.
Why it matters more than it looks
JWT_SECRET/SESSION_SECRET— every session invalidated on each sync.AI_ENCRYPTION_KEY— anything already encrypted with the old key becomes
unrecoverable. There is no recovery path once the old value is gone.
A GitOps user with automated sync would rotate all three on every reconcile without
any signal that it happened.
Proposed fix
Support consuming these from an existing Secret, exactly mirroring the pattern the
chart already has for OIDC (patchmon.oidc.clientSecretFromSecret), which works well:
patchmon:
existingSecret:
name: "" # when set, the chart does NOT render its own Secret
keys:
jwt: JWT_SECRET
session: SESSION_SECRET
aiKey: AI_ENCRYPTION_KEYWhen name is set:
templates/secret.yamlskips those keys entirely (or skips rendering altogether if
OIDC is also external);deployment-server.yamlpoints eachsecretKeyRefat the named Secret.
That lets the secrets live in a real secret store (we use OpenBao + External Secrets)
and keeps them out of both git and the chart's render, which makes the manifest
deterministic and the ArgoCD diff clean.
Notes
- The
lookupbehaviour is correct forhelm upgradeusers and should stay as the
default — this is purely an additional opt-in path. - Worth a README section on GitOps deployment once implemented, since the failure mode
is silent and expensive. - Related: Fix postgres StatefulSet crash under readOnlyRootFilesystem #7 (added the value knobs), Expose OIDC_TRUST_UNVERIFIED_EMAIL and document the 2.1.x verified-email requirement #8 (OIDC_TRUST_UNVERIFIED_EMAIL).
Activity
MartinHell commentedon Aug 20, 2026
The blast radius is wider than the variable name suggests
Checked the environment variables reference.
AI_ENCRYPTION_KEYis not AI-specific — it is the application's general at-rest encryption key:So a GitOps sync that regenerates it does not just break an optional AI integration. It takes out:
SESSION_SECRETis also worth naming correctly: it is not a cookie signer but a fallback encryption key for the same data, kept "for backward compatibility with early 1.x installs." Both keys therefore need the same protection, and the proposedexistingSecretshould cover all three.Related footgun the docs call out, relevant to anyone using an external database with rotating credentials:
The chart always sets both, so chart users are not exposed to that — but it is another argument for making these explicitly pinnable rather than generated, and probably worth a line in the README when this lands.