Skip to content

fix(secrets): seed never creates a secret before its value exists - #2132

Open
Smana wants to merge 1 commit into
mainfrom
fix/secret-store-seed-empty-version
Open

Smana wants to merge 1 commit into
mainfrom
fix/secret-store-seed-empty-version

Conversation

@Smana

@Smana Smana commented Sep 29, 2026

Copy link
Copy Markdown
Owner

The bug

secret-store.sh seed's cmd_seed ran, for every generatable key:

seed_body "$name" | store_create "$name"

store_create runs unconditionally regardless of whether seed_body produced
a good value:

  • GCP: store_create first runs gcp_sm create (creates the Secret
    Manager secret), then gcp_sm versions add --data-file=- (reads the value
    from stdin). If seed_body failed or printed nothing, the secret is
    created anyway, then gets a version-less or empty add.
  • AWS: store_create buffers stdin into a 0600 temp file, then calls
    create-secret --secret-string file://... in one call. If seed_body
    failed or printed nothing, the secret is still created, holding an empty
    string.

set -o pipefail does not catch this: pipefail reports the pipeline's exit
status as the rightmost failing command. If seed_body fails but
store_create itself exits 0, the pipeline reports success. Only when
store_create's own calls eventually fail (e.g. GCP's versions add
rejecting an empty payload) does errexit abort the run -- by which point
the broken secret already exists.

Failure scenario

  1. A rebuild seeds cnpg-xplane-zitadel-superuser, whose body is derived
    from zitadel-envvars (not generated). zitadel-envvars hasn't been
    seeded yet, so the derive step fails.
  2. The secret is created anyway (empty/version-less), and the run aborts.
  3. Every later run's store_has now finds the secret present and skips it
    forever -- the entry never self-heals, even after zitadel-envvars is
    seeded.

The fix

cmd_seed now captures seed_body's output into a variable first, checks
that the command succeeded and produced a non-empty value, and only then
pipes that value into store_create on stdin (never on argv -- unchanged
from before). On failure it reports [FAILED ] <key> (never the value),
increments a failed counter, and continues to the next entry -- the same
per-key report-and-continue convention cmd_grant already uses -- with
cmd_seed's own exit status now reflecting whether any entry failed.

AWS's store_create was checked per the review note: it has the same defect
shape (create runs regardless of a bad payload), just folded into one call
instead of GCP's two, so the same outer guard fixes it too. No change was
needed inside store_create itself.

Gates

Gate Result
bash scripts/ci/tests/test-secret-store-seed.sh (new, TDD) exit 0 -- fails against the pre-fix script (reproduces [created] test-key despite a failing/empty seed_body), passes against the fix
bash scripts/ci/tests/test-no-secret-argv.sh exit 0
shellcheck -x -S warning scripts/provision/secret-store.sh exit 0
task check exit 0 -- 40 passed, 1 skipped (missing vector, unrelated), 0 failed

Not a draft -- ready for review, not for merge.

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Rendered manifest diff — this PR vs main (desired state)

No changes to the rendered desired state. ✅

cmd_seed ran seed_body | store_create for each key. GCP's store_create
creates the Secret Manager secret, then adds a version from stdin --
two calls -- and AWS's buffers stdin into a temp file before its one
create-secret call. Either way, a failing or empty seed_body still let
store_create run: a version-less secret on GCP, or one holding an
empty string on AWS. set -o pipefail doesn't catch this -- pipefail
reports the pipeline's rightmost failing command, so a failing
seed_body piped into a succeeding store_create reports success.
Worse, store_has then sees the secret on every later run and skips it,
so the entry stays broken forever.

Capture seed_body's output into a variable first, check it is
non-empty and that seed_body succeeded, and only then call
store_create with the value on stdin. On failure, report which key
failed (never the value) and continue to the next entry, tallying a
failed count -- matching cmd_grant's existing per-key convention --
and make cmd_seed's own exit status reflect it.
@Smana
Smana force-pushed the fix/secret-store-seed-empty-version branch from 8641275 to 7c08974 Compare September 29, 2026 16:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant