Skip to content

fix(config): never generate a secret the staging/prod validators reject - #948

Merged
acamarata merged 1 commit into
mainfrom
fix/flaky-tests-wave6
Oct 26, 2026
Merged

acamarata merged 1 commit into
mainfrom
fix/flaky-tests-wave6

Conversation

@acamarata

Copy link
Copy Markdown
Collaborator

Root cause (product bug, not a test race)

TestSurfaceIdentity failed on the arm runner in build --plan (staging fixture) with POSTGRES_PASSWORD contains insecure pattern (E400, run 40198819060). config.generateSecureRandom returned raw base64url text, so a random draw sometimes spells one of insecurePatterns (test, admin, hasura, 12345, secret, ...). In staging/prod validatePasswords then rejects a value the CLI itself just generated. Build persists generated secrets to .env.secrets before it validates, so a real first nself build in staging could leave a project permanently failing with E400.

Measured on the unfixed generator, 5,000,000 draws each: 107 rejected at 24 chars (POSTGRES_PASSWORD), 138 at 32, 195 at 44 (admin secret, JWT key). About 1 in 10,000 fresh staging/prod projects, and the surface test builds several.

Fix

generateSecureRandom discards a draw that matches insecurePatterns or a placeholder and takes the next bytes of the source. A seeded source (plan and apply share one stream) skips the same draw both times, so they still agree. A source stuck on a blocked value errors after 32 draws.

Evidence

  • 1 in 10,000 cannot be reproduced by repetition, so the regression tests force the draw: TestGenerateSecureRandomSkipsBlockedPatterns (5 blocked prefixes x 24/44 chars), ...SkipIsDeterministic, ...StuckSourceErrors. Before the fix: 2 of the 3 fail (the generated value is xxTeStxxAAAA...); after: all pass, also with -race.
  • go test ./internal/build/... ./internal/config/... ./cmd/commands/: 5595 passed.
  • TestSurfaceIdentity -count=15 after the fix: pass. This is a sanity check only, at 1 in 10,000 it cannot prove the flake gone; the forced-draw tests do.

generateSecureRandom returned base64url text straight from the random
source. About 1 draw in 50,000 (24 chars) to 25,000 (44 chars) contains a
blocked substring (test, admin, hasura, 12345, change-me ...), so a fresh
staging/prod project got a POSTGRES_PASSWORD or admin secret that
validatePasswords refused (E400), after build had already persisted it to
.env.secrets. TestSurfaceIdentity hit this on the arm runner (build --plan).

Discard a draw that matches insecurePatterns or a placeholder and take the
next bytes of the source. A seeded source (plan and apply) skips the same
draw both times, so they still agree. A source stuck on a blocked value
errors after 32 draws instead of looping.
@acamarata
acamarata force-pushed the fix/flaky-tests-wave6 branch from 92ac349 to 677785b Compare October 25, 2026 20:03
@acamarata

Copy link
Copy Markdown
Collaborator Author

Rebased onto origin/main (9 commits behind) and force-pushed (677785b). Full go test ./... passes locally on the rebased head, including the nginx call-site, docker funnel, 300-line and panic-baseline ratchets; none needed loosening. The earlier gate failure ('Host key verification failed') is an ssh failure on the gate host, not a Go test failure, so a gate re-run on this head is needed.

@acamarata
acamarata merged commit b42bc40 into main Oct 26, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant