Allow db creds declared in Cfn to be secrets

[nicklathe]

This PR adds db secrets declared in a Cloud Formation Stack to be propagated to the cluster secrets so they are available in a Deployment/Pod

Why

Currently, we're not able to connect to the MySQL database from our cluster. Root cause is a gap between how dashboard declares its database configuration and what we sync into Kubernetes.

dashboard's DB endpoints and credentials — db_endpoint_proxy_reporting, db_credential_reader, and ten sibling keys — are declared !StackSecret in code-dot-org's config.yml.erb. A !StackSecret resolves to the AWS Secrets Manager path CfnStack/<stack-name>/<key>, where the stack name is discovered by reading the aws:cloudformation:stack-name tag off the EC2 instance the process is running on.

There is no EC2 instance in Kubernetes. So that resolution cannot work, and the config loader falls back to a live GetSecretValue call at boot. That call has no credentials to sign with: no workload manifest sets serviceAccountName (no IRSA), and the worker explicitly sets AWS_EC2_METADATA_DISABLED=true.

The existing ExternalSecret syncs only <env>/cdo/*, which supplies the URL-style keys (db_writer, db_reader, reporting_db_reader, …) but none of the CloudFormation-provisioned ones. Confirmed in the staging namespace:

$ kubectl exec deploy/cdo-active-job-worker -n staging -- sh -c 'env | grep ^CDO_db | cut -d= -f1'
CDO_db_admin
CDO_db_admin_password
CDO_db_reader
CDO_db_reader_password
CDO_db_writer
CDO_db_writer_password

No CDO_db_endpoint_*, no CDO_db_credential_*.

What

Add a second dataFrom.find to the per-namespace ExternalSecret so CfnStack/<env>/* is synced into cdo external-secrets alongside <env>/cdo/*, with the same prefix-stripping rewrite.

Once those keys arrive as CDO_* environment variables, they win the config loader's precedence (env is source #1, and the merge keeps the first writer), so the !StackSecret placeholder is never constructed and no Secrets Manager call
happens at boot. No code-dot-org change is required.

This also fixes bin/mysql-client-dashboard-reader in-cluster, which needs CDO.db_endpoint_proxy_reader and CDO.db_credential_reader for the same reason.

Scope

Single-namespace env types only — staging, test, levelbuilder, production.

The adhoc ClusterExternalSecret is deliberately unchanged: adhoc deployments get one CloudFormation stack each, so there is no CfnStack/adhoc/* path to sync, and the multi-namespace IAM policy does not grant one. Adhoc namespaces still need either the in-cluster mysql component or a code-dot-org-side change.

Why this is low risk

• No IAM change needed. apps/infra/standard-envtypes/chart/templates/aws/single-namespace-envtypes-iam.yaml already grants the ESO service account secret:CfnStack/{{ $env }}/*. The permission was provisioned in anticipation of exactly this.
• The path is already proven. bootstrap/codeai-k8s/cluster-infra-argocd/test/test-external-secrets.yaml fetches CfnStack/staging/db_endpoint_proxy_reader_port through the same aws-secrets-manager-store SecretStore.
• Additive. Existing keys and their values are unchanged; this only adds keys that are absent today.

Verification

helm template on standard-envtypes with all four single-namespace env types renders valid YAML, with dataFrom paths:

ClusterExternalSecret  cdo-external-secrets-adhoc   ["adhoc/cdo/"]
ExternalSecret         cdo-external-secrets         ["staging/cdo/", "CfnStack/staging/"]
ExternalSecret         cdo-external-secrets         ["test/cdo/", "CfnStack/test/"]
ExternalSecret         cdo-external-secrets         ["levelbuilder/cdo/", "CfnStack/levelbuilder/"]
ExternalSecret         cdo-external-secrets         ["production/cdo/", "CfnStack/production/"]

After sync (allow one 5m refreshInterval), confirm in the staging namespace:

kubectl get secret cdo-external-secrets -n staging -o jsonpath='{.data}' | jq -r 'keys[]' | grep db_endpoint
kubectl rollout restart deploy/cdo-active-job-worker -n staging

envFrom values are injected once at container start, so the restart is required.

Notes / follow-ups

• apps/infra/standard-envtypes/chart/charts/eso-per-envtype-0.1.0.tgz is tracked
  and stale — the _envtype.tpl inside it still contains the creationPolicy /
  deletionPolicy / conversionStrategy / decodingStrategy fields that 1021fba
  removed from source. If argocd-repo-server runs helm dependency build before
  templating (expected), the vendored copy is dead weight and should be untracked.
  If it does not, that commit never took effect and neither will this one. Worth
  confirming in the rendered diff on first sync.
• Assumes environment_type equals the CloudFormation stack name for these four
  env types. Same assumption the IAM template already makes; worth a spot-check for
  production.
• dataFrom entries merge in order, so ordering is now load-bearing if the two
  paths ever share a key name. They are disjoint today.
• Three code-dot-org bugs surfaced while diagnosing this, none blocking:
  lib/cdo/secrets.rb:68 mutates a frozen exception message and destroys the real
  error; StackSecret#key in lib/cdo/secrets_config.rb shadows Secret#key so
  the environment-type fallback is unreachable and requests an empty secret id off
  EC2; and lib/cdo/mysql_console_helper.rb:23,31 resolve StackSecrets in every
  environment because the rack_env?(:production) guard is last in the && chain.

🤖 Generated with Claude Code