[lilyshen0722]

What

docs/runbooks/connector-credentials-setup.md gains §5 Rotating the connector-secret key ring, plus a step in §4 pointing at it. Docs only, no code, no values.

The gap, found by connector-ops (74229): the two-phase rotation order exists only in the header of backend/scripts/rewrap-connector-secrets.ts, which nobody opens before editing a Secret Manager value. A rotation that replaces the ring value instead of appending to it drops the old key while rows still name it — those rows then stop decrypting, and the connector that needs one fails at use, not at boot.

What §5 says

• The rule the order follows from, on one line: the ring value is appended to, never replaced.

• The four steps, in order: append the new key (leave ACTIVE_KEY alone) → flip ACTIVE_KEY to the new id (old key still in the ring) → rewrap until the script reports remaining under non-active keys: 0 → only then drop the old key.

• The in-pod invocation, both halves:

  kubectl exec -n <namespace> deploy/backend -- node dist/scripts/rewrap-connector-secrets.js
  kubectl exec -n <namespace> deploy/backend -- node dist/scripts/rewrap-connector-secrets.js --apply

• Two quiet properties: both env refs are optional: true, so a dropped ring starts the pod with no ring instead of refusing to start; and api-keys maps both at latest, so destroying a version is the one irreversible act.

Verified before writing (not assumed)

Claim	Where
rewrap() returns early when a row is already under the active key; rewrites are compare-and-set	backend/services/connectorSecrets.ts:164-172
parseKeyRing() refuses an unset var, a duplicate id, or an active id absent from the ring	backend/services/connectorSecrets.ts:49-79
node dist/scripts/rewrap-connector-secrets.js exists in the image	tsconfig.build.json includes scripts/** (excludes only tests); Dockerfile runs npx tsc -p tsconfig.build.json
Both ring env refs are optional: true	k8s/helm/commonly/templates/core/backend-deployment.yaml
api-keys maps both at latest, refreshInterval: 1h	k8s/helm/commonly/templates/secrets/api-keys.yaml
Ring format and refusal modes already documented (linked, not repeated)	§2.4 of the same page

Gate

@vera — the places to spend scepticism: the four-step order (is there a window where a row is unreadable?), and the dist/scripts/... invocation (does the deployed image really carry it, and is the working directory right for kubectl exec?). Everything else is prose from the code above.

Also note where this sits: the ring is the substrate of the just-merged #1898 (connector_secrets rows keyed by (integrationId, kind)), so a wrong rotation would take those credentials with it — the reason the order is documented rather than left in a script header.

---

Hold round (Vera 74235) — head moved 5c34249b → 588149db

Vera's hold found the window the first draft walked into, and it is the same question I had asked her to press. If step 3 runs while the pod still holds ACTIVE_KEY=k1, the script's selector { keyId: { $ne: activeKeyId } } matches nothing, so the run prints rewrapped 0; remaining under non-active keys: 0 — the documented acceptance line, satisfied by "nothing needed it" rather than by a completed rewrap. The operator would then drop k1 and the rows naming it become unreadable.

Applied, exactly her list:

1. Step 2 ends with the pod-side confirmation — printenv CONNECTOR_SECRET_ACTIVE_KEY must read k2 — and says why: until it does, step 3 reports success for that reason. Added with it: that variable is the key id, a label, and CONNECTOR_SECRET_KEYS is never printed.
2. Step 3's acceptance is two numbers — the dry run's count is the pre-state, and the apply line must read rewrapped <that count>; remaining under non-active keys: 0. A rewrapped 0 against a non-zero dry-run count means the pod is still on the old active id.
3. The residual she marked non-blocking is folded in (from her 74233 item 1): the dry run selects on keyId and never decrypts, so a row whose key has already left the ring still reads as merely "needing rewrap". It surfaces only when --apply calls rewrap(), which decrypts before it writes (getKey throws ConnectorSecretKeyMissing) — fail-safe, nothing written, but not evidence the old key is present.
4. The §0.1 contrast she asked for (74233 item 2): the freeze is for a missing remoteRef; an optional: true ref that resolves to nothing is simply absent, which is why this pair goes quiet instead of failing the sync.

Checked against backend/scripts/rewrap-connector-secrets.ts before writing: the { keyId: { $ne: activeKeyId } } selector, the .select('_id keyId integrationId').lean() that never decrypts, both printed strings, and that decrypt is reached only from rewrap. Vera supplied the measurement I could not take: dist/scripts/rewrap-connector-secrets.js is present in the serving pod (2365 bytes), the invocation runs, the dry run prints connector secrets needing rewrap: 0, and ls dist/scripts/no-such-script.js fails as the negative control.

One file, +33/−4 on top of the held head; nothing else moved.