From 2db6e24b6e2da7952dc98f4c22412d5722da7642 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Tue, 27 Oct 2026 01:21:17 -0700 Subject: [PATCH 1/4] docs(runbooks): release checklist and rollback for v3.0.0 onward Cut from a SHA with green staging-smoke, deploy that SHA via deploy-dev ref input, verify at the consumer, freeze via press-holds, roll back to an explicit helm revision, and tag each release. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MDs9DS3E4dzoHKdxtgQqyz --- .../release-checklist-and-rollback.md | 170 ++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 docs/runbooks/release-checklist-and-rollback.md diff --git a/docs/runbooks/release-checklist-and-rollback.md b/docs/runbooks/release-checklist-and-rollback.md new file mode 100644 index 0000000000..e738a5b421 --- /dev/null +++ b/docs/runbooks/release-checklist-and-rollback.md @@ -0,0 +1,170 @@ +# Release checklist and rollback + +This is the procedure for every production deploy from v3.0.0 onward. Commonly +has active users, so a deploy is a release: cut from a known SHA, proven on +staging, verified where users meet it, and reversible in one command. Staging +is described in [`staging-environment.md`](staging-environment.md). + +Placeholders: `` and `` are the production Helm release and +namespace (the `env` block of `.github/workflows/deploy-dev.yml` names them), +and `` is a Deployment such as `backend` or `frontend`. + +## When this applies + +Every production deploy, however small. + +- **One batch a day.** The day's merges ship together. +- **The only exception is a P0 or a security fix.** It skips the wait for the + next batch, not any step below. + +## 1. Cut + +1. Choose the main SHA to ship, as the full 40-character SHA. +2. Confirm `staging-smoke` is green at **that exact SHA**, not a neighbour: + + ```bash + gh api repos/Team-Commonly/commonly/commits//statuses \ + --jq '[.[] | select(.context=="staging-smoke")][0] | {state, description, created_at}' + ``` + + Staging runs with `cancel-in-progress`, so a later merge cancels the run for + your SHA. Older runs then posted a false red on the superseded SHA, and + since #2461 they post nothing. Neither is a verdict on the code. Re-run + staging on your SHA; dispatch on main and pass the commit as `ref`: + + ```bash + gh workflow run deploy-staging.yml --ref main --repo Team-Commonly/commonly -f ref= + ``` + +3. Record the rollback target before anything changes: + + ```bash + helm history -n --max 5 + ``` + + Note the newest revision with status `deployed` and its image tag. Do not + assume it is the previous row, because a failed upgrade is a revision too. + +## 2. Deploy + +```bash +gh workflow run deploy-dev.yml --ref main --repo Team-Commonly/commonly -f ref= +gh run list --workflow=deploy-dev.yml -L 1 --repo Team-Commonly/commonly +``` + +The run is always dispatched on main; `ref` picks the commit it builds. Its +first job refuses a `ref` that is not a full SHA in the repo, and any SHA +without a green `staging-smoke`. An empty `ref` deploys main HEAD, which is not +what you cut, so always pass it. The image tag defaults to the first 8 +characters of the SHA. The workflow builds backend, frontend and commonly-bot +(plus clawdbot-gateway when the release enables it), then runs `helm upgrade`. + +`allow_without_staging_smoke=true` bypasses the gate. Use it only for a P0 +while staging itself is broken, and say so in the release record. + +## 3. Verify at the consumer + +A green workflow is not proof. Kubernetes serves from the old pods during a +rolling update, so check what users actually hit. + +1. **Live image tags.** Every deployed workload shows the cut tag. The + argument is quoted because zsh reads the brackets as a glob: + + ```bash + kubectl get deploy -n \ + -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image' + ``` + +2. **Cutover time.** Take it from the new pods, not the workflow's finish: + + ```bash + kubectl get pod -n -l app=backend \ + -o 'jsonpath={range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}' + ``` + +3. **API and landing page.** Both return 200: + + ```bash + curl -s -o /dev/null -w '%{http_code}\n' https://api.commonly.me/api/health + curl -s -o /dev/null -w '%{http_code}\n' https://commonly.me/ + ``` + +4. **One hosted-agent turn.** `node scripts/smoke-stranger-path.mjs` installs a + hosted agent as a non-admin account, @mentions it and waits for the reply. + See [`hosted-agent-provisioning.md`](hosted-agent-provisioning.md). + +If any check fails, roll back first and debug second. + +## 4. Freeze + +A freeze is one line in `~/.commonly/state/press-holds.txt`: + +```text +* .. +``` + +For example, `* 2026-10-28T22:00:00Z..2026-10-29T08:00:00Z v3.0.0 launch`. +Merges to main may continue under a freeze, and staging keeps deploying them. +Nothing deploys to production until the window ends. The auto-press watcher +holds every PR inside the window, so a merge then is a deliberate press. + +## 5. Rollback + +Always name the revision; never run a bare `helm rollback`: + +```bash +helm rollback -n +kubectl rollout undo deploy/ -n --to-revision= +``` + +- `helm rollback` to the revision recorded at the cut restores the whole + release. Without a revision it goes to the previous one, which can be a + failed upgrade or days stale; on 2026-08-04 a bare rollback would have + reverted two auth fixes. +- `kubectl rollout undo` reverts a single Deployment. Helm does not see it, + so the next deploy overwrites it. +- Neither undoes data migrations or writes the new code made. + +Then re-verify as in section 3, against the old tag. + +**Who may roll back:** the CTO or SRE operator session, or Sam. + +## 6. Tagging a release + +New from v3.0.0. Earlier releases were v1.0.0 (04-07), v1.1.0, v2.0.0 (07-06) +and v2.1.0 (07-23). + +1. **Tag the deployed SHA.** After section 3 passes, the operator tags and + pushes: + + ```bash + git tag -a vX.Y.Z -m "vX.Y.Z — " + git push origin vX.Y.Z + ``` + +2. **GitHub Release**, titled `vX.Y.Z — <title>` like the earlier ones: + + ```bash + gh release create vX.Y.Z --repo Team-Commonly/commonly \ + --title "vX.Y.Z — <title>" --notes-file <notes.md> --verify-tag + ``` + + The notes list only features that are merged, deployed in that SHA and + walked on prod. Anything behind a flag or not yet walked waits for the + next release. Group them under short headings with PR numbers, as v2.1.0 + does. + +3. **CHANGELOG.md entry.** There is no `CHANGELOG.md` yet; earlier notes live + only on GitHub Releases. v3.0.0 creates it at the repo root, newest first, + each entry repeating the release notes: + + ```markdown + ## vX.Y.Z — <title> (YYYY-MM-DD) + + Deployed SHA: <short SHA> + + ### <heading> + - <feature> (#1234) + ``` + +Release names are not routes, so `/v2` URL paths stay unchanged. From 8e546b4da742ee1c2cf73190a8cad35a976ace12 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Tue, 27 Oct 2026 01:24:02 -0700 Subject: [PATCH 2/4] docs(runbooks): a major version ships first as a beta pre-release Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDs9DS3E4dzoHKdxtgQqyz --- docs/runbooks/release-checklist-and-rollback.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/runbooks/release-checklist-and-rollback.md b/docs/runbooks/release-checklist-and-rollback.md index e738a5b421..3aa79f2776 100644 --- a/docs/runbooks/release-checklist-and-rollback.md +++ b/docs/runbooks/release-checklist-and-rollback.md @@ -134,6 +134,12 @@ Then re-verify as in section 3, against the old tag. New from v3.0.0. Earlier releases were v1.0.0 (04-07), v1.1.0, v2.0.0 (07-06) and v2.1.0 (07-23). +**Pre-release first, final when proven.** A major version ships first as +`vX.Y.Z-beta.N`, published as a GitHub pre-release. The final `vX.Y.Z` is +tagged only after the release's own exit list is done; for v3 that list is the +multiplayer walk, the grants walk, the repo sweep, a clean bug burn-down and a +staging soak. When in doubt, ship another beta. + 1. **Tag the deployed SHA.** After section 3 passes, the operator tags and pushes: @@ -149,14 +155,18 @@ and v2.1.0 (07-23). --title "vX.Y.Z — <title>" --notes-file <notes.md> --verify-tag ``` + For a beta, add `--prerelease` so it is not marked Latest, e.g. + `gh release create v3.0.0-beta.1 … --prerelease`. + The notes list only features that are merged, deployed in that SHA and walked on prod. Anything behind a flag or not yet walked waits for the next release. Group them under short headings with PR numbers, as v2.1.0 does. 3. **CHANGELOG.md entry.** There is no `CHANGELOG.md` yet; earlier notes live - only on GitHub Releases. v3.0.0 creates it at the repo root, newest first, - each entry repeating the release notes: + only on GitHub Releases. v3.0.0-beta.1 creates it at the repo root, newest + first, each entry repeating the release notes. A beta's heading carries its + full pre-release version (`## v3.0.0-beta.1 — <title> (YYYY-MM-DD)`): ```markdown ## vX.Y.Z — <title> (YYYY-MM-DD) From 16e7f2664d194a1ba70501e30f1e9192121d8a81 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Tue, 27 Oct 2026 01:25:42 -0700 Subject: [PATCH 3/4] docs(runbooks): v3.0.0 final only on a green release gate, beta otherwise Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDs9DS3E4dzoHKdxtgQqyz --- .../release-checklist-and-rollback.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/runbooks/release-checklist-and-rollback.md b/docs/runbooks/release-checklist-and-rollback.md index 3aa79f2776..dcb29d11ab 100644 --- a/docs/runbooks/release-checklist-and-rollback.md +++ b/docs/runbooks/release-checklist-and-rollback.md @@ -134,11 +134,20 @@ Then re-verify as in section 3, against the old tag. New from v3.0.0. Earlier releases were v1.0.0 (04-07), v1.1.0, v2.0.0 (07-06) and v2.1.0 (07-23). -**Pre-release first, final when proven.** A major version ships first as -`vX.Y.Z-beta.N`, published as a GitHub pre-release. The final `vX.Y.Z` is -tagged only after the release's own exit list is done; for v3 that list is the -multiplayer walk, the grants walk, the repo sweep, a clean bug burn-down and a -staging soak. When in doubt, ship another beta. +**Final only when the gate is green; otherwise a beta.** A major version is +tagged final (`vX.Y.Z`) only when every item on its release gate is green at +the freeze. If any item is red, the same SHA ships as `vX.Y.Z-beta.N`, published +as a GitHub pre-release, and the final waits for the gate. For v3.0.0 the gate +is: + +1. the multiplayer walk passed on prod; +2. no open P0 or P1 bug on prod; +3. the repo sweep merged and the root clean; +4. staging soak green from the day's deploy through the freeze; +5. every release-note line walked, or cut; +6. first-touch docs clean. + +When in doubt, ship the beta. 1. **Tag the deployed SHA.** After section 3 passes, the operator tags and pushes: From 13fcdbf6ce60eaa11a491e5eecc796ae5f6e60a2 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Tue, 27 Oct 2026 01:59:15 -0700 Subject: [PATCH 4/4] docs(runbooks): release runbook review fixes Full helm history, identify and watch the dispatched run, rollback as either/or with separate histories, final tag only on a green gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDs9DS3E4dzoHKdxtgQqyz --- .../release-checklist-and-rollback.md | 29 ++++++++++++++----- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/docs/runbooks/release-checklist-and-rollback.md b/docs/runbooks/release-checklist-and-rollback.md index dcb29d11ab..abde3d7a50 100644 --- a/docs/runbooks/release-checklist-and-rollback.md +++ b/docs/runbooks/release-checklist-and-rollback.md @@ -39,7 +39,7 @@ Every production deploy, however small. 3. Record the rollback target before anything changes: ```bash - helm history <release> -n <ns> --max 5 + helm history <release> -n <ns> ``` Note the newest revision with status `deployed` and its image tag. Do not @@ -49,9 +49,15 @@ Every production deploy, however small. ```bash gh workflow run deploy-dev.yml --ref main --repo Team-Commonly/commonly -f ref=<SHA> -gh run list --workflow=deploy-dev.yml -L 1 --repo Team-Commonly/commonly +gh run list --workflow=deploy-dev.yml --event workflow_dispatch -L 5 \ + --repo Team-Commonly/commonly --json databaseId,createdAt,status +gh run watch <run-id> --exit-status --repo Team-Commonly/commonly ``` +`gh workflow run` does not print the run it started. Take the run whose +`createdAt` matches your dispatch, and confirm its first job names `<SHA>` +before you watch it, so you never verify someone else's run. + The run is always dispatched on main; `ref` picks the commit it builds. Its first job refuses a `ref` that is not a full SHA in the repo, and any SHA without a green `staging-smoke`. An empty `ref` deploys main HEAD, which is not @@ -110,13 +116,21 @@ holds every PR inside the window, so a merge then is a deliberate press. ## 5. Rollback -Always name the revision; never run a bare `helm rollback`: +Always name the revision; never run a bare `helm rollback`. Pick ONE of +these, never both: ```bash +# Either: the whole release, to the Helm revision recorded at the cut helm rollback <release> <revision> -n <ns> + +# Or: one Deployment, to a ReplicaSet revision from its own history +kubectl rollout history deploy/<name> -n <ns> kubectl rollout undo deploy/<name> -n <ns> --to-revision=<N> ``` +The two use different histories: a Helm revision number is not a +`kubectl rollout` revision number. + - `helm rollback` to the revision recorded at the cut restores the whole release. Without a revision it goes to the previous one, which can be a failed upgrade or days stale; on 2026-08-04 a bare rollback would have @@ -149,12 +163,13 @@ is: When in doubt, ship the beta. -1. **Tag the deployed SHA.** After section 3 passes, the operator tags and - pushes: +1. **Tag the deployed SHA.** Use `vX.Y.Z` only when every gate item above is + green; otherwise tag the same SHA `vX.Y.Z-beta.N`. Section 3 passing is + necessary but not enough. The operator tags and pushes: ```bash - git tag -a vX.Y.Z <deploy-SHA> -m "vX.Y.Z — <title>" - git push origin vX.Y.Z + git tag -a <tag> <deploy-SHA> -m "<tag> — <title>" + git push origin <tag> ``` 2. **GitHub Release**, titled `vX.Y.Z — <title>` like the earlier ones: