feat: backup job deployment script added (STIT-787) by mbarlow12 · Pull Request #325 · RMI/stitch

mbarlow12 · 2026-10-13T16:48:15Z

Summary

Adds a developer focused script to deploy/update a minimal ACA Job that dumps a single db then uploads it to blob storage. This allows us to easily update the job config and quickly redeploy the job, we can also be a little more agile in quickly setting up temporary backups. But it's also a good faith effort to more narrowly implement the ACs, that also highlighted a small tension in our CI/CD workflows: we can't easily run them ourselves locally.

I realized that it's usually pretty straightforward to automate a local script in CI but the inverse isn't necessarily true at all, so this is a bit of a plug to reassess how we've designed our CI/CD flows and consider piecemeal if there's value to shifting the approach to "locally runnable first" -> "automated in CI/CD".

Details

The job backs up 1 db at a time, uses prebuilt images (that we don't need to maintain), and runs nightly
Locally, it relies on user permissions through az and will fail if the user doesn't have the required Azure capabilities (This seemed like a small concerns since it'll be me and Alex almost exclusively)
The job configuration yaml is housed alongside the script. Though it's technically another deployment, it seemed like overkill to add an entire new deployments/ directory for something so small.
Currently no "destroy" option, but that can still be done through Azure Portal or az

Related issues

Closes: STIT-787

Testing

Tested locally with a manually triggered job config
confirmed the resources were created
confirmed that the sql dump was written to the expected location
successfully restored local db from dump

AI Usage

Claude drafted early iteration & handled git commands to split feat: automated db backups #315
Subsequent edits were a mix of meatbag updates and targeted agent guidance
Claude authored docs update. I reviewed all copy and was satisfied (yay simple-english skill)

Checklist

PR is focused on a single concern
~~- [ ] Tests pass locally and in CI~~
Docs updated for user-visible changes
AI-assisted portions declared (see CONTRIBUTING.md)

github-actions · 2026-10-13T16:50:58Z

CD summary `cb8e9ae`

Deployments (1)

service	url	fqdn
api	open	`pr-0325-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io`

Database (1)

db_name	postgres_host	postgres_port	postgres_db
pr_0325	`stitch-dev.postgres.database.azure.com`	`5432`	`pr_0325`

Jobs (1)

job	image	postgres_db
db-migrations	`ghcr.io/rmi/stitch-api:pr-0325@sha256:04b242e2290a024a4f7adfe2a9f4681e45123a1474b79edba2ae6511a1a05c97`	`pr_0325`

Images (3)

build_time	commit_time	git_sha	image	image_digest
2026-10-13T16:48:37Z	2026-10-13T16:48:15Z	`c7b5627`	`ghcr.io/rmi/stitch-api:pr-0325`	`ghcr.io/rmi/stitch-api:pr-0325@sha256:04b242e2290a024a4f7adfe2a9f4681e45123a1474b79edba2ae6511a1a05c97`
2026-10-13T16:48:37Z	2026-10-13T16:48:15Z	`c7b5627`	`ghcr.io/rmi/stitch-seed:pr-0325`	`ghcr.io/rmi/stitch-seed:pr-0325@sha256:51f9ec738f8a3200ea0e0a44f3c0a337f26529b2c8b94b9bac5e8868b766a212`
2026-10-13T16:48:33Z	2026-10-13T16:48:15Z	`c7b5627`	`ghcr.io/rmi/stitch-stitch-llm:pr-0325`	`ghcr.io/rmi/stitch-stitch-llm:pr-0325@sha256:cfdc79106b7b6dfd456378eeda1ab1879e321053db8c984f231fdcabbf4ff32b`

github-actions · 2026-10-13T16:55:20Z

CD summary `abaf22b`

Frontend: https://witty-mushroom-017a3dc1e-325.westus2.1.azurestaticapps.net

Deployments (3)

service	url	fqdn
api	open	`pr-0325-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io`
frontend	https://witty-mushroom-017a3dc1e-325.westus2.1.azurestaticapps.net
stitch-llm	open	`pr-0325-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io`

Database (1)

db_name	postgres_host	postgres_port	postgres_db
pr_0325	`stitch-dev.postgres.database.azure.com`	`5432`	`pr_0325`

Jobs (1)

job	image	postgres_db
db-migrations	`ghcr.io/rmi/stitch-api:pr-0325@sha256:7629ea99d863386a6884c5cafebb03f1e4d1143dbb2f1f9c0260eeaf6bb20fd8`	`pr_0325`

Images (3)

build_time	commit_time	git_sha	image	image_digest
2026-10-13T16:51:17Z	2026-10-13T16:50:07Z	`53e19b7`	`ghcr.io/rmi/stitch-api:pr-0325`	`ghcr.io/rmi/stitch-api:pr-0325@sha256:7629ea99d863386a6884c5cafebb03f1e4d1143dbb2f1f9c0260eeaf6bb20fd8`
2026-10-13T16:51:14Z	2026-10-13T16:50:07Z	`53e19b7`	`ghcr.io/rmi/stitch-seed:pr-0325`	`ghcr.io/rmi/stitch-seed:pr-0325@sha256:783999078a6b3eb52152080a911e9d7dc6fc0a8898edff6a897b54f50398f399`
2026-10-13T16:51:14Z	2026-10-13T16:50:07Z	`53e19b7`	`ghcr.io/rmi/stitch-stitch-llm:pr-0325`	`ghcr.io/rmi/stitch-stitch-llm:pr-0325@sha256:2b0bf4d81da2ddf35a00d88c4d400e60bf3883b1b27b00f9a94a01b6ce0d4b4b`

github-actions · 2026-10-13T17:28:07Z

CD summary `9cea883`

Deployments (2)

service	url	fqdn
api	open	`pr-0325-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io`
stitch-llm	open	`pr-0325-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io`

Database (1)

db_name	postgres_host	postgres_port	postgres_db
pr_0325	`stitch-dev.postgres.database.azure.com`	`5432`	`pr_0325`

Jobs (1)

job	image	postgres_db
db-migrations	`ghcr.io/rmi/stitch-api:pr-0325@sha256:568c9545fb6ec1d1cf090d1e20a3621a9a80fd9315f7abee3a3f3277550ad853`	`pr_0325`

Images (3)

build_time	commit_time	git_sha	image	image_digest
2026-10-13T17:22:15Z	2026-10-13T17:21:55Z	`d9e415e`	`ghcr.io/rmi/stitch-api:pr-0325`	`ghcr.io/rmi/stitch-api:pr-0325@sha256:568c9545fb6ec1d1cf090d1e20a3621a9a80fd9315f7abee3a3f3277550ad853`
2026-10-13T17:22:13Z	2026-10-13T17:21:55Z	`d9e415e`	`ghcr.io/rmi/stitch-seed:pr-0325`	`ghcr.io/rmi/stitch-seed:pr-0325@sha256:64193084ecc67cf93d1e128fabb57b56c7f740bcfe3af571f987bc807e5c000a`
2026-10-13T17:22:12Z	2026-10-13T17:21:55Z	`d9e415e`	`ghcr.io/rmi/stitch-stitch-llm:pr-0325`	`ghcr.io/rmi/stitch-stitch-llm:pr-0325@sha256:639f240e297eebe106fec4e4c6710cc6f789483d1b47cc31ff1d94f357734ff3`

AlexAxthelm

overall a solid concept, but the big things I would change are:

Center around an on-demand backup action/job (that we can run whenever, but gets scripted to run nightly)
Scheduling in GH not Azure
Not depending on EmptyDir (which will be insufficient when we get bigger data)

AlexAxthelm · 2026-10-16T14:41:20Z

+ROLE="Storage Blob Data Contributor"
+SPEC="$(cd "$(dirname "$0")" && pwd)/templates/db-backup-job-config.yml"
+
+resource_group=


style: I generally prefer envvars over script args, mostly out of a mix of "parsing script args can get weird" and "most scripts usually get run in a place where temporarily setting envvars is easy"

The intent is to make this locally runnable. The almost ubiquitous pattern is to use both where CLI args/options take precedence. For the sake of brevity, I opted to skip env vars since those can still be used in a GH runner, but env vars locally require export MY_VAR="..." or reading from a specified env file.

AlexAxthelm · 2026-10-16T15:05:41Z

+    initContainers:
+      - name: dump
+        image: postgres:17-alpine
+        resources: { cpu: 0.5, memory: 1.0Gi }
+        volumeMounts:
+          - { volumeName: scratch, mountPath: /scratch }
+        env:
+          - { name: PGHOST, value: "${JOB_PGHOST}" }
+          - { name: PGUSER, value: "${JOB_PGUSER}" }
+          - { name: PGDATABASE, value: "${JOB_DATABASE}" }
+          - { name: PGSSLMODE, value: require }
+          - { name: PGPASSWORD, secretRef: pgpassword }
+        command: ["pg_dump"]
+        args: ["-Fc", "-f", "/scratch/dump"]


This is a good use of init containers, but I'm not a fan of using EmptyDir for communication of large files, because it can be hard to reason about, espescially for out-of-space concerns. Not an immediate problem, but also one that we probably will have to face at some point (for example, since we have a 0.5 cpu instance, this whole thing will fail in the init if the dup is bigger than 2 GiB) https://learn.microsoft.com/en-us/azure/container-apps/storage-mounts?tabs=smb&pivots=azure-cli#ephemeral-storage

Would prefer to handle everything on one instance, if possible

Current db is <200mb, we have some time before the size is a problem. The separate containers mean we don't need to manage our own images (no Dockerfile, no build, no push to registry).

AlexAxthelm · 2026-10-16T15:08:23Z

+    scheduleTriggerConfig:
+      cronExpression: "0 3 * * *"
+      parallelism: 1
+      replicaCompletionCount: 1


Could we move scheduling out of Azure Container Jobs and into GH Actions? This is mostly one where we aren't currently using Jobs (for anything), and having a scheduled action there is a new control surface (and place to check if jobs failed), while GH Actions is something we're already using daily, so backup failure notifications will be surfaced along with other failed actions.

If we want visibility in GH, why not use GH actions to poll the logs? Beyond dependabot and admin checks (neither of which touch azure), we're not using scheduled workflows in GH.

Regarding our azure resources, az is still the control plane, not GitHub. GH connects repo events to workflows/actions (that execute az commands), and this is the first real instance where there's overlapping capability between GH runners and our Azure resources. Moving configuration into the resource itself let's GH treat all deployments equivalently, it doesn't distinguish between how to deploy a Container Application vs how to deploy an ACA Job, it's their respective configuration that determines the details appropriate for that resource.

…ript

- add `run` to start a backup on demand, leaving the schedule in place - render the job spec with bash parameter expansion and {{ var }} placeholders - read the identity with --output json instead of splitting tsv with IFS - de-duplicate the job-name default and the job-existence probe - pass the blob container to the template instead of naming it in two files

mbarlow12 added 2 commits October 13, 2026 10:37

chore: ignore local env files and script working data

3755e19

feat(db): add deploy script and spec for the nightly backup job

cb8e9ae

mbarlow12 deployed to development October 13, 2026 16:48 — with GitHub Actions Active

mbarlow12 deployed to development October 13, 2026 16:49 — with GitHub Actions Active

mbarlow12 had a problem deploying to development October 13, 2026 16:49 — with GitHub Actions Error

mbarlow12 deployed to development October 13, 2026 16:49 — with GitHub Actions Active

mbarlow12 changed the title ~~Feat/backup deploy script~~ feat: backup job deployment script added (STIT-787) Oct 13, 2026

refactor: move the backup job spec next to its deploy script

abaf22b

mbarlow12 deployed to development October 13, 2026 16:51 — with GitHub Actions Active

mbarlow12 deployed to development October 13, 2026 16:52 — with GitHub Actions Active

mbarlow12 deployed to development October 13, 2026 16:53 — with GitHub Actions Active

mbarlow12 deployed to development October 13, 2026 16:54 — with GitHub Actions Active

docs(db): document the backup job deploy script

9cea883

mbarlow12 deployed to development October 13, 2026 17:22 — with GitHub Actions Active

mbarlow12 deployed to development October 13, 2026 17:23 — with GitHub Actions Active

mbarlow12 marked this pull request as ready for review October 13, 2026 17:25

mbarlow12 had a problem deploying to development October 13, 2026 17:26 — with GitHub Actions Error

mbarlow12 added no-deploy dev-tools labels Oct 13, 2026

mbarlow12 deployed to development October 13, 2026 17:26 — with GitHub Actions Active

mbarlow12 requested a review from AlexAxthelm October 13, 2026 17:26

docs(db): tighten the backups section and cover job deletion

a9a7e32

mbarlow12 mentioned this pull request Oct 13, 2026

feat(ci): add GH action for calling backup deploy script #326

Open

3 tasks done

feat(db): add a delete subcommand to the backup job script

1f10840

mbarlow12 added this pull request to stack #327 October 14, 2026 04:39

AlexAxthelm requested changes Oct 16, 2026

View reviewed changes

mbarlow12 added 3 commits October 16, 2026 11:33

Merge remote-tracking branch 'origin/main' into feat/backup-deploy-sc…

612692a

…ript

rename script

d8e261c

Navigation Menu

Repository navigation

feat: backup job deployment script added (STIT-787) - #325

feat: backup job deployment script added (STIT-787)#325
mbarlow12 wants to merge 9 commits into
mainfrom
feat/backup-deploy-script

mbarlow12 commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

AlexAxthelm left a comment

Uh oh!

AlexAxthelm Oct 16, 2026

Uh oh!

mbarlow12 Oct 16, 2026

Uh oh!

Uh oh!

AlexAxthelm Oct 16, 2026

Uh oh!

mbarlow12 Oct 16, 2026

Uh oh!

AlexAxthelm Oct 16, 2026

Uh oh!

mbarlow12 Oct 16, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mbarlow12 commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Related issues

Testing

AI Usage

Checklist

Uh oh!

github-actions Bot commented Oct 13, 2026

CD summary cb8e9ae

Uh oh!

github-actions Bot commented Oct 13, 2026

CD summary abaf22b

Uh oh!

github-actions Bot commented Oct 13, 2026

CD summary 9cea883

Uh oh!

AlexAxthelm left a comment

Choose a reason for hiding this comment

Uh oh!

AlexAxthelm Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

mbarlow12 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AlexAxthelm Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

mbarlow12 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

AlexAxthelm Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

mbarlow12 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mbarlow12 commented Oct 13, 2026 •

edited

Loading

CD summary `cb8e9ae`

CD summary `abaf22b`

CD summary `9cea883`