Demo/read state

.github/scripts/build_pr_summary_comment.py

@@ -30,7 +30,9 @@ def build_comment_body(root: Path) -> str:
     for item in items:
         for warning in item.get("warnings", []):
             warnings.append(f"{item.get('name', 'unknown')}: {warning}")
-        sections.setdefault(item.get("section", "other"), []).append(item)
+        # Entries with no table data only contribute warnings.
+        if item.get("data"):
+            sections.setdefault(item.get("section", "other"), []).append(item)
 
     short_sha = ""
     for item in items:

.github/workflows/build-and-deploy.yml

@@ -289,6 +289,27 @@ jobs:
       postgres-db: ${{ needs.deploy-db.outputs.postgres-db }}
       postgres-sslmode: ${{ needs.deploy-db.outputs.postgres-sslmode }}
 
+  # Rebuilds the API's resource-state read model for this commit (a no-op if
+  # it's already current). Runs after the API deploy, which waits until the
+  # previous revision has stopped serving (wait-for-cutover): that revision's
+  # writes refresh rows with the old code's rules, and could otherwise land
+  # after this rebuild marks the table ready.
+  # Nothing waits on this job: until it finishes, list reads are served live
+  # (correct, just slower), so a slow or failed rebuild never blocks a deploy.
+  rebuild-read-model:
+    name: "Rebuild Read Model"
+    needs: [resolve-context, build-api-docker-image, deploy-db, run-db-migrations, deploy-api]
+    uses: ./.github/workflows/rebuild-read-model.yml
+    secrets:
+      stitch-app-password: ${{ secrets.STITCH_APP_PASSWORD }}
+    with:
+      deployment-lane: ${{ needs.resolve-context.outputs.deployment-lane }}
+      full-image-name: ${{ needs.build-api-docker-image.outputs.digest-image-name }}
+      postgres-host: ${{ needs.deploy-db.outputs.postgres-host }}
+      postgres-port: ${{ needs.deploy-db.outputs.postgres-port }}
+      postgres-db: ${{ needs.deploy-db.outputs.postgres-db }}
+      postgres-sslmode: ${{ needs.deploy-db.outputs.postgres-sslmode }}
+
   check-db-migrations-live:
     name: "Check DB Migrations (live)"
     needs: [resolve-context, build-api-docker-image, deploy-db, run-db-migrations]
@@ -337,6 +358,9 @@ jobs:
         OTEL_SERVICE_NAME=stitch-api
         OTEL_RESOURCE_ATTRIBUTES=deployment.name=${{ needs.resolve-context.outputs.deployment-name }},deployment.lane=${{ needs.resolve-context.outputs.deployment-lane }},service.version=${{ github.sha }}
       deployment-label: api
+      # The read-model rebuild that runs after this deploy must not overlap
+      # the previous API revision's writes.
+      wait-for-cutover: true
       # Only the production release stays warm; everything else scales to zero.
       min-replicas: ${{ needs.resolve-context.outputs.always-on }}
 
@@ -470,4 +494,4 @@ jobs:
       && !contains(github.event.pull_request.labels.*.name, 'no-deploy') }}
     uses: ./.github/workflows/comment-pr-summary.yml
     secrets: inherit
-    needs: [resolve-context, build-api-docker-image, build-stitch-llm-docker-image, build-seed-docker-image, deploy-db, run-db-migrations, deploy-api, deploy-stitch-llm, deploy-etl, run-seed, deploy-frontend]
+    needs: [resolve-context, build-api-docker-image, build-stitch-llm-docker-image, build-seed-docker-image, deploy-db, run-db-migrations, deploy-api, rebuild-read-model, deploy-stitch-llm, deploy-etl, run-seed, deploy-frontend]

.github/workflows/deploy-container.yml

@@ -68,6 +68,11 @@ on:
         required: false
         default: ""
         type: string
+      wait-for-cutover:
+        description: "Optional. After deploying, wait until every active revision runs this image (the previous revision no longer receives requests), plus time for its in-flight requests to finish. Use when something run after this deploy must not overlap the previous version's writes (e.g. the API's read-model rebuild). Fails the deploy if the new revision fails to provision or doesn't take over within 10 minutes."
+        required: false
+        default: false
+        type: boolean
     secrets:
       azure-client-id:
         required: true
@@ -345,6 +350,72 @@ jobs:
             --resource-group "${AZURE_RESOURCE_GROUP}" \
             --yaml /tmp/app-cooldown.yaml
 
+      - name: Wait for the new revision to replace the previous one
+        # The deploy action can return while the new revision is still
+        # provisioning, with the previous revision still serving requests (see
+        # the cooldown step above). Callers that set wait-for-cutover need the
+        # previous version to have stopped handling requests -- the API's
+        # read-model rebuild must not overlap the old code's writes (see
+        # deployments/api/READ_MODEL.md). Done when no active revision runs
+        # another image and one running this image has finished provisioning;
+        # then wait for the old revision's in-flight requests (deactivated
+        # replicas get a 30 s termination grace period by default).
+        if: ${{ inputs.wait-for-cutover }}
+        shell: bash
+        env:
+          AZURE_RESOURCE_GROUP: ${{ vars.AZURE_RESOURCE_GROUP }}
+          APP_NAME: ${{ inputs.container-app-name }}
+          IMAGE: ${{ inputs.full-image-name }}
+          TIMEOUT_SECONDS: "600"
+          POLL_SECONDS: "10"
+          DRAIN_SECONDS: "45"
+        run: |
+          set -euo pipefail
+          deadline=$(( $(date +%s) + TIMEOUT_SECONDS ))
+          az_errors="$(mktemp)"
+          while true; do
+            # A failed poll (ARM throttling, a network blip, an unexpected
+            # payload) is logged and retried until the deadline rather than
+            # failing the deploy; only the outcomes below end the step.
+            if ! revisions="$(
+              az containerapp revision list \
+                --name "$APP_NAME" \
+                --resource-group "$AZURE_RESOURCE_GROUP" \
+                --output json 2> "$az_errors"
+            )"; then
+              state="waiting: couldn't list revisions ($(tail -n 1 "$az_errors"))"
+            elif ! state="$(
+              jq -r --arg image "$IMAGE" '
+                [.[] | select(.properties.active)] as $active
+                | [$active[] | select(any((.properties.template.containers // [])[]; .image == $image))] as $new
+                | [$active[] | select(all((.properties.template.containers // [])[]; .image != $image))] as $old
+                | if any($new[]; .properties.provisioningState == "Failed") then "failed"
+                  elif ($old | length) == 0 and any($new[]; .properties.provisioningState == "Provisioned") then "done"
+                  else "waiting: \($old | length) active revision(s) on another image; \($new | length) on this image (\([$new[].properties.provisioningState] | join(", ")))"
+                  end
+              ' <<< "$revisions" 2>&1
+            )"; then
+              state="waiting: couldn't read the revision list ($state)"
+            fi
+            case "$state" in
+              done)
+                echo "Only revisions running $IMAGE are active; waiting ${DRAIN_SECONDS}s for the previous revision's in-flight requests."
+                sleep "$DRAIN_SECONDS"
+                exit 0
+                ;;
+              failed)
+                echo "::error::The new revision of $APP_NAME failed to provision; the previous revision is still serving."
+                exit 1
+                ;;
+            esac
+            if [ "$(date +%s)" -ge "$deadline" ]; then
+              echo "::error::The new revision of $APP_NAME did not replace the previous one within ${TIMEOUT_SECONDS}s ($state)."
+              exit 1
+            fi
+            echo "$state"
+            sleep "$POLL_SECONDS"
+          done
+
       - name: Export application URL
         id: export-app-url
         shell: bash

.github/workflows/python-test.yml

@@ -5,6 +5,26 @@ jobs:
   test:
     name: "tests (pytest)"
     runs-on: ubuntu-latest
+    # Throwaway Postgres for the API tests SQLite can't exercise (row locks under
+    # concurrency, JSONB): see deployments/api/tests/db/read_model/postgres.py.
+    # Each test creates and drops its own schema.
+    services:
+      postgres:
+        image: postgres:17-alpine
+        env:
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: stitch_test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres -d stitch_test"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+    env:
+      STITCH_TEST_POSTGRES_URL: postgresql+psycopg://postgres:postgres@localhost:5432/stitch_test
+      # Fail rather than skip if the URL above ever goes missing.
+      STITCH_REQUIRE_POSTGRES_TESTS: "1"
     steps:
       - name: Checkout code
         uses: actions/checkout@v7

.github/workflows/rebuild-read-model.yml

@@ -0,0 +1,121 @@
+---
+# Post-deploy step: rebuild the resource-state read model if it isn't already
+# built by this commit. The API only serves list/filter-options from the read
+# model when a full rebuild by its own commit (GIT_SHA, baked into the image)
+# has completed; until then it answers live (correct, just slower). Running the
+# deployed image here records the same commit the API runs.
+#
+# See deployments/api/READ_MODEL.md.
+on:
+  workflow_call:
+    inputs:
+      deployment-lane:
+        description: "Deployment lane and GitHub Environment name"
+        required: true
+        type: string
+      full-image-name:
+        description: "Full pushed API image name including host/registry, name, and digest"
+        required: true
+        type: string
+      postgres-host:
+        required: true
+        type: string
+      postgres-port:
+        required: false
+        default: "5432"
+        type: string
+      postgres-db:
+        required: true
+        type: string
+      postgres-sslmode:
+        required: false
+        default: "require"
+        type: string
+    secrets:
+      stitch-app-password:
+        required: true
+
+name: Rebuild Read Model
+
+jobs:
+  rebuild-read-model:
+    runs-on: ubuntu-latest
+    environment: ${{ inputs.deployment-lane }}
+    permissions:
+      contents: read
+      packages: read
+    timeout-minutes: 30
+
+    steps:
+      - name: Validate lane secrets
+        shell: bash
+        env:
+          STITCH_APP_PASSWORD: ${{ secrets.stitch-app-password }}
+        run: |
+          set -euo pipefail
+          : "${STITCH_APP_PASSWORD:?Missing required secret STITCH_APP_PASSWORD}"
+
+      - name: Log in to GHCR
+        run: echo "${{ github.token }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+
+      - name: Pull API image
+        run: docker pull "${{ inputs.full-image-name }}"
+
+      - name: Rebuild read model if stale
+        id: rebuild
+        env:
+          POSTGRES_HOST: ${{ inputs.postgres-host }}
+          POSTGRES_PORT: ${{ inputs.postgres-port }}
+          POSTGRES_DB: ${{ inputs.postgres-db }}
+          # Reads and rewrites rows only, so the app's own role is enough.
+          POSTGRES_USER: stitch_app
+          POSTGRES_PASSWORD: ${{ secrets.stitch-app-password }}
+          PGSSLMODE: ${{ inputs.postgres-sslmode }}
+          # A batch job: don't instrument every query for tracing.
+          OTEL_TRACES_EXPORTER: none
+        run: |
+          set -euo pipefail
+          docker run --rm \
+            -e POSTGRES_HOST \
+            -e POSTGRES_PORT \
+            -e POSTGRES_DB \
+            -e POSTGRES_USER \
+            -e POSTGRES_PASSWORD \
+            -e PGSSLMODE \
+            -e OTEL_TRACES_EXPORTER \
+            "${{ inputs.full-image-name }}" \
+            python -m stitch.api.db.read_model.rebuild --if-stale
+
+      # Only when the rebuild didn't succeed (failed, cancelled, or never ran):
+      # a warning-only entry for the PR summary comment, so reviewers know the
+      # preview's lists are on the slower live path.
+      - name: Write comment JSON
+        if: ${{ always() && github.event_name == 'pull_request' && steps.rebuild.outcome != 'success' }}
+        shell: bash
+        env:
+          OUTCOME: ${{ steps.rebuild.outcome }}
+        run: |
+          set -euo pipefail
+          mkdir -p /tmp/comment-json
+          json_file="/tmp/comment-json/comment-json-read-model.json"
+
+          jq -n \
+            --arg outcome "$OUTCOME" \
+            '{
+              "kind": "read_model_rebuild",
+              "name": "rebuild-read-model",
+              "warnings": [
+                "the read model was not rebuilt (\($outcome)); list reads use the slower live path until it is"
+              ]
+            }' > "$json_file"
+
+          jq empty "$json_file"
+          cat "$json_file"
+
+      - name: Upload comment JSON artifact
+        if: ${{ always() && github.event_name == 'pull_request' && steps.rebuild.outcome != 'success' }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: comment-json-read-model
+          path: /tmp/comment-json/*.json
+          if-no-files-found: error

HACKING.md

@@ -96,6 +96,29 @@ Depending on which entrypoint you use, these are the main local endpoints:
 * API docs: `http://localhost:8000/docs`
 * Adminer: `http://localhost:8081`
 
+## Fast list reads locally
+
+The list and filter-options endpoints read from a precomputed table, but only
+after a full rebuild *by the same code* has marked it ready; until then they use
+the slower live path, with the same results. Run the rebuild where the API runs:
+
+- API in Docker (`make frontend-dev`, `make dev-docker`): the image records your
+  commit, so rebuild inside it, and again after rebuilding the image:
+
+  ```bash
+  docker compose exec api python -m stitch.api.db.read_model.rebuild
+  ```
+
+- API on your machine (`make api-dev`): the code version is `unknown`, so a
+  rebuild stays valid until you rebuild again (do so after changing how answers
+  are computed):
+
+  ```bash
+  POSTGRES_HOST=127.0.0.1 POSTGRES_USER=stitch_app uv run --env-file .env -- python -m stitch.api.db.read_model.rebuild
+  ```
+
+See [`deployments/api/READ_MODEL.md`](deployments/api/READ_MODEL.md) for details.
+
 ## Other useful targets
 
 You do not need to memorize the whole Makefile, but these are worth knowing.

deployments/CI_DEPLOYMENTS.md

@@ -324,7 +324,15 @@ deployments, `pr-*`, `main` deploy to the same PostgreSQL instance, with
 separate logical DBs)
 
 It also handles running `db-migrations` (via Alembic in the `api` image) and
-`seed`, both directly from GH Actions (rather than starting on Azure).
+`seed`, both directly from GH Actions (rather than starting on Azure). The API
+deploy waits until its new revision has fully replaced the previous one
+(`wait-for-cutover` in `deploy-container.yml`; it fails the deploy if the new
+revision fails to provision or doesn't take over within 10 minutes). After the
+API deploy, `rebuild-read-model` runs the API's resource-state read-model
+rebuild from the same `api` image (a no-op when it's already current for this
+commit); nothing waits on it, so a failure leaves list reads on the slower live
+path without blocking the deploy. See
+[`api/READ_MODEL.md`](./api/READ_MODEL.md).
 
 ### Custom domains (RMI hostnames)
 

deployments/api/README.md

@@ -1,5 +1,13 @@
 # api
 
+## Resource-state read model
+
+The list and filter-options endpoints are served from a precomputed table that
+must be **rebuilt after each deploy of new code** (CI does this automatically)
+and after a data restore. Until then they use the slower live path, with the
+same results. See [`READ_MODEL.md`](./READ_MODEL.md) for rollout, rebuild, and
+verification steps.
+
 ## Deployment: Azure Container App
 
 ### Prerequisites