Skip to content

Feat/stit 766 resource state - #324

Closed
AlexAxthelm wants to merge 41 commits into
mainfrom
feat/stit-766-resource-state
Closed

Feat/stit 766 resource state#324
AlexAxthelm wants to merge 41 commits into
mainfrom
feat/stit-766-resource-state

Conversation

@AlexAxthelm

@AlexAxthelm AlexAxthelm commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Introduce a new read model table, which holds the current coalesced state of resources (winners only), to allow for fast reads on the list and filter-options endpoints. The table grain also includes "permission sets", to account for the possibility that the winning coalesced values may be different for different users.

The "live" coalescer still exists, and is still used for the details page, and also for the list and filter-options endpoints for requests that have JWTs that have non-standard permission sets (in practice, this shouldn't happen). This preserves the "you can only see data that you have been explicitly granted read permission for" behavior, but makes use of the fact that all the actual tokens we use will be a combination of [6 public sources] + (maybe wm) + (maybe ccr), so those collapse to 4 actual profiles.

The table is kept up to date by the API's application logic (where the updating queries are defined), on all the resource operations (merge/attach/create/etc), so after a rebuild, the application should keep the whole table up to date.

Cache Invalidation & Rebuild: Because the new read table is functionally a cache, it needs to be rebuilt sometimes. There is a full-rebuild helper (python -m stitch.api.db.read_model.rebuild) that includes some helpful options like --if-stale, see docs/script for details.
NOTE: The code uses the git SHA as the validity key, and as a consequence of that, runs the rebuild script on every new deploy (busy on PRs, but mandatory on PROD)

Readiness: this also includes a OGFieldResourceStateStatus table, which the application uses to track the bookkeeping/status of the read model

  • Run tests on CI in postgres (some PG specific functionality introduced)
  • Some adjustments to CI:
    • Deploy container action now optionally waits for cutover to complete before reporting green

Summary

The oil & gas field list and filter-options endpoints now read from a precomputed, permission-scoped table (og_field_resource_state) instead of working out every field's winning value on each request. At ~50K resources, list requests drop from ~2–3 s to under 10 ms and filter-options from ~0.5 s to ~20 ms. The results are identical.

The table is a cache, not a source of truth. When it can't be trusted, the API answers the existing way ("live"), with the same results. The runbook is deployments/api/READ_MODEL.md; start there.

How it works

  • One row per resource per permission profile. There are four profiles: all public sources, plus each combination of the restricted wm / ccr. Each row holds every field's winning value and which source it came from. A request whose licensed sources exactly match a profile gets the fast path; any other grant set gets live.
  • Kept current by writes. Every write that can change a list answer (creating or attaching sources, re-ordering a field's sources, merging) recomputes the affected resources' rows in the same transaction.
    • Those writes now lock their resources first, in id order, to avoid deadlocks between concurrent writers.
    • The detail endpoint and per-field source listings are unchanged; they always compute live.
  • Readiness is tied to the commit that built it. The API uses the table only when a complete full rebuild by the same commit it is running (GIT_SHA) has finished. Each row also records the commit that computed it. As a result:
    • every deploy of new code reads live until that code has rebuilt the table, so a change to how answers are computed is never served from older rows;
    • rows written by an old revision during a deploy are found and redone.
  • Operator switch-off: UPDATE og_field_resource_state_status SET disabled = true; sends reads back to live within ~5 s. It stays off across rebuilds and deploys until it is set back to false.

Rebuild CLI

python -m stitch.api.db.read_model.rebuild:

  • Full rebuild (rebuild): works in committed batches and never empties the table, so it is safe while the API is serving. It takes ~97 s at 50K.
  • --if-stale (what CI runs): only recomputes rows not already computed by this code, so it resumes after a failed or cancelled run.
  • --ids: repairs specific resources.
  • No overlapping rebuilds: a Postgres advisory lock prevents two from running at once. --if-stale waits up to 10 minutes for a running rebuild, then checks again.

CI / deploy

  • New Rebuild Read Model job (rebuild-read-model.yml): runs rebuild --if-stale from the deployed API image after the API deploy. If it fails, the deploy still goes out, lists stay on live, and the PR summary comment shows a warning.
  • deploy-container.yml gains wait-for-cutover (used for the API): the deploy step finishes only once the previous revision has stopped serving, plus 45 s for in-flight requests. This keeps the old code's writes from landing after the rebuild marks the table ready.
  • python-test.yml gets a Postgres service for the tests SQLite can't exercise: row locks under concurrency and JSONB. These tests fail rather than skip if the database URL goes missing.

Migrations

  • 76427dd73cbc: the read-model and status tables
  • ebd0fff3d941: the status row's code_version
  • 2a1562956888: each row's code_version
  • eb52164663b1: the status row's disabled flag

The tables start empty and not ready, so the API reads live until the first rebuild (CI runs it after deploy). The downgrades refuse to run.

Also on this branch

The design docs this work was planned from: docs/STIT-766-design.md, docs/draft-schema-operations-api.md, docs/draft-schema.sql.

Related issues

Closes: STIT-766
Follow-up: STIT-809 (incremental --since repair)

Related: #322. Its unique (resource_id, source_pk) index on og_field_memberships cuts the read model's per-write refresh from ~85 ms to ~16 ms at 50K. The two PRs will conflict; that will be resolved when #322 lands.

Testing

  • Automated:
    • New tests under deployments/api/tests/db/read_model/ cover:
      • write paths: every one, checked against the live answer;
      • permission profiles and the read path: fast path vs. live fallback (not ready, built by other code, switched off);
      • rebuilds: batched, resumable, repeated passes for rows rewritten by other code;
      • --ids guards.
    • Postgres-only tests cover concurrent writers (no deadlocks, no lost refreshes), a read during a rebuild, the rebuild lock, and keepalives on the lock's connection.
    • Full API suite: 686 passed locally with Postgres; make check passes.
    • New behaviour was mutation-checked: each new guard was broken by hand to confirm a test fails.
  • Migrations: applied to a fresh SQLite database and to two Postgres databases; the models match the migrations.
  • 50K benchmark: results are in the runbook's Performance section. It was measured against a restored copy of the 50K-resource seed, and live and read-model answers were compared for equality.
    • A rebuild killed partway through (18,500 resources) was resumed by --if-stale, which recomputed only the remaining 33,517.
  • Deploy: a real PR deploy showed the old revision still active when the deploy action returned; the cutover wait held until it stopped.
  • Not covered by automated tests: the cutover check's jq filter (the repo has no home for CI-script tests yet). It was checked against simulated revision lists locally and in the real deploy above. If it breaks, API deploys would time out after 10 minutes.

Checklist

  • PR is focused on a single concern
  • Tests pass locally and in CI
  • Docs updated for user-visible changes
  • AI-assisted portions declared (see CONTRIBUTING.md)

AI use: Claude Code wrote most of the implementation, tests and docs, in small steps. I reviewed each step before committing, and several review passes (Claude and GPT) fed the later fixes. Verified by: the tests above, the 50K benchmark, and the real PR deploy.

🤖 Generated with Claude Code

mbarlow12 and others added 18 commits September 17, 2026 20:25
@mbarlow12
initial draft
91b1223
@mbarlow12
structural revisions, and wording
3bbbe6a
@mbarlow12
shorten sections, add optional tables and query/API explanation
ed07a6f
@mbarlow12
delete outdated/unused files
bf8e329
@mbarlow12
delete mermaid test doc
8399666
@mbarlow12
fix section numbering
3799140
@mbarlow12
Merge remote-tracking branch 'origin/main' into design/db-state-refactor
ce2e06f
@mbarlow12
rename supporting docs
761c823
@mbarlow12
Merge branch 'main' into design/db-state-refactor
cf21ca3
@mbarlow12 @AlexAxthelm
Update docs/STIT-766-design.md
c2b2e63 
Co-authored-by: Alex Axthelm <alex@alexaxthelm.com>
@AlexAxthelm
Merge branch 'main' into design/db-state-refactor
a518b5b
@AlexAxthelm @claude
Add PLAN
00089a8 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: ChatGPT noreply@openai.com
@AlexAxthelm @claude
feat(api): add resource-state read model schema
f600032 
Add og_field_resource_state (one typed row per resource x permission
profile, plus provenance) and its single-row readiness record, with a
schema-only migration that starts not-ready. Declare the four exact
permission profiles explicitly; any other grant set falls back to live
coalescing. Guard value columns against ATTRIBUTE_NAMES and the source
universe against OGSISrcKey so drift fails loudly.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexAxthelm @claude
feat(api): build and rebuild resource read state
6955833 
Compute og_field_resource_state with the live coalescer, once per exact
permission profile, so cached and live ranking cannot drift. Targeted
refreshes lock the resource rows in id order before recomputing, inside
the caller's transaction, and drop rows for resources that stop being
listable. A full rebuild clears readiness, rebuilds in committed batches,
integrity-checks the result and only then marks the model ready; a
Postgres advisory lock keeps rebuilds from overlapping. Add a CLI for
full rebuilds and --ids repairs.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexAxthelm @claude
feat(api): refresh resource state on mutations
6266b7f 
Recompute a resource's read-model rows in the same transaction as every
write that can change its coalesced answer: attaching sources (all
attach paths share _attach_source_models), re-prioritizing a field, and
merging. Merge now locks the originals in id order and re-reads them
before checking they are mergeable, so overlapping merges serialize
instead of deadlocking or double-merging a resource.

Add mutation-vs-recompute parity tests for each write path, an
in-place-source-edit contract test, lock-ordering checks that run on
SQLite, and opt-in Postgres tests (STITCH_TEST_POSTGRES_URL) that
exercise real concurrent writers.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexAxthelm @claude
perf(api): serve resource list reads from precomputed state
ed5952c 
When the caller's licensed sources exactly match a read-model profile
and the model is ready, list and filter-options read the stored,
already-coalesced rows: filters and sort reuse the shared clause
builders on typed columns, and filter-options is a profile-scoped
DISTINCT per field. Every other caller, and any not-ready state, uses
the unchanged live path. Detail stays live. Wire models and query
labels are unchanged; list provenance is re-keyed in attribute order
because JSONB does not preserve key order.

Add read-parity tests against live for each profile, scenario and sort,
fallback tests (including partially populated, not-ready state),
byte-identical HTTP responses, label checks for the fast path, and an
opt-in Postgres serialization-parity test.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexAxthelm @claude
docs(api): document resource-state rollout and verification
ca004df 
Add a runbook for the resource-state read model: how it works, the
required post-deploy rebuild, verification SQL, when a rebuild is needed
again, rules for future write paths/fields/source keys, the opt-in
Postgres tests, 50K-resource measurements and deferred findings. Link it
from the API README and HACKING.

From the 50K query plans: count read-model list results without the
ORDER BY (Postgres otherwise sorts every matching row just to count),
and turn off per-statement SQL echo in the rebuild CLI, which would
otherwise log every statement and parameter of a full rebuild outside
prod.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexAxthelm
chore: rm PLAN.md
1abd90c
@AlexAxthelm
AlexAxthelm changed the base branch from main to design/db-state-refactor October 12, 2026 15:50
@github-actions

github-actions Bot commented Oct 12, 2026

Copy link
Copy Markdown

CD summary 1abd90c

Deployments (3)
service url fqdn
api open pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
entity-linkage open pr-0324-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io
stitch-llm open pr-0324-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0324 stitch-dev.postgres.database.azure.com 5432 pr_0324
Jobs (2)
job image postgres_db api_url auth_mode
db-migrations ghcr.io/rmi/stitch-api:pr-0324@sha256:420b920d6d15bef2b5fd05e38aeb20ee593dd7260af1e08e4b5122399a334cc1 pr_0324
seed ghcr.io/rmi/stitch-seed:pr-0324@sha256:fbb3d47d9381b539991f9c5cef22a02ff2a6099057635f9b3b5d0388450a98d0 https://pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 stitch-client-bearer-token
Images (4)
build_time commit_time git_sha image image_digest
2026-10-12T15:50:31Z 2026-10-12T15:50:10Z 0cdda1a ghcr.io/rmi/stitch-api:pr-0324 ghcr.io/rmi/stitch-api:pr-0324@sha256:420b920d6d15bef2b5fd05e38aeb20ee593dd7260af1e08e4b5122399a334cc1
2026-10-12T15:50:31Z 2026-10-12T15:50:10Z 0cdda1a ghcr.io/rmi/stitch-entity-linkage:pr-0324 ghcr.io/rmi/stitch-entity-linkage:pr-0324@sha256:62bbc669dc8db7922697d99c21aa89586768821470c636bad8b8d15fcf3339cf
2026-10-12T15:50:32Z 2026-10-12T15:50:10Z 0cdda1a ghcr.io/rmi/stitch-seed:pr-0324 ghcr.io/rmi/stitch-seed:pr-0324@sha256:fbb3d47d9381b539991f9c5cef22a02ff2a6099057635f9b3b5d0388450a98d0
2026-10-12T15:50:32Z 2026-10-12T15:50:10Z 0cdda1a ghcr.io/rmi/stitch-stitch-llm:pr-0324 ghcr.io/rmi/stitch-stitch-llm:pr-0324@sha256:824f3dfd4f94fe371f04616024a5e043ad37936f1a41ec2c32863dfff5e07195

@AlexAxthelm @claude
fix(api): lock the resource before writing to avoid refresh deadlocks
a3c9e4c 
Attach and field re-prioritization took the resource's FOR UPDATE lock
only when refreshing read-model state, after inserting rows whose
foreign keys already hold FOR KEY SHARE on the same row. Two concurrent
writers could each hold that and wait to upgrade it, which Postgres
resolves by aborting one (a 500). Lock the resource at the start of
both actions instead, and re-read it after locking, which also stops an
attach that waited on a merge from attaching to the merged-away
resource.

Add Postgres tests that hold both writers just before their refresh
(attach/attach, attach/reprioritize, reprioritize/reprioritize,
attach-after-merge), make the no-lock collision test deterministic, and
require the lock to be the first thing each write action does.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

CD summary 928442b

Frontend: https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net

Deployments (3)
service url fqdn
api open pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net
stitch-llm open pr-0324-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0324 stitch-dev.postgres.database.azure.com 5432 pr_0324
Jobs (1)
job image postgres_db
db-migrations ghcr.io/rmi/stitch-api:pr-0324@sha256:948499356820d2ca42ed2439e52903bfe4b0bffc18f64284a77d6eb2cd3d24fe pr_0324
Images (3)
build_time commit_time git_sha image image_digest
2026-10-14T10:45:41Z 2026-10-14T10:45:26Z 8f87704 ghcr.io/rmi/stitch-api:pr-0324 ghcr.io/rmi/stitch-api:pr-0324@sha256:948499356820d2ca42ed2439e52903bfe4b0bffc18f64284a77d6eb2cd3d24fe
2026-10-14T10:45:47Z 2026-10-14T10:45:26Z 8f87704 ghcr.io/rmi/stitch-seed:pr-0324 ghcr.io/rmi/stitch-seed:pr-0324@sha256:44daee9529168050a7cc5fc0ed1310b54e336cb87bcad4cb8f1b09247b04c92b
2026-10-14T10:45:44Z 2026-10-14T10:45:26Z 8f87704 ghcr.io/rmi/stitch-stitch-llm:pr-0324 ghcr.io/rmi/stitch-stitch-llm:pr-0324@sha256:4cdf05fa09092c6b3fcea712559d0603e657eca54e2a6ef43bfef21db6d3d6f6

@AlexAxthelm @claude
don't freeze the shared clock in the readiness-cache test; docs
f621062 
The readiness-cache test simulated the window passing by patching
time.monotonic, which the asyncio event loop shares. Shrink
READINESS_CACHE_SECONDS to 0 instead. In the runbook, mention the PR
summary warning for a failed rebuild, and that clearing readiness takes
effect within a few seconds (the readiness cache), not immediately.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

CD summary f621062

Frontend: https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net

Deployments (3)
service url fqdn
api open pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net
stitch-llm open pr-0324-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0324 stitch-dev.postgres.database.azure.com 5432 pr_0324
Jobs (1)
job image postgres_db
db-migrations ghcr.io/rmi/stitch-api:pr-0324@sha256:64c8c141f0ad44de1c54e25c7641641acb1ed6d398d1305764d67e2f6c281278 pr_0324
Images (3)
build_time commit_time git_sha image image_digest
2026-10-14T11:09:41Z 2026-10-14T11:09:25Z 5cc1713 ghcr.io/rmi/stitch-api:pr-0324 ghcr.io/rmi/stitch-api:pr-0324@sha256:64c8c141f0ad44de1c54e25c7641641acb1ed6d398d1305764d67e2f6c281278
2026-10-14T11:09:39Z 2026-10-14T11:09:25Z 5cc1713 ghcr.io/rmi/stitch-seed:pr-0324 ghcr.io/rmi/stitch-seed:pr-0324@sha256:c039f4ebf7dd3d634a36614b0a3c3f0987ecf36e15913f635a6891e7c3ae4339
2026-10-14T11:09:40Z 2026-10-14T11:09:25Z 5cc1713 ghcr.io/rmi/stitch-stitch-llm:pr-0324 ghcr.io/rmi/stitch-stitch-llm:pr-0324@sha256:76af5b75f66d52ba3414eda9f81371684e891a0f3178a58fbd365099b3cc5c65

AlexAxthelm and others added 3 commits October 14, 2026 13:21
@AlexAxthelm @claude ChatGPT 5.5
refuse an --ids repair from code other than the served build
faacb70 
The API trusts a ready read model only as built entirely by its own
commit, but a targeted --ids repair recomputed rows with whatever code
ran it and left the status untouched, so rows from another version
could be served under the wrong commit. When the table is ready and was
built by different code, refuse the repair (nothing changed, status 1)
instead of rewriting rows; a share lock on the status row keeps
readiness from flipping mid-repair. Not-ready tables, which aren't
served, can still be repaired from any code.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: ChatGPT 5.5 <noreply@oopenai.com>
@AlexAxthelm @claude
record which code computed each read-model row
5556422 
During a deploy the previous API revision can keep serving writes while
the new commit rebuilds the read model; its refreshes rewrite rows with
the old code's rules, and the rebuild would then mark the table ready
for the new commit with those rows in it. Stamp every row with the
commit that computed it, and have the rebuild redo rows computed by
other code in further passes until none are left (giving up after 5,
not ready, if something keeps rewriting them); the integrity check now
also requires every listable resource's rows to come from this code.

--if-stale now only recomputes what isn't already computed by this
commit, so re-running it after a failed or cancelled run resumes where
that run stopped; a plain full rebuild still recomputes everything (for
restores and manual SQL).

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexAxthelm @claude
ci: wait for the API's new revision to take over before rebuilding
b15f840 
The Container Apps deploy action can return while the new revision is
still provisioning and the previous one still serves requests, so the
read-model rebuild could finish while old code was still writing. Add
an opt-in wait-for-cutover to deploy-container.yml, enabled for the
API: poll the app's active revisions until only ones running this image
remain and it has provisioned, then allow 45 s for the old revision's
in-flight requests. It fails the deploy if the new revision fails to
provision or doesn't take over within 10 minutes, rather than leaving
the old code serving behind a green deploy.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

CD summary b15f840

Frontend: https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net

Deployments (3)
service url fqdn
api open pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net
stitch-llm open pr-0324-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0324 stitch-dev.postgres.database.azure.com 5432 pr_0324
Jobs (1)
job image postgres_db
db-migrations ghcr.io/rmi/stitch-api:pr-0324@sha256:c3eba2cbb7a462ebd2b4fcd489f9c4466624e931560e6389449a96f162f1f40a pr_0324
Images (3)
build_time commit_time git_sha image image_digest
2026-10-14T11:47:56Z 2026-10-14T11:47:37Z 686c839 ghcr.io/rmi/stitch-api:pr-0324 ghcr.io/rmi/stitch-api:pr-0324@sha256:c3eba2cbb7a462ebd2b4fcd489f9c4466624e931560e6389449a96f162f1f40a
2026-10-14T11:48:00Z 2026-10-14T11:47:37Z 686c839 ghcr.io/rmi/stitch-seed:pr-0324 ghcr.io/rmi/stitch-seed:pr-0324@sha256:4271ece5d8e4f62a1841acf5361c1d93f9882c6b687e51a75541be28e0310315
2026-10-14T11:47:55Z 2026-10-14T11:47:37Z 686c839 ghcr.io/rmi/stitch-stitch-llm:pr-0324 ghcr.io/rmi/stitch-stitch-llm:pr-0324@sha256:12b314676086c14c0981aaf2c3915a78e5b7e16d1ccda3fd6783c7b9202e3954

AlexAxthelm and others added 2 commits October 14, 2026 14:16
@AlexAxthelm @claude
ci: keep waiting through transient errors in the API cutover check
1d1f7c9 
A single failed `az containerapp revision list` (throttling, a network
blip) or unreadable payload aborted the cutover wait under set -e,
failing the API deploy and everything that depends on it. Log such a
poll and keep waiting until the deadline; keep az's stderr out of the
JSON; and tolerate a revision with no container list. The step still
ends only on cutover, failed provisioning, or the 10-minute timeout.

STIT-766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexAxthelm @claude
keep an operator's read-model switch-off across rebuilds
4206fc5 
Switching the fast path off with `ready = false` was undone by the next
deploy's `--if-stale` rebuild. Add a separate `disabled` flag on the
status row: readers fall back to live while it is set, and rebuilds
keep the rows current but never clear it (the CLI warns that it is
set). `--if-stale` still decides on "built by this code", so a
switched-off table isn't rebuilt on every deploy, and `--ids` repairs
are allowed on a switched-off table since nothing serves it. The
runbook now switches off with `disabled = true` and says how to turn
it back on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

CD summary 4206fc5

Frontend: https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net

Deployments (3)
service url fqdn
api open pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net
stitch-llm open pr-0324-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0324 stitch-dev.postgres.database.azure.com 5432 pr_0324
Jobs (1)
job image postgres_db
db-migrations ghcr.io/rmi/stitch-api:pr-0324@sha256:46b84a0160ab6eb6ea605f5a498adae04b6cf879373e8b5b95aeaa4f77e81be0 pr_0324
Images (3)
build_time commit_time git_sha image image_digest
2026-10-14T12:40:05Z 2026-10-14T12:39:45Z c2ddd2c ghcr.io/rmi/stitch-api:pr-0324 ghcr.io/rmi/stitch-api:pr-0324@sha256:46b84a0160ab6eb6ea605f5a498adae04b6cf879373e8b5b95aeaa4f77e81be0
2026-10-14T12:40:06Z 2026-10-14T12:39:45Z c2ddd2c ghcr.io/rmi/stitch-seed:pr-0324 ghcr.io/rmi/stitch-seed:pr-0324@sha256:b4df6e7bd5145c793511ff96266dd3e2ce070abb8be64729b3be78a4942e624c
2026-10-14T12:40:01Z 2026-10-14T12:39:45Z c2ddd2c ghcr.io/rmi/stitch-stitch-llm:pr-0324 ghcr.io/rmi/stitch-stitch-llm:pr-0324@sha256:3e3eb1c11e16da1bc5c9795953d6341410f338da902151f3f35ad1da15435226

Copilot AI reviewed Oct 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The final rebuild validation can race an old-code targeted repair and mark mixed-version rows ready.

2 open findings
What changed in this PR

Adds a permission-scoped resource-state read model to accelerate list and filter-options endpoints while preserving live-query fallback behavior.

Changes:

  • Introduces read-model schema, rebuild tooling, readiness/version controls, and write-path refreshes.
  • Adds PostgreSQL concurrency, parity, schema, and rebuild tests.
  • Updates deployment workflows and operational documentation.
File Description
HACKING.md Documents local rebuilds.
docs/​STIT-766-design.md Adds design proposal.
docs/​draft-schema.sql Adds draft EAV schema.
docs/​draft-schema-operations-api.md Adds draft operational API.
deployments/​CI_DEPLOYMENTS.md Documents deploy sequencing.
deployments/​api/​tests/​observability/​test_query_name_actions.py Updates query-label tests.
deployments/​api/​tests/​db/​read_model/​test_state.py Tests rebuild and state lifecycle.
deployments/​api/​tests/​db/​read_model/​test_resource_state_schema.py Tests schema invariants.
deployments/​api/​tests/​db/​read_model/​test_rebuild_lock_postgres.py Tests PostgreSQL rebuild locking.
deployments/​api/​tests/​db/​read_model/​test_read_path.py Tests read-path parity and fallback.
deployments/​api/​tests/​db/​read_model/​test_read_path_postgres.py Tests PostgreSQL output consistency.
deployments/​api/​tests/​db/​read_model/​test_permissions.py Tests permission profiles.
deployments/​api/​tests/​db/​read_model/​test_mutations.py Tests write-path refreshes.
deployments/​api/​tests/​db/​read_model/​test_concurrency_postgres.py Tests concurrent writers.
deployments/​api/​tests/​db/​read_model/​scenarios.py Adds shared rebuild scenarios.
deployments/​api/​tests/​db/​read_model/​postgres.py Configures PostgreSQL-only tests.
deployments/​api/​tests/​db/​read_model/​dataset.py Provides deterministic test data.
deployments/​api/​tests/​db/​read_model/​conftest.py Adds read-model fixtures.
deployments/​api/​tests/​db/​read_model/​__init__.py Marks the test package.
deployments/​api/​tests/​conftest.py Resets readiness cache per test.
deployments/​api/​src/​stitch/​api/​db/​read_model/​state.py Implements refreshes and rebuilds.
deployments/​api/​src/​stitch/​api/​db/​read_model/​rebuild.py Adds rebuild CLI.
deployments/​api/​src/​stitch/​api/​db/​read_model/​permissions.py Defines permission profiles.
deployments/​api/​src/​stitch/​api/​db/​read_model/​__init__.py Defines the read-model package.
deployments/​api/​src/​stitch/​api/​db/​queries.py Adds state-table queries.
deployments/​api/​src/​stitch/​api/​db/​og_field_source_actions.py Refreshes state after attachments.
deployments/​api/​src/​stitch/​api/​db/​og_field_resource_actions.py Adds fast reads and mutation refreshes.
deployments/​api/​src/​stitch/​api/​db/​model/​og_field_resource_state.py Defines read-model ORM tables.
deployments/​api/​src/​stitch/​api/​db/​model/​__init__.py Exports new models.
deployments/​api/​README.md Links operational guidance.
deployments/​api/​READ_MODEL.md Adds the operational runbook.
deployments/​api/​alembic/​versions/​76427dd73cbc_add_resource_state_read_model.py Creates read-model tables.
deployments/​api/​alembic/​versions/​ebd0fff3d941_resource_state_code_version.py Versions readiness state.
deployments/​api/​alembic/​versions/​2a1562956888_resource_state_row_code_version.py Versions individual rows.
deployments/​api/​alembic/​versions/​eb52164663b1_resource_state_disabled_flag.py Adds operator disable flag.
.gitignore Allows documentation tracking.
.github/​workflows/​rebuild-read-model.yml Adds post-deploy rebuild job.
.github/​workflows/​python-test.yml Adds PostgreSQL CI service.
.github/​workflows/​deploy-container.yml Adds cutover waiting.
.github/​workflows/​build-and-deploy.yml Integrates rebuild deployment flow.
.github/​scripts/​build_pr_summary_comment.py Supports warning-only artifacts.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread deployments/api/src/stitch/api/db/read_model/state.py
Comment on lines +568 to +575
async with session_factory.begin() as session:
await _delete_unlistable_state(session)
if problems := await _integrity_problems(session):
raise ResourceStateRebuildError(
"Resource-state rebuild failed its integrity check: "
+ "; ".join(problems)
)
await _set_ready(session, True)
Comment thread docs/draft-schema-operations-api.md
Comment on lines +3 to +5
This document describes how application code should operate on the schema in
`01_schema.sql`. It treats the database model as an API: the supported commands,
read semantics, invariants, transaction boundaries, and expected failure modes.
@AlexAxthelm

AlexAxthelm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

FWIW, I think it'd be safe to drop the _live fallbacks and set the mask to 0/"public" if the mask function returns None.

I also don't know if I'd communicated this fully, but my bitmask idea is completely wrong and can't be indexed, so using strings or an enum instead of an int is another valid option.

I agree with changing the bitmasks, no strong opinion. I'd like to keep the _live for now, so that we aren't changing too much around how we think about Auth, while we're figuring out what the next step for that looks like (it's nice to have the slow path as a fallback)

@AlexAxthelm @claude
chore(api): collapse the read-model migrations into one
879a34c 
The branch has not shipped, so fold the code_version and disabled
columns into the migration that creates the read-model tables.
Databases already migrated on this branch (the PR environment, local
dev) need resetting, or their alembic_version set to 76427dd73cbc:
the schema is the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

CD summary 879a34c

Frontend: https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net

Deployments (3)
service url fqdn
api open pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io
frontend https://witty-mushroom-017a3dc1e-324.westus2.1.azurestaticapps.net
stitch-llm open pr-0324-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io
Database (1)
db_name postgres_host postgres_port postgres_db
pr_0324 stitch-dev.postgres.database.azure.com 5432 pr_0324
Jobs (2)
job image postgres_db api_url auth_mode
db-migrations ghcr.io/rmi/stitch-api:pr-0324@sha256:23eb4f0408db257190cd46dfcc01ec22e6169d6f8587b6efd0e8a2f7782d9f7e pr_0324
seed ghcr.io/rmi/stitch-seed:pr-0324@sha256:d0f349b90b88ac3ce684ddb45dbd1f5edebf0ff8cad43077014a19c367e92258 https://pr-0324-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 stitch-client-bearer-token
Images (3)
build_time commit_time git_sha image image_digest
2026-10-14T18:22:52Z 2026-10-14T18:22:32Z 7e3a969 ghcr.io/rmi/stitch-api:pr-0324 ghcr.io/rmi/stitch-api:pr-0324@sha256:23eb4f0408db257190cd46dfcc01ec22e6169d6f8587b6efd0e8a2f7782d9f7e
2026-10-14T18:22:50Z 2026-10-14T18:22:32Z 7e3a969 ghcr.io/rmi/stitch-seed:pr-0324 ghcr.io/rmi/stitch-seed:pr-0324@sha256:d0f349b90b88ac3ce684ddb45dbd1f5edebf0ff8cad43077014a19c367e92258
2026-10-14T18:22:50Z 2026-10-14T18:22:32Z 7e3a969 ghcr.io/rmi/stitch-stitch-llm:pr-0324 ghcr.io/rmi/stitch-stitch-llm:pr-0324@sha256:beafe15de1e5687b8712c0de3ffb5e00208e436ccc419dd080c09db21fa9040f

This was referenced Oct 15, 2026
Draft
Draft
Draft
Draft
Draft
@AlexAxthelm

AlexAxthelm commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Closing in favor of the stack comprised of #330, #331, #332, #333, #334, which is at parity with this branch, but in smaller chunks for easier review.

This branch was successfully deployed

1 active deployment
development — 879a34c7 Deployed Oct 16, 2026 by AlexAxthelm via Delete stitch-etl container app / delete-container-app #350
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

@AlexAxthelm AlexAxthelm

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AlexAxthelm @mbarlow12