Repository navigation
Conversation
GitHub Actions
Error
CD summary
|
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0279 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0279 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-09-18T17:52:26Z | 2026-09-18T17:52:07Z | 3a41c37 | ghcr.io/rmi/stitch-api:pr-0279 |
ghcr.io/rmi/stitch-api:pr-0279@sha256:ea4eebf824117a19d566cb9c8f9d5edce8eeb44e15275ad177503c273c75382d |
| 2026-09-18T17:52:25Z | 2026-09-18T17:52:07Z | 3a41c37 | ghcr.io/rmi/stitch-entity-linkage:pr-0279 |
ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:c45c7eae3d959263de3c5a2d716b703691655b765bab7cd56f677a55a916c5b0 |
| 2026-09-18T17:52:26Z | 2026-09-18T17:52:07Z | 3a41c37 | ghcr.io/rmi/stitch-seed:pr-0279 |
ghcr.io/rmi/stitch-seed:pr-0279@sha256:2e36f7b1441fd214a2a2caf5db59404fccd9fe5d380a15905ab199ccf85a6501 |
| 2026-09-18T17:52:25Z | 2026-09-18T17:52:07Z | 3a41c37 | ghcr.io/rmi/stitch-stitch-llm:pr-0279 |
ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:970d5c451bf8493d2ea1c7dc28620ff0fd5707f4cb5e4eb80225ed9347aa3b8c |
CD summary
|
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0279 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0279 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-09-18T17:55:18Z | 2026-09-18T17:54:35Z | f1cb57c | ghcr.io/rmi/stitch-api:pr-0279 |
ghcr.io/rmi/stitch-api:pr-0279@sha256:e2cea949199016e474d2f15cb5b42e281d31c479a8c940a750eb9912a26a6643 |
| 2026-09-18T17:55:19Z | 2026-09-18T17:54:35Z | f1cb57c | ghcr.io/rmi/stitch-entity-linkage:pr-0279 |
ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:450bc7e9c8bb0c30d3ff6d0f5591140106eee11951d93e9c3dec5d2d2e64b8d9 |
| 2026-09-18T17:55:18Z | 2026-09-18T17:54:35Z | f1cb57c | ghcr.io/rmi/stitch-seed:pr-0279 |
ghcr.io/rmi/stitch-seed:pr-0279@sha256:fbeed77c90f8d1a8ae10ee06d2dde35b18187a78061b628561b88ea362cfdebd |
| 2026-09-18T17:55:20Z | 2026-09-18T17:54:35Z | f1cb57c | ghcr.io/rmi/stitch-stitch-llm:pr-0279 |
ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:5d3a8479df3800c2c21f23817b77c30349a4c54e848fc752812abd85f3157baa |
GitHub Actions
Error
CD summary
|
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0279 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0279 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0279@sha256:f2707c67a42f86b6da89ddbb65864b8bb041b9403f25416824310292c8c4183d |
pr_0279 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-09-18T17:56:27Z | 2026-09-18T17:55:42Z | 5b43d27 | ghcr.io/rmi/stitch-api:pr-0279 |
ghcr.io/rmi/stitch-api:pr-0279@sha256:f2707c67a42f86b6da89ddbb65864b8bb041b9403f25416824310292c8c4183d |
| 2026-09-18T17:56:27Z | 2026-09-18T17:55:42Z | 5b43d27 | ghcr.io/rmi/stitch-entity-linkage:pr-0279 |
ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:de6c95e3d392dbd6353c9396882d07641a86d30d35eb2106ef2047c44a7e05e1 |
| 2026-09-18T17:56:28Z | 2026-09-18T17:55:42Z | 5b43d27 | ghcr.io/rmi/stitch-seed:pr-0279 |
ghcr.io/rmi/stitch-seed:pr-0279@sha256:2b9b6bacc12ce6f47c9a6942b9a512e4ee15b8216fcd215b136dd2f1600f04b6 |
| 2026-09-18T17:56:29Z | 2026-09-18T17:55:42Z | 5b43d27 | ghcr.io/rmi/stitch-stitch-llm:pr-0279 |
ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:6bdb510aeb48f43e39615fb06020f2438b3c0fffc77cc16250cf46f22b0aa469 |
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0279-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0279-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-279.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0279-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0279 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0279 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0279@sha256:539a0c72039d84f6e4b8a12bb346c209e6ea787341350008706cbda4d0257bcf |
pr_0279 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-09-18T17:57:38Z | 2026-09-18T17:57:00Z | 18df0f8 | ghcr.io/rmi/stitch-api:pr-0279 |
ghcr.io/rmi/stitch-api:pr-0279@sha256:539a0c72039d84f6e4b8a12bb346c209e6ea787341350008706cbda4d0257bcf |
| 2026-09-18T17:57:34Z | 2026-09-18T17:57:00Z | 18df0f8 | ghcr.io/rmi/stitch-entity-linkage:pr-0279 |
ghcr.io/rmi/stitch-entity-linkage:pr-0279@sha256:02f78ae66c7fd0e6252d784a739434a82c37850640ea5071149beeb7be5d5de2 |
| 2026-09-18T17:57:37Z | 2026-09-18T17:57:00Z | 18df0f8 | ghcr.io/rmi/stitch-seed:pr-0279 |
ghcr.io/rmi/stitch-seed:pr-0279@sha256:c308e54799ac800f27c50540a562768e170cb7cd99f743066d58706a343d9dbf |
| 2026-09-18T17:57:35Z | 2026-09-18T17:57:00Z | 18df0f8 | ghcr.io/rmi/stitch-stitch-llm:pr-0279 |
ghcr.io/rmi/stitch-stitch-llm:pr-0279@sha256:ea0a6018690b07af41bb6c47bcf13c759f72967636725050ccbee35fc66e1680 |
| - user with `wm + cc` sees `wm` | ||
| - user with `wm` sees `wm` | ||
| - user with `cc` sees `cc` | ||
| - remaining users see `gem` |
There was a problem hiding this comment.
Should this be gem, or "all combined public sources?" Like why couldnt' they see GEM+BC? Same question repeats below. You say it above, just confirming in the bullets too.
Re-read, think I get it--the presumption is if both BC and GEM have coords, we'll defaul to GEM?
There was a problem hiding this comment.
Right. This was one spot where I suspected a diagram/table would be more illustrative.
|
|
||
| The purpose is to provide a durable store that houses the flattened/coalesced resources. We're effectively precomputing the coalescing logic and saving it to a single table. The actual schema is less important than its function and the constraints we place on it. | ||
|
|
||
| It must: |
There was a problem hiding this comment.
We talked about this and it's inherent, but just to be clear/pedantic, it must also be re-creatable. Like if this table is inadvertently dropped, it would straightforward to recreate it by marching through the resources and re-precomputing. We should be careful to maintain that, ie not add to the table in the future in a way that it would hold data that could be recontructed--so we will be adding complexity, but it's low-risk complexity.
There was a problem hiding this comment.
@john Is that sufficiently captured in:
- be able to be rebuilt from scratch at any time, we should be able to derive the data for the table easily & quickly
Or is there some rewording you think it needs?
AlexAxthelm
left a comment
AlexAxthelm
left a comment
There was a problem hiding this comment.
Including my thoughts from working through an draft implementation (#290).
Overall, the idea works well (that draft is showing big performance gains), but I think that there's a lot of this docset that can get dropped/deferred, since the actual working feature requires only a little bit of replumbing.
| Combined with serializing to/from Pydantic models, this has led to poor response times that | ||
| negatively impact user experience. Furthermore, given that our entire db is less than 100MB | ||
| in size, this raises questions about our overall db design and access patterns. |
There was a problem hiding this comment.
| Combined with serializing to/from Pydantic models, this has led to poor response times that | |
| negatively impact user experience. Furthermore, given that our entire db is less than 100MB | |
| in size, this raises questions about our overall db design and access patterns. |
Perf testing I ran in support of testing #291 (an alternate version of #290) indicated that we're actually doing pretty well on serialization, since most of the stuff is paginated, and we're only serializing values when we're going to display them (i.e. the detail view which is 1 at a time, or list view hydration)
There was a problem hiding this comment.
Can you share some of the perf testing details and implementation?
There was a problem hiding this comment.
Read-path perf: read-model (#290) vs resolve-at-read (#291) vs main
TL;DR — #290 (pre-resolved winner rows) is 14–21× faster on list and 10–21× on filter-options at prod volume (50k) and stays ~flat as data grows; #291 (resolve-at-read) is ~2–3× on list and no win on filter-options. Both return byte-identical responses to main. One general index gap surfaced (bottom).
Method
- Dataset: deterministic seed (RANDOM_SEED=8675309), complex resources — multi-source spanning all 8 OGSI source keys, field-priority overrides (~15%), merges (~4% → repointed). ~49k unmerged resources at the top rung.
- Volume ladder: 1k / 10k / 50k resources (50k ≈ prod), grown cumulatively via a deterministic offset (nested datasets).
- Permission profiles (real Auth0 tokens): public-6, +wm, +ccr, all-8. Edge cases (0-source, 3-source) via a token-free in-process sweep.
- Protocol: local docker, single-stream, warm-up discarded then 50 measured requests per (branch × volume × profile × endpoint); medians reported. Latency + per-query timings from the app's observability logs (X-Stitch-Perf-Scenario tagging).
- Baseline: the branches' merge-base main (isolates the materialization change).
Headline — median ms @ 50k (× = vs main)
| endpoint | main | #290 | #291 |
|---|---|---|---|
| list (all8) | 2741 | 130 (21×) | 1222 (2.2×) |
| list (public6) | 1906 | 132 (14×) | 1064 (1.8×) |
| filter-options | 332–403 | **~19 (18–21×)**¹ | ~490 (~0.8×, slower) |
| detail / id² | ~10–11 | ~10 | ~11 |
¹ restricted profiles ~19ms; the all8 mask is noisier (~73ms median). ² after the index fix below.
Scaling — list all8, median ms
| vol | main | #290 | #291 |
|---|---|---|---|
| 1k | 85 | 11 (8×) | 42 (2.1×) |
| 10k | 794 | 23 (35×) | 344 (2.3×) |
| 50k | 2741 | 130 (21×) | 1222 (2.2×) |
main scales ~linearly with volume; #290 stays ~flat (pre-resolved keyed reads); #291 tracks main ~2.5× lower (skips the join but still ranks+pivots a candidate set that grows with data).
Why it's faster (it's not the pivot)
The cost is producing the winners — the 5-table join + ROW_NUMBER() winner-pick — not the wide pivot. Evidence: the count query (same coalesce, no pivot) is as expensive as list_ids; and list_ids pivots only the sort column — the full 18-column pivot is a separate query over just the 50 paged rows (~25ms). #290 pre-resolves the winner per permission mask so reads are keyed lookups; #291 avoids the join but still does the winner-pick + pivot at read time.
Consistency / parity
Byte-identical responses to main across all 4 profiles at 1k and 10k (seed metadata normalized). At 50k #291 matched main; #290 differed in a single cell due to a 2-resource seed/merge drift (dataset artifact, not a coalescing error).
Actionable DB finding (independent of which PR)
og_field_memberships has no index on resource_id (only its PK), forcing a ~104k-row seq scan in the single-resource source_data join:
CREATE INDEX ix_membership_resource_active ON og_field_memberships (resource_id, status);detail @50k, real HTTP, before → after: #291 107 → 11 ms, #290 35 → 10 ms (query-level: seq scan 17ms → index scan 1.4ms). It speeds up main's live paths too, so it's worth shipping regardless. (Secondary: ~100ms/query JIT+planning overhead on the big coalesce queries — a prepared-statement/plan-cache opportunity.)
Adoption cost (one-time full rebuild) & repro
- rebuild_all: Refactor/db mv #290 ~47s, Refactor/db ranking mv #291 ~29s at 50k.
- Not yet committed/pushed: the load-test harness (deployments/loadtest/: tagged single-stream driver + aggregator + token-free in-process sweep + consistency differ) and the seed complexity extension; results reconstructed on branch perf/materialization-loadtest. The seeded 50k docker volumes are retained for re-measurement.
(All numbers are local-docker single-stream medians, single pass — directional at the ~5–10% level, not CI-grade.)
| With the newly added attribute-level priority feature ([STIT-494](https://rmi1.atlassian.net/browse/STIT-494)), we now have a more deterministic and bounded way to establish the different variations we display for a flat resource. Our permissions model only has 2 restricted options: `wm` and `cc`. All the remaining sources are public, and ALL users have read permissions for them. | ||
|
|
||
| > [!NOTE] | ||
| > It was more straightforward to treat these individually when implemented, but reviewing our permissions model may help simplify in the future. However, it's not strictly necessary for this work. |
There was a problem hiding this comment.
Up to this point, we've treated this as an invariant that stitch needs to build at its core: "If you don't explicitly have permission to see X, then you don't see X". When working on a draft implementation in #290, I've got it still maintaining that behavior, while still hitting the big points of the design below. We treat the "public" as a group of permissions, and basically don't do any of the pre-compute/caching for permissions combos that don't include all of the public sources, then if a token ever comes in that doesn't have the full public set, it falls back to the slow (current main coalescing code path)
All to say that if we want to keep that invariant we can (I have like that stitch doesn't play favorites by design), but if we are actually willing to introduce the concept of a public source (that can be read without explicitly granting permissions) then that opens up some different possibilities for solutions.
| > - 3 licensed sources = 8 variants | ||
| > - 4 licensed sources = 16 variants | ||
| > | ||
| > so even if we add 2 proprietary sources, this structure grows to a max of 16 x the number of top-level (unmerged) resources. I don't see this as a real issue even in the long term. How many licensed sources are even realistic? 6? 10? Even at 10 proprietary sources, the view/table would be 2^10 = 1024 x resources. The upper bound of **all** named fields is probably somewhere in the ~65k range, so ~65M rows is still manageable...if we'd ever even get close to that. |
There was a problem hiding this comment.
Practically, I would say we can make this scale nicely up to 4 or 5, but if we go past 32 rows per resource, that would be a good time to rethink the design, not for perf reasons, but it suggests that we need to think a bit more about how we're caching, since most of those rows would end up being the same (there would be very few resources with all source types attached)
|
|
||
| **Goals** | ||
|
|
||
| 1. Serve `list` and `filter-options` with <100ms latency |
There was a problem hiding this comment.
| 1. Serve `list` and `filter-options` with <100ms latency | |
| 1. Serve `list` and `filter-options` with <100ms latency. | |
| `detail` and `GET {id}` can stay on the live path (`source_data` shouldn't be included in the MV, and these are single-resource endpoints, so shouldn't be taxing queries). |
We can sketch out feeding them from the MV if we want that consistency, but I didn't see any great perf changes on them when I included them in #291. No strong opinion on this in practice
There was a problem hiding this comment.
My thought here was more about consistency and maintainability. The state table becomes the only way users read resources, which makes all the queries we've written thus far implementation details. So long as we preserve invariants on the state table, the details of how we maintain the table are less consequential, giving us more freedom & flexibility to make changes, address tech debt, and/or experiment with alternatives.
There was a problem hiding this comment.
having them along the same path would provide nice symmetry, but the detail endpoint builds the source_data object (which has all the component sources, not just the winner), which is going to hit the live coalescer in any case, So I would say it's better to keep it all there, rather than reading some off the MV.
| - `oil_gas_field_source_values` currently has an `id` primary key, so we could replace `(attribute_id, source_id)` with `source_value_id` | ||
| - we set `is_curated` to `True` when a user makes an update | ||
|
|
||
| ### Optional Additional Tables |
There was a problem hiding this comment.
Similarly, I don't think these optional tables add a whole lot for us, and we would probably be better off letting them arise naturally out of the implementation if that's what it needs to have. 🤷
| ## 5. Open Issues | ||
|
|
||
| 1. Priority row count at current and projected resource volume. | ||
| 2. How to handle priorities upon merge? |
There was a problem hiding this comment.
We should handle priorities on merge the way we do now: by dropping all the overrides/reordering and it all falls back to default
| 2. How to handle priorities upon merge? |
| Caches must include the user's effective visibility profile in the cache key or | ||
| be invalidated when permissions change. | ||
|
|
||
| ## 9. Resolving one resource for one user |
There was a problem hiding this comment.
for single-resource/single-user, I would say keeping the current path is simple enough. We haven't noticed any performance issues on that path, only on bulk paths (where we have to process the whole table before getting results)
| Do not reveal hidden candidates or their keys unless the user has separate | ||
| permission to inspect restricted provenance. | ||
|
|
||
| ## 12. Dynamic search by unknown attributes |
There was a problem hiding this comment.
Introducing a new query format/language seems way out of scope for materializing the coalesced values
|
|
||
| Choose one strategy and use it consistently in all priority mutation paths. | ||
|
|
||
| ## 15. Row-level security |
There was a problem hiding this comment.
This could be implemented, but I'm not sure it's worth the effort/headache, given that we're already handling all the DB connections as a user that can read all data (stitch_app_user)
Co-authored-by: Alex Axthelm <alex@alexaxthelm.com>
This branch was successfully deployed
3 participants
Attempt 1 at a design doc for a DB architecture proposal.
Design Doc (rendered)
Would welcome feedback along 2 separate axes:
I've tried to hit a relatively high level of abstraction to provide freedom in implementation for more minute details. Any glaring oversights? Potential bugs? Missing components?
Unclear wording or missing sections? Too detailed or not detailed enough?
AI helped with an initial draft, but I essentially overwrote the entire thing. I also used AI to draft
draft-schema.sqlanddraft-schema-operations-api.mdfiles. They're very early iterations and are largely outdated, but they represent a rough initial solution to the db structural issues.