From f17fb27d789690e67db616042943fef98d18bf5f Mon Sep 17 00:00:00 2001 From: Pranshu Maheshwari Date: Thu, 3 Sep 2026 14:05:47 -0400 Subject: [PATCH 1/8] docs: describe cluster lifecycle states on the clusters concept page (#38629) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### Motivation It is not obvious from the docs that a cluster moves through distinct lifecycle states, nor how to tell whether it is making progress (vs. stuck) in each one. This came out of a Slack discussion on cluster lifecycle and freshness/"total lag": https://materializeinc.slack.com/archives/CU7ELJ6E9/p1788365831558719?thread_ts=1788365056.126659&cid=CU7ELJ6E9 ### Description Adds a **Lifecycle of a cluster** section to the [clusters concept page](https://materialize.com/docs/concepts/clusters/). It briefly describes the states a cluster's replicas move through and, for each state, recommends how to monitor progress: - **Provisioning** — replicas are scheduled and brought online. Monitor via `mz_cluster_replica_statuses` and `mz_clusters`. - **Hydrating** — replicas reconstruct in-memory state from Materialize's storage. Monitor via `mz_hydration_statuses` (and `mz_compute_hydration_statuses` for indexes/materialized views). - **Catching up** — the cluster works through the accumulated backlog and total lag comes down. Monitor via `mz_wallclock_global_lag_recent_history` and `mz_materialization_lag`. - **Steady state** — lag holds low and roughly constant. Monitor via the same lag relation to catch regressions. A note also points to the snapshotting step sources go through on first start. ### Verification Docs-only change. Verified all system-catalog and concept-page links resolve against the repo, and ran `bin/format-docs` (no changes). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015919agXAtbTfF1iHNpGQoj --- _Generated by [Claude Code](https://claude.ai/code/session_015919agXAtbTfF1iHNpGQoj)_ --------- Co-authored-by: Claude --- doc/user/CLAUDE.md | 8 ++ doc/user/content/concepts/clusters.md | 140 ++++++++++++++++++++++++++ 2 files changed, 148 insertions(+) diff --git a/doc/user/CLAUDE.md b/doc/user/CLAUDE.md index b818c4fffabd8..0c46c7474d1b4 100644 --- a/doc/user/CLAUDE.md +++ b/doc/user/CLAUDE.md @@ -137,6 +137,14 @@ with two dead sections: - **Symlinked content**: `content/` contains no symlinks today (only `sql-grammar/keywords.txt` is one). +## Verifying docs + +- Test documentation by running its SQL against the Materialize emulator (the + `materialize/materialized` Docker image) before committing, rather than + assuming a query works. +- When writing syntax demos, include both the query and its output, and confirm + the syntax and shown output match what the emulator actually returns. + ## Reviewing docs changes The `docs-content-reviewer` agent and `.prompts/docs_pre_pr_review.md` encode the diff --git a/doc/user/content/concepts/clusters.md b/doc/user/content/concepts/clusters.md index bbb8f0af79237..05f5f542f0162 100644 --- a/doc/user/content/concepts/clusters.md +++ b/doc/user/content/concepts/clusters.md @@ -103,6 +103,146 @@ When provisioning replicas, See also [Hydration considerations](#hydration-considerations). +## Lifecycle of a cluster + +Whenever a cluster starts running a workload (after you create it, resize it, +or one of its replicas restarts), its replicas move through a sequence of states +before results are fully up to date. Knowing which state a cluster is in tells +you whether it is making progress or is stuck. + +The queries below monitor a cluster named `lifecycle_demo` that hosts the +materialized view `bids_by_auction` and its index `bids_by_auction_idx`, both +built on a continuously-updating `AUCTION` load-generator source. Substitute +your own cluster and object names. + +### Provisioning + +Replicas are scheduled and brought online. A cluster with a [replication +factor](#cluster-replicas) of `0` has no compute and never leaves this state. To +monitor progress, check that replicas report `online` in +[`mz_cluster_replica_statuses`](/reference/system-catalog/mz_internal/#mz_cluster_replica_statuses), +and confirm the cluster has replicas via +[`mz_clusters`](/reference/system-catalog/mz_catalog/#mz_clusters). + +```mzsql +SELECT c.name AS cluster, r.name AS replica, r.size, st.status, st.reason +FROM mz_internal.mz_cluster_replica_statuses st +JOIN mz_catalog.mz_cluster_replicas r ON r.id = st.replica_id +JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id +WHERE c.name = 'lifecycle_demo' +ORDER BY r.name; +``` + +```none + cluster | replica | size | status | reason +----------------+---------+------+--------+-------- + lifecycle_demo | r1 | 25cc | online | +(1 row) +``` + +The `reason` column is empty while the replica is `online`, and reports why a +replica is unavailable otherwise. + +### Hydrating + +Each replica reconstructs its in-memory state by reading from Materialize's +storage layer (see [hydration](/concepts/hydration/)). While an object is +hydrating, its `hydrated` flag reads `f` and its lag is reported as `NULL`. To +monitor progress, check the `hydrated` flag per object in +[`mz_hydration_statuses`](/reference/system-catalog/mz_internal/#mz_hydration_statuses), +where the `replica_id` stays blank until the object attaches to a replica. For +indexes and materialized views, +[`mz_compute_hydration_statuses`](/reference/system-catalog/mz_internal/#mz_compute_hydration_statuses) +also reports how long hydration took. + +```mzsql +SELECT o.name AS object, o.type, r.name AS replica, ch.hydrated, ch.hydration_time +FROM mz_internal.mz_compute_hydration_statuses ch +JOIN mz_objects o ON o.id = ch.object_id +JOIN mz_catalog.mz_cluster_replicas r ON r.id = ch.replica_id +WHERE o.name IN ('bids_by_auction', 'bids_by_auction_idx', 'bids_load') +ORDER BY o.name; +``` + +```none + object | type | replica | hydrated | hydration_time +---------------------+-------------------+---------+----------+----------------- + bids_by_auction | materialized-view | r1 | t | 00:00:00.000074 + bids_by_auction_idx | index | r1 | t | 00:00:00.000019 + bids_load | materialized-view | r1 | t | 00:00:05.6032 +(3 rows) +``` + +The light view and index hydrate in microseconds, while the larger `bids_load` +view takes about 5.6 seconds. A larger object with more state to reconstruct +shows a longer, more visible hydration window. + +### Catching up + +Once hydrated, the cluster processes the backlog of input updates that +accumulated while it was unavailable, so its total lag starts high and comes +down. To monitor progress, watch `lag` decrease in +[`mz_wallclock_global_lag_recent_history`](/reference/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history), +or break the lag down by input with +[`mz_materialization_lag`](/reference/system-catalog/mz_internal/#mz_materialization_lag). + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name IN ('bids_by_auction', 'bids_load') +ORDER BY o.name; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+------------------+------------------+---------------------+---------------------- + bids_by_auction | 00:00:34.001 | 00:00:34.001 | bids | bids + bids_load | 00:00:41.001 | 00:00:41.001 | bids | bids +``` + +Both objects trail their slowest input, the `bids` source, by tens of seconds. +As the cluster works through the backlog, these lags fall. + +### Steady state + +The cluster has caught up and its lag holds low and roughly constant, typically +a few seconds. Re-running the lag query confirms the objects have caught up to +their input. + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name = 'bids_by_auction'; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+-----------+------------+---------------------+---------------------- + bids_by_auction | 00:00:00 | 00:00:00 | bids | bids +(1 row) +``` + +Wallclock lag in +[`mz_wallclock_global_lag_recent_history`](/reference/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history) +holds near-constant at a few seconds. A lag that instead climbs steadily, at +about one minute per minute, means the cluster has stopped making progress. + +{{< note >}} +Sources go through an additional +[snapshotting](/concepts/snapshotting/) step the first time they run, reading the +initial state of the upstream system before the states above apply. See +[Troubleshooting](/transform-data/freshness-troubleshooting/) for how to +diagnose a cluster that is not progressing through these states. +{{< /note >}} + ## Cluster sizing From 4533dc579a275fe5d63563ff6a3254c80997190b Mon Sep 17 00:00:00 2001 From: Jon Currey Date: Thu, 3 Sep 2026 14:28:53 -0400 Subject: [PATCH 2/8] v26.40.0 release notes (#38552) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Release notes for the v26.40.0 train. Dates in RC snapshots are provisional until the release ships. One commit is added per snapshot; the final snapshot sets the real dates. **Applied so far: `rc.1`, `rc.2`, `rc.3`, `final` — the train is complete.** `final` adds no entries: `v26.40.0` is exactly one commit ahead of `v26.40.0-rc.3`, and that commit is the version bump. Nothing shipped between the last RC and the release tag, so the increment is genuinely empty rather than a gap; the release's 19 published entries all live in the `rc.1`, `rc.2`, and `rc.3` increments. **Dates are now real, not provisional — please confirm them.** Cloud **2026-09-02**, Self-Managed **2026-09-03**. No dates were supplied, so these come from the `v26.40.0` tag's commit date (2026-09-02T17:06:24Z) and `+1 day`. Two checks support them: v26.39.0's tag date was exactly its published Cloud date, and the `+1` Self-Managed offset matches v26.39.0, v26.37.0, v26.35.0, and v26.33.0. They supersede the provisional pair (Cloud 2026-09-03 / Self-Managed 2026-09-04) that all three RC snapshots carried, anchored on the `rc.1` tag. The release shipped **one day earlier** than the cadence estimate — the direction `rc.1` predicted, since v26.37.0, v26.38.0, and v26.39.0 each did the same. **Merge ordering — this PR currently conflicts with `main`, by design of how the branch was cut.** It was branched from a `main` that contained none of the v26.38.0, v26.38.1, or v26.39.0 sections, so `## v26.40.0` sits directly above `## v26.38.2` here. Since then #38204 (v26.38.0) and #38383 (v26.39.0) have merged, and `main` now opens with `## v26.39.0` — hence the conflict. #38365 (v26.38.1) is **still open**, so resolving the conflict now would only reintroduce it when that PR merges. The resolution is mechanical and belongs to whichever of these merges last: `## v26.40.0` must end up **above** all of those sections, and its operator-compatibility row at the **top** of `rows:`. Left unresolved deliberately rather than by oversight. **For the release owner — decisions worth a look before merge.** - **RESOLVED: #38556 was never picked, and `rc.3`'s narrowed bug fix is the correct final statement.** `rc.3` published #38558, the second of two [MZC-59](https://linear.app/materializeinc/issue/MZC-59) commits, but not #38556 ("return 202 for notifications so MCP SDK clients complete the handshake"), and deliberately narrowed its entry to the transport-level fact that build delivers, naming no client as fixed. The open question was whether #38556 would land in an `rc.4` or in `final`. **It did not:** no `v26.40.0-rc.4` tag exists, the final tag adds only the version bump, and #38556 is still not an ancestor of it. So no supersession and no merged bullet — the entry stands as written. **[MZC-59](https://linear.app/materializeinc/issue/MZC-59) remains open for SDK-based MCP clients such as Codex in v26.40.0** and should carry to a later train. - **RESOLVED: #38462 is in the release**, which retires a caveat `rc.3` could not settle at the time. [agent-skills#55](https://github.com/MaterializeInc/agent-skills/pull/55) flagged that its claim about `mz_dataflow_arrangement_sizes` on pre-v26.40 servers *assumed* #38462 would ship in v26.40, when no v26.40 tag yet existed. The tag now exists and #38462 is an ancestor of it, so the `mz-optimize-memory` entry needs no change. #38462 itself stays omitted — `rc.1` classified it Borderline as instruction text with no server behavior change, and seeing the whole release does not disturb that call. - **The Iceberg options this train publishes are undocumented at the tag.** `grep -rn "ACCESS DELEGATION\|OAUTH2 SERVER URL" doc/user/content/` returns nothing, though `doc/user/content/sql/create-connection.md` is where they belong. The feature section is published anyway because the SQL surface is reachable, unflagged, and tested against a real catalog — but if the docs are not landing in this release, hold the section for the release that documents them. - **#38480 is a privilege tightening that can break working deployments on upgrade.** `CREATE TABLE ... FROM SOURCE` now requires `SELECT` on the source and `USAGE` on its schema. It is published as a bug fix; you may want it called out more prominently than the Bug Fixes list, since affected admins have to grant `SELECT` before upgrading. - **`rc.2`'s recovery path is deliberately not in the release note.** #38545 fixes the root cause of a production incident in which `environmentd` entered a deterministic crash loop. A deployment that applied a replacement on v26.39.x or earlier keeps the stale cache entries until it upgrades, and one already crash-looping needs an `enable_expression_cache=false` boot to come up at all. Consider whether the upgrade guide or a support note should carry that. - **Nine flag-gated items were held as Borderline in `rc.1`** pending a call on whether their flags are on in production: `CREATE METRIC SINK` (`enable_metric_sink`, `false` at the tag), object hydration history (`hydration_history_collection_interval`, `Duration::ZERO`, which disables collection), hedged persist blob gets (`persist_blob_hedged_get_enabled`, `false`), three upsert/batcher spill optimizations (`enable_column_paged_batcher` / `enable_upsert_chunked_stash` / `enable_upsert_paged_spill`, all `false`), the columnar merge batcher (`enable_columnar_merge_batcher`, `false`), the peek row-iteration limit (`enable_compute_peek_row_iteration_limit`, `false`), parallel MySQL snapshots (`mysql_source_snapshot_parallelism`, `false`), and Console cluster-list usage metrics (`usage-metrics-in-cluster-list-CNS121`, off). No published entry in any snapshot is behind an off-by-default flag. - **#38399 is omitted as a cross-train duplicate.** It appears in the v26.40.0 tracking issue as its `main` merge, but v26.39.0 `rc.2` already published it (#38383), so restating it under v26.40.0 would show the reader the same fix twice in `_index.md`. Override if you would rather it appear here. - **Cosmetic, your call:** the **Agent Skills** list reads `mz-optimize-memory` then `mcp-developer-analysis` — newest increment first, an artifact of each snapshot prepending into the existing list. Both entries are correct; flagging only in case you prefer them reordered before merge. - **Per-snapshot inclusion:** `rc.1` 23 of 86 PRs (26.7% — a property of a small single-week denominator carrying two penetration-test security fixes, a GA graduation, and two multi-PR capabilities, not a loosened bar); `rc.2` 1 of 1; `rc.3` 2 of 4; `final` 0 of 3. - **The `final` Step 5 verification pass ran in its widened form** over all four snapshot directories and the assembled section on this branch, since the increment had no entries of its own: **0 cut, 0 rewritten, 0 merged.** All 19 published entries stand as reviewed, and no RC-time Borderline call flipped when re-read with the full release in view. Snapshot drafts (with PR links) in mz-skills: - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.1/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.1/omitted.md) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.2/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.2/omitted.md) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.3/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.3/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/final/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 17 — [review borderline calls](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.1/omitted.md#borderline), rc.2: 0 — [none](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.2/omitted.md#borderline), rc.3: 0 — [none](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.3/omitted.md#borderline), final: 0 — [none](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/final/omitted.md#borderline) --------- Co-authored-by: Claude Co-authored-by: Pranshu Maheshwari Co-authored-by: Pranshu Maheshwari --- doc/user/content/releases/_index.md | 70 +++++++++++++++++++ .../self_managed_operator_compatibility.yml | 5 ++ 2 files changed, 75 insertions(+) diff --git a/doc/user/content/releases/_index.md b/doc/user/content/releases/_index.md index ad4bc15367714..ab52ca41c6f91 100644 --- a/doc/user/content/releases/_index.md +++ b/doc/user/content/releases/_index.md @@ -19,6 +19,76 @@ Starting with the v26.1.0 release, Materialize releases on a weekly schedule for both Cloud and Self-Managed. See [Release schedule](/releases/schedule) for details. {{}} +## v26.40.0 +*Released to Materialize Cloud: 2026-09-02*
+*Released to Materialize Self-Managed: 2026-09-03*
+ +### Iceberg support for Databricks on AWS {#v26.40-iceberg-support-for-databricks-on-aws} + +{{< public-preview />}} + +Iceberg sinks can now write to Apache Iceberg tables registered in [Databricks +Unity Catalog](/serve-results/sink/iceberg-databricks/) on AWS, reached through +Unity Catalog's Iceberg REST catalog endpoint. Two new Iceberg catalog +connection options make this work: `OAUTH2 SERVER URL`, which names a token +endpoint that does not sit under the catalog URI, and `ACCESS DELEGATION = +'vended-credentials'`, which asks the catalog for temporary, table-scoped +storage credentials instead of static keys. Materialize refreshes both the +OAuth2 token and the vended credentials while the sink runs, so a sink that +outlives one credential lifetime keeps writing. + +```mzsql +CREATE SECRET databricks_oauth + AS ':'; + +CREATE CONNECTION iceberg_catalog_connection TO ICEBERG CATALOG ( + CATALOG TYPE = 'rest', + URL = 'https://.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest', + WAREHOUSE = '', + CREDENTIAL = SECRET databricks_oauth, + OAUTH2 SERVER URL = 'https://.cloud.databricks.com/oidc/v1/token', + SCOPE = 'all-apis', + ACCESS DELEGATION = 'vended-credentials' +); + +CREATE SINK + IN CLUSTER + FROM + INTO ICEBERG CATALOG CONNECTION iceberg_catalog_connection ( + NAMESPACE = '', + TABLE = '' + ) + MODE APPEND + WITH (COMMIT INTERVAL = ''); +``` + +For more information, see: +- [Guide: Databricks on AWS](/serve-results/sink/iceberg-databricks/) +- [`CREATE CONNECTION`: Iceberg catalog](/sql/create-connection/#iceberg-catalog), including [storage access delegation](/sql/create-connection/#iceberg-catalog-access-delegation) +- [`CREATE SINK`: Iceberg](/sql/create-sink/iceberg/), including [append mode](/sql/create-sink/iceberg/#append-mode) + +### Improvements {#v26.40-improvements} +- **Query History for Self-Managed**: Self-Managed deployments can now use the Console's Query History view to debug latency and performance. +- **Bounded staleness isolation is generally available**: The [`bounded staleness `](/reference/isolation-level/#bounded-staleness) transaction [isolation level](/reference/isolation-level/) is out of public preview and is now a supported part of the [`transaction_isolation`](/reference/isolation-level/#setting-isolation-level) surface. See [when to use bounded staleness](/reference/isolation-level/#when-to-use-bounded-staleness) and its [restrictions](/reference/isolation-level/#restrictions). +- **Replica resource usage introspection**: A new `mz_introspection.mz_cluster_replica_resource_usage` relation reports each replica process's own memory, swap, and disk observations at a higher cadence than the roughly once-a-minute orchestrator samples, so a spike between two samples is no longer invisible. + +### Agent Skills {#v26.40-agent-skills} +- **mz-optimize-memory**: A new skill that works top-down from a cluster's largest arrangements to a table of memory-reducing fixes — index changes, outer-join and subquery rewrites, window-function patterns, and arrangement size hints — with rules for estimating each saving before making the change and verifying it afterwards. + +### Bug Fixes {#v26.40-bug-fixes} +- Fixed the MCP servers rejecting `tools/list`, `ping`, and `notifications/initialized` requests that carry any `params` — including the `_meta` field MCP clients are allowed to attach — with a non-JSON-RPC HTTP 422 that clients read as the server being unavailable. +- Fixed `ALTER MATERIALIZED VIEW ... APPLY REPLACEMENT` leaving a stale cached query plan behind, which put `environmentd` into a crash loop on every subsequent restart once a dependency of the replaced definition was dropped, and could otherwise leave the view computing its old definition. +- `CREATE TABLE ... FROM SOURCE` now requires `SELECT` on the source and `USAGE` on its schema, closing a case where `CREATE` on any schema a role controlled was enough to read a source that role had been denied; deployments where a platform team owns sources and application teams attach tables into their own schemas will need those `SELECT` grants added. +- Fixed `ALTER CONNECTION` letting a connection owner keep secrets and connections they lack `USAGE` privileges on, and read those secrets during content checks or connection validation. +- Fixed `ANY`/`ALL` over a `NULL` array or list returning the empty-set answer instead of `NULL`, and `array_position` failing to find `NULL` elements, both now matching PostgreSQL. +- Fixed `round(numeric, scale)` erroring on a scale that reaches past the value's fractional digits, such as `round(123::numeric, 38)`, which could also let filter pushdown discard data a query matched. +- Fixed identity-provider group sync silently finding nothing when `oidc_group_claim` names a claim the token declares directly, such as `roles`, so group-to-role mapping now works without a custom claim or JWT prehook. +- Fixed Arrow ADBC clients failing to connect by adding the missing `pg_type.typsend` column, and fixed `typreceive` rendering as a numeric OID rather than the function name, so those clients resolve every column to its real type. +- Fixed Iceberg sinks ignoring a table's `write.data.path` property and always writing data files to the default location. +- Fixed sources on the Console's Objects page being described in replica-hydration terms, which could show a healthy source as `Not Hydrated`, leave a webhook source with no status, and hold a completed snapshot at 99%. +- Fixed several `dbt-materialize` error paths reporting confusing failures instead of the intended messages, covering the index config parser, renaming a view, dropping an unsupported relation type, connection option parsing, and the version check against a server that is not Materialize. +- Fixed the `mz` CLI failing on read-only commands such as `mz sql` when `mz.toml` sits on a read-only mount. + ## v26.39.0 *Released to Materialize Cloud: 2026-08-26*
*Released to Materialize Self-Managed: 2026-08-27*
diff --git a/doc/user/data/self_managed/self_managed_operator_compatibility.yml b/doc/user/data/self_managed/self_managed_operator_compatibility.yml index 96fda410754a0..fa91e90709909 100644 --- a/doc/user/data/self_managed/self_managed_operator_compatibility.yml +++ b/doc/user/data/self_managed/self_managed_operator_compatibility.yml @@ -5,6 +5,11 @@ columns: - column: Release date - column: Notes rows: + - Materialize Operator: v26.40 + orchestratord version: v26.40 + environmentd version: v26.40 + Release date: "2026-09-03" + Notes: "See [v26.40 release notes](/releases/#v26400)" - Materialize Operator: v26.39 orchestratord version: v26.39 environmentd version: v26.39 From a4cd865ad44ac8711d25956fae84d7ae521cc472 Mon Sep 17 00:00:00 2001 From: Jeremy Donelson Date: Thu, 3 Sep 2026 15:58:47 -0400 Subject: [PATCH 3/8] console: CNS-38 explain why an interrupted shell command needs a retry (#38610) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### Motivation Fixes [CNS-38](https://linear.app/materializeinc/issue/CNS-38) Instead of simply displaying "Retry" after a disconnection, add an error message. ### Description Investigation revealed that we can mitigate the disconnection problem at its source. This UX improvement is a first step. Screenshot 2026-09-01 at 4 19 47 PM ### Verification New test case asserting CONNECTION_CLOSED mid-command sets both interrupted and the error. This event had no coverage at all before. Assert that an interrupted command renders the message and an enabled Retry button, and that a normal command renders neither. Manually tested with a non-trivial bit of chicanery. --- console/src/platform/shell/CommandResult.tsx | 12 +- .../src/platform/shell/HistoryOutput.test.tsx | 114 ++++++++++++++++++ console/src/platform/shell/constants.ts | 6 + .../shell/machines/webSocketFsm.test.ts | 51 +++++++- .../platform/shell/machines/webSocketFsm.ts | 18 +++ 5 files changed, 197 insertions(+), 4 deletions(-) create mode 100644 console/src/platform/shell/HistoryOutput.test.tsx diff --git a/console/src/platform/shell/CommandResult.tsx b/console/src/platform/shell/CommandResult.tsx index 65c065539992d..0ed4bcb76371a 100644 --- a/console/src/platform/shell/CommandResult.tsx +++ b/console/src/platform/shell/CommandResult.tsx @@ -115,15 +115,21 @@ const CommandResult = ({ ? `Returned in ${formatCommandDuration(timeTaken)}` : null; - // If the error comes from a query cancellation, we don't want to display it as - // an error + // A cancelled query and a dropped connection both surface as errors so that + // they get an explanation in the command block, but neither is a failure the + // server reported. Drop the "Error:" prefix for both. const errorOutputPropOverrides = - error && error.code === ErrorCode.QUERY_CANCELED + error && + (error.code === ErrorCode.QUERY_CANCELED || + error.code === ErrorCode.CONNECTION_FAILURE) ? { errorMessageOverride: capitalizeSentence(error.message, false), } : {}; + // NOTE: this deliberately does not include CONNECTION_FAILURE. An + // interrupted command never receives an end timestamp, so `timeTakenStr` is + // null and the element these props apply to is not rendered at all. const codePropOverrides = error?.code === ErrorCode.QUERY_CANCELED ? { diff --git a/console/src/platform/shell/HistoryOutput.test.tsx b/console/src/platform/shell/HistoryOutput.test.tsx new file mode 100644 index 0000000000000..383544a7a3c78 --- /dev/null +++ b/console/src/platform/shell/HistoryOutput.test.tsx @@ -0,0 +1,114 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { screen } from "@testing-library/react"; +import { Setter } from "jotai"; +import mitt from "mitt"; +import React from "react"; + +import { + defaultRegionId, + healthyEnvironment, + renderComponent, + setFakeEnvironment, +} from "~/test/utils"; + +import { COMMAND_INTERRUPTED_MESSAGE } from "./constants"; +import HistoryOutput from "./HistoryOutput"; +import { createInterruptedCommandError } from "./machines/webSocketFsm"; +import { CommandOutput, historyItemAtom, shellStateAtom } from "./store/shell"; +import { + ShellWebsocketContext, + ShellWebsocketContextType, + ShellWSEmitterEvents, +} from "./useShellWebsocket"; + +// The command echo reads Chakra's color mode, which the test providers do not +// install. Nothing here asserts on syntax highlighting. +vi.mock("./SyntaxHighlightedBlock", () => ({ + default: ({ value }: { value: string }) => 
{value}
, +})); + +const HISTORY_ID = "1"; + +const emitter = mitt(); + +const websocketContext: ShellWebsocketContextType = { + send: vi.fn(), + commitToHistory: vi.fn(), + isSocketInitializing: false, + isSocketError: false, + isSocketAvailable: true, + cacheCommand: vi.fn(), + on: emitter.on, + off: emitter.off, +}; + +function commandOutput(overrides: Partial = {}): CommandOutput { + return { + kind: "command", + historyId: HISTORY_ID, + command: "SHOW MATERIALIZED VIEWS;", + statements: [{ query: "SHOW MATERIALIZED VIEWS;" }], + commandSentTimeMs: 0, + commandResults: [{ notices: [] }], + commandResultsDisplayStates: [ + { + isSubscribeManager: false, + isFollowingSubscribeManager: true, + currentTablePage: 0, + currentSubscribeManagerTablePage: 0, + }, + ], + ...overrides, + }; +} + +async function renderHistoryOutput(historyItem: CommandOutput) { + return renderComponent( + + + , + { + initializeState: async ({ set }: { set: Setter }) => { + await setFakeEnvironment(set, defaultRegionId, healthyEnvironment); + // Retry is only enabled once the socket is idle again. + set(shellStateAtom, (prev) => ({ + ...prev, + webSocketState: "readyForQuery", + })); + set(historyItemAtom(HISTORY_ID), historyItem); + }, + }, + ); +} + +describe("HistoryOutput", () => { + it("explains why an interrupted command needs to be retried", async () => { + await renderHistoryOutput( + commandOutput({ + interrupted: true, + error: createInterruptedCommandError(), + }), + ); + + expect(screen.getByText(COMMAND_INTERRUPTED_MESSAGE)).toBeVisible(); + + const retry = screen.getByRole("button", { name: "Retry" }); + expect(retry).toBeVisible(); + expect(retry).toBeEnabled(); + }); + + it("shows neither the message nor Retry for a command that was not interrupted", async () => { + await renderHistoryOutput(commandOutput()); + + expect(screen.queryByText(COMMAND_INTERRUPTED_MESSAGE)).toBeNull(); + expect(screen.queryByRole("button", { name: "Retry" })).toBeNull(); + }); +}); diff --git a/console/src/platform/shell/constants.ts b/console/src/platform/shell/constants.ts index 407c2f606a423..1dd0f58644627 100644 --- a/console/src/platform/shell/constants.ts +++ b/console/src/platform/shell/constants.ts @@ -19,3 +19,9 @@ export const TUTORIAL_WIDTH = "600px"; export const CONNECTION_LOST_NOTICE_MESSAGE = "The connection was interrupted. Some session state may have been lost."; + +// Shown on a command that was in flight when the WebSocket closed. The socket +// has no heartbeat, so a connection that died while idle is only discovered +// when the next command is sent into it. +export const COMMAND_INTERRUPTED_MESSAGE = + "The connection to Materialize was lost. This command may not have run. Press Retry to send it again."; diff --git a/console/src/platform/shell/machines/webSocketFsm.test.ts b/console/src/platform/shell/machines/webSocketFsm.test.ts index 29ab3d2b582d1..e155f339c6e8b 100644 --- a/console/src/platform/shell/machines/webSocketFsm.test.ts +++ b/console/src/platform/shell/machines/webSocketFsm.test.ts @@ -12,7 +12,7 @@ import { interpret } from "@xstate/fsm"; import { ErrorCode, MzDataType } from "~/api/materialize/types"; import { createHistoryId } from "../historyId"; -import webSocketFsm from "./webSocketFsm"; +import webSocketFsm, { createInterruptedCommandError } from "./webSocketFsm"; const MOCKED_HISTORY_ID = "1"; @@ -229,6 +229,55 @@ describe("webSocketFsm", () => { machine.state.context.latestCommandOutput?.commandResults[0].notices[0], ).toEqual(notice); }); + + it("connection closed while a command is in flight marks it interrupted and explains why", () => { + const machine = interpret(webSocketFsm); + machine.start(); + machine.send("READY_FOR_QUERY"); + machine.send({ + type: "SEND", + command: "SELECT 5;", + statements: [{ query: "SELECT 5;" }], + }); + machine.send("COMMAND_STARTING_HAS_ROWS"); + + machine.send({ type: "CONNECTION_CLOSED" }); + + expect(machine.state.matches("interrupted")).toBeTruthy(); + expect( + machine.state.context.latestCommandOutput?.interrupted, + ).toBeTruthy(); + expect(machine.state.context.latestCommandOutput?.error).toEqual( + createInterruptedCommandError(), + ); + }); + + it("connection closed after a server error keeps the server error", () => { + const machine = interpret(webSocketFsm); + machine.start(); + machine.send("READY_FOR_QUERY"); + machine.send({ + type: "SEND", + command: "SELECT nuke_materialize();", + statements: [{ query: "SELECT nuke_materialize();" }], + }); + machine.send("COMMAND_STARTING_HAS_ROWS"); + + const error = { + message: "function nuke_materialize does not exist", + code: ErrorCode.UNDEFINED_FUNCTION, + }; + machine.send({ type: "ERROR", error }); + + // The socket can close before READY_FOR_QUERY arrives, which leaves the + // machine in a state that still counts as processing. + machine.send({ type: "CONNECTION_CLOSED" }); + + expect( + machine.state.context.latestCommandOutput?.interrupted, + ).toBeTruthy(); + expect(machine.state.context.latestCommandOutput?.error).toEqual(error); + }); }); it("should merge a notice into a command that results in an error", () => { diff --git a/console/src/platform/shell/machines/webSocketFsm.ts b/console/src/platform/shell/machines/webSocketFsm.ts index 9a716fbf6eac8..e79b91b7103b9 100644 --- a/console/src/platform/shell/machines/webSocketFsm.ts +++ b/console/src/platform/shell/machines/webSocketFsm.ts @@ -12,11 +12,13 @@ import { assign, createMachine, StateMachine } from "@xstate/fsm"; import { Column, Error, + ErrorCode, ExtendedRequestItem, Notice, } from "~/api/materialize/types"; import { assert } from "~/util"; +import { COMMAND_INTERRUPTED_MESSAGE } from "../constants"; import { CommandOutput, CommandResult, @@ -115,6 +117,19 @@ function getLatestCommandResult(latestCommandOutput?: CommandOutput) { return latestCommandResult; } +/** + * The error a command carries when the connection dropped before it finished. + * + * Materialize never sends 08006 itself, so this code unambiguously marks an + * error the client synthesized rather than one the server reported. + */ +export function createInterruptedCommandError(): Error { + return { + message: COMMAND_INTERRUPTED_MESSAGE, + code: ErrorCode.CONNECTION_FAILURE, + }; +} + const markCommandAsInterrupted = assign< WebSocketFsmContext, ConnectionClosedEvent @@ -122,6 +137,9 @@ const markCommandAsInterrupted = assign< latestCommandOutput: ({ latestCommandOutput }) => { if (latestCommandOutput) { latestCommandOutput.interrupted = true; + if (!latestCommandOutput.error) { + latestCommandOutput.error = createInterruptedCommandError(); + } } return latestCommandOutput; }, From 2dbeae007a74efa231b70b36adc3d3d67bf5e77d Mon Sep 17 00:00:00 2001 From: Patrick Butler Date: Thu, 3 Sep 2026 16:58:38 -0400 Subject: [PATCH 4/8] SS-414 Add schema validation to load_or_create_table, add test (#38639) Addresses [SS-414](https://linear.app/materializeinc/issue/SS-414/sentry-panic-itertools-zip-eq-reached-end-of-one-iterator-before-the) by adding schema validation to `load_or_create_table` in `iceberg.rs` that checks that the schema materialize has built for this sink matches the schema of the existing iceberg table, if one exists already (in the case of cluster/sink restart, 0dt upgrade, etc). This avoids the panic caused during parquet writing when there is a schema mismatch. Adds the repro test case from [SS-414](https://linear.app/materializeinc/issue/SS-414) with a small tweak to verify that the change works as intended --- src/storage/src/sink/iceberg.rs | 14 ++++ test/iceberg/alter-table-add-column.td | 96 ++++++++++++++++++++++++++ test/iceberg/mzcompose.py | 20 ++++++ 3 files changed, 130 insertions(+) create mode 100644 test/iceberg/alter-table-add-column.td diff --git a/src/storage/src/sink/iceberg.rs b/src/storage/src/sink/iceberg.rs index c416e450ee33d..eb6526aea0ddf 100644 --- a/src/storage/src/sink/iceberg.rs +++ b/src/storage/src/sink/iceberg.rs @@ -927,6 +927,20 @@ async fn load_or_create_table( Ok(table) => { // Table exists, return it // TODO: Add proper schema evolution/validation to ensure compatibility + let current_schema = table.metadata().current_schema(); + if !(current_schema.as_struct().eq(schema.as_struct()) + && current_schema + .identifier_field_ids() + .eq(schema.identifier_field_ids())) + { + anyhow::bail!( + "Iceberg table '{}' schema does not match expected schema. \ + Current schema: {:?}, expected schema: {:?}", + table_name, + current_schema, + schema + ); + } Ok(table) } Err(err) => { diff --git a/test/iceberg/alter-table-add-column.td b/test/iceberg/alter-table-add-column.td new file mode 100644 index 0000000000000..a9df7a3bbc8d4 --- /dev/null +++ b/test/iceberg/alter-table-add-column.td @@ -0,0 +1,96 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# An Iceberg sink whose input relation has more columns than the Iceberg table it +# writes to must not panic the storage worker. +# +# `load_or_create_table` hands back a pre-existing Iceberg table without checking +# that its schema still matches the sink's input, and the writer derives its Arrow +# column builders from that table schema (`merge_materialize_metadata_into_iceberg_schema`). +# Once the Materialize relation has gained a column, every row carries more datums +# than the builder has columns, and `ArrowBuilder::add_row` reaches +# `zip_eq` with iterators of different lengths: +# +# itertools: .zip_eq() reached end of one iterator before the other +# mz_arrow_util::builder::ArrowBuilder::add_row +# mz_storage::sink::iceberg::AppendEnvelopeHandler::row_to_batch +# mz_storage::sink::iceberg::write_data_files + +$ postgres-connect name=mz_system url=postgres://mz_system:materialize@${testdrive.materialize-internal-sql-addr} + +$ postgres-execute connection=mz_system +ALTER SYSTEM SET enable_alter_table_add_column = true + +> CREATE SECRET IF NOT EXISTS evolve_access_key_secret AS '${arg.s3-access-key}' + +> CREATE CONNECTION evolve_polaris TO ICEBERG CATALOG ( + CATALOG TYPE = 'REST', + URL = 'http://polaris:8181/api/catalog', + CREDENTIAL = 'root:root', + WAREHOUSE = 'default_catalog', + SCOPE = 'PRINCIPAL_ROLE:ALL' + ); + +> CREATE TABLE evolve_src (a int); + +> INSERT INTO evolve_src VALUES (1); + +# Creates the Iceberg table with columns (a, _mz_diff, _mz_timestamp). +> CREATE SINK evolve_sink_v1 + FROM evolve_src + INTO ICEBERG CATALOG CONNECTION evolve_polaris ( + NAMESPACE 'default_namespace', + TABLE 'evolve_table' + ) + MODE APPEND + WITH (COMMIT INTERVAL '1s'); + +# Iceberg sinks commit asynchronously; wait for the table to be created and the +# first batch committed before dropping the sink. +$ sleep-is-probably-flaky-i-have-justified-my-need-with-a-comment duration=5s + +$ duckdb-execute name=evolve_iceberg +CREATE SECRET s3_secret_evolve (TYPE S3, KEY_ID 'tduser', SECRET '${arg.s3-access-key}', ENDPOINT '${arg.aws-endpoint}', URL_STYLE 'path', USE_SSL false, REGION 'minio'); +SET unsafe_enable_version_guessing = true; + +$ duckdb-query name=evolve_iceberg +SELECT a, _mz_diff FROM iceberg_scan('s3://test-bucket/default_namespace/evolve_table') ORDER BY a +1 1 + +> DROP SINK evolve_sink_v1; + +> ALTER TABLE evolve_src ADD COLUMN b text; + +# The new sink's input has two columns, the Iceberg table still has one. +> CREATE SINK evolve_sink_v2 + FROM evolve_src + INTO ICEBERG CATALOG CONNECTION evolve_polaris ( + NAMESPACE 'default_namespace', + TABLE 'evolve_table' + ) + MODE APPEND + WITH (COMMIT INTERVAL '1s'); + +> INSERT INTO evolve_src VALUES (2, 'two'); + +$ sleep-is-probably-flaky-i-have-justified-my-need-with-a-comment duration=10s + +# The sink must not have taken the process down with it. Any outcome that keeps +# the storage worker alive (an error status, or a schema evolution that lets the +# write through) is acceptable here, so only assert that we can still talk to the +# cluster hosting the sink. +> SELECT count(*) FROM evolve_src; +2 + +> SELECT count(*) > 0 + FROM mz_internal.mz_sink_status_history h + JOIN mz_sinks s ON s.id = h.sink_id + WHERE s.name = 'evolve_sink_v2' + AND h.error LIKE '%schema does not match expected schema%'; +true diff --git a/test/iceberg/mzcompose.py b/test/iceberg/mzcompose.py index 37f1bb460a22e..d0da220de564e 100644 --- a/test/iceberg/mzcompose.py +++ b/test/iceberg/mzcompose.py @@ -204,6 +204,26 @@ def workflow_finite_source(c: Composition) -> None: ) +def workflow_alter_table_add_column(c: Composition) -> None: + """A sink pointed at an Iceberg table whose schema is narrower than the + sink's input relation must not panic the storage worker. The writer builds + its Arrow column builders from the Iceberg table's schema, so a row with + more datums than that schema has columns reaches `zip_eq` in + `ArrowBuilder::add_row`.""" + key = _setup(c) + + c.run_testdrive_files( + f"--var=s3-access-key={key}", + "--var=aws-endpoint=minio:9000", + "alter-table-add-column.td", + ) + + logs = c.invoke("logs", "materialized", capture=True) + assert ( + "zip_eq" not in logs.stdout + ), "storage worker panicked in ArrowBuilder::add_row" + + def _polaris_get(table_url: str, access_token: str) -> dict: """GET table metadata from Polaris REST API (always returns latest).""" req = urllib.request.Request( From 62f55a3fd4536e552463fea0934cf19283f95d70 Mon Sep 17 00:00:00 2001 From: Kay Kim Date: Thu, 3 Sep 2026 16:59:08 -0400 Subject: [PATCH 5/8] docs: update indexes w. mz material + corrections (#38641) --- doc/user/content/concepts/clusters.md | 34 +-- doc/user/content/concepts/indexes.md | 231 ++++++++++-------- doc/user/content/headless/index-ordering.md | 16 -- doc/user/content/transform-data/faq.md | 26 +- .../content/transform-data/optimization.md | 2 +- doc/user/data/index_details.yml | 102 +++++++- .../static/images/indexes/index_in_memory.svg | 231 +++++++++--------- 7 files changed, 377 insertions(+), 265 deletions(-) delete mode 100644 doc/user/content/headless/index-ordering.md diff --git a/doc/user/content/concepts/clusters.md b/doc/user/content/concepts/clusters.md index 05f5f542f0162..cfa8cbd9b533d 100644 --- a/doc/user/content/concepts/clusters.md +++ b/doc/user/content/concepts/clusters.md @@ -16,6 +16,21 @@ aliases: Clusters are pools of compute resources (CPU, memory, and scratch disk space) for running your workloads. +## Resource isolation + +Clusters provide **resource isolation.** Each cluster provisions dedicated +compute resources and can fail independently from other clusters. All workloads +on a given cluster compete for access to that cluster's compute resources. + +Workloads on different clusters are strictly isolated from one another. That is, +a given workload has access only to the CPU, memory, and scratch disk of the +cluster it runs on. + +Resource isolation lets you place workloads on separate clusters to prevent +them from competing for compute resources. + +See also [three-tier architecture](#three-tier-architecture-in-production). + ## Clusters and workloads The following operations require a cluster in Materialize: @@ -37,7 +52,8 @@ SET CLUSTER = 'my_transform_cluster'; [`SELECT`] and [`SUBSCRIBE`] statements run in the session's active cluster. Objects that require compute (e.g., indexes, materialized views, sources) are -associated with a cluster when they are created, either: +associated with a cluster when they are created. The associated cluster is +either: - the session's active cluster by default, or @@ -54,22 +70,6 @@ associated with a cluster when they are created, either: For more on indexes and clusters, see [Indexes](/concepts/indexes/). -## Resource isolation - -Clusters provide **resource isolation.** Each cluster provisions dedicated -compute resources and can fail independently from other clusters. All workloads -on a given cluster compete for access to that cluster's compute resources. - -Workloads on different clusters are strictly isolated from one another. That is, -a given workload has access only to the CPU, memory, and scratch disk of the -cluster it runs on. - -Resource isolation lets you place workloads on separate clusters to prevent -them from competing for compute resources: for example, sources in one -cluster, materialized views in a second, and indexes that serve queries in a -third, as in the recommended [three-tier -architecture](#three-tier-architecture-in-production). - ## Cluster replicas The [replication factor](/sql/create-cluster/#replication-factor) of a cluster diff --git a/doc/user/content/concepts/indexes.md b/doc/user/content/concepts/indexes.md index 9d2b69cde128a..286ba55b7b923 100644 --- a/doc/user/content/concepts/indexes.md +++ b/doc/user/content/concepts/indexes.md @@ -12,15 +12,25 @@ aliases: - /self-managed/v25.2/concepts/indexes/ --- -In Materialize, you can create indexes on [views](/concepts/views/#views) and -[materialized views](/concepts/views/#materialized-views) as well as tables, -[sources](/concepts/sources/), and subsources. - ## Overview {{% include-from-yaml data="index_details" name="definition" %}} -## Indexes on sources, tables, and subsources +## Creating indexes on objects + +In Materialize, you can create indexes on [views](/concepts/views/#views) and +[materialized views](/concepts/views/#materialized-views) as well as on +[sources, tables, and subsources](/concepts/sources/). + +To create indexes on an object, use the [`CREATE INDEX`](/sql/create-index/) +command. To create the index in a cluster other than the active cluster, include +the `IN CLUSTER` clause in the `CREATE INDEX` statement. + +{{% include-example file="examples/create_index" example="syntax" %}} + +See [`CREATE INDEX`](/sql/create-index/) for the syntax details. + +### Indexes on sources, tables, and subsources {{< note >}} @@ -29,132 +39,141 @@ or subsources without performing some transformation using a view, etc. {{}} -In Materialize, you can create indexes on a [source and its tables or -subsources](/concepts/sources/) to maintain in-memory up-to-date data within the -cluster you create the index. This can help improve [query -performance](#indexes-and-query-optimizations) such as when [using +In Materialize, you can create indexes on [sources, tables, or +subsources](/concepts/sources/) to maintain up-to-date data in the memory of +the cluster where you create the index. This can help improve [query +performance](#indexes-and-query-optimizations), for example when [using joins](/transform-data/optimization/#join) in your transformation. However, in practice, you may find that you rarely need to index these objects directly. ```mzsql -CREATE INDEX idx_on_my_source_table ON my_source_table (...); +CREATE INDEX idx_on_my_source_table ON my_source_table(...); ``` -## Indexes on views +### Indexes on views -In Materialize, you can create indexes on a [view](/concepts/views/#views "query -saved under a name") to maintain **up-to-date view results in memory** within -the [cluster](/concepts/clusters/) you create the index. +In Materialize, you can [create indexes](/sql/create-index/) on a +[view](/concepts/views/#views "query saved under a name") to maintain +**up-to-date view results in memory** within the [cluster](/concepts/clusters/) +where you create the index. -```mzsql -CREATE INDEX idx_on_my_view ON my_view_name(...) ; -``` +- To create the index in the current active cluster (you can use the `SET + CLUSTER` command to change the active cluster): + + ```mzsql + CREATE INDEX idx_on_my_view ON my_view_name(...); + ``` -During the index creation on a [view](/concepts/views/#views "query saved under -a name"), the view is executed and the view results are stored in memory within -the cluster. **As new data arrives**, the index **incrementally updates** the -view results in memory. +- To create the index in a specified cluster: -Within the cluster, querying an indexed view is **fast** because the results are -already computed and are served from memory. + ```mzsql + CREATE INDEX idx_on_my_view IN CLUSTER serving_cluster ON my_view_name(...); + ``` -For best practices on using indexes, and understanding when to use indexed views -vs. materialized views, see [Usage patterns](#usage-patterns). +During the index creation, the view is executed and the view results are stored +in memory within the cluster. **As new data arrives**, the index **incrementally +updates** the view results in memory. -## Indexes on materialized views +Querying a view from a cluster where the view is indexed is **fast** because +the results are already computed and are served from memory. Querying a view +from a cluster where the view isn't indexed requires executing the view each +time you query it. + +### Indexes on materialized views In Materialize, materialized view results are stored in durable storage and -**incrementally updated** as new data arrives. Indexing a materialized view -makes the already up-to-date view results available **in memory** within the -[cluster](/concepts/clusters/) you create the index. That is, indexes on -materialized views require no additional computation to keep results up-to-date. +**incrementally updated** as new data arrives. [Indexing](/sql/create-index/) a +materialized view makes the already up-to-date view results available **in +memory** within the [cluster](/concepts/clusters/) where you create the index. +That is, indexes on materialized views require no additional computation to keep +results up-to-date. {{< note >}} A materialized view can be queried from any cluster whereas its indexed results -are available only within the cluster you create the index. Querying a -materialized view, whether indexed or not, from any cluster is fast since the -results are already computed. However, querying an indexed materialized view -within the cluster where the index is created is faster since the results are -served from memory rather than from storage. +are available only within the cluster where you create the index. Querying a +materialized view from any cluster, whether the materialized view is indexed or +not, is fast because the results are already computed. However, querying an +indexed materialized view from a cluster where the materialized view is indexed +is faster since the results are served from memory rather than from storage. {{}} -For best practices on using indexes, and understanding when to use indexed views -vs. materialized views, see [Usage patterns](#usage-patterns). +- To create the index in the current active cluster (you can use the `SET + CLUSTER` command to change the active cluster): -```mzsql -CREATE INDEX idx_on_my_mat_view ON my_mat_view_name(...) ; -``` + ```mzsql + CREATE INDEX idx_on_my_mat_view ON my_mat_view_name(...); + ``` -## Indexes and clusters +- To create the index in a specified cluster: -{{% include-from-yaml data="index_details" name="index-cluster-local" %}} + ```mzsql + CREATE INDEX idx_on_my_mat_view IN CLUSTER serving_cluster ON my_mat_view_name(...); + ``` -As such, queries issued from a different cluster cannot use the index. +## Properties -For example, to create an index in the current cluster: +### Cluster-local -```mzsql -CREATE INDEX idx_on_my_view ON my_view_name(...) ; -``` +{{% include-from-yaml data="index_details" name="index-cluster-local" %}} As +such, references to the indexed object from a different cluster cannot use the +index. -You can also explicitly specify the cluster: +### Data distribution and ordering -```mzsql -CREATE INDEX idx_on_my_view IN CLUSTER active_cluster ON my_view (...); -``` +{{% include-from-yaml data="index_details" name="index-key-distribution" %}} -## Usage patterns +{{% include-from-yaml data="index_details" name="index-key-ordering-within-workers" %}} -### Index usage +### Serving ad-hoc queries -{{% important %}} -Indexes are local to a cluster. Queries in one cluster cannot use the indexes in another, different cluster. -{{% /important %}} +Within a cluster, all ad-hoc queries that reference an indexed object read from +the index, regardless of whether the index is optimized for the query. This +includes queries that do not specify a `WHERE` condition on the index key. +Because the indexed results are already up-to-date and in memory, reading from +an index avoids recomputing the results. -Unlike some other databases, Materialize can use an index to serve query results -even if the query does not specify a `WHERE` condition on the index key. Serving -queries from an index is fast since the results are already up-to-date and in -memory. +- **Point lookups**: For queries that specify an equality condition on the full + index key, Materialize can perform a point lookup, reading only the matching + records from the index. Point lookups are the most efficient use of an index. + See [Point lookups](#point-lookups) for the exact requirements. -For example, consider the following index: +- **Index scans**: Otherwise, Materialize scans the index. Although the indexed + results are already up-to-date and in memory, a full index scan must examine + the indexed results and is less efficient than a point lookup. The performance + of full index scans degrades with data volume. -```mzsql -CREATE INDEX idx_orders_view_qty ON orders_view (quantity); -``` +### Index use by objects -Materialize will maintain the `orders_view` in memory in `idx_orders_view_qty`, -and it will be able to use the index to serve a various queries on the -`orders_view` (and not just queries that specify conditions on -`orders_view.quantity`). +{{% include-from-yaml data="index_details" name="index-reuse" %}} -Materialize can use the index for the following queries (issued from the same -cluster as the index) on `orders_view`: +To inspect index reuse and dependencies: -```mzsql -SELECT * FROM orders_view; -- scans the index -SELECT * FROM orders_view WHERE status = 'shipped'; -- scans the index -SELECT * FROM orders_view WHERE quantity = 10; -- point lookup on the index -``` +- To check whether a new index would reuse an existing index before creating + it, use [`EXPLAIN CREATE INDEX`](/sql/explain-plan/). + +- To find which indexes and materialized views use an index, query + [`mz_internal.mz_materialization_dependencies`](/reference/system-catalog/mz_internal/#mz_materialization_dependencies). -For the queries that do not specify a condition on the indexed field, -Materialize scans the index. For the query that specifies an equality condition -on the indexed field, Materialize performs a **point lookup** on the index -(i.e., reads just the matching records from the index). Point lookups are the -most efficient use of an index. +### Limitations -#### Point lookups +{{% include-from-yaml data="index_details" name="index-not-optimized" %}} -Materialize performs **point lookup** (i.e., reads just the matching records -from the index) on the index if the query's `WHERE` clause: +## Point lookups vs index scans + +### Point lookups + +Point lookups read just the matching records from the index and are the most +efficient use of an index. Materialize performs a point lookup if the query's +`WHERE` clause: - Specifies equality (`=` or `IN`) condition and **only** equality conditions on **all** the indexed fields. The equality conditions must specify the **exact** index key expression (including type) for point lookups. For example: - If the index is on `round(quantity)`, the query must specify equality - condition on `round(quantity)` (and not just `quanity`) for Materialize to + condition on `round(quantity)` (and not just `quantity`) for Materialize to perform a point lookup. - If the index is on `quantity * price`, the query must specify equality @@ -166,39 +185,55 @@ from the index) on the index if the query's `WHERE` clause: - Only uses `AND` (conjunction) to combine conditions for **different** fields. -Point lookups are the most efficient use of an index. - For queries whose `WHERE` clause meets the point lookup criteria and includes conditions on additional fields (also using `AND` conjunction), Materialize performs a point lookup on the index keys and then filters the results using the additional conditions on the non-indexed fields. -For queries that do not meet the point lookup criteria, Materialize performs a -full index scan (including for range queries). That is, Materialize performs a -full index scan if the `WHERE` clause: +### Index scans + +For queries that do not meet the [point lookup criteria](#point-lookups), +Materialize performs a full index scan (including for range queries). That is, +Materialize performs a full index scan if the `WHERE` clause: - Does not specify **all** the indexed fields. - Does not specify only equality conditions on the index fields or specifies an equality condition that specifies a different value type than the index key type. -- Uses OR (disjunction) to combine conditions for **different** fields. +- Uses `OR` (disjunction) to combine conditions for **different** fields. -Full index scans are less efficient than point lookups. The performance of full +Full index scans are less efficient than point lookups. The performance of full index scans will degrade with data volume; i.e., as you get more data, full scans will get slower. -#### Examples +### Examples + +Within a cluster, indexes can serve queries that reference an indexed object, +regardless of whether the index is optimized for the query. -Consider again the following index on a view: +Consider the following index on the `orders_view`: ```mzsql -CREATE INDEX idx_orders_view_qty on orders_view (quantity); +CREATE INDEX idx_orders_view_qty ON orders_view (quantity); +``` + +Materialize can use the index to serve various queries on the `orders_view` +(and not just queries that specify conditions on `orders_view.quantity`). For +example: + +```mzsql +SELECT * FROM orders_view; -- scans the index +SELECT * FROM orders_view WHERE status = 'shipped'; -- scans the index +SELECT * FROM orders_view WHERE quantity = 10; -- point lookup on the index ``` +For the queries that do not satisfy the [point-lookup +conditions](#point-lookups), Materialize scans the index. + The following table shows various queries and whether Materialize performs a point lookup or an index scan. -{{< yaml-table data="examples/index_usage/index_usage_key_quantity" >}} +{{% yaml-table data="examples/index_usage/index_usage_key_quantity" %}} Consider that the view has an index on the `quantity` and `price` fields instead of an index on the `quantity` field: @@ -208,11 +243,9 @@ DROP INDEX idx_orders_view_qty; CREATE INDEX idx_orders_view_qty_price on orders_view (quantity, price); ``` -{{< yaml-table data="examples/index_usage/index_usage_key_quantity_price" >}} - -#### Limitations +{{% yaml-table data="examples/index_usage/index_usage_key_quantity_price" %}} -{{% include-headless "/headless/index-ordering" %}} +## Usage ### Indexes on views vs. materialized views diff --git a/doc/user/content/headless/index-ordering.md b/doc/user/content/headless/index-ordering.md deleted file mode 100644 index 2489a2afee243..0000000000000 --- a/doc/user/content/headless/index-ordering.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -headless: true ---- - -Indexes in Materialize do not order their keys using the data type's natural -ordering and instead orders by its internal representation of the key (the tuple -of key length and value). - -As such, indexes in Materialize currently do not provide optimizations for: - -- Range queries; that is queries using `>`, `>=`, - `<`, `<=`, `BETWEEN` clauses (e.g., `WHERE - quantity > 10`, `price >= 10 AND price <= 50`, and `WHERE quantity - BETWEEN 10 AND 20`). - -- `GROUP BY`, `ORDER BY` and `LIMIT` clauses. diff --git a/doc/user/content/transform-data/faq.md b/doc/user/content/transform-data/faq.md index 4b921c6b0ddb8..755f921b9c261 100644 --- a/doc/user/content/transform-data/faq.md +++ b/doc/user/content/transform-data/faq.md @@ -11,14 +11,28 @@ aliases: - /self-managed/v25.2/transform-data/faq/ --- -## Do indexes in Materialize support `ORDER BY`? +## Are indexes in Materialize optimized for `ORDER BY`? -No. Indexes in Materialize do not support `ORDER BY` clauses. +No. -{{% include-headless "/headless/index-ordering" %}} +{{% include-from-yaml data="index_details" name="index-key-distribution" %}} -## Do indexes in Materialize support range queries? +{{% include-from-yaml data="index_details" name="index-key-ordering-within-workers" %}} -No. Indexes in Materialize do not support range queries. +As such, Materialize indexes are not optimized for ordered access, including +`ORDER BY` clauses. -{{% include-headless "/headless/index-ordering" %}} +## Are indexes in Materialize optimized for range queries? + +No. + +{{% include-from-yaml data="index_details" name="index-key-distribution" %}} + +{{% include-from-yaml data="index_details" name="index-key-ordering-within-workers" %}} + +As such, Materialize indexes are not optimized for ordered access, including +range queries. + +## Are indexes in Materialize optimized for `GROUP BY` aggregations? + +No. {{< include-from-yaml data="index_details" name="index-groupby-not-optimized" >}} diff --git a/doc/user/content/transform-data/optimization.md b/doc/user/content/transform-data/optimization.md index 0063cfb439f2d..2d7c76f87fdf0 100644 --- a/doc/user/content/transform-data/optimization.md +++ b/doc/user/content/transform-data/optimization.md @@ -448,7 +448,7 @@ The following are the possible index usage types: ### Limitations -{{% include-headless "/headless/index-ordering" %}} +{{% include-from-yaml data="index_details" name="index-not-optimized" %}} ## Query hints diff --git a/doc/user/data/index_details.yml b/doc/user/data/index_details.yml index a5fd97610f84c..bad35ecfa3ca5 100644 --- a/doc/user/data/index_details.yml +++ b/doc/user/data/index_details.yml @@ -6,9 +6,12 @@ - name: definition content: | Materialize indexes maintain the full result set of the indexed object in - the memory of the [cluster](/concepts/clusters/) where the index is - created.[^db-term] The indexed results are kept up-to-date as new data - arrives. + the memory of the [cluster](/concepts/clusters/) where the index is created. + The cluster's workers keep the indexed results up-to-date as new data + arrives. Like clustered[^db-term] hash indexes, Materialize indexes store + the indexed results themselves and are efficient for equality lookups on the + full index key. Materialize indexes are not themselves hash indexes; hashing + is used only to distribute the index across the cluster's workers. ![Materialize index maintains the full result set in memory](/images/indexes/index_in_memory.svg) @@ -17,14 +20,14 @@ ![Materialize indexes do not use a key-pointer structure.](/images/indexes/index_not_key_pointer.svg) - [^db-term]: Materialize indexes are like clustered hash indexes. The term - *clustered index* is a database term unrelated to Materialize clusters, - which are compute resources. + [^db-term]: The term *clustered + index* is a database term unrelated to Materialize clusters, which are + compute resources. # Shared with the clusters concept page, which composes it after the # cross-cluster-accessibility entry in cluster_details.yml. Renders on both # the clusters and indexes pages, so avoid page-root links to either page; -# deep anchor links (e.g., #resource-isolation) are fine — on clusters.md +# deep anchor links (e.g., #resource-isolation) are fine. On clusters.md # they act as same-page jumps. - name: index-cluster-local content: | @@ -32,3 +35,88 @@ in the memory of the cluster where the index is created, and a [cluster's memory](/concepts/clusters/#resource-isolation) cannot be accessed from another cluster. + +# Creation-order semantics for index reuse. The hydration page's "index-order" +# strategy bullet (concepts/hydration.md) states the same create-before- +# consumers rule; keep them consistent. +- name: index-reuse + content: | + Within a cluster, an index can be used not only by ad-hoc queries but also + by other indexes and materialized views. For an index or materialized view + to use another index, however, that index must exist when the dependent + object is created. That is: + + - When you create an index or a materialized view, Materialize plans how + to compute its results at creation time. As part of planning, Materialize + checks whether it can reuse an **existing** index in the **same** + cluster. + + - Because the plan is bound at creation time, creation order matters. An + index or materialized view that is already running will **not** adopt an + index created afterward. To have an existing index or materialized view + use a newer index, drop and recreate the existing object. However, + recreating an index or a materialized view triggers + [hydration](/concepts/hydration/#when-hydration-occurs). + + Ad-hoc queries, by contrast, are planned at query time. They can use any + index that exists in the cluster when the query runs. + + {{< note >}} + Reusing an index saves computation since the dependent objects read the + index's maintained results instead of recomputing them from the base data. + However, each new index has costs related to cluster memory and ongoing + maintenance, especially indexes on regular views. + {{< /note >}} + +- name: index-key-distribution + content: | + The index data is distributed across the cluster's + workers by a hash of the key, which spreads the maintenance and lookup work + across the cluster. + +- name: index-key-ordering-within-workers + content: | + Within each worker, index keys are ordered by their internal representation + (the encoded key's length, then its bytes), not by the data types' natural + ordering. + +# GROUP BY gets no index-specific optimization: the reduce operator always +# builds and maintains its own arrangement keyed by the grouping key, so an +# index on the grouping key does not reduce aggregation work. Single +# paragraph (>-): nested via the {{< >}} angle form inside the GROUP BY +# bullet of the index-not-optimized entry below, and rendered inline after +# "No." on transform-data/faq.md. +- name: index-groupby-not-optimized + content: >- + An index on the grouping key does not reduce the work of computing the + aggregation: Materialize reads the full index and maintains the + aggregation separately. + +# Consumers: concepts/indexes.md and transform-data/optimization.md, both +# under a "Limitations" heading. On optimization.md the list renders with no +# supporting context, so each bullet must stay self-contained and carry its +# own reason where one is needed. The GROUP BY bullet nests the +# index-groupby-not-optimized entry above; it must use the {{< >}} angle +# form (the {{% %}} form ejects the nested content out of the list). +- name: index-not-optimized + content: | + Materialize indexes are not optimized for: + + - Ordered access, including: + + - Range queries, that is, queries using `>`, `>=`, `<`, `<=`, or `BETWEEN` + (e.g., `WHERE quantity > 10`, `WHERE price >= 10 AND price <= 50`, and + `WHERE quantity BETWEEN 10 AND 20`). + + - Queries that use `ORDER BY` on the index key. + + - Lookups on a prefix of a multi-column index key. For example, an index + with the key `(a, b)` is not optimized for a query that specifies an + equality condition on `a` but not on `b`. + + - Lookups that do not match the exact index key expression. For example, + for an index with the key `lower(a)`, an equality condition on `a` does + not match the index key; the query must specify an equality condition on + `lower(a)` for a point lookup. + + - `GROUP BY` aggregations. {{< include-from-yaml data="index_details" name="index-groupby-not-optimized" >}} diff --git a/doc/user/static/images/indexes/index_in_memory.svg b/doc/user/static/images/indexes/index_in_memory.svg index ed6676610cc73..0def5002bd683 100644 --- a/doc/user/static/images/indexes/index_in_memory.svg +++ b/doc/user/static/images/indexes/index_in_memory.svg @@ -1,121 +1,114 @@ - - -In Memory - -cart_items_subtotals_idx -cart_totals_idx - - -hash -cart_id -user_id -cart_item_id -product_id -product_name -quantity -price -line_total -alert -7b4e1d52 -2 -3 -3 -7 -Rye Bread -1 -7.49 -7.49 -Notice: 0 in stock. -a3f2c891 -1 -2 -1 -3 -Baguette -1 -4.5 -4.5 -null -a3f2c891 -1 -2 -2 -6 -Chocolate Eclair -5 -4.99 -24.95 -null -f1c08e3a -3 -5 -6 -2 -Croissant -3 -3.49 -10.47 -null -f1c08e3a -3 -5 -4 -4 -Cinnamon Roll -3 -5.99 -17.97 -null -f1c08e3a -3 -5 -5 -5 -Blueberry Muffin -10 -3.99 -39.9 -null + + +In Memory + +cart_items_subtotals_idx +cart_totals_idx + + + + +cart_id +user_id +cart_item_id +product_id +product_name +quantity +price +line_total +alert +2 +3 +3 +7 +Rye Bread +1 +7.49 +7.49 +Notice: 0 in stock. +1 +2 +1 +3 +Baguette +1 +4.5 +4.5 +null +1 +2 +2 +6 +Chocolate Eclair +5 +4.99 +24.95 +null +3 +5 +6 +2 +Croissant +3 +3.49 +10.47 +null +3 +5 +4 +4 +Cinnamon Roll +3 +5.99 +17.97 +null +3 +5 +5 +5 +Blueberry Muffin +10 +3.99 +39.9 +null - - - - - - - - - - - - - - - - - - - - -hash -cart_id -cart_total -7b4e1d52 -2 -7.49 -a3f2c891 -1 -29.45 -f1c08e3a -3 -68.34 - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + +cart_id +cart_total +2 +7.49 +1 +29.45 +3 +68.34 + + + + + + + + + +index key column From 7053f0b019617fd1d0730b22905d290148429ebe Mon Sep 17 00:00:00 2001 From: Sang Jun Bak Date: Thu, 3 Sep 2026 17:19:29 -0400 Subject: [PATCH 6/8] sql-499: Convert mz_object_dependencies (#38252) I've tried my best to explain each change in the commit messages, but I've also shared some nicely formatted explanations that I used to understand the changes myself: - Why we don't resolve an array type to both its array type (e.g. _int4) and its element type (int4) anymore (commit 1): https://claude.ai/code/artifact/f9d39cf2-eff9-4a80-b5e0-c712183a24bd - How we were able to recreate the dependency graph via a SQL function (commits >= 2): https://claude.ai/code/artifact/f3b59447-9924-49d6-8085-1db9dff69048 . Ignore the diff harness comment. You might notice some code is different, but the motivation/high level approach is the same. ### Motivation Partially does sql-499 ### Verification - More SLTs for testing the ported builtin more in depth - Unit tests for the parser changes - I created a harness that diffs builtins for all system objects, and user objects created from Dennis' workload replay. It's in draft https://github.com/MaterializeInc/materialize/pull/38489 but used it and things were 1:1 --- .../reference/system-catalog/mz_internal.md | 5 +- .../materialize/parallel_workload/action.py | 23 +- src/adapter/src/catalog.rs | 71 +++ .../src/catalog/builtin_table_updates.rs | 29 +- .../catalog/open/builtin_schema_migration.rs | 11 + src/catalog/src/builtin.rs | 125 ++++- src/catalog/src/builtin/mz_catalog.rs | 14 +- src/catalog/src/builtin/mz_internal.rs | 93 +--- .../src/builtin/mz_object_dependencies.rs | 465 ++++++++++++++++++ src/compute-types/tests/snapshots/lir_v1.json | 6 + src/expr/src/scalar/func/impls/jsonb.rs | 121 ++++- src/expr/src/scalar/func/unary.rs | 1 + src/pgrepr-consts/src/oid.rs | 4 +- src/pgrepr-consts/src/regproc.rs | 1 + src/sql-parser/src/ast.rs | 1 + src/sql-parser/src/ast/item_refs.rs | 389 +++++++++++++++ src/sql/src/func.rs | 5 + src/sql/src/names.rs | 39 +- .../autogenerated/mz_internal.slt | 1 + test/sqllogictest/catalog_server_explain.slt | 224 ++++++++- test/sqllogictest/funcs.slt | 19 + .../information_schema_tables.slt | 6 +- .../mz_catalog_server_index_accounting.slt | 166 ++++--- test/sqllogictest/mz_object_dependencies.slt | 313 ++++++++++++ test/sqllogictest/oid.slt | 2 + test/testdrive/catalog.td | 5 +- test/testdrive/mz-depends.td | 30 ++ 27 files changed, 1907 insertions(+), 262 deletions(-) create mode 100644 src/catalog/src/builtin/mz_object_dependencies.rs create mode 100644 src/sql-parser/src/ast/item_refs.rs create mode 100644 test/sqllogictest/mz_object_dependencies.slt diff --git a/doc/user/content/reference/system-catalog/mz_internal.md b/doc/user/content/reference/system-catalog/mz_internal.md index 76d7e78b76ae4..40165fecba5cf 100644 --- a/doc/user/content/reference/system-catalog/mz_internal.md +++ b/doc/user/content/reference/system-catalog/mz_internal.md @@ -649,8 +649,8 @@ has no replicas — needs operator action." ## `mz_object_dependencies` -The `mz_object_dependencies` table describes the dependency structure between -all database objects in the system. +The `mz_object_dependencies` materialized view describes the dependency +structure between all database objects in the system. | Field | Type | Meaning | @@ -1623,6 +1623,7 @@ The `mz_webhook_sources` table contains a row for each webhook source in the sys + diff --git a/misc/python/materialize/parallel_workload/action.py b/misc/python/materialize/parallel_workload/action.py index 194d7b2f31516..fcb1f2a980898 100644 --- a/misc/python/materialize/parallel_workload/action.py +++ b/misc/python/materialize/parallel_workload/action.py @@ -1753,18 +1753,17 @@ def run(self, exe: Executor) -> bool: if index not in exe.db.indexes: return False - query = f"DROP INDEX {index}" - try: - exe.execute(query, http=Http.RANDOM) - except QueryError: - # The indexed object or its schema may have been dropped - # concurrently, taking the index with it. Untrack the index - # either way so stale entries don't fill up the set and choke - # off CreateIndexAction. Use discard, not remove: a concurrent - # CASCADE drop's untrack_objects_in_schemas may have already - # removed it, and remove would raise KeyError. - exe.db.indexes.discard(index) - raise + # The indexed object or its schema may have dropped + # concurrently, taking the index with it. Thus whether + # we drop it successfully or find it doesn't exist, we + # untrack its entry as to not fill up the set and choke + # off CreateIndexAction. An RBAC error however (in case + # a non-owner tries this drop) will raise an error and + # we'll retry the DropIndexAction. + exe.execute(f"DROP INDEX IF EXISTS {index}", http=Http.RANDOM) + # Use discard, not remove: a concurrent + # CASCADE drop's untrack_objects_in_schemas may have already + # removed it, and remove would raise KeyError. exe.db.indexes.discard(index) return True diff --git a/src/adapter/src/catalog.rs b/src/adapter/src/catalog.rs index b29d44a2ad12f..38d125ae21c56 100644 --- a/src/adapter/src/catalog.rs +++ b/src/adapter/src/catalog.rs @@ -2836,6 +2836,77 @@ mod tests { .await; } + /// Resolving a statement and resolving its normalized `create_sql` must + /// yield the same `ResolvedIds`. + /// + /// The in-memory catalog records the ids from the first resolution, while + /// a catalog reload re-derives them from the stored `create_sql`. Any + /// disagreement makes the reloaded catalog differ from the in-memory one. + #[mz_ore::test(tokio::test)] + #[cfg_attr(miri, ignore)] // slow + async fn test_resolved_ids_survive_create_sql_round_trip() { + use mz_ore::collections::CollectionExt; + Catalog::with_debug(|catalog| async move { + let conn_catalog = catalog.for_system_session(); + let scx = &mut StatementContext::new(None, &conn_catalog); + + let resolve_and_normalize = |scx: &mut StatementContext, sql: &str| { + let parsed = mz_sql_parser::parser::parse_statements(sql) + .expect("parses") + .into_element() + .ast; + let (stmt, ids) = names::resolve(scx.catalog, parsed).expect("resolves"); + let normalized = + mz_sql::normalize::create_statement(scx, stmt).expect("normalizes"); + (ids, normalized) + }; + + const ARRAY_SUFFIX: &str = "create table public.t (a pg_catalog.int4[])"; + const ARRAY_TYPE: &str = "create table public.t (a pg_catalog._int4)"; + const ELEMENT_TYPE: &str = "create table public.t (a pg_catalog.int4)"; + + let mut ids_by_spelling: BTreeMap<&str, Vec> = BTreeMap::new(); + for sql in [ + ARRAY_SUFFIX, + ARRAY_TYPE, + ELEMENT_TYPE, + "create table public.t (a pg_catalog.int4 list)", + "create view public.v as select null::pg_catalog.text[]", + ] { + let (ids, normalized) = resolve_and_normalize(scx, sql); + let (round_tripped_ids, _) = resolve_and_normalize(scx, &normalized); + assert_eq!( + ids.items().collect::>(), + round_tripped_ids.items().collect::>(), + "resolving {normalized:?} produced different ids than {sql:?}", + ); + ids_by_spelling.insert(sql, ids.items().copied().collect()); + } + + // `int4[]` and `_int4` name the same type, so both spellings must + // record the same ids. `int4[]` is only ever stored as `_int4`, + // so this is what keeps the reloaded catalog identical to the + // in-memory one. + assert_eq!( + ids_by_spelling[ARRAY_SUFFIX], ids_by_spelling[ARRAY_TYPE], + "{ARRAY_SUFFIX:?} and {ARRAY_TYPE:?} must resolve to the same ids", + ); + + // Neither spelling records the element type: an array reference + // names the array type alone. + let element_ids = &ids_by_spelling[ELEMENT_TYPE]; + assert!( + ids_by_spelling[ARRAY_SUFFIX] + .iter() + .all(|id| !element_ids.contains(id)), + "{ARRAY_SUFFIX:?} recorded an element type id from {ELEMENT_TYPE:?}", + ); + + catalog.expire().await; + }) + .await; + } + // Test that if a large catalog item is somehow committed, then we can still load the catalog. #[mz_ore::test(tokio::test)] #[cfg_attr(miri, ignore)] // slow diff --git a/src/adapter/src/catalog/builtin_table_updates.rs b/src/adapter/src/catalog/builtin_table_updates.rs index b71258e3898f0..b05dc8a6a1de4 100644 --- a/src/adapter/src/catalog/builtin_table_updates.rs +++ b/src/adapter/src/catalog/builtin_table_updates.rs @@ -18,10 +18,10 @@ use mz_catalog::builtin::{ BuiltinTable, MZ_AGGREGATES, MZ_ARRAY_TYPES, MZ_BASE_TYPES, MZ_CLUSTER_REPLICA_SIZE_INTERNAL, MZ_CLUSTER_REPLICA_SIZES, MZ_COLUMNS, MZ_EGRESS_IPS, MZ_FUNCTIONS, MZ_HISTORY_RETENTION_STRATEGIES, MZ_INDEX_COLUMNS, MZ_LICENSE_KEYS, MZ_LIST_TYPES, - MZ_MAP_TYPES, MZ_MATERIALIZED_VIEW_REFRESH_STRATEGIES, MZ_OBJECT_DEPENDENCIES, - MZ_OBJECT_GLOBAL_IDS, MZ_OPERATORS, MZ_PSEUDO_TYPES, MZ_REPLACEMENTS, MZ_ROLE_AUTH, - MZ_SESSIONS, MZ_SOURCE_REFERENCES, MZ_STORAGE_USAGE_BY_SHARD, MZ_SUBSCRIPTIONS, - MZ_TYPE_PG_METADATA, MZ_TYPES, MZ_WEBHOOKS_SOURCES, + MZ_MAP_TYPES, MZ_MATERIALIZED_VIEW_REFRESH_STRATEGIES, MZ_OBJECT_GLOBAL_IDS, MZ_OPERATORS, + MZ_PSEUDO_TYPES, MZ_REPLACEMENTS, MZ_ROLE_AUTH, MZ_SESSIONS, MZ_SOURCE_REFERENCES, + MZ_STORAGE_USAGE_BY_SHARD, MZ_SUBSCRIPTIONS, MZ_TYPE_PG_METADATA, MZ_TYPES, + MZ_WEBHOOKS_SOURCES, }; use mz_catalog::durable::SourceReferences; use mz_catalog::memory::error::Error; @@ -103,19 +103,6 @@ impl CatalogState { BuiltinTableUpdate { id, data } } - pub fn pack_depends_update( - &self, - depender: CatalogItemId, - dependee: CatalogItemId, - diff: Diff, - ) -> BuiltinTableUpdate<&'static BuiltinTable> { - let row = Row::pack_slice(&[ - Datum::String(&depender.to_string()), - Datum::String(&dependee.to_string()), - ]); - BuiltinTableUpdate::row(&*MZ_OBJECT_DEPENDENCIES, row, diff) - } - pub(super) fn pack_role_auth_update( &self, id: RoleId, @@ -206,14 +193,6 @@ impl CatalogState { CatalogItem::Connection(_) => vec![], }; - if !entry.item().is_temporary() { - // Populate or clean up the `mz_object_dependencies` table. - // TODO(jkosh44) Unclear if this table wants to include all uses or only references. - for dependee in entry.item().references().items() { - updates.push(self.pack_depends_update(id, *dependee, diff)) - } - } - // Always report the latest for an objects columns. if let Some(desc) = entry.relation_desc_latest() { let defaults = match entry.item() { diff --git a/src/adapter/src/catalog/open/builtin_schema_migration.rs b/src/adapter/src/catalog/open/builtin_schema_migration.rs index 1754ccbf26250..34653cd13aad0 100644 --- a/src/adapter/src/catalog/open/builtin_schema_migration.rs +++ b/src/adapter/src/catalog/open/builtin_schema_migration.rs @@ -451,6 +451,17 @@ static MIGRATIONS: LazyLock> = LazyLock::new(|| { MZ_CATALOG_SCHEMA, "mz_sources", ), + // Converting mz_object_dependencies from a builtin table to a + // materialized view over mz_catalog_raw changes its catalog + // fingerprint, so it needs an explicit replacement step. See the + // NOTE above: this version must stay at the workspace's current dev + // version until the change ships. + MigrationStep::replacement( + "26.41.0-dev.0", + CatalogItemType::MaterializedView, + MZ_INTERNAL_SCHEMA, + "mz_object_dependencies", + ), ] }); diff --git a/src/catalog/src/builtin.rs b/src/catalog/src/builtin.rs index e70d02816250f..fae6d3e356974 100644 --- a/src/catalog/src/builtin.rs +++ b/src/catalog/src/builtin.rs @@ -31,6 +31,8 @@ mod mz_catalog; pub use mz_catalog::*; mod mz_internal; pub use mz_internal::*; +mod mz_object_dependencies; +pub use mz_object_dependencies::*; mod mz_introspection; pub use mz_introspection::*; mod information_schema; @@ -221,25 +223,20 @@ fn is_false(v: &bool) -> bool { /// /// Choosing the right variant matters: /// -/// - [`LinkProperties::ForeignKey`]: the source entity has a column whose -/// value is an ID that directly references a row in the target entity. -/// Use this when there is an explicit FK column (e.g. `schema_id` -> -/// `schema`). -/// - [`LinkProperties::DependsOn`]: this entity logically depends on the -/// target entity via a graph-edge table (e.g. `mz_compute_dependencies` -/// records that a compute object depends on another object). The -/// `source_column` is the column **in this entity** that holds the -/// dependent's ID; `target_column` is the column in the target entity -/// being depended upon. Use this for dependency-graph tables, **not** -/// `ForeignKey`. -/// - [`LinkProperties::Union`]: the source entity is a superset view that -/// contains the target entity as a subset, optionally filtered by a -/// discriminator column. -/// - [`LinkProperties::MapsTo`]: the source entity provides an ID translation -/// to the target entity, possibly via an intermediate table or across ID -/// namespaces. -/// - [`LinkProperties::Measures`]: the source entity records metric -/// measurements about the target entity. +/// - [`LinkProperties::ForeignKey`]: the source entity has a column whose value is an ID that +/// directly references a row in the target entity. Use this when there is an explicit FK column +/// (e.g. `schema_id` -> `schema`). +/// - [`LinkProperties::DependsOn`]: this entity logically depends on the target entity via a +/// graph-edge table (e.g. `mz_compute_dependencies` records that a compute object depends on +/// another object). The `source_column` is the column **in this entity** that holds the +/// dependent's ID; `target_column` is the column in the target entity being depended upon. Use +/// this for dependency-graph tables, **not** `ForeignKey`. +/// - [`LinkProperties::Union`]: the source entity is a superset view that contains the target +/// entity as a subset, optionally filtered by a discriminator column. +/// - [`LinkProperties::MapsTo`]: the source entity provides an ID translation to the target entity, +/// possibly via an intermediate table or across ID namespaces. +/// - [`LinkProperties::Measures`]: the source entity records metric measurements about the target +/// entity. #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, serde::Serialize)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum LinkProperties { @@ -783,6 +780,20 @@ impl Fingerprint for SqlRelationType { } } +/// Asserts that `name` is safe to embed unquoted inside a `'...'`-quoted SQL literal +/// or inside a `"..."`-quoted SQL identifier. Generated builtin relations +/// (`make_mz_indexes`, `make_mz_object_dependencies_raw`, ...) concatenate builtin names +/// into SQL fragments, so a quote or backslash would produce malformed SQL. Builtin +/// names should always be plain ASCII identifiers. +pub(super) fn assert_safe_builtin_name(name: &str, kind: &str) { + assert!( + !name.contains('\'') && !name.contains('"') && !name.contains('\\'), + "builtin {kind} name {name:?} contains an unsupported character; \ + generated builtin relations reconstruct SQL via string \ + concatenation and assume names contain no quotes or backslashes" + ); +} + pub(super) const PUBLIC_SELECT: MzAclItem = MzAclItem { grantee: RoleId::Public, grantor: MZ_SYSTEM_ROLE_ID, @@ -1137,7 +1148,9 @@ pub static BUILTINS_STATIC: LazyLock>> = LazyLock::ne Builtin::MaterializedView(&MZ_KAFKA_SINKS), Builtin::MaterializedView(&MZ_KAFKA_CONNECTIONS), Builtin::MaterializedView(&MZ_KAFKA_SOURCES), - Builtin::Table(&MZ_OBJECT_DEPENDENCIES), + // mz_object_dependencies_raw is generated dynamically below with inlined + // builtin VALUES and inserted directly before this entry. + Builtin::MaterializedView(&MZ_OBJECT_DEPENDENCIES), Builtin::MaterializedView(&MZ_ICEBERG_SINKS), Builtin::MaterializedView(&MZ_DATABASES), Builtin::MaterializedView(&MZ_SCHEMAS), @@ -1555,6 +1568,38 @@ pub static BUILTINS_STATIC: LazyLock>> = LazyLock::ne builtin_items.insert(insert_pos, Builtin::MaterializedView(mz_indexes_ref)); } + // Generate mz_object_dependencies_raw, which inlines every builtin's + // dependency edges as VALUES. + // + // It has to run before ontology::generate_views and builtin::builtins, because those consume + // mz_object_dependencies. However, because the views those two generators produce + // have dependency edges of their own, it must use them as inputs. So this block runs both + // generators early, keeps the edges of the views they hand back, and discards the views + // themselves. The real ones are built below. + { + let ontology_preview = ontology::generate_views(&builtin_items); + let builtin_reporter_preview: Vec<_> = builtin::builtins(&builtin_items).collect(); + let generator_input: Vec<_> = builtin_types + .iter() + .chain(builtin_funcs.iter()) + .chain(builtin_reporter_preview.iter()) + .chain(builtin_items.iter()) + .chain(ontology_preview.iter()) + .cloned() + .collect(); + let mz_object_dependencies_raw = + mz_object_dependencies::make_mz_object_dependencies_raw(&generator_input); + let mz_object_dependencies_raw_ref: &'static BuiltinView = + Box::leak(Box::new(mz_object_dependencies_raw)); + // The view goes directly before mz_object_dependencies, which reads it, + // because BUILTINS_STATIC must list dependencies before their dependents. + let insert_pos = builtin_items + .iter() + .position(|b| b.name() == "mz_object_dependencies") + .expect("mz_object_dependencies must be present in builtin_items"); + builtin_items.insert(insert_pos, Builtin::View(mz_object_dependencies_raw_ref)); + } + // Generate ontology views by enumerating existing builtins. builtin_items.extend(ontology::generate_views(&builtin_items)); @@ -2004,8 +2049,8 @@ mod tests { // avoid noise. // // Exemptions: - // - Column at index 0 named "id": almost always the entity's own PK, - // not a FK (e.g. mz_objects.id, mz_functions.id). + // - Column at index 0 named "id": almost always the entity's own PK, not a FK (e.g. + // mz_objects.id, mz_functions.id). // - Columns in the relation's declared key set. // // "Reference" types are ID types that imply a FK. Discriminators @@ -2429,4 +2474,40 @@ mod tests { "mz_indexes fingerprint must change when a builtin log is added" ); } + + /// Because `mz_object_dependencies_raw`` is built from prior copies of the ontology and builtin + /// reporter views to get their edges since they must be built afterwards. Thus we test + /// that regenerating the view using all builtins (excluding itself) equals the SQL of the real + /// view. + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // slow: parses every builtin's SQL twice + fn test_mz_object_dependencies_raw_sql_is_stable() { + let without_self: Vec> = BUILTINS_STATIC + .iter() + .filter(|b| { + !matches!(b, Builtin::View(v) if v.name == mz_object_dependencies::MZ_OBJECT_DEPENDENCIES_RAW) + }) + .cloned() + .collect(); + + let regenerated = mz_object_dependencies::make_mz_object_dependencies_raw(&without_self); + + let from_static = BUILTINS_STATIC + .iter() + .find_map(|b| match b { + Builtin::View(v) + if v.name == mz_object_dependencies::MZ_OBJECT_DEPENDENCIES_RAW => + { + Some(*v) + } + _ => None, + }) + .expect("mz_object_dependencies_raw must be present in BUILTINS_STATIC"); + + assert_eq!( + regenerated.sql, from_static.sql, + "regenerating mz_object_dependencies_raw from the final builtin \ + list must reproduce the SQL generated during static init" + ); + } } diff --git a/src/catalog/src/builtin/mz_catalog.rs b/src/catalog/src/builtin/mz_catalog.rs index 3874ac8d7d177..bdd9cd3969b5e 100644 --- a/src/catalog/src/builtin/mz_catalog.rs +++ b/src/catalog/src/builtin/mz_catalog.rs @@ -29,6 +29,7 @@ use mz_storage_client::controller::IntrospectionType; use super::{ BuiltinIndex, BuiltinLog, BuiltinMaterializedView, BuiltinSource, BuiltinTable, BuiltinType, BuiltinView, Cardinality, LinkProperties, Ontology, OntologyLink, PUBLIC_SELECT, + assert_safe_builtin_name, }; pub const TYPE_LIST: BuiltinType = BuiltinType { @@ -676,19 +677,6 @@ pub static MZ_COLUMNS: LazyLock = LazyLock::new(|| BuiltinTable { }); // mz_indexes is generated dynamically in BUILTINS_STATIC via mz_catalog::make_mz_indexes() -/// Asserts that `name` is safe to embed unquoted inside a `'...'`-quoted SQL literal -/// or inside a `"..."`-quoted SQL identifier. Builtin index/log/object names are -/// concatenated into SQL fragments below, so a quote or backslash would produce -/// malformed SQL. Builtin names should always be plain ASCII identifiers. -fn assert_safe_builtin_name(name: &str, kind: &str) { - assert!( - !name.contains('\'') && !name.contains('"') && !name.contains('\\'), - "builtin {kind} name {name:?} contains an unsupported character; \ - mz_indexes reconstructs SQL via string concatenation and assumes \ - names contain no quotes or backslashes" - ); -} - /// User-created indexes, sourced from `mz_catalog_raw` `Item` entries with /// `parse_catalog_create_sql(...)` yielding `type = 'index'`. const USER_INDEXES_CTE: &str = "\ diff --git a/src/catalog/src/builtin/mz_internal.rs b/src/catalog/src/builtin/mz_internal.rs index e8e154850df90..006492fbdf5d2 100644 --- a/src/catalog/src/builtin/mz_internal.rs +++ b/src/catalog/src/builtin/mz_internal.rs @@ -394,64 +394,7 @@ WHERE }), } }); -pub static MZ_OBJECT_DEPENDENCIES: LazyLock = LazyLock::new(|| BuiltinTable { - name: "mz_object_dependencies", - schema: MZ_INTERNAL_SCHEMA, - oid: oid::TABLE_MZ_OBJECT_DEPENDENCIES_OID, - desc: RelationDesc::builder() - .with_column("object_id", SqlScalarType::String.nullable(false)) - .with_column( - "referenced_object_id", - SqlScalarType::String.nullable(false), - ) - .finish(), - column_comments: BTreeMap::from_iter([ - ( - "object_id", - "The ID of the dependent object. Corresponds to `mz_objects.id`.", - ), - ( - "referenced_object_id", - "The ID of the referenced object. Corresponds to `mz_objects.id`.", - ), - ]), - is_retained_metrics_object: true, - access: vec![PUBLIC_SELECT], - ontology: Some(Ontology { - entity_name: "object_dependency", - description: "A dependency edge: one object depends on another", - links: &const { - [ - OntologyLink { - name: "depends_on", - target: "object", - properties: LinkProperties::DependsOn { - source_column: "object_id", - target_column: "id", - source_id_type: Some(mz_repr::SemanticType::CatalogItemId), - requires_mapping: None, - }, - }, - OntologyLink { - name: "dependency_is", - target: "object", - properties: LinkProperties::DependsOn { - source_column: "referenced_object_id", - target_column: "id", - source_id_type: Some(mz_repr::SemanticType::CatalogItemId), - requires_mapping: None, - }, - }, - ] - }, - column_semantic_types: &const { - [ - ("object_id", SemanticType::CatalogItemId), - ("referenced_object_id", SemanticType::CatalogItemId), - ] - }, - }), -}); + pub static MZ_COMPUTE_DEPENDENCIES: LazyLock = LazyLock::new(|| BuiltinSource { name: "mz_compute_dependencies", schema: MZ_INTERNAL_SCHEMA, @@ -4979,9 +4922,10 @@ ON mz_internal.pg_attrdef_all_databases (oid, adrelid, adnum, adbin, adsrc)", pub static MZ_COMPUTE_ERROR_COUNTS_RAW_UNIFIED: LazyLock = LazyLock::new(|| BuiltinSource { - // TODO(database-issues#8173): Rename this source to `mz_compute_error_counts_raw`. Currently this causes a - // naming conflict because the resolver stumbles over the source with the same name in - // `mz_introspection` due to the automatic schema translation. + // TODO(database-issues#8173): Rename this source to `mz_compute_error_counts_raw`. + // Currently this causes a naming conflict because the resolver stumbles over the + // source with the same name in `mz_introspection` due to the automatic schema + // translation. name: "mz_compute_error_counts_raw_unified", schema: MZ_INTERNAL_SCHEMA, oid: oid::SOURCE_MZ_COMPUTE_ERROR_COUNTS_RAW_UNIFIED_OID, @@ -8000,12 +7944,11 @@ fn console_cluster_utilization_overview_desc() -> RelationDesc { /// `console/src/api/materialize/cluster/replicaUtilizationHistory.ts`). /// /// * `bin`: the `date_bin` bucket width, e.g. `1 MINUTE`. -/// * `retention`: how much history the view retains, e.g. `3 HOURS`, enforced -/// with a temporal `mz_now()` filter so the maintained arrangement stays -/// bounded. -/// * `group_size`: the expected number of metric samples per (replica, bucket), -/// used for the `DISTINCT ON INPUT GROUP SIZE` top-k hint. Replica metrics are -/// scraped roughly once per minute, so this is the bucket width in minutes. +/// * `retention`: how much history the view retains, e.g. `3 HOURS`, enforced with a temporal +/// `mz_now()` filter so the maintained arrangement stays bounded. +/// * `group_size`: the expected number of metric samples per (replica, bucket), used for the +/// `DISTINCT ON INPUT GROUP SIZE` top-k hint. Replica metrics are scraped roughly once per +/// minute, so this is the bucket width in minutes. fn console_cluster_utilization_overview_sql(bin: &str, retention: &str, group_size: u32) -> String { format!( r#"WITH replica_history AS ( @@ -8359,12 +8302,11 @@ pub static MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_24H: LazyLock = * cluster_name: The name of the cluster. * The approach taken is as follows. First, find all extant clusters and add them * to the result set. Per cluster, we do the following: - * 1. Find the most recent create or rename event. This moment represents when the - * cluster took on its final logical identity. - * 2. Look for a cluster that had the same name (or the same name with `_dbt_deploy` - * appended) that was dropped within one minute of that moment. That cluster is - * almost certainly the logical predecessor of the current cluster. Add the cluster - * to the result set. + * 1. Find the most recent create or rename event. This moment represents when the cluster took + * on its final logical identity. + * 2. Look for a cluster that had the same name (or the same name with `_dbt_deploy` appended) + * that was dropped within one minute of that moment. That cluster is almost certainly the + * logical predecessor of the current cluster. Add the cluster to the result set. * 3. Repeat the procedure until a cluster with no logical predecessor is discovered. * Limiting the search for a dropped cluster to a window of one minute is a heuristic, * but one that's likely to be pretty good one. If a name is reused after more @@ -8728,8 +8670,9 @@ ON mz_internal.mz_sink_status_history (sink_id)", // underlying relation. // // We append WITH_HISTORY because we want to build a separate view + index that doesn't -// retain history. This is because retaining its history causes MZ_SOURCE_STATISTICS_WITH_HISTORY_IND -// to hold all records/updates, which causes CPU and latency of querying it to spike. +// retain history. This is because retaining its history causes +// MZ_SOURCE_STATISTICS_WITH_HISTORY_IND to hold all records/updates, which causes CPU and latency +// of querying it to spike. pub static MZ_SOURCE_STATISTICS_WITH_HISTORY: LazyLock = LazyLock::new(|| BuiltinView { name: "mz_source_statistics_with_history", diff --git a/src/catalog/src/builtin/mz_object_dependencies.rs b/src/catalog/src/builtin/mz_object_dependencies.rs new file mode 100644 index 0000000000000..a03d46a87dc56 --- /dev/null +++ b/src/catalog/src/builtin/mz_object_dependencies.rs @@ -0,0 +1,465 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +//! The `mz_internal.mz_object_dependencies` materialized view and the +//! generated view it reads. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::LazyLock; + +use itertools::Itertools; +use mz_catalog_protos::objects::CatalogItemType as ProtoCatalogItemType; +use mz_ore::collections::CollectionExt; +use mz_pgrepr::oid; +use mz_repr::namespaces::MZ_INTERNAL_SCHEMA; +use mz_repr::{RelationDesc, SemanticType, SqlScalarType}; +use mz_sql::catalog::{CatalogType, NameReference}; +use mz_sql_parser::ast::UnresolvedItemName; +use mz_sql_parser::ast::item_refs::collect_item_references; + +use super::{ + Builtin, BuiltinMaterializedView, BuiltinView, LinkProperties, Ontology, OntologyLink, + PUBLIC_SELECT, assert_safe_builtin_name, +}; + +pub(super) const MZ_OBJECT_DEPENDENCIES_RAW: &str = "mz_object_dependencies_raw"; + +/// The durable encoding of a `GidMapping` key's `object_type`, as it appears in +/// `mz_catalog_raw`'s JSON. +fn object_type_code(object_type: ProtoCatalogItemType) -> String { + serde_json::to_string(&object_type).expect("CatalogItemType is serializable") +} + +/// One builtin dependency edge, inlined as a `VALUES` row of +/// `mz_object_dependencies_raw`. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +struct BuiltinEdgeRow { + object_schema: String, + object_name: String, + object_type: String, + ref_schema: String, + ref_name: String, + ref_kind: &'static str, +} + +impl BuiltinEdgeRow { + /// Renders the row as a SQL `VALUES` tuple. + /// + /// The literals are not escaped. `BuiltinEdgeCollector::collect` runs + /// every name through `assert_safe_builtin_name` before building a row. + fn to_sql_row(&self) -> String { + let BuiltinEdgeRow { + object_schema, + object_name, + object_type, + ref_schema, + ref_name, + ref_kind, + } = self; + format!( + "('{object_schema}', '{object_name}', '{object_type}', \ + '{ref_schema}', '{ref_name}', '{ref_kind}')" + ) + } +} + +/// Collects builtin dependency edges by parsing builtin SQL. +struct BuiltinEdgeCollector<'a> { + /// Builtin name to its schema, for resolving unqualified references. + schema_by_name: BTreeMap<&'a str, &'a str>, + /// Element type name to the `(schema, name)` of its array type. e.g. `T[]` + /// maps to the qualified Postgres compatible type `_T`. + array_type_by_elem: BTreeMap<&'a str, (&'a str, &'a str)>, + rows: BTreeSet, +} + +impl<'a> BuiltinEdgeCollector<'a> { + fn new(builtins: &'a [Builtin]) -> Self { + let mut schema_by_name: BTreeMap<&str, &str> = BTreeMap::new(); + for b in builtins { + if let Some(prev) = schema_by_name.insert(b.name(), b.schema()) { + assert_eq!( + prev, + b.schema(), + "builtin name {} appears in multiple schemas; unqualified references are ambiguous", + b.name() + ); + } + } + + let mut array_type_by_elem: BTreeMap<&str, (&str, &str)> = BTreeMap::new(); + for b in builtins { + if let Builtin::Type(t) = b { + if let CatalogType::Array { element_reference } = &t.details.typ { + array_type_by_elem.insert(element_reference, (t.schema, t.name)); + } + } + } + + BuiltinEdgeCollector { + schema_by_name, + array_type_by_elem, + rows: BTreeSet::new(), + } + } + + fn resolve(&self, object: &str, name: &UnresolvedItemName) -> (String, String) { + match &name.0[..] { + [.., schema, item] => (schema.as_str().to_string(), item.as_str().to_string()), + [item] => { + let item = item.as_str(); + let schema = self.schema_by_name.get(item).unwrap_or_else(|| { + panic!( + "cannot resolve unqualified reference {item:?} in builtin {object}; \ + qualify the reference or check that the referenced builtin exists" + ) + }); + (schema.to_string(), item.to_string()) + } + [] => panic!("empty item reference in builtin {object}"), + } + } + + /// Parses `create_sql` and records one row per catalog item it references. + fn collect( + &mut self, + object_schema: &str, + object_name: &str, + object_type: ProtoCatalogItemType, + create_sql: &str, + ) { + assert_safe_builtin_name(object_schema, "object schema"); + assert_safe_builtin_name(object_name, "object"); + + let stmt = mz_sql::parse::parse(create_sql) + .unwrap_or_else(|e| panic!("invalid sql for builtin {object_name}: {e}")) + .into_element() + .ast; + let refs = collect_item_references(&stmt); + + // Builtin SQL is name-based. An id reference here means somebody + // hardcoded a system id, which is not stable across versions. + assert!( + refs.ids.is_empty(), + "builtin {} references items by id: {:?}", + object_name, + refs.ids + ); + + let mut referenced: Vec<(String, String, &'static str)> = Vec::new(); + for item_name in &refs.named_relations { + let (schema, name) = self.resolve(object_name, item_name); + referenced.push((schema, name, "rel")); + } + for item_name in &refs.named_funcs { + let (schema, name) = self.resolve(object_name, item_name); + referenced.push((schema, name, "func")); + } + for item_name in &refs.named_types { + let (schema, name) = self.resolve(object_name, item_name); + referenced.push((schema, name, "type")); + } + // Map the array type `T[]` to its Postgres compatible type `_T`. + for elem in &refs.named_array_elements { + // `T[]` references the array type paired with `T`, not `T` itself. + let (_, name) = self.resolve(object_name, elem); + let (array_schema, array_name) = self + .array_type_by_elem + .get(name.as_str()) + .unwrap_or_else(|| { + panic!( + "builtin {object_name} uses {name}[], but no builtin array type has \ + element {name}" + ) + }); + referenced.push((array_schema.to_string(), array_name.to_string(), "type")); + } + + let object_type = object_type_code(object_type); + for (ref_schema, ref_name, ref_kind) in referenced { + assert_safe_builtin_name(&ref_schema, "referenced schema"); + assert_safe_builtin_name(&ref_name, "referenced object"); + assert!( + !(object_schema == ref_schema && object_name == ref_name), + "builtin {object_schema}.{object_name} references itself" + ); + self.rows.insert(BuiltinEdgeRow { + object_schema: object_schema.to_string(), + object_name: object_name.to_string(), + object_type: object_type.clone(), + ref_schema, + ref_name, + ref_kind, + }); + } + } +} + +/// Renders the `mz_object_dependencies_raw` view body with `rows` inlined as +/// the builtin edge table. +fn mz_object_dependencies_raw_sql(rows: &BTreeSet) -> String { + let builtin_ref_values = rows.iter().map(BuiltinEdgeRow::to_sql_row).join(","); + + let func_type = object_type_code(ProtoCatalogItemType::Func); + let type_type = object_type_code(ProtoCatalogItemType::Type); + let source_type = object_type_code(ProtoCatalogItemType::Source); + // Every kind a name in relation position can denote. + let relation_types = format!("NOT IN ('{type_type}', '{func_type}')"); + + format!( + " +WITH + user_items AS ( + SELECT + mz_internal.parse_catalog_id(data->'key'->'gid') AS id, + mz_internal.parse_catalog_item_references(data->'value'->'definition'->'V1'->>'create_sql') AS refs + FROM mz_internal.mz_catalog_raw + WHERE + data->>'kind' = 'Item' AND + -- Exclude temporary objects + data->'value'->>'ephemeral_owner_session' IS NULL + ), + gid_mappings AS ( + SELECT + 's' || (data->'value'->>'catalog_id') AS id, + data->'key'->>'schema_name' AS schema_name, + data->'key'->>'object_name' AS object_name, + data->'key'->>'object_type' AS object_type + FROM mz_internal.mz_catalog_raw + WHERE data->>'kind' = 'GidMapping' + ), + user_id_edges AS ( + SELECT u.id AS object_id, r.ref AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements_text(u.refs->'ids') AS r(ref) + ), + user_func_edges AS ( + SELECT u.id AS object_id, gm.id AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements(u.refs->'named_funcs') AS f(func) + JOIN gid_mappings gm ON + gm.object_type = '{func_type}' AND + gm.schema_name = f.func->>'schema' AND + gm.object_name = f.func->>'name' + ), + user_type_edges AS ( + SELECT u.id AS object_id, gm.id AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements(u.refs->'named_types') AS t(typ) + JOIN gid_mappings gm ON + gm.object_type = '{type_type}' AND + gm.schema_name = t.typ->>'schema' AND + gm.object_name = t.typ->>'name' + ), + user_relation_edges AS ( + SELECT u.id AS object_id, gm.id AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements(u.refs->'named_relations') AS n(rel) + JOIN gid_mappings gm ON + gm.object_type {relation_types} AND + gm.schema_name = n.rel->>'schema' AND + gm.object_name = n.rel->>'name' + ), + builtin_edges AS ( + SELECT obj.id AS object_id, ref.id AS referenced_object_id + FROM + (VALUES {builtin_ref_values}) + AS bv(object_schema, object_name, object_type, ref_schema, ref_name, ref_kind) + JOIN gid_mappings obj ON + obj.schema_name = bv.object_schema AND + obj.object_name = bv.object_name AND + obj.object_type = bv.object_type + JOIN gid_mappings ref ON + ref.schema_name = bv.ref_schema AND + ref.object_name = bv.ref_name AND + CASE bv.ref_kind + WHEN 'func' THEN ref.object_type = '{func_type}' + WHEN 'type' THEN ref.object_type = '{type_type}' + ELSE ref.object_type {relation_types} + END + ), + introspection_source_index_edges AS ( + SELECT + 'si' || (isi.data->'value'->>'catalog_id') AS object_id, + 's' || (gm.data->'value'->>'catalog_id') AS referenced_object_id + FROM mz_internal.mz_catalog_raw AS isi + JOIN mz_internal.mz_catalog_raw AS gm ON + gm.data->>'kind' = 'GidMapping' AND + gm.data->'key'->>'object_type' = '{source_type}' AND + gm.data->'key'->>'schema_name' = 'mz_introspection' AND + gm.data->'key'->>'object_name' = isi.data->'key'->>'name' + WHERE isi.data->>'kind' = 'ClusterIntrospectionSourceIndex' + ) +-- UNION rather than UNION ALL: the branches are disjoint for the most part, but a +-- reference can appear both by id and by name in one statement (a table function +-- spelled `FROM [sNN AS pg_catalog.generate_series](...)` next to a plain call to +-- the same function, which resolves by name) +SELECT object_id, referenced_object_id FROM user_id_edges +UNION +SELECT object_id, referenced_object_id FROM user_func_edges +UNION +SELECT object_id, referenced_object_id FROM user_type_edges +UNION +SELECT object_id, referenced_object_id FROM user_relation_edges +UNION +SELECT object_id, referenced_object_id FROM builtin_edges +UNION +SELECT object_id, referenced_object_id FROM introspection_source_index_edges +" + ) +} + +/// Generate the `mz_internal.mz_object_dependencies_raw` builtin view with +/// builtin dependency edges inlined as VALUES clauses. +/// +/// The view unions these edge sources: +/// +/// - User items: references extracted from stored `create_sql` via `parse_catalog_item_references`. +/// Id references are used directly. Function references are recovered by joining +/// `(schema, name)` against `GidMapping` rows, which is sound by construction: name resolution +/// never prints an id for a function (`print_id` is false for `Func`) and a function's +/// `full_name` always carries its schema, so a function reference persists as exactly +/// `"schema"."name"`. Temporary items are excluded. +/// - Builtin items: name references joined to `GidMapping` at query time. +/// - Introspection source indexes: `si -> s` edges from +/// `ClusterIntrospectionSourceIndex` entries. +pub(super) fn make_mz_object_dependencies_raw(builtins: &[Builtin]) -> BuiltinView { + let mut collector = BuiltinEdgeCollector::new(builtins); + for b in builtins { + let (object_type, create_sql) = match b { + Builtin::View(v) => (ProtoCatalogItemType::View, v.create_sql()), + Builtin::MaterializedView(mv) => { + (ProtoCatalogItemType::MaterializedView, mv.create_sql()) + } + Builtin::Index(i) => (ProtoCatalogItemType::Index, i.create_sql()), + // The user_items branch already covers runtime alterable connections, thus we skip as + // to not double-count. + Builtin::Connection(c) if c.runtime_alterable => continue, + Builtin::Connection(c) => (ProtoCatalogItemType::Connection, c.sql.to_string()), + // Skip objects constructed without SQL. + Builtin::Log(_) + | Builtin::Table(_) + | Builtin::Type(_) + | Builtin::Func(_) + | Builtin::Source(_) => continue, + }; + collector.collect(b.schema(), b.name(), object_type, &create_sql); + } + + // Append `mz_object_dependencies_raw`'s own outgoing edges to its VALUES + // rows. + let self_edges = |sql: &str| { + let mut collector = BuiltinEdgeCollector::new(builtins); + collector.collect( + MZ_INTERNAL_SCHEMA, + MZ_OBJECT_DEPENDENCIES_RAW, + ProtoCatalogItemType::View, + &format!("CREATE VIEW {MZ_INTERNAL_SCHEMA}.{MZ_OBJECT_DEPENDENCIES_RAW} AS {sql}"), + ); + collector.rows + }; + + let self_rows = self_edges(&mz_object_dependencies_raw_sql(&collector.rows)); + let mut rows = collector.rows; + rows.extend(self_rows.iter().cloned()); + let sql = mz_object_dependencies_raw_sql(&rows); + + BuiltinView { + name: MZ_OBJECT_DEPENDENCIES_RAW, + schema: MZ_INTERNAL_SCHEMA, + oid: oid::VIEW_MZ_OBJECT_DEPENDENCIES_RAW_OID, + desc: RelationDesc::builder() + .with_column("object_id", SqlScalarType::String.nullable(true)) + .with_column("referenced_object_id", SqlScalarType::String.nullable(true)) + .with_key(vec![0, 1]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "object_id", + "The ID of the dependent object. Corresponds to `mz_objects.id`.", + ), + ( + "referenced_object_id", + "The ID of the referenced object. Corresponds to `mz_objects.id`.", + ), + ]), + sql: Box::leak(sql.into_boxed_str()), + access: vec![PUBLIC_SELECT], + ontology: None, + } +} + +pub static MZ_OBJECT_DEPENDENCIES: LazyLock = + LazyLock::new(|| BuiltinMaterializedView { + name: "mz_object_dependencies", + schema: MZ_INTERNAL_SCHEMA, + oid: oid::MV_MZ_OBJECT_DEPENDENCIES_OID, + desc: RelationDesc::builder() + .with_column("object_id", SqlScalarType::String.nullable(false)) + .with_column( + "referenced_object_id", + SqlScalarType::String.nullable(false), + ) + .with_key(vec![0, 1]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "object_id", + "The ID of the dependent object. Corresponds to `mz_objects.id`.", + ), + ( + "referenced_object_id", + "The ID of the referenced object. Corresponds to `mz_objects.id`.", + ), + ]), + sql: " +IN CLUSTER mz_catalog_server +WITH ( + ASSERT NOT NULL object_id, + ASSERT NOT NULL referenced_object_id +) AS +SELECT object_id, referenced_object_id +FROM mz_internal.mz_object_dependencies_raw", + is_retained_metrics_object: true, + access: vec![PUBLIC_SELECT], + ontology: Some(Ontology { + entity_name: "object_dependency", + description: "A dependency edge: one object depends on another", + links: &const { + [ + OntologyLink { + name: "depends_on", + target: "object", + properties: LinkProperties::DependsOn { + source_column: "object_id", + target_column: "id", + source_id_type: Some(mz_repr::SemanticType::CatalogItemId), + requires_mapping: None, + }, + }, + OntologyLink { + name: "dependency_is", + target: "object", + properties: LinkProperties::DependsOn { + source_column: "referenced_object_id", + target_column: "id", + source_id_type: Some(mz_repr::SemanticType::CatalogItemId), + requires_mapping: None, + }, + }, + ] + }, + column_semantic_types: &const { + [ + ("object_id", SemanticType::CatalogItemId), + ("referenced_object_id", SemanticType::CatalogItemId), + ] + }, + }), + }); diff --git a/src/compute-types/tests/snapshots/lir_v1.json b/src/compute-types/tests/snapshots/lir_v1.json index 311d1fbdb95c4..e15f7e8b893c5 100644 --- a/src/compute-types/tests/snapshots/lir_v1.json +++ b/src/compute-types/tests/snapshots/lir_v1.json @@ -3067,6 +3067,7 @@ "ParseCatalogAuditLogDetails": "UNITSTRUCT", "ParseCatalogCreateSql": "UNITSTRUCT", "ParseCatalogId": "UNITSTRUCT", + "ParseCatalogItemReferences": "UNITSTRUCT", "ParseCatalogPrivileges": "UNITSTRUCT", "ParseConnectionDetails": "UNITSTRUCT", "ParseIdent": "UNITSTRUCT", @@ -5086,6 +5087,11 @@ "TYPENAME": "ParseCatalogId" } }, + "ParseCatalogItemReferences": { + "NEWTYPE": { + "TYPENAME": "ParseCatalogItemReferences" + } + }, "ParseCatalogPrivileges": { "NEWTYPE": { "TYPENAME": "ParseCatalogPrivileges" diff --git a/src/expr/src/scalar/func/impls/jsonb.rs b/src/expr/src/scalar/func/impls/jsonb.rs index 287027f3eedca..f21b7109b1b00 100644 --- a/src/expr/src/scalar/func/impls/jsonb.rs +++ b/src/expr/src/scalar/func/impls/jsonb.rs @@ -17,6 +17,7 @@ use mz_repr::adt::numeric::{self, Numeric, NumericMaxScale}; use mz_repr::role_id::RoleId; use mz_repr::{ArrayRustType, Datum, Row, RowPacker, SqlColumnType, SqlScalarType, strconv}; use mz_sql_parser::ast::display::AstDisplay; +use mz_sql_parser::ast::item_refs::collect_item_references; use mz_sql_parser::ast::{ AstInfo, AvroSchema, ConnectionOption, ConnectionOptionName, CreateConnectionType, CreateSinkConnection, CreateSubsourceOptionName, Format, FormatSpecifier, @@ -408,7 +409,6 @@ fn option_string(value: &WithOptionValue) -> Option { /// /// The returned JSONB does not fully reflect the parsed SQL and instead contains only fields /// required by current callers. -/// // TODO: This function isn't parsing JSONB and therefore shouldn't live in the `jsonb` module. // Consider moving all the `parse_catalog_*` functions into their own module. #[sqlfunc] @@ -815,6 +815,51 @@ fn parse_catalog_create_sql<'a>(a: &'a str) -> Result { Ok(jsonb) } +/// Extracts the catalog item references from a catalog `create_sql` string as a JSONB object. +/// +/// - `ids`: array of catalog id strings from `[ AS ]` references. +/// - `named_funcs`: array of function references such as "pg_catalog"."max". +/// - `named_types`: array of type references such as "pg_catalog"."int4", exclusive from the ones +/// in `ids`. +/// - `named_relations`: array of relation references, exclusive from the ones in `ids`. +#[sqlfunc] +fn parse_catalog_item_references<'a>(a: &'a str) -> Result { + fn qualified(name: &UnresolvedItemName) -> serde_json::Value { + match &name.0[..] { + [.., schema, item] => json!({"schema": schema.as_str(), "name": item.as_str()}), + [item] => json!({"schema": serde_json::Value::Null, "name": item.as_str()}), + [] => json!({"schema": serde_json::Value::Null, "name": ""}), + } + } + + let parse = || -> Result { + let mut stmts = mz_sql_parser::parser::parse_statements(a) + .map_err(|e| format!("failed to parse create_sql: {e}"))?; + let stmt = match stmts.len() { + 1 => stmts.remove(0).ast, + n => return Err(format!("expected a single statement, found {n}")), + }; + + let refs = collect_item_references(&stmt); + mz_ore::soft_assert_or_log!( + refs.named_array_elements.is_empty(), + "persisted create_sql should never carry an array type T[] \ + and resolve its array type in `ids`: {:?}", + refs.named_array_elements + ); + Ok(json!({ + "ids": refs.ids.iter().collect::>(), + "named_funcs": refs.named_funcs.iter().map(qualified).collect::>(), + "named_types": refs.named_types.iter().map(qualified).collect::>(), + "named_relations": refs.named_relations.iter().map(qualified).collect::>(), + })) + }; + + let val = parse().map_err(|e| EvalError::InvalidCatalogJson(e.into()))?; + let jsonb = Jsonb::from_serde_json(val).expect("valid JSONB"); + Ok(jsonb) +} + /// Minimal decoder for `ProtoPostgresSourcePublicationDetails`. The /// canonical proto lives in `mz-storage-types`, which depends on /// `mz-expr`, so we redeclare the two tags we read here. Upstream tag @@ -831,8 +876,8 @@ struct PostgresPublicationDetailsSubset { /// Extracts postgres source publication details (slot, timeline_id) from a /// catalog `create_sql`. Returns: /// -/// - jsonb `{"slot": , "timeline_id": }` for -/// `CREATE SOURCE ... FROM POSTGRES CONNECTION ... (DETAILS = ...)` statements. +/// - jsonb `{"slot": , "timeline_id": }` for `CREATE SOURCE ... FROM POSTGRES +/// CONNECTION ... (DETAILS = ...)` statements. /// - jsonb `null` for any other statement. /// /// Errors if the statement fails to parse, is a postgres source without @@ -887,8 +932,8 @@ fn parse_postgres_source_details<'a>(a: &'a str) -> Result { /// Extracts kafka source configuration (topic, group id prefix, connection /// id) from a catalog `create_sql`. Returns: /// -/// - jsonb `{"topic": , "group_id_prefix": , "connection_id": }` -/// for `CREATE SOURCE ... FROM KAFKA CONNECTION ... (TOPIC = ..., [GROUP ID PREFIX = ...])` +/// - jsonb `{"topic": , "group_id_prefix": , "connection_id": }` for +/// `CREATE SOURCE ... FROM KAFKA CONNECTION ... (TOPIC = ..., [GROUP ID PREFIX = ...])` /// statements. /// - jsonb `null` for any other statement. /// @@ -2327,4 +2372,70 @@ mod tests { Err("unresolved item name".to_string()) ); } + + // --- parse_catalog_item_references ---------------------------------------- + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_view() { + let sql = "CREATE VIEW \"materialize\".\"public\".\"v\" AS \ + SELECT \"pg_catalog\".\"abs\"(\"a\"::[s20 AS \"pg_catalog\".\"int4\"]) \ + FROM [u1 AS \"materialize\".\"public\".\"t\"]"; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!( + out, + json!({ + "ids": ["s20", "u1"], + "named_funcs": [{"schema": "pg_catalog", "name": "abs"}], + "named_types": [], + "named_relations": [], + }) + ); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_connection() { + let sql = "CREATE CONNECTION \"materialize\".\"public\".\"kc\" TO KAFKA (\ + BROKER 'kafka:9092', \ + SSH TUNNEL [u5 AS \"materialize\".\"public\".\"ssh\"], \ + SASL PASSWORD = SECRET [u7 AS \"materialize\".\"public\".\"pw\"])"; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!(out["ids"], json!(["u5", "u7"])); + assert_eq!(out["named_funcs"], json!([])); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_cte_shadowing() { + let sql = "CREATE VIEW \"materialize\".\"public\".\"v\" AS \ + WITH \"c\" AS (SELECT * FROM [u1 AS \"materialize\".\"public\".\"t\"]) \ + SELECT * FROM \"c\""; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!(out["ids"], json!(["u1"])); + assert_eq!(out["named_relations"], json!([])); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_name_based_relation_surfaces() { + // Stored SQL always prints relations as id references. A name-based + // relation must surface in "named_relations" rather than vanish. + let sql = "CREATE VIEW \"materialize\".\"public\".\"v\" AS \ + SELECT * FROM \"mz_catalog\".\"mz_tables\""; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!( + out["named_relations"], + json!([{"schema": "mz_catalog", "name": "mz_tables"}]) + ); + } + + #[mz_ore::test] + fn item_references_parse_error() { + let err = super::parse_catalog_item_references("NOT SQL").unwrap_err(); + assert!( + matches!(err, EvalError::InvalidCatalogJson(msg) if msg.contains("failed to parse")), + "wrong error variant/message" + ); + } } diff --git a/src/expr/src/scalar/func/unary.rs b/src/expr/src/scalar/func/unary.rs index 4607716af19c5..c53f9d8ec60ef 100644 --- a/src/expr/src/scalar/func/unary.rs +++ b/src/expr/src/scalar/func/unary.rs @@ -467,6 +467,7 @@ derive_unary!( ParseCatalogAuditLogDetails, ParseCatalogCreateSql, ParseCatalogId, + ParseCatalogItemReferences, ParseCatalogPrivileges, ParseConnectionDetails, ParseKafkaSourceDetails, diff --git a/src/pgrepr-consts/src/oid.rs b/src/pgrepr-consts/src/oid.rs index 25eb0d8f7c98b..d768fbba66338 100644 --- a/src/pgrepr-consts/src/oid.rs +++ b/src/pgrepr-consts/src/oid.rs @@ -420,7 +420,7 @@ pub const MV_MZ_KAFKA_SINKS_OID: u32 = 16694; pub const MV_MZ_KAFKA_CONNECTIONS_OID: u32 = 16695; pub const MV_MZ_KAFKA_SOURCES_OID: u32 = 16696; pub const MV_MZ_POSTGRES_SOURCES_OID: u32 = 16697; -pub const TABLE_MZ_OBJECT_DEPENDENCIES_OID: u32 = 16698; +pub const MV_MZ_OBJECT_DEPENDENCIES_OID: u32 = 16698; pub const SOURCE_MZ_COMPUTE_DEPENDENCIES_OID: u32 = 16699; pub const SOURCE_MZ_COMPUTE_HYDRATION_STATUSES_OID: u32 = 16700; pub const LOG_MZ_COMPUTE_OPERATOR_HYDRATION_STATUSES_PER_WORKER_OID: u32 = 16701; @@ -831,3 +831,5 @@ pub const INDEX_MZ_METRIC_SINKS_IND_OID: u32 = 17121; pub const TABLE_MZ_OBJECT_HYDRATION_HISTORY_OID: u32 = 17122; pub const LOG_MZ_CLUSTER_REPLICA_RESOURCE_USAGE_OID: u32 = 17123; pub const TABLE_MZ_REPLICA_HYDRATION_HISTORY_OID: u32 = 17124; +pub const FUNC_PARSE_CATALOG_ITEM_REFERENCES_OID: u32 = 17125; +pub const VIEW_MZ_OBJECT_DEPENDENCIES_RAW_OID: u32 = 17126; diff --git a/src/pgrepr-consts/src/regproc.rs b/src/pgrepr-consts/src/regproc.rs index d3ad9c9c76674..63cc28e2ad5e2 100644 --- a/src/pgrepr-consts/src/regproc.rs +++ b/src/pgrepr-consts/src/regproc.rs @@ -685,6 +685,7 @@ pub const NAMES: &[(u32, &str)] = &[ (17113, "mz_aws_account_id"), (17114, "mz_aws_external_id_prefix"), (17115, "mz_aws_connection_role_arn"), + (17125, "mz_internal.parse_catalog_item_references"), // END GENERATED ]; diff --git a/src/sql-parser/src/ast.rs b/src/sql-parser/src/ast.rs index 268edd724fec7..85124990fcb66 100644 --- a/src/sql-parser/src/ast.rs +++ b/src/sql-parser/src/ast.rs @@ -23,6 +23,7 @@ #[macro_use] pub mod display; pub mod fold; +pub mod item_refs; pub mod visit; pub mod visit_mut; diff --git a/src/sql-parser/src/ast/item_refs.rs b/src/sql-parser/src/ast/item_refs.rs new file mode 100644 index 0000000000000..aa76b97be4bea --- /dev/null +++ b/src/sql-parser/src/ast/item_refs.rs @@ -0,0 +1,389 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software shall be governed +// by the Apache License, Version 2.0. + +//! Extraction of catalog item references from a raw SQL AST. + +use std::collections::BTreeSet; + +use crate::ast::visit::Visit; +use crate::ast::{ + CteBlock, Function, Ident, Query, Raw, RawDataType, RawItemName, Statement, UnresolvedItemName, +}; + +/// The catalog item references appearing in a statement. +/// +/// User statements are normalized and have IDs in its relation and type references (e.g. `[u1 AS +/// db.schema.name]`). For references that don't have IDs, we create separate `named_*` buckets that +/// contain their qualified names. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub struct ItemReferences { + /// Catalog IDs from `RawItemName::Id` references + pub ids: BTreeSet, + /// Named relations that don't carry an ID. Most commonly seen in builtin statements since they + /// are not normalized. CTE names are excluded: a name bound by a WITH clause refers to the + /// query-local binding rather than a catalog item, so recording it would fabricate an edge to + /// any catalog object that happens to share the name. + pub named_relations: BTreeSet, + pub named_funcs: BTreeSet, + pub named_types: BTreeSet, + /// Data-type references written `T[]`. These are + /// reported separately from `named_types` since `T[]` maps to the named type `_T`. + pub named_array_elements: BTreeSet, +} + +/// Collects all catalog item references from a statement. +pub fn collect_item_references(stmt: &Statement) -> ItemReferences { + let mut collector = ReferenceCollector { + refs: ItemReferences::default(), + cte_names: Vec::new(), + }; + collector.visit_statement(stmt); + collector.refs +} + +struct ReferenceCollector { + refs: ItemReferences, + /// Stack of CTE names currently in scope. + cte_names: Vec, +} + +impl ReferenceCollector { + fn record(&mut self, name: &RawItemName, position: Position) { + match name { + RawItemName::Id(id, _, _) => { + self.refs.ids.insert(id.clone()); + } + RawItemName::Name(name) => match position { + Position::Relation => { + // Filter out CTEs. + if let [only] = &name.0[..] { + if self.cte_names.contains(only) { + return; + } + } + self.refs.named_relations.insert(name.clone()); + } + Position::Func => { + self.refs.named_funcs.insert(name.clone()); + } + Position::Type => { + self.refs.named_types.insert(name.clone()); + } + }, + } + } +} + +enum Position { + Relation, + Func, + Type, +} + +impl<'ast> Visit<'ast, Raw> for ReferenceCollector { + fn visit_item_name(&mut self, node: &'ast RawItemName) { + self.record(node, Position::Relation); + } + + fn visit_function(&mut self, node: &'ast Function) { + let Function { + name, + args, + filter, + over, + distinct: _, + } = node; + self.record(name, Position::Func); + self.visit_function_args(args); + if let Some(filter) = filter { + self.visit_expr(filter); + } + if let Some(over) = over { + self.visit_window_spec(over); + } + } + + fn visit_data_type(&mut self, node: &'ast RawDataType) { + match node { + RawDataType::Array(elem_type) => match &**elem_type { + // `T[]` references the array type paired with `T`, not `T` + // itself. That pairing lives in `T`'s catalog details, which + // the parser cannot read, so the element is reported on its + // own field for the caller to map. + RawDataType::Other { + name: RawItemName::Name(name), + typ_mod: _, + } => { + self.refs.named_array_elements.insert(name.clone()); + } + // `[ AS ][]`. The id in hand names the element, and + // mapping it to the array type needs the catalog, so nothing + // is reported rather than an edge to the wrong type. Stored + // SQL never carries this spelling: name resolution prints a + // resolved array type as a single id reference. + RawDataType::Other { + name: RawItemName::Id(..), + typ_mod: _, + } => {} + // An array of a list, map, or array. Name resolution rejects + // the first two and the parser collapses `T[][]` into a 1D + // array, so none reach stored SQL. Recurse anyway so that no + // reference is dropped unreported. + elem_type => self.visit_data_type(elem_type), + }, + RawDataType::List(elem_type) => self.visit_data_type(elem_type), + RawDataType::Map { + key_type, + value_type, + } => { + self.visit_data_type(key_type); + self.visit_data_type(value_type); + } + RawDataType::Other { name, typ_mod: _ } => self.record(name, Position::Type), + } + } + + fn visit_query(&mut self, node: &'ast Query) { + let Query { + ctes, + body, + order_by, + limit, + offset, + } = node; + + let scope_depth = self.cte_names.len(); + // Keep track of CTEs such that we don't record them + // in `named_relations` + match ctes { + CteBlock::Simple(ctes) => { + for cte in ctes { + self.visit_cte(cte); + self.cte_names.push(cte.alias.name.clone()); + } + } + CteBlock::MutuallyRecursive(block) => { + for cte in &block.ctes { + self.cte_names.push(cte.name.clone()); + } + self.visit_mut_rec_block(block); + } + } + self.visit_set_expr(body); + for order_by in order_by { + self.visit_order_by_expr(order_by); + } + if let Some(limit) = limit { + self.visit_limit(limit); + } + if let Some(offset) = offset { + self.visit_expr(offset); + } + // Pop CTEs once visited. + self.cte_names.truncate(scope_depth); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::parser::parse_statements; + + #[track_caller] + fn collect(sql: &str) -> ItemReferences { + let stmts = parse_statements(sql).expect("valid sql"); + assert_eq!(stmts.len(), 1, "expected a single statement"); + collect_item_references(&stmts.into_iter().next().unwrap().ast) + } + + fn names(parts: &[&[&str]]) -> BTreeSet { + parts + .iter() + .map(|name| UnresolvedItemName(name.iter().map(|p| Ident::new_unchecked(*p)).collect())) + .collect() + } + + fn strings(items: &[&str]) -> BTreeSet { + items.iter().map(|s| s.to_string()).collect() + } + + #[mz_ore::test] + fn ids_from_bracketed_references() { + let refs = collect( + r#"CREATE VIEW v AS + SELECT a::[s20 AS "pg_catalog"."int4"] FROM [u1 AS "materialize"."public"."t"]"#, + ); + assert_eq!(refs.ids, strings(&["s20", "u1"])); + assert!(refs.named_relations.is_empty()); + assert!(refs.named_types.is_empty()); + } + + #[mz_ore::test] + fn versioned_id_reference() { + let refs = collect( + r#"CREATE VIEW v AS SELECT * FROM [u3 AS "materialize"."public"."t" VERSION 2]"#, + ); + assert_eq!(refs.ids, strings(&["u3"])); + } + + #[mz_ore::test] + fn function_names_classified() { + let refs = collect( + r#"CREATE VIEW v AS SELECT "pg_catalog"."abs"(1), count(*) FROM "mz_catalog"."mz_tables""#, + ); + assert_eq!( + refs.named_funcs, + names(&[&["pg_catalog", "abs"], &["count"]]) + ); + assert_eq!(refs.named_relations, names(&[&["mz_catalog", "mz_tables"]])); + } + + #[mz_ore::test] + fn function_args_still_visited() { + let refs = collect( + r#"CREATE VIEW v AS SELECT "pg_catalog"."abs"(a::"pg_catalog"."int8") + FROM "mz_catalog"."mz_tables""#, + ); + assert_eq!(refs.named_funcs, names(&[&["pg_catalog", "abs"]])); + assert_eq!(refs.named_types, names(&[&["pg_catalog", "int8"]])); + } + + #[mz_ore::test] + fn nested_data_types() { + let refs = collect( + r#"CREATE TABLE t (a "pg_catalog"."int4"[], b "pg_catalog"."text" list, + c map["pg_catalog"."text" => "pg_catalog"."int8" list])"#, + ); + // int4 sits under `[]`, so it is reported as an array element rather + // than as a type reference in its own right. + assert_eq!( + refs.named_types, + names(&[&["pg_catalog", "text"], &["pg_catalog", "int8"]]) + ); + assert_eq!(refs.named_array_elements, names(&[&["pg_catalog", "int4"]])); + } + + #[mz_ore::test] + fn array_nested_in_list() { + // A list of arrays still references the array type. + let refs = collect(r#"CREATE TABLE t (a "pg_catalog"."int4"[] list)"#); + assert_eq!(refs.named_array_elements, names(&[&["pg_catalog", "int4"]])); + assert!(refs.named_types.is_empty()); + } + + #[mz_ore::test] + fn array_of_id_reference_reports_nothing() { + // Mapping the element id to its array type needs the catalog, so + // nothing is reported rather than an edge to the element. + let refs = collect(r#"CREATE TABLE t (a [s20 AS "pg_catalog"."int4"][])"#); + assert!(refs.named_array_elements.is_empty()); + assert!(refs.named_types.is_empty()); + assert!(refs.ids.is_empty()); + } + + #[mz_ore::test] + fn array_of_list_still_visits_the_inner_type() { + // Name resolution rejects `T list[]`, but the collector must not drop + // the inner reference unreported. + let refs = collect(r#"CREATE TABLE t (a "pg_catalog"."int4" list[])"#); + assert_eq!(refs.named_types, names(&[&["pg_catalog", "int4"]])); + assert!(refs.named_array_elements.is_empty()); + } + + #[mz_ore::test] + fn cte_names_excluded() { + let refs = collect( + r#"CREATE VIEW v AS + WITH c AS (SELECT * FROM "mz_catalog"."mz_tables") + SELECT * FROM c JOIN "mz_catalog"."mz_views" ON true"#, + ); + assert_eq!( + refs.named_relations, + names(&[&["mz_catalog", "mz_tables"], &["mz_catalog", "mz_views"]]) + ); + } + + #[mz_ore::test] + fn mut_rec_cte_names_excluded() { + let refs = collect( + r#"CREATE VIEW v AS + WITH MUTUALLY RECURSIVE + reach (a int) AS (SELECT a FROM "mz_catalog"."mz_tables", reach) + SELECT * FROM reach"#, + ); + assert_eq!(refs.named_relations, names(&[&["mz_catalog", "mz_tables"]])); + // The int column type on the WMR CTE is still a type reference. + assert_eq!(refs.named_types, names(&[&["int4"]])); + } + + #[mz_ore::test] + fn qualified_cte_lookalike_not_excluded() { + // Only 1-part references can refer to a CTE. + let refs = collect( + r#"CREATE VIEW v AS + WITH mz_tables AS (SELECT 1 AS a) + SELECT * FROM mz_tables JOIN "mz_catalog"."mz_tables" ON true"#, + ); + assert_eq!(refs.named_relations, names(&[&["mz_catalog", "mz_tables"]])); + } + + #[mz_ore::test] + fn cte_scoping_is_lexical() { + // The CTE binding is scoped to the subquery that declares it: the + // same name in a sibling scope is a real relation reference. + let refs = collect( + r#"CREATE VIEW v AS + SELECT * FROM (WITH tbl AS (SELECT 1 AS a) SELECT * FROM tbl) x + JOIN tbl ON true"#, + ); + assert_eq!(refs.named_relations, names(&[&["tbl"]])); + } + + #[mz_ore::test] + fn simple_cte_definition_does_not_see_itself() { + // Mirroring the name resolver: a plain WITH binds a CTE's name only + // after its own definition, so the inner `tbl` is a real relation. + let refs = collect( + r#"CREATE VIEW v AS + WITH tbl AS (SELECT * FROM tbl) SELECT * FROM tbl"#, + ); + assert_eq!(refs.named_relations, names(&[&["tbl"]])); + } + + #[mz_ore::test] + fn later_simple_cte_sees_earlier() { + let refs = collect( + r#"CREATE VIEW v AS + WITH a AS (SELECT 1 AS x), b AS (SELECT * FROM a) + SELECT * FROM b"#, + ); + assert!(refs.named_relations.is_empty()); + } + + #[mz_ore::test] + fn connection_and_secret_references() { + let refs = collect( + r#"CREATE CONNECTION kc TO KAFKA ( + BROKER 'kafka:9092', + SSH TUNNEL [u5 AS "materialize"."public"."ssh"], + SASL PASSWORD = SECRET [u7 AS "materialize"."public"."pw"] + )"#, + ); + assert_eq!(refs.ids, strings(&["u5", "u7"])); + } + + #[mz_ore::test] + fn index_references() { + let refs = collect( + r#"CREATE INDEX i IN CLUSTER [u1] ON [u2 AS "materialize"."public"."t"] ("pg_catalog"."abs"(a))"#, + ); + assert_eq!(refs.ids, strings(&["u2"])); + assert_eq!(refs.named_funcs, names(&[&["pg_catalog", "abs"]])); + } +} diff --git a/src/sql/src/func.rs b/src/sql/src/func.rs index 8ef18c0c19c69..3cad3f2269471 100644 --- a/src/sql/src/func.rs +++ b/src/sql/src/func.rs @@ -5355,6 +5355,11 @@ pub static MZ_INTERNAL_BUILTINS: LazyLock> = LazyLo params!(Jsonb) => UnaryFunc::ParseCatalogId(func::ParseCatalogId) => String, oid::FUNC_PARSE_CATALOG_ID_OID; }, + "parse_catalog_item_references" => Scalar { + params!(String) => UnaryFunc::ParseCatalogItemReferences( + func::ParseCatalogItemReferences, + ) => Jsonb, oid::FUNC_PARSE_CATALOG_ITEM_REFERENCES_OID; + }, "parse_catalog_privileges" => Scalar { params!(Jsonb) => UnaryFunc::ParseCatalogPrivileges(func::ParseCatalogPrivileges) => SqlScalarType::Array(Box::new(SqlScalarType::MzAclItem)), diff --git a/src/sql/src/names.rs b/src/sql/src/names.rs index 3c50e68b255fb..1fd703a9d8885 100644 --- a/src/sql/src/names.rs +++ b/src/sql/src/names.rs @@ -40,7 +40,7 @@ use crate::ast::{ RawDataType, RawItemName, Statement, UnresolvedItemName, UnresolvedObjectName, }; use crate::catalog::{ - CatalogError, CatalogItem, CatalogItemType, CatalogType, CatalogTypeDetails, SessionCatalog, + CatalogError, CatalogItem, CatalogItemType, CatalogTypeDetails, SessionCatalog, }; use crate::normalize; use crate::plan::PlanError; @@ -1355,10 +1355,21 @@ impl<'a> NameResolver<'a> { } fn resolve_data_type(&mut self, data_type: RawDataType) -> Result { + self.resolve_data_type_inner(data_type, true) + } + + fn resolve_data_type_inner( + &mut self, + data_type: RawDataType, + should_insert_id: bool, + ) -> Result { match data_type { RawDataType::Array(elem_type) => { let name = elem_type.to_string(); - match self.resolve_data_type(*elem_type)? { + // `T[]` recursively resolves to its element type, but we + // only want to insert its array type `_T` and not its element + // type `T`. Thus we set `should_insert_id` to `false`. + match self.resolve_data_type_inner(*elem_type, false)? { ResolvedDataType::AnonymousList(_) | ResolvedDataType::AnonymousMap { .. } => { sql_bail!("type \"{}[]\" does not exist", name) } @@ -1384,6 +1395,11 @@ impl<'a> NameResolver<'a> { ); } }; + + // We don't need to check `should_insert_id` here since we're + // guaranteed to insert just one array item type for an + // array. For multi-dimensional arrays, the parser always + // collapses it into a 1D array. self.ids.insert(array_item.id(), BTreeSet::new()); Ok(ResolvedDataType::Named { id: array_item.id(), @@ -1397,15 +1413,15 @@ impl<'a> NameResolver<'a> { } } RawDataType::List(elem_type) => { - let elem_type = self.resolve_data_type(*elem_type)?; + let elem_type = self.resolve_data_type_inner(*elem_type, should_insert_id)?; Ok(ResolvedDataType::AnonymousList(Box::new(elem_type))) } RawDataType::Map { key_type, value_type, } => { - let key_type = self.resolve_data_type(*key_type)?; - let value_type = self.resolve_data_type(*value_type)?; + let key_type = self.resolve_data_type_inner(*key_type, should_insert_id)?; + let value_type = self.resolve_data_type_inner(*value_type, should_insert_id)?; Ok(ResolvedDataType::AnonymousMap { key_type: Box::new(key_type), value_type: Box::new(value_type), @@ -1433,17 +1449,8 @@ impl<'a> NameResolver<'a> { (full_name, item) } }; - self.ids.insert(item.id(), BTreeSet::new()); - // If this is a named array type, then make sure to include the element reference - // in the resolved IDs. This helps ensure that named array types are resolved the - // same as an array type with the same element type. For example, `int4[]` and - // `_int4` should have the same set of resolved IDs. - if let Some(CatalogTypeDetails { - typ: CatalogType::Array { element_reference }, - .. - }) = item.type_details() - { - self.ids.insert(*element_reference, BTreeSet::new()); + if should_insert_id { + self.ids.insert(item.id(), BTreeSet::new()); } Ok(ResolvedDataType::Named { id: item.id(), diff --git a/test/sqllogictest/autogenerated/mz_internal.slt b/test/sqllogictest/autogenerated/mz_internal.slt index 62a3ee04f6785..66cd3a52e3a4e 100644 --- a/test/sqllogictest/autogenerated/mz_internal.slt +++ b/test/sqllogictest/autogenerated/mz_internal.slt @@ -859,6 +859,7 @@ mz_notices_redacted mz_object_arrangement_size_history mz_object_arrangement_sizes mz_object_dependencies +mz_object_dependencies_raw mz_object_fully_qualified_names mz_object_global_ids mz_object_graph_edges diff --git a/test/sqllogictest/catalog_server_explain.slt b/test/sqllogictest/catalog_server_explain.slt index 54e9301255e67..4fb1faa189f0a 100644 --- a/test/sqllogictest/catalog_server_explain.slt +++ b/test/sqllogictest/catalog_server_explain.slt @@ -5127,7 +5127,7 @@ mz_catalog.mz_materialized_views: Project: #4, #0, #5, #1, #6, #2, #9, #8, #3, #7 Map: {=r/s1, s1=r/s1}, "s1" →Arrange (#1{schema_name}, #2{name}) - →Constant (43 rows) + →Constant (44 rows) →Arrange (empty key) (#0{schema_name}) (#0{schema_name}, #1{name}) →Fused with Child Map/Filter/Project Project: #4, #3, #5 @@ -5472,7 +5472,7 @@ mz_catalog.mz_tables: Project: #3, #0, #4, #1, #5, #2, #6, #6, #6 Map: "s1", null →Arrange (#1{schema_name}, #2{name}) - →Constant (31 rows) + →Constant (30 rows) →Arrange (#0{schema_name}) (#0{schema_name}, #1{name}) →Fused with Child Map/Filter/Project Project: #5, #4, #6 @@ -5527,7 +5527,7 @@ mz_catalog.mz_views: Project: #5, #0, #6, #1, #2, #8, #3, #4, #7 Map: "s1" →Arrange (#1{schema_name}, #2{name}) - →Constant (193 rows) + →Constant (194 rows) →Arrange (#0{schema_name}) (#0{schema_name}, #1{name}) →Fused with Child Map/Filter/Project Project: #5, #4, #6 @@ -5815,6 +5815,116 @@ Target cluster: mz_catalog_server EOF +query T multiline +EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_object_dependencies"; +---- +mz_internal.mz_object_dependencies: + →With + cte l0 = + →Fused with Child Map/Filter/Project + Project: #3, #4 + Filter: ("Item" = (#0{data} ->> "kind")) AND ((#2 ->> "ephemeral_owner_session")) IS NULL + Map: (#0{data} -> "value"), parse_catalog_id((#1 -> "gid")), parse_catalog_item_references((((#2 -> "definition") -> "V1") ->> "create_sql")) + →Read mz_internal.mz_catalog_raw + cte l1 = + →Fused with Child Map/Filter/Project + Project: #0..=#4 + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Return + →Distinct GroupAggregate + →Union + →Fused with Child Map/Filter/Project + Project: #0, #2 + →Table Function jsonb_array_elements_text((#1{refs} -> "ids")) + →Stream l0 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{func} ->> "name"), (#1{func} ->> "schema")] + →Arrange ((#1{func} ->> "name"), (#1{func} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{func} ->> "name")) IS NOT NULL AND ((#2{func} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_funcs")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("8" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{typ} ->> "name"), (#1{typ} ->> "schema")] + →Arrange ((#1{typ} ->> "name"), (#1{typ} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{typ} ->> "name")) IS NOT NULL AND ((#2{typ} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_types")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("7" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:mz_catalog_raw[#2{object_name}, #1{schema_name}] » %0[(#1{rel} ->> "name"), (#1{rel} ->> "schema")] + →Arrange ((#1{rel} ->> "name"), (#1{rel} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{rel} ->> "name")) IS NOT NULL AND ((#2{rel} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_relations")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2 != "7") AND (#2 != "8") AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_type"), (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Delta Join [%0[#0{object_schema}..=#2{object_type}] » %1:mz_catalog_raw[#1{schema_name}..=#3{object_type}] » %2:mz_catalog_raw[#1{schema_name}, #2{object_name}]] [%1:mz_catalog_raw[#1{schema_name}..=#3{object_type}] » %0[#0{object_schema}..=#2{object_type}] » %2:mz_catalog_raw[#1{schema_name}, #2{object_name}]] [%2:mz_catalog_raw[#1{schema_name}, #2{object_name}] » %0[#3{ref_schema}, #4{ref_name}] » %1:mz_catalog_raw[#1{schema_name}..=#3{object_type}]] + path %0: + after %2: + Project: #3, #4 + Filter: case when (#2{ref_kind} = "func") then (#5{object_type} = "8") else case when (#2{ref_kind} = "type") then (#5{object_type} = "7") else ((#5{object_type} != "7") AND (#5{object_type} != "8")) end end + path %1: + after %2: + Project: #3, #4 + Filter: case when (#2{ref_kind} = "func") then (#5{object_type} = "8") else case when (#2{ref_kind} = "type") then (#5{object_type} = "7") else ((#5{object_type} != "7") AND (#5{object_type} != "8")) end end + path %2: + after %0: + Project: #4..=#6, #2 + Filter: case when (#7{ref_kind} = "func") then (#3{object_type} = "8") else case when (#7{ref_kind} = "type") then (#3{object_type} = "7") else ((#3{object_type} != "7") AND (#3{object_type} != "8")) end end + →Arrange (#0{object_schema}..=#2{object_type}) (#3{ref_schema}, #4{ref_name}) + →Constant (1024 rows) + →Arrange (#1{schema_name}..=#3{object_type}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #2, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "object_type"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Arrange (#1{schema_name}, #2{object_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2, #5 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")), (#1 ->> "object_type") + →Read mz_internal.mz_catalog_raw + →Differential Join %0:mz_catalog_raw[((#0{data} -> "key") ->> "name")] » %1:mz_catalog_raw[((#0{data} -> "key") ->> "object_name")] + after %1: + Project: #3, #4 + Map: ("si" || ((#1{data} -> "value") ->> "catalog_id")), ("s" || ((#2{data} -> "value") ->> "catalog_id")) + →Arrange (((#0{data} -> "key") ->> "name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("ClusterIntrospectionSourceIndex" = (#0{data} ->> "kind")) AND ((#1 ->> "name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arrange (((#0{data} -> "key") ->> "object_name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("2" = (#1 ->> "object_type")) AND ("mz_introspection" = (#1 ->> "schema_name")) AND ((#1 ->> "object_name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + +Source mz_internal.mz_catalog_raw + project=(#0, #1) + map=((#0{data} -> "key")) + +Target cluster: mz_catalog_server + +EOF + query T multiline EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_overridden_system_parameters"; ---- @@ -8226,7 +8336,7 @@ query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_builtin_materialized_views"; ---- Explained Query (fast path): - →Constant (43 rows) + →Constant (44 rows) Target cluster: mz_catalog_server @@ -8246,7 +8356,7 @@ query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_builtin_tables"; ---- Explained Query (fast path): - →Constant (31 rows) + →Constant (30 rows) Target cluster: mz_catalog_server @@ -8256,7 +8366,7 @@ query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_builtin_views"; ---- Explained Query (fast path): - →Constant (193 rows) + →Constant (194 rows) Target cluster: mz_catalog_server @@ -10108,6 +10218,108 @@ Target cluster: mz_catalog_server EOF +query T multiline +EXPLAIN SELECT * FROM "mz_internal"."mz_object_dependencies_raw"; +---- +Explained Query: + →With + cte l0 = + →Fused with Child Map/Filter/Project + Project: #3, #4 + Filter: ("Item" = (#0{data} ->> "kind")) AND ((#2 ->> "ephemeral_owner_session")) IS NULL + Map: (#0{data} -> "value"), parse_catalog_id((#1 -> "gid")), parse_catalog_item_references((((#2 -> "definition") -> "V1") ->> "create_sql")) + →Read mz_internal.mz_catalog_raw + cte l1 = + →Fused with Child Map/Filter/Project + Project: #0..=#4 + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Return + →Distinct GroupAggregate + →Union + →Fused with Child Map/Filter/Project + Project: #0, #2 + →Table Function jsonb_array_elements_text((#1{refs} -> "ids")) + →Stream l0 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{func} ->> "name"), (#1{func} ->> "schema")] + →Arrange ((#1{func} ->> "name"), (#1{func} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{func} ->> "name")) IS NOT NULL AND ((#2{func} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_funcs")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("8" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{typ} ->> "name"), (#1{typ} ->> "schema")] + →Arrange ((#1{typ} ->> "name"), (#1{typ} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{typ} ->> "name")) IS NOT NULL AND ((#2{typ} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_types")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("7" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:mz_catalog_raw[#2{object_name}, #1{schema_name}] » %0[(#1{rel} ->> "name"), (#1{rel} ->> "schema")] + →Arrange ((#1{rel} ->> "name"), (#1{rel} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{rel} ->> "name")) IS NOT NULL AND ((#2{rel} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_relations")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2 != "7") AND (#2 != "8") AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_type"), (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →One-Shot Delta Join [%0 » %1:mz_catalog_raw[#1{schema_name}..=#3{object_type}] » %2:mz_catalog_raw[#1{schema_name}, #2{object_name}]] + path %0: + after %2: + Project: #3, #4 + Filter: case when (#2{ref_kind} = "func") then (#5{object_type} = "8") else case when (#2{ref_kind} = "type") then (#5{object_type} = "7") else ((#5{object_type} != "7") AND (#5{object_type} != "8")) end end + →Unarranged Raw Stream + →Constant (1024 rows) + →Arrange (#1{schema_name}..=#3{object_type}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #2, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "object_type"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Arrange (#1{schema_name}, #2{object_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2, #5 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")), (#1 ->> "object_type") + →Read mz_internal.mz_catalog_raw + →Differential Join %0:mz_catalog_raw[((#0{data} -> "key") ->> "name")] » %1:mz_catalog_raw[((#0{data} -> "key") ->> "object_name")] + after %1: + Project: #3, #4 + Map: ("si" || ((#1{data} -> "value") ->> "catalog_id")), ("s" || ((#2{data} -> "value") ->> "catalog_id")) + →Arrange (((#0{data} -> "key") ->> "name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("ClusterIntrospectionSourceIndex" = (#0{data} ->> "kind")) AND ((#1 ->> "name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arrange (((#0{data} -> "key") ->> "object_name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("2" = (#1 ->> "object_type")) AND ("mz_introspection" = (#1 ->> "schema_name")) AND ((#1 ->> "object_name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + +Source mz_internal.mz_catalog_raw + project=(#0, #1) + map=((#0{data} -> "key")) + +Target cluster: mz_catalog_server + +EOF + query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_object_fully_qualified_names"; ---- diff --git a/test/sqllogictest/funcs.slt b/test/sqllogictest/funcs.slt index 96cd26a74403d..63f5c442fb789 100644 --- a/test/sqllogictest/funcs.slt +++ b/test/sqllogictest/funcs.slt @@ -1769,3 +1769,22 @@ SELECT textrecv() query error cannot call function with arguments of type internal SELECT textrecv('abc') + +# `mz_internal.parse_catalog_item_references` extracts the catalog item +# references from a stored `create_sql` string. + +query T +SELECT mz_internal.parse_catalog_item_references('CREATE VIEW "materialize"."public"."v" AS SELECT "pg_catalog"."abs"("a"::[s20 AS "pg_catalog"."int4"]) FROM [u1 AS "materialize"."public"."t"]') +---- +{"ids":["s20","u1"],"named_funcs":[{"name":"abs","schema":"pg_catalog"}],"named_relations":[],"named_types":[]} + +# Name-based relation references never appear in stored SQL, which prints +# relations as id references. Reporting them keeps a statement that violates +# that from silently losing its dependency edges. +query T +SELECT mz_internal.parse_catalog_item_references('CREATE VIEW "materialize"."public"."v" AS SELECT * FROM "mz_catalog"."mz_tables"') +---- +{"ids":[],"named_funcs":[],"named_relations":[{"name":"mz_tables","schema":"mz_catalog"}],"named_types":[]} + +query error invalid catalog JSON: failed to parse create_sql +SELECT mz_internal.parse_catalog_item_references('NOT SQL') diff --git a/test/sqllogictest/information_schema_tables.slt b/test/sqllogictest/information_schema_tables.slt index 881b0b0b89c44..5b27ee141814f 100644 --- a/test/sqllogictest/information_schema_tables.slt +++ b/test/sqllogictest/information_schema_tables.slt @@ -494,7 +494,11 @@ SOURCE materialize mz_internal mz_object_dependencies -BASE TABLE +MATERIALIZED VIEW +materialize +mz_internal +mz_object_dependencies_raw +VIEW materialize mz_internal mz_object_fully_qualified_names diff --git a/test/sqllogictest/mz_catalog_server_index_accounting.slt b/test/sqllogictest/mz_catalog_server_index_accounting.slt index 05c34d29a228c..75b12a7d85855 100644 --- a/test/sqllogictest/mz_catalog_server_index_accounting.slt +++ b/test/sqllogictest/mz_catalog_server_index_accounting.slt @@ -37,110 +37,110 @@ mz_arrangement_heap_capacity_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangemen mz_arrangement_heap_size_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_heap_size_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_heap_size_raw"␠("operator_id",␠"worker_id") mz_arrangement_records_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_records_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_records_raw"␠("operator_id",␠"worker_id") mz_arrangement_sharing_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_sharing_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_sharing_raw"␠("operator_id",␠"worker_id") -mz_cluster_auto_scaling_strategies_ind CREATE␠INDEX␠"mz_cluster_auto_scaling_strategies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s521␠AS␠"mz_internal"."mz_cluster_auto_scaling_strategies"]␠("cluster_id") -mz_cluster_deployment_lineage_ind CREATE␠INDEX␠"mz_cluster_deployment_lineage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s770␠AS␠"mz_internal"."mz_cluster_deployment_lineage"]␠("cluster_id") +mz_cluster_auto_scaling_strategies_ind CREATE␠INDEX␠"mz_cluster_auto_scaling_strategies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s523␠AS␠"mz_internal"."mz_cluster_auto_scaling_strategies"]␠("cluster_id") +mz_cluster_deployment_lineage_ind CREATE␠INDEX␠"mz_cluster_deployment_lineage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s772␠AS␠"mz_internal"."mz_cluster_deployment_lineage"]␠("cluster_id") mz_cluster_prometheus_metrics_s2_primary_idx CREATE␠INDEX␠"mz_cluster_prometheus_metrics_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_cluster_prometheus_metrics"␠("process_id",␠"metric_name",␠"labels") -mz_cluster_reconfigurations_ind CREATE␠INDEX␠"mz_cluster_reconfigurations_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s520␠AS␠"mz_internal"."mz_cluster_reconfigurations"]␠("cluster_id") -mz_cluster_replica_frontiers_ind CREATE␠INDEX␠"mz_cluster_replica_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s764␠AS␠"mz_catalog"."mz_cluster_replica_frontiers"]␠("object_id") -mz_cluster_replica_history_ind CREATE␠INDEX␠"mz_cluster_replica_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s619␠AS␠"mz_internal"."mz_cluster_replica_history"]␠("dropped_at") -mz_cluster_replica_metrics_history_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s526␠AS␠"mz_internal"."mz_cluster_replica_metrics_history"]␠("replica_id") -mz_cluster_replica_metrics_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s527␠AS␠"mz_internal"."mz_cluster_replica_metrics"]␠("replica_id") -mz_cluster_replica_name_history_ind CREATE␠INDEX␠"mz_cluster_replica_name_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s620␠AS␠"mz_internal"."mz_cluster_replica_name_history"]␠("id") +mz_cluster_reconfigurations_ind CREATE␠INDEX␠"mz_cluster_reconfigurations_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s522␠AS␠"mz_internal"."mz_cluster_reconfigurations"]␠("cluster_id") +mz_cluster_replica_frontiers_ind CREATE␠INDEX␠"mz_cluster_replica_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s766␠AS␠"mz_catalog"."mz_cluster_replica_frontiers"]␠("object_id") +mz_cluster_replica_history_ind CREATE␠INDEX␠"mz_cluster_replica_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s621␠AS␠"mz_internal"."mz_cluster_replica_history"]␠("dropped_at") +mz_cluster_replica_metrics_history_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s528␠AS␠"mz_internal"."mz_cluster_replica_metrics_history"]␠("replica_id") +mz_cluster_replica_metrics_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s529␠AS␠"mz_internal"."mz_cluster_replica_metrics"]␠("replica_id") +mz_cluster_replica_name_history_ind CREATE␠INDEX␠"mz_cluster_replica_name_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s622␠AS␠"mz_internal"."mz_cluster_replica_name_history"]␠("id") mz_cluster_replica_resource_usage_s2_primary_idx CREATE␠INDEX␠"mz_cluster_replica_resource_usage_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_cluster_replica_resource_usage"␠("process_id",␠"source",␠"metric") -mz_cluster_replica_size_internal_ind CREATE␠INDEX␠"mz_cluster_replica_size_internal_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s516␠AS␠"mz_internal"."mz_cluster_replica_size_internal"]␠("size") -mz_cluster_replica_sizes_ind CREATE␠INDEX␠"mz_cluster_replica_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s515␠AS␠"mz_catalog"."mz_cluster_replica_sizes"]␠("size") -mz_cluster_replica_status_history_ind CREATE␠INDEX␠"mz_cluster_replica_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s528␠AS␠"mz_internal"."mz_cluster_replica_status_history"]␠("replica_id") -mz_cluster_replica_statuses_ind CREATE␠INDEX␠"mz_cluster_replica_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s529␠AS␠"mz_internal"."mz_cluster_replica_statuses"]␠("replica_id") -mz_cluster_replicas_ind CREATE␠INDEX␠"mz_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s525␠AS␠"mz_catalog"."mz_cluster_replicas"]␠("id") -mz_clusters_ind CREATE␠INDEX␠"mz_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s517␠AS␠"mz_catalog"."mz_clusters"]␠("id") -mz_columns_ind CREATE␠INDEX␠"mz_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s488␠AS␠"mz_catalog"."mz_columns"]␠("name") -mz_comments_ind CREATE␠INDEX␠"mz_comments_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s544␠AS␠"mz_internal"."mz_comments"]␠("id") +mz_cluster_replica_size_internal_ind CREATE␠INDEX␠"mz_cluster_replica_size_internal_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s518␠AS␠"mz_internal"."mz_cluster_replica_size_internal"]␠("size") +mz_cluster_replica_sizes_ind CREATE␠INDEX␠"mz_cluster_replica_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s517␠AS␠"mz_catalog"."mz_cluster_replica_sizes"]␠("size") +mz_cluster_replica_status_history_ind CREATE␠INDEX␠"mz_cluster_replica_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s530␠AS␠"mz_internal"."mz_cluster_replica_status_history"]␠("replica_id") +mz_cluster_replica_statuses_ind CREATE␠INDEX␠"mz_cluster_replica_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s531␠AS␠"mz_internal"."mz_cluster_replica_statuses"]␠("replica_id") +mz_cluster_replicas_ind CREATE␠INDEX␠"mz_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s527␠AS␠"mz_catalog"."mz_cluster_replicas"]␠("id") +mz_clusters_ind CREATE␠INDEX␠"mz_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s519␠AS␠"mz_catalog"."mz_clusters"]␠("id") +mz_columns_ind CREATE␠INDEX␠"mz_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s490␠AS␠"mz_catalog"."mz_columns"]␠("name") +mz_comments_ind CREATE␠INDEX␠"mz_comments_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s546␠AS␠"mz_internal"."mz_comments"]␠("id") mz_compute_dataflow_global_ids_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_dataflow_global_ids_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_dataflow_global_ids_per_worker"␠("id",␠"worker_id",␠"global_id") -mz_compute_dependencies_ind CREATE␠INDEX␠"mz_compute_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s745␠AS␠"mz_internal"."mz_compute_dependencies"]␠("dependency_id") +mz_compute_dependencies_ind CREATE␠INDEX␠"mz_compute_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s747␠AS␠"mz_internal"."mz_compute_dependencies"]␠("dependency_id") mz_compute_error_counts_raw_s2_primary_idx CREATE␠INDEX␠"mz_compute_error_counts_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_error_counts_raw"␠("export_id",␠"worker_id") mz_compute_exports_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_exports_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_exports_per_worker"␠("export_id",␠"worker_id") mz_compute_frontiers_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_frontiers_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_frontiers_per_worker"␠("export_id",␠"worker_id") -mz_compute_hydration_times_ind CREATE␠INDEX␠"mz_compute_hydration_times_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s755␠AS␠"mz_internal"."mz_compute_hydration_times"]␠("replica_id") +mz_compute_hydration_times_ind CREATE␠INDEX␠"mz_compute_hydration_times_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s757␠AS␠"mz_internal"."mz_compute_hydration_times"]␠("replica_id") mz_compute_hydration_times_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_hydration_times_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_hydration_times_per_worker"␠("export_id",␠"worker_id") mz_compute_import_frontiers_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_import_frontiers_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_import_frontiers_per_worker"␠("export_id",␠"import_id",␠"worker_id") mz_compute_lir_mapping_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_lir_mapping_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_lir_mapping_per_worker"␠("global_id",␠"lir_id",␠"worker_id") mz_compute_operator_durations_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_compute_operator_durations_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_operator_durations_histogram_raw"␠("id",␠"worker_id",␠"duration_ns") mz_compute_operator_hydration_statuses_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_operator_hydration_statuses_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_operator_hydration_statuses_per_worker"␠("export_id",␠"lir_id",␠"worker_id") -mz_connections_ind CREATE␠INDEX␠"mz_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s523␠AS␠"mz_catalog"."mz_connections"]␠("schema_id") -mz_console_cluster_utilization_overview_24h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_24h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s751␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_24h"]␠("cluster_id") -mz_console_cluster_utilization_overview_3h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_3h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s750␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_3h"]␠("cluster_id") -mz_console_cluster_utilization_overview_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s749␠AS␠"mz_internal"."mz_console_cluster_utilization_overview"]␠("cluster_id") -mz_databases_ind CREATE␠INDEX␠"mz_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s486␠AS␠"mz_catalog"."mz_databases"]␠("name") +mz_connections_ind CREATE␠INDEX␠"mz_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s525␠AS␠"mz_catalog"."mz_connections"]␠("schema_id") +mz_console_cluster_utilization_overview_24h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_24h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s753␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_24h"]␠("cluster_id") +mz_console_cluster_utilization_overview_3h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_3h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s752␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_3h"]␠("cluster_id") +mz_console_cluster_utilization_overview_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s751␠AS␠"mz_internal"."mz_console_cluster_utilization_overview"]␠("cluster_id") +mz_databases_ind CREATE␠INDEX␠"mz_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s488␠AS␠"mz_catalog"."mz_databases"]␠("name") mz_dataflow_addresses_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_addresses_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_addresses_per_worker"␠("id",␠"worker_id") mz_dataflow_channels_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_channels_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_channels_per_worker"␠("id",␠"worker_id") mz_dataflow_operator_reachability_raw_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_operator_reachability_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_operator_reachability_raw"␠("id",␠"worker_id",␠"source",␠"port",␠"update_type",␠"time") mz_dataflow_operators_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_operators_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_operators_per_worker"␠("id",␠"worker_id") -mz_frontiers_ind CREATE␠INDEX␠"mz_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s736␠AS␠"mz_internal"."mz_frontiers"]␠("object_id") -mz_hydration_statuses_ind CREATE␠INDEX␠"mz_hydration_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s766␠AS␠"mz_internal"."mz_hydration_statuses"]␠("object_id",␠"replica_id") -mz_indexes_ind CREATE␠INDEX␠"mz_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s489␠AS␠"mz_catalog"."mz_indexes"]␠("id") -mz_kafka_sources_ind CREATE␠INDEX␠"mz_kafka_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s483␠AS␠"mz_catalog"."mz_kafka_sources"]␠("id") -mz_materialized_views_ind CREATE␠INDEX␠"mz_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s548␠AS␠"mz_catalog"."mz_materialized_views"]␠("id") +mz_frontiers_ind CREATE␠INDEX␠"mz_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s738␠AS␠"mz_internal"."mz_frontiers"]␠("object_id") +mz_hydration_statuses_ind CREATE␠INDEX␠"mz_hydration_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s768␠AS␠"mz_internal"."mz_hydration_statuses"]␠("object_id",␠"replica_id") +mz_indexes_ind CREATE␠INDEX␠"mz_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s491␠AS␠"mz_catalog"."mz_indexes"]␠("id") +mz_kafka_sources_ind CREATE␠INDEX␠"mz_kafka_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s484␠AS␠"mz_catalog"."mz_kafka_sources"]␠("id") +mz_materialized_views_ind CREATE␠INDEX␠"mz_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s550␠AS␠"mz_catalog"."mz_materialized_views"]␠("id") mz_message_batch_counts_received_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_batch_counts_received_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_batch_counts_received_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_batch_counts_sent_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_batch_counts_sent_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_batch_counts_sent_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_counts_received_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_counts_received_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_counts_received_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_counts_sent_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_counts_sent_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_counts_sent_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") -mz_metric_sinks_ind CREATE␠INDEX␠"mz_metric_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s546␠AS␠"mz_internal"."mz_metric_sinks"]␠("id") -mz_notices_ind CREATE␠INDEX␠"mz_notices_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s844␠AS␠"mz_internal"."mz_notices"]␠("id") -mz_object_arrangement_size_history_object_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_object_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s758␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("object_id") -mz_object_arrangement_size_history_ts_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_ts_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s758␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("collection_timestamp") -mz_object_arrangement_sizes_ind CREATE␠INDEX␠"mz_object_arrangement_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s756␠AS␠"mz_internal"."mz_object_arrangement_sizes"]␠("replica_id") -mz_object_dependencies_ind CREATE␠INDEX␠"mz_object_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s484␠AS␠"mz_internal"."mz_object_dependencies"]␠("object_id") -mz_object_graph_edges_ind CREATE␠INDEX␠"mz_object_graph_edges_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s747␠AS␠"mz_internal"."mz_object_graph_edges"]␠("object_id") -mz_object_history_ind CREATE␠INDEX␠"mz_object_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s559␠AS␠"mz_internal"."mz_object_history"]␠("id") -mz_object_lifetimes_ind CREATE␠INDEX␠"mz_object_lifetimes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s560␠AS␠"mz_internal"."mz_object_lifetimes"]␠("id") -mz_object_transitive_dependencies_ind CREATE␠INDEX␠"mz_object_transitive_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s576␠AS␠"mz_internal"."mz_object_transitive_dependencies"]␠("object_id") -mz_objects_ind CREATE␠INDEX␠"mz_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s556␠AS␠"mz_catalog"."mz_objects"]␠("schema_id") +mz_metric_sinks_ind CREATE␠INDEX␠"mz_metric_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s548␠AS␠"mz_internal"."mz_metric_sinks"]␠("id") +mz_notices_ind CREATE␠INDEX␠"mz_notices_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s846␠AS␠"mz_internal"."mz_notices"]␠("id") +mz_object_arrangement_size_history_object_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_object_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s760␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("object_id") +mz_object_arrangement_size_history_ts_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_ts_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s760␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("collection_timestamp") +mz_object_arrangement_sizes_ind CREATE␠INDEX␠"mz_object_arrangement_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s758␠AS␠"mz_internal"."mz_object_arrangement_sizes"]␠("replica_id") +mz_object_dependencies_ind CREATE␠INDEX␠"mz_object_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s486␠AS␠"mz_internal"."mz_object_dependencies"]␠("object_id") +mz_object_graph_edges_ind CREATE␠INDEX␠"mz_object_graph_edges_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s749␠AS␠"mz_internal"."mz_object_graph_edges"]␠("object_id") +mz_object_history_ind CREATE␠INDEX␠"mz_object_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s561␠AS␠"mz_internal"."mz_object_history"]␠("id") +mz_object_lifetimes_ind CREATE␠INDEX␠"mz_object_lifetimes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s562␠AS␠"mz_internal"."mz_object_lifetimes"]␠("id") +mz_object_transitive_dependencies_ind CREATE␠INDEX␠"mz_object_transitive_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s578␠AS␠"mz_internal"."mz_object_transitive_dependencies"]␠("object_id") +mz_objects_ind CREATE␠INDEX␠"mz_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s558␠AS␠"mz_catalog"."mz_objects"]␠("schema_id") mz_peek_durations_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_peek_durations_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_peek_durations_histogram_raw"␠("worker_id",␠"type",␠"duration_ns") -mz_recent_activity_log_thinned_ind CREATE␠INDEX␠"mz_recent_activity_log_thinned_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s720␠AS␠"mz_internal"."mz_recent_activity_log_thinned"]␠("sql_hash") -mz_recent_sql_text_ind CREATE␠INDEX␠"mz_recent_sql_text_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s716␠AS␠"mz_internal"."mz_recent_sql_text"]␠("sql_hash") -mz_recent_storage_usage_ind CREATE␠INDEX␠"mz_recent_storage_usage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s835␠AS␠"mz_catalog"."mz_recent_storage_usage"]␠("object_id") -mz_roles_ind CREATE␠INDEX␠"mz_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s507␠AS␠"mz_catalog"."mz_roles"]␠("id") +mz_recent_activity_log_thinned_ind CREATE␠INDEX␠"mz_recent_activity_log_thinned_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s722␠AS␠"mz_internal"."mz_recent_activity_log_thinned"]␠("sql_hash") +mz_recent_sql_text_ind CREATE␠INDEX␠"mz_recent_sql_text_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s718␠AS␠"mz_internal"."mz_recent_sql_text"]␠("sql_hash") +mz_recent_storage_usage_ind CREATE␠INDEX␠"mz_recent_storage_usage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s837␠AS␠"mz_catalog"."mz_recent_storage_usage"]␠("object_id") +mz_roles_ind CREATE␠INDEX␠"mz_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s509␠AS␠"mz_catalog"."mz_roles"]␠("id") mz_scheduling_elapsed_raw_s2_primary_idx CREATE␠INDEX␠"mz_scheduling_elapsed_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_scheduling_elapsed_raw"␠("id",␠"worker_id") mz_scheduling_parks_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_scheduling_parks_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_scheduling_parks_histogram_raw"␠("worker_id",␠"slept_for_ns",␠"requested_ns") -mz_schemas_ind CREATE␠INDEX␠"mz_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s487␠AS␠"mz_catalog"."mz_schemas"]␠("database_id") -mz_secrets_ind CREATE␠INDEX␠"mz_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s522␠AS␠"mz_catalog"."mz_secrets"]␠("name") -mz_show_all_objects_ind CREATE␠INDEX␠"mz_show_all_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s604␠AS␠"mz_internal"."mz_show_all_objects"]␠("schema_id") -mz_show_cluster_replicas_ind CREATE␠INDEX␠"mz_show_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s768␠AS␠"mz_internal"."mz_show_cluster_replicas"]␠("cluster") -mz_show_clusters_ind CREATE␠INDEX␠"mz_show_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s606␠AS␠"mz_internal"."mz_show_clusters"]␠("name") -mz_show_columns_ind CREATE␠INDEX␠"mz_show_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s605␠AS␠"mz_internal"."mz_show_columns"]␠("id") -mz_show_connections_ind CREATE␠INDEX␠"mz_show_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s614␠AS␠"mz_internal"."mz_show_connections"]␠("schema_id") -mz_show_databases_ind CREATE␠INDEX␠"mz_show_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s608␠AS␠"mz_internal"."mz_show_databases"]␠("name") -mz_show_indexes_ind CREATE␠INDEX␠"mz_show_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s618␠AS␠"mz_internal"."mz_show_indexes"]␠("schema_id") -mz_show_materialized_views_ind CREATE␠INDEX␠"mz_show_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s617␠AS␠"mz_internal"."mz_show_materialized_views"]␠("schema_id") -mz_show_roles_ind CREATE␠INDEX␠"mz_show_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s613␠AS␠"mz_internal"."mz_show_roles"]␠("name") -mz_show_schemas_ind CREATE␠INDEX␠"mz_show_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s609␠AS␠"mz_internal"."mz_show_schemas"]␠("database_id") -mz_show_secrets_ind CREATE␠INDEX␠"mz_show_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s607␠AS␠"mz_internal"."mz_show_secrets"]␠("schema_id") -mz_show_sinks_ind CREATE␠INDEX␠"mz_show_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s616␠AS␠"mz_internal"."mz_show_sinks"]␠("schema_id") -mz_show_sources_ind CREATE␠INDEX␠"mz_show_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s615␠AS␠"mz_internal"."mz_show_sources"]␠("schema_id") -mz_show_tables_ind CREATE␠INDEX␠"mz_show_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s610␠AS␠"mz_internal"."mz_show_tables"]␠("schema_id") -mz_show_types_ind CREATE␠INDEX␠"mz_show_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s612␠AS␠"mz_internal"."mz_show_types"]␠("schema_id") -mz_show_views_ind CREATE␠INDEX␠"mz_show_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s611␠AS␠"mz_internal"."mz_show_views"]␠("schema_id") -mz_sink_statistics_ind CREATE␠INDEX␠"mz_sink_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s733␠AS␠"mz_internal"."mz_sink_statistics"]␠("id",␠"replica_id") -mz_sink_status_history_ind CREATE␠INDEX␠"mz_sink_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s705␠AS␠"mz_internal"."mz_sink_status_history"]␠("sink_id") -mz_sink_statuses_ind CREATE␠INDEX␠"mz_sink_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s706␠AS␠"mz_internal"."mz_sink_statuses"]␠("id") -mz_sinks_ind CREATE␠INDEX␠"mz_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s499␠AS␠"mz_catalog"."mz_sinks"]␠("id") -mz_source_statistics_ind CREATE␠INDEX␠"mz_source_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s731␠AS␠"mz_internal"."mz_source_statistics"]␠("id",␠"replica_id") -mz_source_statistics_with_history_ind CREATE␠INDEX␠"mz_source_statistics_with_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s729␠AS␠"mz_internal"."mz_source_statistics_with_history"]␠("id",␠"replica_id") -mz_source_status_history_ind CREATE␠INDEX␠"mz_source_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s707␠AS␠"mz_internal"."mz_source_status_history"]␠("source_id") -mz_source_statuses_ind CREATE␠INDEX␠"mz_source_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s724␠AS␠"mz_internal"."mz_source_statuses"]␠("id") -mz_sources_ind CREATE␠INDEX␠"mz_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s492␠AS␠"mz_catalog"."mz_sources"]␠("id") -mz_tables_ind CREATE␠INDEX␠"mz_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s491␠AS␠"mz_catalog"."mz_tables"]␠("schema_id") -mz_types_ind CREATE␠INDEX␠"mz_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s501␠AS␠"mz_catalog"."mz_types"]␠("schema_id") -mz_views_ind CREATE␠INDEX␠"mz_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s500␠AS␠"mz_catalog"."mz_views"]␠("schema_id") -mz_wallclock_global_lag_recent_history_ind CREATE␠INDEX␠"mz_wallclock_global_lag_recent_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s740␠AS␠"mz_internal"."mz_wallclock_global_lag_recent_history"]␠("object_id") -mz_webhook_sources_ind CREATE␠INDEX␠"mz_webhook_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s545␠AS␠"mz_internal"."mz_webhook_sources"]␠("id") -pg_attrdef_all_databases_ind CREATE␠INDEX␠"pg_attrdef_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s645␠AS␠"mz_internal"."pg_attrdef_all_databases"]␠("oid",␠"adrelid",␠"adnum",␠"adbin",␠"adsrc") -pg_attribute_all_databases_ind CREATE␠INDEX␠"pg_attribute_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s638␠AS␠"mz_internal"."pg_attribute_all_databases"]␠("attrelid",␠"attname",␠"atttypid",␠"attlen",␠"attnum",␠"atttypmod",␠"attnotnull",␠"atthasdef",␠"attidentity",␠"attgenerated",␠"attisdropped",␠"attcollation",␠"database_name",␠"pg_type_database_name") -pg_authid_core_ind CREATE␠INDEX␠"pg_authid_core_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s655␠AS␠"mz_internal"."pg_authid_core"]␠("rolname") -pg_class_all_databases_ind CREATE␠INDEX␠"pg_class_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s626␠AS␠"mz_internal"."pg_class_all_databases"]␠("relname") -pg_description_all_databases_ind CREATE␠INDEX␠"pg_description_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s635␠AS␠"mz_internal"."pg_description_all_databases"]␠("objoid",␠"classoid",␠"objsubid",␠"description",␠"oid_database_name",␠"class_database_name") -pg_namespace_all_databases_ind CREATE␠INDEX␠"pg_namespace_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s623␠AS␠"mz_internal"."pg_namespace_all_databases"]␠("nspname") -pg_type_all_databases_ind CREATE␠INDEX␠"pg_type_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s632␠AS␠"mz_internal"."pg_type_all_databases"]␠("oid") +mz_schemas_ind CREATE␠INDEX␠"mz_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s489␠AS␠"mz_catalog"."mz_schemas"]␠("database_id") +mz_secrets_ind CREATE␠INDEX␠"mz_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s524␠AS␠"mz_catalog"."mz_secrets"]␠("name") +mz_show_all_objects_ind CREATE␠INDEX␠"mz_show_all_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s606␠AS␠"mz_internal"."mz_show_all_objects"]␠("schema_id") +mz_show_cluster_replicas_ind CREATE␠INDEX␠"mz_show_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s770␠AS␠"mz_internal"."mz_show_cluster_replicas"]␠("cluster") +mz_show_clusters_ind CREATE␠INDEX␠"mz_show_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s608␠AS␠"mz_internal"."mz_show_clusters"]␠("name") +mz_show_columns_ind CREATE␠INDEX␠"mz_show_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s607␠AS␠"mz_internal"."mz_show_columns"]␠("id") +mz_show_connections_ind CREATE␠INDEX␠"mz_show_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s616␠AS␠"mz_internal"."mz_show_connections"]␠("schema_id") +mz_show_databases_ind CREATE␠INDEX␠"mz_show_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s610␠AS␠"mz_internal"."mz_show_databases"]␠("name") +mz_show_indexes_ind CREATE␠INDEX␠"mz_show_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s620␠AS␠"mz_internal"."mz_show_indexes"]␠("schema_id") +mz_show_materialized_views_ind CREATE␠INDEX␠"mz_show_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s619␠AS␠"mz_internal"."mz_show_materialized_views"]␠("schema_id") +mz_show_roles_ind CREATE␠INDEX␠"mz_show_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s615␠AS␠"mz_internal"."mz_show_roles"]␠("name") +mz_show_schemas_ind CREATE␠INDEX␠"mz_show_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s611␠AS␠"mz_internal"."mz_show_schemas"]␠("database_id") +mz_show_secrets_ind CREATE␠INDEX␠"mz_show_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s609␠AS␠"mz_internal"."mz_show_secrets"]␠("schema_id") +mz_show_sinks_ind CREATE␠INDEX␠"mz_show_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s618␠AS␠"mz_internal"."mz_show_sinks"]␠("schema_id") +mz_show_sources_ind CREATE␠INDEX␠"mz_show_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s617␠AS␠"mz_internal"."mz_show_sources"]␠("schema_id") +mz_show_tables_ind CREATE␠INDEX␠"mz_show_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s612␠AS␠"mz_internal"."mz_show_tables"]␠("schema_id") +mz_show_types_ind CREATE␠INDEX␠"mz_show_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s614␠AS␠"mz_internal"."mz_show_types"]␠("schema_id") +mz_show_views_ind CREATE␠INDEX␠"mz_show_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s613␠AS␠"mz_internal"."mz_show_views"]␠("schema_id") +mz_sink_statistics_ind CREATE␠INDEX␠"mz_sink_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s735␠AS␠"mz_internal"."mz_sink_statistics"]␠("id",␠"replica_id") +mz_sink_status_history_ind CREATE␠INDEX␠"mz_sink_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s707␠AS␠"mz_internal"."mz_sink_status_history"]␠("sink_id") +mz_sink_statuses_ind CREATE␠INDEX␠"mz_sink_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s708␠AS␠"mz_internal"."mz_sink_statuses"]␠("id") +mz_sinks_ind CREATE␠INDEX␠"mz_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s501␠AS␠"mz_catalog"."mz_sinks"]␠("id") +mz_source_statistics_ind CREATE␠INDEX␠"mz_source_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s733␠AS␠"mz_internal"."mz_source_statistics"]␠("id",␠"replica_id") +mz_source_statistics_with_history_ind CREATE␠INDEX␠"mz_source_statistics_with_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s731␠AS␠"mz_internal"."mz_source_statistics_with_history"]␠("id",␠"replica_id") +mz_source_status_history_ind CREATE␠INDEX␠"mz_source_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s709␠AS␠"mz_internal"."mz_source_status_history"]␠("source_id") +mz_source_statuses_ind CREATE␠INDEX␠"mz_source_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s726␠AS␠"mz_internal"."mz_source_statuses"]␠("id") +mz_sources_ind CREATE␠INDEX␠"mz_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s494␠AS␠"mz_catalog"."mz_sources"]␠("id") +mz_tables_ind CREATE␠INDEX␠"mz_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s493␠AS␠"mz_catalog"."mz_tables"]␠("schema_id") +mz_types_ind CREATE␠INDEX␠"mz_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s503␠AS␠"mz_catalog"."mz_types"]␠("schema_id") +mz_views_ind CREATE␠INDEX␠"mz_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s502␠AS␠"mz_catalog"."mz_views"]␠("schema_id") +mz_wallclock_global_lag_recent_history_ind CREATE␠INDEX␠"mz_wallclock_global_lag_recent_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s742␠AS␠"mz_internal"."mz_wallclock_global_lag_recent_history"]␠("object_id") +mz_webhook_sources_ind CREATE␠INDEX␠"mz_webhook_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s547␠AS␠"mz_internal"."mz_webhook_sources"]␠("id") +pg_attrdef_all_databases_ind CREATE␠INDEX␠"pg_attrdef_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s647␠AS␠"mz_internal"."pg_attrdef_all_databases"]␠("oid",␠"adrelid",␠"adnum",␠"adbin",␠"adsrc") +pg_attribute_all_databases_ind CREATE␠INDEX␠"pg_attribute_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s640␠AS␠"mz_internal"."pg_attribute_all_databases"]␠("attrelid",␠"attname",␠"atttypid",␠"attlen",␠"attnum",␠"atttypmod",␠"attnotnull",␠"atthasdef",␠"attidentity",␠"attgenerated",␠"attisdropped",␠"attcollation",␠"database_name",␠"pg_type_database_name") +pg_authid_core_ind CREATE␠INDEX␠"pg_authid_core_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s657␠AS␠"mz_internal"."pg_authid_core"]␠("rolname") +pg_class_all_databases_ind CREATE␠INDEX␠"pg_class_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s628␠AS␠"mz_internal"."pg_class_all_databases"]␠("relname") +pg_description_all_databases_ind CREATE␠INDEX␠"pg_description_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s637␠AS␠"mz_internal"."pg_description_all_databases"]␠("objoid",␠"classoid",␠"objsubid",␠"description",␠"oid_database_name",␠"class_database_name") +pg_namespace_all_databases_ind CREATE␠INDEX␠"pg_namespace_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s625␠AS␠"mz_internal"."pg_namespace_all_databases"]␠("nspname") +pg_type_all_databases_ind CREATE␠INDEX␠"pg_type_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s634␠AS␠"mz_internal"."pg_type_all_databases"]␠("oid") # Record all transitive dependencies (tables, sources, views, mvs) of indexes on # the mz_catalog_server cluster. @@ -558,6 +558,8 @@ mz_object_arrangement_sizes replica_id mz_object_arrangement_sizes size mz_object_dependencies object_id mz_object_dependencies referenced_object_id +mz_object_dependencies_raw object_id +mz_object_dependencies_raw referenced_object_id mz_object_graph_edges dependency_id mz_object_graph_edges object_id mz_object_history cluster_id diff --git a/test/sqllogictest/mz_object_dependencies.slt b/test/sqllogictest/mz_object_dependencies.slt new file mode 100644 index 0000000000000..fe7463f151f90 --- /dev/null +++ b/test/sqllogictest/mz_object_dependencies.slt @@ -0,0 +1,313 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Behavioural tests for mz_internal.mz_object_dependencies. + +mode cockroach + +reset-server + +# All user objects live in a scratch schema owned by the test role, so the +# cascade at the end of the file can clean up everything in one statement. +# The public schema is owned by mz_system, so it cannot play that role. +statement ok +CREATE SCHEMA deps + +statement ok +SET search_path = deps + +statement ok +CREATE TABLE t (a int, b text NOT NULL) + +statement ok +CREATE VIEW v AS SELECT pg_catalog.abs(a), b::text FROM t WHERE a > 42 + +# --- relation and function edges from user items ------------------------------ + +# v depends on t (relation reference, stored as an id in create_sql). +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'v') + AND referenced_object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +1 + +# v depends on pg_catalog.abs. Function references print as plain names in +# stored SQL and are recovered by name from the GidMapping rows. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'v') + AND referenced_object_id IN (SELECT id FROM mz_functions WHERE name = 'abs') +---- +1 + +# v depends on the type named in its definition (text from the cast), which +# prints as an id reference in stored SQL. +query T rowsort +SELECT ty.name +FROM mz_internal.mz_object_dependencies dep +JOIN mz_types ty ON ty.id = dep.referenced_object_id +WHERE dep.object_id = (SELECT id FROM mz_views WHERE name = 'v') +---- +text + +# t depends on its column types. +query T rowsort +SELECT ty.name +FROM mz_internal.mz_object_dependencies dep +JOIN mz_types ty ON ty.id = dep.referenced_object_id +WHERE dep.object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +int4 +text + +# --- array types --------------------------------------------------------------- + +statement ok +CREATE TABLE arr (a int4[]) + +# int4[] yields an edge to the array type _int4 and none to the element type +# int4, the same set a direct reference to _int4 produces. Name resolution +# treats both spellings as referencing the array type alone. +query T rowsort +SELECT ty.name +FROM mz_internal.mz_object_dependencies dep +JOIN mz_types ty ON ty.id = dep.referenced_object_id +WHERE dep.object_id = (SELECT id FROM mz_tables WHERE name = 'arr') +---- +_int4 + +# --- connection and secret edges ------------------------------------------------ + +statement ok +CREATE SECRET pw AS 'hunter2' + +statement ok +CREATE CONNECTION kc TO KAFKA (BROKER 'kafka:9092', SASL MECHANISMS 'PLAIN', SASL USERNAME 'u', SASL PASSWORD = SECRET pw) WITH (VALIDATE = false) + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_connections WHERE name = 'kc') + AND referenced_object_id = (SELECT id FROM mz_secrets WHERE name = 'pw') +---- +1 + +# --- temporary items are excluded ---------------------------------------------- + +statement ok +CREATE TEMPORARY VIEW tv AS SELECT * FROM t + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'tv') +---- +0 + +# --- materialized view edges ----------------------------------------------------- + +statement ok +CREATE MATERIALIZED VIEW umv AS SELECT a + 1 AS a1 FROM t + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_materialized_views WHERE name = 'umv') + AND referenced_object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +1 + +# --- index edges ------------------------------------------------------------------- + +statement ok +CREATE INDEX fidx ON t (abs(a)) + +# The index depends on the relation it indexes and on the function its key +# calls. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_indexes WHERE name = 'fidx') + AND referenced_object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +1 + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_indexes WHERE name = 'fidx') + AND referenced_object_id IN (SELECT id FROM mz_functions WHERE name = 'abs') +---- +1 + +# --- custom types ------------------------------------------------------------------ + +statement ok +CREATE TYPE custom_list AS LIST (ELEMENT TYPE = int4) + +# A custom type depends on its element type: a type -> type edge. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_types WHERE name = 'custom_list') + AND referenced_object_id = (SELECT id FROM mz_types WHERE name = 'int4') +---- +1 + +# --- tables from sources ------------------------------------------------------------- + +statement ok +CREATE SOURCE lg FROM LOAD GENERATOR COUNTER + +statement ok +CREATE TABLE lg_tbl FROM SOURCE lg (REFERENCE counter) + +# The table's create_sql references its source by id outside any query. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_tables WHERE name = 'lg_tbl') + AND referenced_object_id = (SELECT id FROM mz_sources WHERE name = 'lg') +---- +1 + +# --- builtin edges --------------------------------------------------------------- + +# mz_catalog.mz_indexes is itself a materialized view over mz_catalog_raw. +# Its edge comes from the VALUES inlined at generation time, joined to +# GidMapping rows at query time. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +JOIN mz_objects r ON r.id = dep.referenced_object_id +JOIN mz_schemas rs ON rs.id = r.schema_id +WHERE os.name = 'mz_catalog' AND o.name = 'mz_indexes' + AND rs.name = 'mz_internal' AND r.name = 'mz_catalog_raw' +---- +1 + +# Builtin views also depend on the functions they call, recorded by name at +# generation time. mz_catalog.mz_indexes calls mz_internal.parse_catalog_id. +query T +SELECT count(*) > 0 +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +WHERE os.name = 'mz_catalog' AND o.name = 'mz_indexes' + AND dep.referenced_object_id IN (SELECT id FROM mz_functions WHERE name = 'parse_catalog_id') +---- +true + +# Introspection source indexes depend on their log. +query T +SELECT count(*) > 0 FROM mz_internal.mz_object_dependencies WHERE object_id LIKE 'si%' +---- +true + +# The relation itself must be reported as a materialized view. +query T +SELECT count(*) FROM mz_materialized_views WHERE name = 'mz_object_dependencies' +---- +1 + +query T +SELECT count(*) FROM mz_internal.mz_ontology_entity_types WHERE name = 'object_dependency' +---- +1 + +# The generating view carries its own outgoing edges. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +JOIN mz_objects r ON r.id = dep.referenced_object_id +JOIN mz_schemas rs ON rs.id = r.schema_id +WHERE os.name = 'mz_internal' AND o.name = 'mz_object_dependencies_raw' + AND rs.name = 'mz_internal' AND r.name = 'mz_catalog_raw' +---- +1 + +query T +SELECT count(*) > 0 +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +WHERE os.name = 'mz_internal' AND o.name = 'mz_object_dependencies_raw' + AND dep.referenced_object_id IN ( + SELECT id FROM mz_functions WHERE name = 'parse_catalog_item_references' + ) +---- +true + +# mz_object_dependencies reads mz_object_dependencies_raw, so the edge between +# the two is recorded like any other. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +JOIN mz_objects r ON r.id = dep.referenced_object_id +JOIN mz_schemas rs ON rs.id = r.schema_id +WHERE os.name = 'mz_internal' AND o.name = 'mz_object_dependencies' + AND rs.name = 'mz_internal' AND r.name = 'mz_object_dependencies_raw' +---- +1 + +# --- dropped items leave no edges -------------------------------------------------- + +statement ok +CREATE VIEW dropme AS SELECT a FROM t + +query T +SELECT count(*) > 0 +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'dropme') +---- +true + +statement ok +DROP VIEW dropme + +# --- structural invariants ------------------------------------------------------- + +# No duplicate edges: ResolvedIds was a map, so the packed table never +# produced duplicates, and consumers (e.g. transitive closures) assume this. +query T +SELECT count(*) FROM ( + SELECT object_id, referenced_object_id + FROM mz_internal.mz_object_dependencies + GROUP BY object_id, referenced_object_id + HAVING count(*) > 1 +) +---- +0 + +# No self-edges. +query T +SELECT count(*) FROM mz_internal.mz_object_dependencies WHERE object_id = referenced_object_id +---- +0 + +# No edge outlives its object: cascading away the whole scratch schema drops +# every user object created above, so any user edge still present is a leak. +statement ok +DROP SCHEMA deps CASCADE + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id LIKE 'u%' OR referenced_object_id LIKE 'u%' +---- +0 diff --git a/test/sqllogictest/oid.slt b/test/sqllogictest/oid.slt index fe4fdaaebd174..8bea7512a2c74 100644 --- a/test/sqllogictest/oid.slt +++ b/test/sqllogictest/oid.slt @@ -1254,3 +1254,5 @@ SELECT oid, name FROM mz_objects WHERE id LIKE 's%' AND oid < 20000 ORDER BY oid 17122 mz_object_hydration_history 17123 mz_cluster_replica_resource_usage 17124 mz_replica_hydration_history +17125 parse_catalog_item_references +17126 mz_object_dependencies_raw diff --git a/test/testdrive/catalog.td b/test/testdrive/catalog.td index 12074488a6a79..645afa00580b8 100644 --- a/test/testdrive/catalog.td +++ b/test/testdrive/catalog.td @@ -608,7 +608,6 @@ mz_history_retention_strategies "" mz_license_keys "" mz_materialized_view_refresh_strategies "" mz_object_arrangement_size_history "" -mz_object_dependencies "" mz_object_global_ids "" mz_object_hydration_history "" mz_optimizer_notices "" @@ -649,6 +648,7 @@ mz_mcp_data_product_details "" mz_mcp_data_products "" mz_notices "" mz_notices_redacted "" +mz_object_dependencies_raw "" mz_object_fully_qualified_names "" mz_object_graph_edges "" mz_object_history "" @@ -734,6 +734,7 @@ mz_metric_sinks mz_catalog_server "" mz_mysql_source_tables mz_catalog_server "" mz_network_policies mz_catalog_server "" mz_network_policy_rules mz_catalog_server "" +mz_object_dependencies mz_catalog_server "" mz_overridden_system_parameters mz_catalog_server "" mz_pending_cluster_replicas mz_catalog_server "" mz_postgres_source_tables mz_catalog_server "" @@ -837,7 +838,7 @@ test_table "" # `SHOW TABLES` and `mz_tables` should agree. > SELECT COUNT(*) FROM mz_tables WHERE id LIKE 's%' -31 +30 # There is one entry in mz_indexes for each field_number/expression of the index. > SELECT COUNT(id) FROM mz_indexes WHERE id LIKE 's%' diff --git a/test/testdrive/mz-depends.td b/test/testdrive/mz-depends.td index 4f3b29b2ab948..6ba32dfec7a6e 100644 --- a/test/testdrive/mz-depends.td +++ b/test/testdrive/mz-depends.td @@ -65,6 +65,36 @@ conn dep_conn pg_conn ssh_conn +# A function referenced both by id and by name in one statement yields a +# single edge. + +$ set-from-sql var=gs-id +SELECT DISTINCT id FROM mz_functions +WHERE name = 'generate_series' + AND schema_id IN (SELECT id FROM mz_schemas WHERE name = 'pg_catalog') + +> CREATE VIEW dup_func_ref AS + SELECT by_id.g, by_name.h + FROM [${gs-id} AS pg_catalog.generate_series](1, 3) AS by_id(g), + generate_series(4, 6) AS by_name(h); + +# Both spellings persist and land in different reference buckets. +> SELECT refs->'ids' ? '${gs-id}' AS by_id, + refs->'named_funcs' @> '[{"schema": "pg_catalog", "name": "generate_series"}]' AS by_name + FROM ( + SELECT mz_internal.parse_catalog_item_references(create_sql) AS refs + FROM mz_views WHERE name = 'dup_func_ref' + ); +by_id by_name +------------- +true true + +# ... yet exactly one edge to the function. +> SELECT count(*) FROM mz_internal.mz_object_dependencies + WHERE object_id = (SELECT id FROM mz_views WHERE name = 'dup_func_ref') + AND referenced_object_id = '${gs-id}'; +1 + # Assert that we actually are populating all the system objects. > SELECT COUNT(*) > 200 FROM mz_internal.mz_object_dependencies WHERE object_id LIKE 's%' true From 26fa51cfed3943e2379f6e1d472122bbcba419eb Mon Sep 17 00:00:00 2001 From: Qindeel Ishtiaq Date: Thu, 3 Sep 2026 14:20:15 -0700 Subject: [PATCH 7/8] console: fix stale catalog data across region switches (#38631) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Switching regions in the console left pages fed by the app-wide SUBSCRIBEs (cluster list, object explorer) showing the previous region's catalog until a full page refresh, and the sync-engine localStorage cache could be poisoned with another region's rows. Fixing this properly also motivated restructuring how the subscribe stack is wired, and documenting the resulting architecture. Fixes: [CNS-157 ](https://linear.app/materializeinc/issue/CNS-157/fix-sync-engine-reseeding-when-changing-regions) Fixes: [CNS-155](https://linear.app/materializeinc/issue/CNS-155/clusters-do-not-update-when-switching-regions) ## The commits (one concern each; the branch squash-merges) 1. **Sync-engine cache scope fixes** (`subscribeCollection.ts`): a scope change drops in-memory rows and any pending persist before re-seeding, so one tenant's catalog is never written under another's cache key, and a reconnect's empty pre-snapshot clears a stale error so the UI falls back to loading during backoff. (Both flagged by the QA review on #38609.) 2. **Region-aware connection manager** (`WebsocketConnectionManager.ts`): tracks the address and region of the most recent connection attempt. Reconnects when the region's address changes (including mid-handshake), and tears down a paused socket only when it points at a region the user has left — keyed on region so disabled regions (whose address never updates) are covered, while a same-region health blip leaves a working socket alone (an intentional disconnect fires no close callback, so unconditional teardown would silently replace the SQL Shell's session). 3. **Subscribe sessions** (`subscribeSession.ts` + hooks + stores): the wiring that had accumulated as one React effect per concern moves into plain session objects that subscribe to the jotai store directly, the pattern the connection manager already used. Sessions reset their manager and sink on region change and hydrate collections from the scoped cache via a new cache-scope atom. Hooks keep a single lifecycle effect, cannot suspend, and take referentially stable module-scope options. The allObjects-to-collection bridge runs off a store subscription instead of re-renders. 4. **Documentation** (`doc/design/20260902_sync_engine.md`, `doc/architecture.md`, `AGENTS.md`): a design doc explaining the sync engine end to end (feed, sessions, both sinks, the collection bridge and its scoped instant-load cache, region-switch data flow, boundaries, adoption recipes), plus updates to the data-layer sections of the architecture doc and the agent guidance so they describe the server-state split: React Query for polled/one-shot data, the sync engine for live catalog data. ## Tips for reviewer Every behavioral claim has a regression test that was verified to fail against the code it fixes: cache scope and error-clear tests in `subscribeCollection.test.ts`, the region round-trip / same-region-blip / disabled-region / mid-handshake tests in `WebsocketConnectionManager.test.ts`, and headless session tests in `useSubscribe.test.tsx`. Known unrelated failure: `PlanDetails.test.tsx` (billing) fails identically on current main — it started failing on 2026-09-03 with no billing changes here, so it looks date-dependent and is not introduced by this branch. ## Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [x] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. - [ ] If this PR evolves [an existing $T ⇔ Proto$T mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- console/AGENTS.md | 7 +- console/doc/architecture.md | 8 + console/doc/design/20260902_sync_engine.md | 290 ++++++++++++++++++ console/package.json | 11 + console/src/api/auth.tsx | 22 +- .../WebsocketConnectionManager.test.ts | 165 ++++++++++ .../materialize/WebsocketConnectionManager.ts | 35 ++- .../materialize/subscribeCollection.test.ts | 120 +++++++- .../api/materialize/subscribeCollection.ts | 83 ++++- .../src/api/materialize/subscribeSession.ts | 222 ++++++++++++++ .../src/api/materialize/useSubscribe.test.tsx | 237 ++++++++++++++ console/src/api/materialize/useSubscribe.ts | 131 +++----- .../allNamespacesCollection.ts | 42 ++- console/src/store/allClusters.ts | 31 +- console/src/store/allObjects.ts | 23 +- console/src/store/allObjectsCollection.ts | 21 +- console/src/store/allRoles.ts | 24 +- console/src/store/allSchemas.ts | 25 +- console/src/store/syncEngineCache.ts | 45 ++- 19 files changed, 1329 insertions(+), 213 deletions(-) create mode 100644 console/doc/design/20260902_sync_engine.md create mode 100644 console/src/api/materialize/subscribeSession.ts create mode 100644 console/src/api/materialize/useSubscribe.test.tsx diff --git a/console/AGENTS.md b/console/AGENTS.md index aa30845198a21..186e37235dd10 100644 --- a/console/AGENTS.md +++ b/console/AGENTS.md @@ -77,7 +77,12 @@ Use `~/` for imports from src: `import { foo } from "~/api/materialize"` ### State Management - **Jotai** for global state (sparingly used - prefer component state) -- **React Query** for server state +- **React Query** for polled/one-shot server state (`useSql` hooks) +- **Sync engine** for live catalog data: SUBSCRIBE-fed sessions reduce into + jotai atoms and TanStack DB collections (`useLiveQuery` consumers, scoped + instant-load cache). See `doc/design/20260902_sync_engine.md`. Wiring between + the engine's singletons belongs in session objects on the jotai store + (`store.sub`), not in React effects. - **Kysely** as query builder (NOT as database driver) for type-safe SQL against Materialize system catalog ### Stacks diff --git a/console/doc/architecture.md b/console/doc/architecture.md index 6c6d20aedb2e9..623b4356fdce8 100644 --- a/console/doc/architecture.md +++ b/console/doc/architecture.md @@ -69,3 +69,11 @@ state is stored in components with the useState hook. Storing state globally can be tricky to get right, in particular when it comes to invalidating / cleaning up the state. With component state, we get that behavior for free because the component gets unmounted. + +Server state is split by shape. Request/response data (polled telemetry, +one-shot queries, mutations) goes through React Query and the `useSql` hooks. +Live catalog data goes through the sync engine: app-session `SUBSCRIBE`s feed +subscribe sessions, which reduce into jotai atoms and TanStack DB collections, +with a per-tenant localStorage cache for instant load. See +[the sync engine design doc](design/20260902_sync_engine.md) for the full +architecture and adoption recipes. diff --git a/console/doc/design/20260902_sync_engine.md b/console/doc/design/20260902_sync_engine.md new file mode 100644 index 0000000000000..88213bba869b0 --- /dev/null +++ b/console/doc/design/20260902_sync_engine.md @@ -0,0 +1,290 @@ +# The sync engine + +- Associated: MaterializeInc/materialize#38231, MaterializeInc/materialize#38631 + +## Overview + +The console keeps a live, client-side replica of parts of the Materialize +catalog. The sync engine is the stack that maintains that replica: a +Materialize `SUBSCRIBE` feeds a reduction layer, plain session objects manage +lifecycle, and the results land in one of two stores that pages read from. The +newer of the two stores is TanStack DB, which adds incrementally maintained +live queries and a per-tenant instant-load cache on top of the same feed. + +Pages never see the plumbing. They read a jotai atom or run a `useLiveQuery` +against a collection, and the data is simply there, current, across +reconnects and region switches. + +``` + Materialize + SUBSCRIBE ... WITH (PROGRESS) ENVELOPE UPSERT (one per data set) + | + v WebSocket diff stream + SubscribeManager folds diffs into the current row set, + | emits one snapshot per closed timestamp + v + subscribe session plain object: lifecycle, region reset, + | cache hydration, sink writes + v + +----+---------------------------+ + | | + v v + jotai atom TanStack DB collection <---> localStorage cache + (SubscribeState) (rows diffed in) (scoped, instant load) + | | + v v + useAtomValue useLiveQuery + consumers consumers (incremental) +``` + +## Why two stores + +The jotai atoms (`allObjects`, `allSchemas`, `allClusters`, `allRoles`) hold +each data set as one array plus status. They are simple and fine for consumers +that render the whole set or run a cheap filter. + +TanStack DB collections hold the same rows keyed, with a live-query engine on +top. A collection is the right store when a surface needs any of: + +- **Incremental recomputation.** A live query (filters, joins, lookups) + recomputes from row deltas instead of re-scanning the full array on every + catalog tick. With atoms, every tick replaces the array and every consumer + re-filters and re-renders. +- **Fine-grained re-renders.** A component watching one key re-renders only + when that row changes, not whenever anything in the set changes. +- **Instant load.** Collections persist complete snapshots to a per-tenant + localStorage cache and seed from it on the next visit, so the surface + renders before the live snapshot arrives. + +Both stores are sinks of the same engine. Adopting a collection for a data set +does not change what the server sees: within a tab, the upstream SUBSCRIBE +count stays the same, and a collection can even be fed from an existing atom +(see the atom-fed session below) at zero additional upstream cost. + +## The layers + +| Layer | File | Owns | +| --- | --- | --- | +| `MaterializeWebsocket` | `api/materialize/MaterializeWebsocket.ts` | The wire: one WebSocket to the SQL HTTP endpoint, one subscribe per connection. | +| `SubscribeManager` | `api/materialize/SubscribeManager.ts` | The SUBSCRIBE protocol: buffers rows per timestamp, folds the upsert stream into the current row set, exposes snapshots via `onChange`/`getSnapshot`. Holds its last snapshot through a resubscribe so the UI does not flash empty. | +| `WebsocketConnectionManager` | `api/materialize/WebsocketConnectionManager.ts` | Connection lifecycle: subscribes to environment health and the current region on the jotai store, connects when healthy, reconnects with backoff, reconnects when the region's address changes, and tears down a socket left pointing at a departed region. | +| Subscribe sessions | `api/materialize/subscribeSession.ts` | Composition: one constructor wires the manager, the connection manager, the sink, the region reset, and cache hydration, in one defined order. | +| Hooks | `api/materialize/useSubscribe.ts` | Lifecycle adapters: a single effect creates the session on mount and destroys it on unmount. | +| Collection bridge | `api/materialize/subscribeCollection.ts` | `createSubscribeCollection`: turns manager snapshots into incremental insert/update/delete writes on a TanStack DB collection, owns the scoped localStorage cache, and mirrors status (error, snapshotComplete) into a companion atom. | +| Cache scope | `store/syncEngineCache.ts` | Derives the cache scope `organizationId\|regionId` as an atom, so hydration can be driven from the store. | + +The design rule behind the split: reactive plumbing between module-level +singletons belongs on the jotai store (`store.sub`), not in React effects. +Effects impose ordering between independently registered callbacks, re-run +under Suspense, and force every cross-cutting concern to be threaded into each +hook variant separately. A session wires everything once, synchronously, and +is testable without a DOM. + +## Subscribe sessions + +A session is a plain object created by a hook's single effect and destroyed on +unmount. Three kinds exist: + +- `createAtomSubscribeSession`: owns a socket, reduces into a jotai atom. + Holds existing atom data through a fresh connection's empty pre-snapshot, + and resets the manager and the atom to the loading state when the region + changes, so no page serves another region's catalog. +- `createCollectionSubscribeSession`: owns a socket, feeds a collection. + Additionally holds a keep-alive subscriber so the collection is not garbage + collected while no component queries it, clears the collection synchronously + on a region change, and hydrates it from its scoped cache via the scope atom. +- `createAtomFedCollectionSession`: no socket. Bridges an already running + subscribe atom into a collection, adding no upstream load, plus the same + region clear and scoped hydration. (The source atom's own reset reaches the + bridge as an empty pre-snapshot, which applySnapshot ignores, so the + collection must clear itself.) + +``` + +--------------------------------------+ + hook | subscribe session (plain object) | + one effect: | | +-----------+ + create / destroy --+-> SubscribeManager <---- diff stream +----- | WebSocket | + | WebsocketConnectionManager --------+----> +-----------+ + | reconnect at addr| + | sink: atom | collection -----------+----> atoms / collections + | | | + | store.sub(regionId) -> reset | v + | store.sub(cacheScope) -> hydrate | localStorage cache + +--------------------------------------+ (scoped persist) +``` + +## The collection bridge + +`createSubscribeCollection` is the seam between the feed and TanStack DB. The +manager already reduces the diff stream to the current row set, so the bridge +diffs each snapshot against what the collection holds and writes only the +delta as insert/update/delete operations. Live queries downstream therefore +recompute incrementally. + +The bridge also owns the instant-load cache: + +- **Persist.** With a `persistName`, complete snapshots (never partial state) + are written to localStorage on a trailing throttle, under the key + `mz-console:sync-engine:|||v`. +- **Hydrate.** Seeding is deferred until the scope is known, then the cached + rows load into the collection and the loading gate opens immediately. A + cached snapshot counts as complete for gating; the live snapshot replaces it + when it arrives. Other scopes' cache entries are pruned on hydrate. +- **Fail closed.** A scope resolution that errors suspends persistence + entirely rather than leaving writes aimed at the previous scope's key. +- **Scope changes.** Re-hydrating under a new scope drops the in-memory rows + and any pending persist first, so one tenant's rows are never shown as, or + written under, another tenant's cache. +- **Status.** Error and snapshotComplete mirror into a companion jotai atom so + consumers keep the same loading/error semantics as atom-backed data. + +An empty pre-snapshot from a fresh connection carries no data: the bridge +ignores it rather than clearing cache-seeded state or counting it as live +data, except that it clears a stale error so the UI falls back to loading +during reconnect backoff. + +## Data flow + +Steady state: the socket delivers the diff stream, the manager folds it and +emits a snapshot per closed timestamp, the sink writes it into the atom or +diffs it into the collection, and `useLiveQuery` consumers recompute +incrementally. + +On a region switch: + +1. The region atom is written by the region selector. +2. The connection manager sees the new address and reconnects the socket + there. Pausing on an unhealthy target tears the departed region's socket + down, so a later return to a healthy region always resumes. A health blip + within the current region leaves a working socket alone. +3. Each session's region subscription fires synchronously: the manager drops + its held rows and atoms go back to loading. +4. The cache scope atom recomputes. Collection sessions hydrate the new + scope's cache, dropping in-memory rows and any pending persist. +5. The new region's snapshot arrives and flows through the sinks as usual. + +On a reconnect within one region, the manager keeps its last snapshot and +re-emits it when the socket reopens, so the UI holds data through the +resubscribe instead of flashing empty. + +## Boundaries + +What the engine is not: + +- **Not a server-load lever by itself.** It is a client store. Within one tab + the upstream SUBSCRIBE count is unchanged by adopting it. Its sessions and + sinks are transport-agnostic, which is the seam a shared upstream + (cross-tab or cross-client dedup) can plug into without touching surfaces, + but the engine alone does not provide that. +- **Not for polled telemetry.** Expensive introspection queries + (utilization history, arrangement sizes) are react-query territory, with + polling intervals and its persistence tooling. The engine serves + catalog-shaped data that a SUBSCRIBE can maintain. +- **Not for page-scoped, request-driven subscribes.** Those use + `useSubscribe`/`useSubscribeManager`, which tie a manager to a component + and expose state via `useSyncExternalStore`. + +## Adopting a collection for a surface + +A surface that moves from an atom read to a collection gets, without further +code: incremental live queries, per-key re-render granularity, instant load +from the scoped cache, keep-alive for the app session, and region-correct +resets. The object explorer is the reference adoption: its tree runs live +queries over the objects collection (fed from the app-wide `allObjects` atom) +and the namespaces collection (its own subscribe), and renders from cache on +revisit before the live snapshot lands. + +Declare configuration once at module scope. The hooks require a referentially +stable options object, and a new object identity restarts the session, which +makes accidental instability visible instead of masked. + +Atom-backed data set: + +```ts +export const allWidgets = atom>({ + data: [], + error: undefined, + snapshotComplete: false, +}); + +const ALL_WIDGETS_SUBSCRIBE_OPTIONS = { + atom: allWidgets, + subscribe: buildSubscribeQuery(buildWidgetsQuery(), { upsertKey: "id" }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; + +export function useSubscribeToAllWidgets() { + useGlobalUpsertSubscribe(ALL_WIDGETS_SUBSCRIBE_OPTIONS); +} +``` + +Collection with its own subscribe and the instant-load cache: + +```ts +export const widgetsCollection = createSubscribeCollection({ + id: "widgets", + getKey: (widget) => widget.id, + persistName: "widgets", +}); + +const WIDGETS_SUBSCRIBE_OPTIONS = { + target: widgetsCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + subscribe: buildSubscribeQuery(buildWidgetsQuery(), { upsertKey: "id" }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; + +export function useSubscribeToWidgetsCollection() { + useGlobalSubscribeCollection(WIDGETS_SUBSCRIBE_OPTIONS); +} +``` + +Collection fed from an existing atom, no second socket: + +```ts +export const widgetsCollection = createSubscribeCollection({ + id: "widgets", + getKey: (widget) => widget.id, + persistName: "widgets", +}); + +export function useSubscribeToWidgetsCollection() { + const store = useStore(); + React.useEffect(() => { + const session = createAtomFedCollectionSession({ + store, + sourceAtom: allWidgets, + target: widgetsCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + }); + return session.destroy; + }, [store]); +} +``` + +Consumers read collections through `useLiveQuery`, with status from the +companion atom: + +```ts +const { data } = useLiveQuery((q) => q.from({ widgets: widgetsCollection.collection })); +const status = useAtomValue(widgetsCollection.statusAtom); +``` + +Mount the activator hook once: in `AppInitializer` for data the whole app +reads, or in a route component for data one surface reads. Note that a route +component keyed by a path pattern survives param-only navigation, including +region switches, which is why region handling lives in the session rather +than in mount/unmount. + +## Testing + +Sessions and the collection bridge are plain objects, so their behavior is +asserted headlessly and synchronously: create them against `getStore()`, +write the region or scope atoms or call `applySnapshot`, and assert on the +sink in the same tick. No rendering, no Suspense, no `waitFor`. See +`useSubscribe.test.tsx` and `subscribeCollection.test.ts`. jsdom-rendered +tests of this stack are unreliable (sockets fail and produce error states, +Suspense tears sessions down mid-switch), so prefer session-level tests plus +component tests that mock the activator hooks. diff --git a/console/package.json b/console/package.json index 0974b9a412806..d5992e98df7be 100644 --- a/console/package.json +++ b/console/package.json @@ -239,6 +239,17 @@ "prettier/prettier": "error", "simple-import-sort/imports": "error", "simple-import-sort/exports": "error", + "no-restricted-syntax": [ + "error", + { + "selector": "CallExpression[callee.name='useGlobalUpsertSubscribe'] > ObjectExpression", + "message": "Declare subscribe options at module scope; an inline object restarts the SUBSCRIBE session on every render." + }, + { + "selector": "CallExpression[callee.name='useGlobalSubscribeCollection'] > ObjectExpression", + "message": "Declare subscribe options at module scope; an inline object restarts the SUBSCRIBE session on every render." + } + ], "@typescript-eslint/no-shadow": "error", "@typescript-eslint/ban-ts-comment": "off", "@typescript-eslint/explicit-module-boundary-types": "off", diff --git a/console/src/api/auth.tsx b/console/src/api/auth.tsx index de32b3e02d1ea..c94ca8578b99d 100644 --- a/console/src/api/auth.tsx +++ b/console/src/api/auth.tsx @@ -14,6 +14,7 @@ import { useQuery } from "@tanstack/react-query"; +import { AppConfig } from "~/config/AppConfig"; import { CloudRuntimeConfig } from "~/config/AppConfigSwitch"; import { useAppConfig } from "~/config/useAppConfig"; import { @@ -170,6 +171,20 @@ export function useSelfManagedSubscription() { return { subscription, isLoading, isError, error }; } +/** + * Whether this deployment fetches the current organization at all. This also + * decides the organization half of the sync-engine cache key + * (store/syncEngineCache.ts), which isolates tenants' caches on a shared + * origin, so every caller must use this one predicate. + */ +export function isOrganizationFetchEnabled(appConfig: AppConfig): boolean { + const isLocalImpersonation = + appConfig.mode === "cloud" && + appConfig.isImpersonating && + appConfig.isLocalImpersonation; + return appConfig.mode !== "self-managed" && !isLocalImpersonation; +} + // Convenience hook for getting the current organization ID based on // the app's deployment mode. Although "organizations" don't really exist in // self managed environments, a lot of our code uses the organization ID as part of a key to cache @@ -177,13 +192,8 @@ export function useSelfManagedSubscription() { // in deployment modes where the organization ID is not available. export function useMaybeCurrentOrganizationId() { const appConfig = useAppConfig(); - const isLocalImpersonation = - appConfig.mode === "cloud" && - appConfig.isImpersonating && - appConfig.isLocalImpersonation; - const isCurrentOrganizationFetchEnabled = - appConfig.mode !== "self-managed" && !isLocalImpersonation; + isOrganizationFetchEnabled(appConfig); const res = useQuery({ queryKey: queryKeys.currentOrganization(), diff --git a/console/src/api/materialize/WebsocketConnectionManager.test.ts b/console/src/api/materialize/WebsocketConnectionManager.test.ts index f71bd63f37659..4fb59194b5775 100644 --- a/console/src/api/materialize/WebsocketConnectionManager.test.ts +++ b/console/src/api/materialize/WebsocketConnectionManager.test.ts @@ -12,9 +12,11 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { getStore } from "~/jotai"; import { + currentRegionIdAtom, EnvironmentsWithHealth, environmentsWithHealth, } from "~/store/environments"; +import { defaultRegionId } from "~/test/utils"; import { Connectable, @@ -234,6 +236,169 @@ describe("WebsocketConnectionManager", () => { }); }); + describe("region switch", () => { + const twoRegions = () => + new Map([ + [defaultRegionId, createHealthyEnvironment("addr-a:6876")], + ["aws/eu-west-1", createHealthyEnvironment("addr-b:6876")], + ]) as unknown as EnvironmentsWithHealth; + + afterEach(() => { + getStore().set(currentRegionIdAtom, defaultRegionId); + }); + + it("reconnects at the new address when the region switches while connected", () => { + const store = getStore(); + store.set(environmentsWithHealth, twoRegions()); + + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + + store.set(currentRegionIdAtom, "aws/eu-west-1"); + + expect(mockTarget.reconnect).toHaveBeenCalledTimes(1); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-b:6876", + undefined, + ); + }); + + it("reconnects after a round trip through an unhealthy region", () => { + const store = getStore(); + const crashed = { + ...createHealthyEnvironment("addr-b:6876"), + status: { health: "crashed" as const, version: "0.0.0", errors: [] }, + }; + store.set( + environmentsWithHealth, + new Map([ + [defaultRegionId, createHealthyEnvironment("addr-a:6876")], + ["aws/eu-west-1", crashed], + ]) as unknown as EnvironmentsWithHealth, + ); + + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + + // Pausing on the unhealthy region tears the old region's socket down. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(mockTarget.disconnect).toHaveBeenCalled(); + expect(mockTarget.reconnect).not.toHaveBeenCalled(); + + // Returning to the healthy region reconnects rather than assuming the + // old socket still serves it. + store.set(currentRegionIdAtom, defaultRegionId); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-a:6876", + undefined, + ); + }); + + it("keeps the socket through a same-region health blip", () => { + const store = getStore(); + store.set(environmentsWithHealth, twoRegions()); + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + vi.mocked(mockTarget.disconnect).mockClear(); + + // The current region's health check fails: the socket is left alone, + // so consumers like the Shell see no silent teardown. + store.set( + environmentsWithHealth, + new Map([ + [ + defaultRegionId, + { + ...createHealthyEnvironment("addr-a:6876"), + status: { + health: "crashed" as const, + version: "0.0.0", + errors: [], + }, + }, + ], + ["aws/eu-west-1", createHealthyEnvironment("addr-b:6876")], + ]) as unknown as EnvironmentsWithHealth, + ); + expect(mockTarget.disconnect).not.toHaveBeenCalled(); + + // Health recovers: the socket is still connected, so no churn either. + store.set(environmentsWithHealth, twoRegions()); + expect(mockTarget.reconnect).not.toHaveBeenCalled(); + }); + + it("reconnects after a round trip through a disabled region", () => { + const store = getStore(); + store.set( + environmentsWithHealth, + new Map([ + [defaultRegionId, createHealthyEnvironment("addr-a:6876")], + ["aws/eu-west-1", { state: "disabled" as const }], + ]) as unknown as EnvironmentsWithHealth, + ); + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + + // A disabled region never updates the address, so the teardown must key + // on the region, not the address. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(mockTarget.disconnect).toHaveBeenCalled(); + + store.set(currentRegionIdAtom, defaultRegionId); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-a:6876", + undefined, + ); + }); + + it("reconnects after a region switch that lands mid-handshake", () => { + const store = getStore(); + store.set(environmentsWithHealth, twoRegions()); + + // Constructor starts a connect to region A; handshake still in flight. + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-a:6876", + undefined, + ); + vi.mocked(mockTarget.reconnect).mockClear(); + + // Switch regions before the handshake completes: deferred, not dropped. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(mockTarget.reconnect).not.toHaveBeenCalled(); + + mockTarget.simulateOpen(); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-b:6876", + undefined, + ); + }); + }); + describe("destroy", () => { it("disconnects and cleans up subscriptions", () => { const store = getStore(); diff --git a/console/src/api/materialize/WebsocketConnectionManager.ts b/console/src/api/materialize/WebsocketConnectionManager.ts index e6884a62a46cd..f766224246d76 100644 --- a/console/src/api/materialize/WebsocketConnectionManager.ts +++ b/console/src/api/materialize/WebsocketConnectionManager.ts @@ -77,6 +77,10 @@ export class WebsocketConnectionManager { private isHealthy = false; private currentHttpAddress?: string; + /** Address used for the most recent connection attempt. */ + private attemptedHttpAddress?: string; + /** Region the most recent connection attempt was made for. */ + private attemptedRegionId?: string | null; private retryAttempt = 0; private hasEverConnected = false; private retryTimer: ReturnType | undefined; @@ -158,11 +162,20 @@ export class WebsocketConnectionManager { } if (nowHealthy) { - if (!this.target.isConnected()) { + // A connected socket may still stream from a previous region's address + // after a region switch; reconnect at the current one. + if ( + !this.target.isConnected() || + this.attemptedHttpAddress !== this.currentHttpAddress + ) { this.resumeConnection(); } } else { - this.pauseConnection(); + // Only tear down a socket pointing at a region the user has left; a + // health blip in the current region leaves a working socket alone + // (consumers like the Shell would get no close callback). + const regionId = this.store.get(currentRegionIdSyncAtom); + this.pauseConnection(regionId !== this.attemptedRegionId); } }; @@ -205,6 +218,14 @@ export class WebsocketConnectionManager { this.retryAttempt = 0; this.clearRetryTimer(); this.notifyStateChange(); + // The region may have switched while this handshake was in flight; the + // socket that just opened points at the old address. + if ( + this.isHealthy && + this.attemptedHttpAddress !== this.currentHttpAddress + ) { + this.attemptConnection(); + } }; /** Tear down and reopen the socket. Used on SQL request changes. */ @@ -239,6 +260,8 @@ export class WebsocketConnectionManager { hasEverConnected: this.hasEverConnected, }); this.connectInFlight = true; + this.attemptedHttpAddress = this.currentHttpAddress; + this.attemptedRegionId = this.store.get(currentRegionIdSyncAtom); try { this.target.reconnect(this.currentHttpAddress, sessionVariables); } catch { @@ -253,9 +276,15 @@ export class WebsocketConnectionManager { this.attemptConnection(); } - private pauseConnection() { + private pauseConnection(disconnect: boolean) { this.clearRetryTimer(); this.retryAttempt = 0; + if (disconnect) { + // Tear down so the resume path reconnects after an unhealthy round trip; + // clear connectInFlight since this socket gets no open/close callback. + this.connectInFlight = false; + this.target.disconnect(); + } this.notifyStateChange(); } diff --git a/console/src/api/materialize/subscribeCollection.test.ts b/console/src/api/materialize/subscribeCollection.test.ts index 8020b3ec04252..0372c7894146d 100644 --- a/console/src/api/materialize/subscribeCollection.test.ts +++ b/console/src/api/materialize/subscribeCollection.test.ts @@ -105,9 +105,8 @@ describe("createSubscribeCollection", () => { }); it("hydrates from cache after an empty pre-snapshot was applied", async () => { - // The mount effect pushes the atom's empty initial state before the async - // auth/region scope resolves, so hydrate always runs after it. The empty - // placeholder must not count as live data, or the cache is never read. + // The empty initial state is pushed before the async scope resolves, so + // hydrate always runs after it; it must not count as live data. const name = "late-hydrate"; const scope = "org1|region1"; localStorage.setItem( @@ -143,6 +142,121 @@ describe("createSubscribeCollection", () => { expect(collection.size).toBe(1); }); + it("does not carry the previous scope's rows into a new scope", async () => { + const name = "scope-switch"; + const keyB = syncEngineCacheKey(name, "org|regionB"); + const { collection, statusAtom, applySnapshot, hydrate } = + freshCollection(name); + + hydrate("org|regionA"); + applySnapshot(state([{ id: "1", name: "region-a" }])); + await flush(); + expect(collection.size).toBe(1); + + vi.useFakeTimers(); + try { + // A region switch re-hydrates with a new scope while region A's rows are + // still in memory. They must not be persisted under region B's key. + hydrate("org|regionB"); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(keyB)).toBeNull(); + } finally { + vi.useRealTimers(); + } + + // Nor served as region B data: the collection empties and the loading + // gate closes until region B's own snapshot arrives. + await flush(); + expect(collection.size).toBe(0); + expect(getStore().get(statusAtom).snapshotComplete).toBe(false); + }); + + it("seeds the new scope from its own cache after a scope switch", async () => { + const name = "scope-switch-seed"; + const keyB = syncEngineCacheKey(name, "org|regionB"); + const { collection, applySnapshot, hydrate } = freshCollection(name); + + hydrate("org|regionA"); + applySnapshot(state([{ id: "1", name: "region-a" }])); + await flush(); + + // Written after region A's hydrate pruned other scopes, as another tab + // already on region B would. + localStorage.setItem(keyB, JSON.stringify([{ id: "9", name: "region-b" }])); + hydrate("org|regionB"); + await flush(); + + expect(collection.has("1")).toBe(false); + expect(collection.get("9")?.name).toBe("region-b"); + }); + + it("reset drops rows, pending persistence, and the loading gate", async () => { + vi.useFakeTimers(); + try { + const name = "reset"; + const scope = "org|regionA"; + const key = syncEngineCacheKey(name, scope); + const { collection, statusAtom, applySnapshot, hydrate, reset } = + freshCollection(name); + hydrate(scope); + applySnapshot(state([{ id: "1", name: "a" }])); + + // Reset before the persist throttle fires: nothing may be written. + reset(); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(key)).toBeNull(); + expect(getStore().get(statusAtom).snapshotComplete).toBe(false); + + vi.useRealTimers(); + await flush(); + expect(collection.size).toBe(0); + } finally { + vi.useRealTimers(); + } + }); + + it("suspendPersistence stops writes until a scope resolves again", () => { + vi.useFakeTimers(); + try { + const name = "suspend"; + const scope = "org|regionA"; + const key = syncEngineCacheKey(name, scope); + const { applySnapshot, hydrate, suspendPersistence } = + freshCollection(name); + hydrate(scope); + applySnapshot(state([{ id: "1", name: "a" }])); + + // Suspend cancels the pending write and later snapshots stay unpersisted. + suspendPersistence(); + applySnapshot(state([{ id: "2", name: "b" }])); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(key)).toBeNull(); + + // A scope resolving again re-arms persistence, even the same scope. + hydrate(scope); + applySnapshot(state([{ id: "3", name: "c" }])); + vi.advanceTimersByTime(1100); + expect(JSON.parse(localStorage.getItem(key) ?? "[]")).toEqual([ + { id: "3", name: "c" }, + ]); + } finally { + vi.useRealTimers(); + } + }); + + it("clears a stale error when an empty pre-snapshot follows a reconnect", () => { + const { applySnapshot, statusAtom } = freshCollection(); + const error = { code: "boom", message: "it failed" }; + applySnapshot(state([], false, error)); + expect(getStore().get(statusAtom).error).toEqual(error); + + // SubscribeManager re-emits an empty pre-snapshot when a reconnect opens; + // the status must fall back to loading, not hold the error. + applySnapshot(state([], false)); + expect(getStore().get(statusAtom).error).toBeUndefined(); + expect(getStore().get(statusAtom).snapshotComplete).toBe(false); + }); + it("surfaces error and snapshotComplete on the status atom", () => { const { applySnapshot, statusAtom } = freshCollection(); diff --git a/console/src/api/materialize/subscribeCollection.ts b/console/src/api/materialize/subscribeCollection.ts index f234f292e2303..c8c79691cfe24 100644 --- a/console/src/api/materialize/subscribeCollection.ts +++ b/console/src/api/materialize/subscribeCollection.ts @@ -72,6 +72,14 @@ export interface SubscribeCollection { * tenant's catalog into another's session after an org or region switch. Only * the first call per scope seeds; later calls for the same scope no-op. */ hydrate: (scope: string) => void; + /** Drop all rows and pending persistence and return to the loading state, + * for when the held rows belong to another region's catalog. A later + * hydrate may still seed the new scope from its cache. */ + reset: () => void; + /** Stop persisting until a scope resolves again. Scope resolution failing + * must fail closed: writes aimed at a previous scope's key would store one + * tenant's rows under another's cache. */ + suspendPersistence: () => void; } /** @@ -213,14 +221,15 @@ export function createSubscribeCollection(options: { }; const applySnapshot = (state: SubscribeState) => { - // An empty pre-snapshot carries no information: ignore it entirely, so it - // neither clears cache-seeded state nor counts as live data. Marking it - // live would block a later hydrate (the scope resolves asynchronously, so - // hydrate always runs after the first empty push) and leave the cache - // permanently unread. + // An empty pre-snapshot carries no data: don't clear cache-seeded state or + // count it as live (that would block the later, async-scoped hydrate). It + // does clear a stale error so a reconnect falls back to loading. const isEmptyPreload = !state.snapshotComplete && state.data.length === 0 && !state.error; - if (isEmptyPreload) return; + if (isEmptyPreload) { + writeStatus({ error: undefined, snapshotComplete: lastSnapshotComplete }); + return; + } liveSnapshotApplied = true; desired = new Map(state.data.map((row) => [getKey(row), row])); @@ -254,9 +263,21 @@ export function createSubscribeCollection(options: { const hydrate = (scope: string) => { if (!persistName || hydratedScope === scope) return; + const scopeChanged = hydratedScope !== undefined; hydratedScope = scope; persistKey = syncEngineCacheKey(persistName, scope); pruneOtherScopes(persistKey); + if (scopeChanged) { + // The rows in memory belong to the previous scope: drop them and the + // pending persist (its timer reads `desired` at fire time). + if (persistTimer) { + clearTimeout(persistTimer); + persistTimer = undefined; + } + desired = new Map(); + liveSnapshotApplied = false; + lastSnapshotComplete = false; + } if (liveSnapshotApplied) { // Live data already arrived; don't overwrite it with the cache, but do // persist it now that the scoped key is known. @@ -264,14 +285,50 @@ export function createSubscribeCollection(options: { return; } const cached = readPersistedRows(persistKey); - if (!cached.length) return; - desired = new Map(cached.map((row) => [getKey(row), row])); - // A cached full snapshot counts as complete for the loading gate; the live - // snapshot refreshes it once it arrives. - lastSnapshotComplete = true; + if (cached.length) { + desired = new Map(cached.map((row) => [getKey(row), row])); + // A cached full snapshot counts as complete for the loading gate; the + // live snapshot refreshes it once it arrives. + lastSnapshotComplete = true; + } else if (!scopeChanged) { + return; + } + // On a scope change this also flushes the emptied set, so the collection + // stops serving the previous scope's rows while the new snapshot loads. + materialize(); + writeStatus({ error: undefined, snapshotComplete: lastSnapshotComplete }); + }; + + const clearPendingPersist = () => { + if (persistTimer) { + clearTimeout(persistTimer); + persistTimer = undefined; + } + }; + + const reset = () => { + clearPendingPersist(); + desired = new Map(); + lastSnapshotComplete = false; + liveSnapshotApplied = false; materialize(); - writeStatus({ error: undefined, snapshotComplete: true }); + writeStatus({ error: undefined, snapshotComplete: false }); + }; + + const suspendPersistence = () => { + clearPendingPersist(); + persistKey = undefined; + // Allow a later successful resolution to hydrate again, even for the + // scope whose resolution just failed. + hydratedScope = undefined; }; - return { collection, statusAtom, applySnapshot, hydrate }; + return { + collection, + statusAtom, + applySnapshot, + hydrate, + reset, + suspendPersistence, + }; } diff --git a/console/src/api/materialize/subscribeSession.ts b/console/src/api/materialize/subscribeSession.ts new file mode 100644 index 0000000000000..6481746a2ebc6 --- /dev/null +++ b/console/src/api/materialize/subscribeSession.ts @@ -0,0 +1,222 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { Atom, createStore, PrimitiveAtom } from "jotai"; + +import { reconnectionStateAtom } from "~/hooks/useAutomaticallyConnectSocket"; +import { currentRegionIdSyncAtom } from "~/store/environments"; + +import { SubscribeCollection } from "./subscribeCollection"; +import { + SelectFunction, + SubscribeManager, + SubscribeRow, + SubscribeState, + UpsertKeyFunction, +} from "./SubscribeManager"; +import { SessionVariables, SqlRequest } from "./types"; +import { WebsocketConnectionManager } from "./WebsocketConnectionManager"; + +type JotaiStore = ReturnType; + +/** The shape jotai's `loadable()` wraps an async atom into. */ +export type ScopeLoadableAtom = Atom< + | { state: "loading" } + | { state: "hasError"; error: unknown } + | { state: "hasData"; data: string | undefined } +>; + +/** Hydrates `target` from its scoped cache whenever the scope settles or + * changes; hydrate itself handles both seeding and the scope-change reset. A + * failed scope resolution fails closed: persistence is suspended rather than + * left aimed at a previous scope's key. */ +function subscribeToScope( + store: JotaiStore, + scopeAtom: ScopeLoadableAtom, + target: { + hydrate: (scope: string) => void; + suspendPersistence: () => void; + }, +) { + const applyScope = () => { + const value = store.get(scopeAtom); + if (value.state === "hasData" && value.data) target.hydrate(value.data); + else if (value.state === "hasError") target.suspendPersistence(); + }; + applyScope(); + return store.sub(scopeAtom, applyScope); +} + +/** Invokes `onRegionChange` when the current region changes. */ +function subscribeToRegionChange( + store: JotaiStore, + onRegionChange: () => void, +) { + let regionId = store.get(currentRegionIdSyncAtom); + return store.sub(currentRegionIdSyncAtom, () => { + const next = store.get(currentRegionIdSyncAtom); + if (next === regionId) return; + regionId = next; + onRegionChange(); + }); +} + +export interface SubscribeSessionOptions { + store: JotaiStore; + request: SqlRequest; + upsertKey: UpsertKeyFunction; + select?: SelectFunction; + sessionVariables?: SessionVariables; + closeSocketOnComplete?: boolean; +} + +export interface SubscribeSession { + manager: SubscribeManager; + destroy: () => void; +} + +/** + * One app-session subscribe, wired outside React: a SubscribeManager kept + * connected by a WebsocketConnectionManager, snapshots pushed into `sink`, and + * a region-change reset subscribed directly on the store. Hooks only own the + * session's lifecycle; the composition is testable headlessly. + */ +function createSubscribeSession( + options: SubscribeSessionOptions, + sink: { + apply: (snapshot: SubscribeState) => void; + onRegionChange?: () => void; + }, +): SubscribeSession { + const { store, request, upsertKey, select } = options; + const manager = new SubscribeManager({ + request, + // The connection manager supplies the address on every attempt. + httpAddress: "", + upsert: { key: upsertKey }, + select, + sessionVariables: options.sessionVariables, + closeSocketOnComplete: options.closeSocketOnComplete, + }); + const unsubscribeChange = manager.onChange(() => + sink.apply(manager.getSnapshot()), + ); + const connection = new WebsocketConnectionManager( + manager, + store, + reconnectionStateAtom, + ); + + // The held rows belong to the previous region's catalog: reset rather than + // serve or replay them while the socket re-subscribes at the new address. + const unsubscribeRegion = subscribeToRegionChange(store, () => { + manager.reset(); + sink.onRegionChange?.(); + }); + + return { + manager, + destroy() { + unsubscribeRegion(); + unsubscribeChange(); + connection.destroy(); + }, + }; +} + +/** A subscribe session that reduces into a jotai atom (allObjects et al.). */ +export function createAtomSubscribeSession< + T extends object, + R = SubscribeRow, +>( + options: SubscribeSessionOptions & { + atom: PrimitiveAtom>; + }, +): SubscribeSession { + const { store, atom } = options; + return createSubscribeSession(options, { + apply: (snapshot) => { + const current = store.get(atom); + if (current === snapshot) return; + // Hold cached atom data through a fresh manager's empty pre-snapshot state. + const snapshotIsEmptyPreload = + !snapshot.snapshotComplete && !snapshot.data.length && !snapshot.error; + if (snapshotIsEmptyPreload && current.data.length) return; + store.set(atom, snapshot); + }, + onRegionChange: () => + store.set(atom, { data: [], snapshotComplete: false, error: undefined }), + }); +} + +/** A subscribe session that feeds a TanStack DB collection (namespaces et al.). + * The region change clears the collection synchronously; hydrate is + * responsible only for cache seeding. */ +export function createCollectionSubscribeSession< + T extends object, + R extends object = SubscribeRow, +>( + options: SubscribeSessionOptions & { + target: SubscribeCollection; + scopeAtom?: ScopeLoadableAtom; + }, +): SubscribeSession { + const { store, target, scopeAtom } = options; + // Hold a subscriber for the session so the collection doesn't pause/GC + // while no component is querying it. + const keepAlive = target.collection.subscribeChanges(() => {}); + const unsubscribeScope = scopeAtom + ? subscribeToScope(store, scopeAtom, target) + : undefined; + const session = createSubscribeSession(options, { + apply: (snapshot) => target.applySnapshot(snapshot), + onRegionChange: () => target.reset(), + }); + target.applySnapshot(session.manager.getSnapshot()); + return { + manager: session.manager, + destroy() { + unsubscribeScope?.(); + keepAlive.unsubscribe(); + session.destroy(); + }, + }; +} + +/** + * Feeds a collection from an already-running subscribe atom rather than its + * own socket, adding no upstream load. The source atom's own session resets + * the atom on a region change, but that reaches this bridge as an empty + * pre-snapshot, which applySnapshot ignores, so the collection clears itself + * on the region change too. + */ +export function createAtomFedCollectionSession(options: { + store: JotaiStore; + sourceAtom: Atom>; + target: SubscribeCollection; + scopeAtom?: ScopeLoadableAtom; +}): { destroy: () => void } { + const { store, sourceAtom, target, scopeAtom } = options; + const unsubscribeScope = scopeAtom + ? subscribeToScope(store, scopeAtom, target) + : undefined; + const unsubscribeRegion = subscribeToRegionChange(store, () => + target.reset(), + ); + const apply = () => target.applySnapshot(store.get(sourceAtom)); + apply(); + const unsubscribeSource = store.sub(sourceAtom, apply); + return { + destroy() { + unsubscribeScope?.(); + unsubscribeRegion(); + unsubscribeSource(); + }, + }; +} diff --git a/console/src/api/materialize/useSubscribe.test.tsx b/console/src/api/materialize/useSubscribe.test.tsx new file mode 100644 index 0000000000000..cbd432a011e02 --- /dev/null +++ b/console/src/api/materialize/useSubscribe.test.tsx @@ -0,0 +1,237 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { atom } from "jotai"; +import { ws } from "msw"; + +import server from "~/api/mocks/server"; +import { getStore } from "~/jotai"; +import { + currentEnvironmentState, + currentRegionIdAtom, + EnvironmentsWithHealth, + environmentsWithHealth, +} from "~/store/environments"; +import { defaultRegionId, healthyEnvironment } from "~/test/utils"; + +import { + createSubscribeCollection, + syncEngineCacheKey, +} from "./subscribeCollection"; +import { SubscribeState } from "./SubscribeManager"; +import { + createAtomFedCollectionSession, + createAtomSubscribeSession, + createCollectionSubscribeSession, +} from "./subscribeSession"; + +// Accept subscribe sockets and stay silent, so connections neither error nor +// deliver data: the only thing that may then change state is the session. +const subscribeSocket = ws.link("ws://*"); + +type Row = { id: string }; + +const request = { queries: [{ query: "SELECT 1", params: [] }] }; +const upsertKey = (row: { data: Row }) => row.data.id; + +/** Let the collection's async sync writes settle before asserting. */ +const flush = () => new Promise((resolve) => setTimeout(resolve, 0)); + +const loadingState: SubscribeState = { + data: [], + snapshotComplete: false, + error: undefined, +}; + +function setTwoRegions() { + const store = getStore(); + store.set( + environmentsWithHealth, + new Map([ + [defaultRegionId, { ...healthyEnvironment, httpAddress: "addr-a:6876" }], + ["aws/eu-west-1", { ...healthyEnvironment, httpAddress: "addr-b:6876" }], + ]) as unknown as EnvironmentsWithHealth, + ); + return store; +} + +describe("subscribeSession", () => { + beforeEach(() => { + server.use(subscribeSocket.addEventListener("connection", () => {})); + }); + + afterEach(async () => { + const store = getStore(); + store.set(currentRegionIdAtom, defaultRegionId); + await store.get(currentEnvironmentState); + }); + + it("resets the atom to loading only when the region actually changes", () => { + const store = setTwoRegions(); + const testAtom = atom>(loadingState); + const session = createAtomSubscribeSession({ + store, + request, + upsertKey, + select: (row) => row.data, + atom: testAtom, + }); + try { + // A completed snapshot from the previous region is in the atom. + store.set(testAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + + // Re-setting the same region is not a change. + store.set(currentRegionIdAtom, defaultRegionId); + expect(store.get(testAtom).data).toEqual([{ id: "1" }]); + + // The previous region's rows must not be served as the new region's data. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(store.get(testAtom).data).toEqual([]); + expect(store.get(testAtom).snapshotComplete).toBe(false); + } finally { + session.destroy(); + } + }); + + it("stops reacting to region changes after destroy", () => { + const store = setTwoRegions(); + const testAtom = atom>(loadingState); + const session = createAtomSubscribeSession({ + store, + request, + upsertKey, + select: (row) => row.data, + atom: testAtom, + }); + session.destroy(); + + store.set(testAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(store.get(testAtom).data).toEqual([{ id: "1" }]); + }); + + it("clears the collection when the region switches", async () => { + const store = setTwoRegions(); + const target = createSubscribeCollection({ + id: "session-test-collection", + getKey: (row) => row.id, + }); + const session = createCollectionSubscribeSession({ + store, + request, + upsertKey, + select: (row) => row.data, + target, + }); + try { + // A completed snapshot from the previous region is in the collection. + target.applySnapshot({ + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + await flush(); + expect(target.collection.size).toBe(1); + + // The rows must be gone and the loading gate closed, not merely a + // manager reset invoked: that reset reaches applySnapshot as an empty + // pre-snapshot, which it ignores by design. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + await flush(); + expect(target.collection.size).toBe(0); + expect(store.get(target.statusAtom).snapshotComplete).toBe(false); + } finally { + session.destroy(); + } + }); + + it("clears an atom-fed collection when the region switches", async () => { + const store = setTwoRegions(); + const sourceAtom = atom>(loadingState); + const target = createSubscribeCollection({ + id: "session-test-atom-fed", + getKey: (row) => row.id, + }); + const session = createAtomFedCollectionSession({ + store, + sourceAtom, + target, + }); + try { + store.set(sourceAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + await flush(); + expect(target.collection.size).toBe(1); + + store.set(currentRegionIdAtom, "aws/eu-west-1"); + await flush(); + expect(target.collection.size).toBe(0); + expect(store.get(target.statusAtom).snapshotComplete).toBe(false); + } finally { + session.destroy(); + } + }); + + it("suspends persistence when the scope fails to resolve", () => { + vi.useFakeTimers(); + const store = setTwoRegions(); + const scopeAtom = atom< + | { state: "loading" } + | { state: "hasError"; error: unknown } + | { state: "hasData"; data: string | undefined } + >({ state: "loading" }); + const sourceAtom = atom>(loadingState); + const target = createSubscribeCollection({ + id: "session-test-scope-error", + getKey: (row) => row.id, + persistName: "session-scope-error", + }); + const session = createAtomFedCollectionSession({ + store, + sourceAtom, + target, + scopeAtom, + }); + const key = syncEngineCacheKey("session-scope-error", "org|regionA"); + try { + store.set(scopeAtom, { state: "hasData", data: "org|regionA" }); + store.set(sourceAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + + // Scope resolution fails before the persist throttle fires: nothing may + // be written under the last known scope's key. + store.set(scopeAtom, { state: "hasError", error: new Error("boom") }); + store.set(sourceAtom, { + data: [{ id: "2" }], + snapshotComplete: true, + error: undefined, + }); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(key)).toBeNull(); + } finally { + vi.useRealTimers(); + session.destroy(); + localStorage.clear(); + } + }); +}); diff --git a/console/src/api/materialize/useSubscribe.ts b/console/src/api/materialize/useSubscribe.ts index c79212b2fd199..50f87b319d92f 100644 --- a/console/src/api/materialize/useSubscribe.ts +++ b/console/src/api/materialize/useSubscribe.ts @@ -7,12 +7,11 @@ // the Business Source License, use of this software will be governed // by the Apache License, Version 2.0. -import { PrimitiveAtom, useSetAtom } from "jotai"; +import { PrimitiveAtom, useStore } from "jotai"; import { RawBuilder } from "kysely"; import React from "react"; import { useAutomaticallyConnectSocket } from "~/hooks/useAutomaticallyConnectSocket"; -import { getStore } from "~/jotai"; import { useCurrentEnvironmentHttpAddress } from "~/store/environments"; import { queryBuilder } from "./db"; @@ -25,6 +24,11 @@ import { SubscribeState, UpsertKeyFunction, } from "./SubscribeManager"; +import { + createAtomSubscribeSession, + createCollectionSubscribeSession, + ScopeLoadableAtom, +} from "./subscribeSession"; export { buildSubscribeQuery } from "~/api/materialize/buildSubscribeQuery"; @@ -102,14 +106,14 @@ export function useSubscribe>( } /** - * Executes a subscribe query, storing state in the provided atom. - * All updates are reduced to the current set of values at latest closed timestamp. + * Runs an app-session subscribe that reduces into the provided atom, for as + * long as the calling component is mounted. All wiring (connection lifecycle, + * atom writes, region-change reset) lives in createAtomSubscribeSession. * - * Note that the subscribe statement must have WITH (PROGRESS) and ENVELOPE UPSERT. + * `options` must be referentially stable (declare it at module scope); a new + * object restarts the subscribe session. * - * The `select` and `upsertKey` functions are not expected to be stable, and new - * function instances for these options will not restart the subscribe. On each render, - * the socket will update the function reference. + * Note that the subscribe statement must have WITH (PROGRESS) and ENVELOPE UPSERT. */ export function useGlobalUpsertSubscribe>( options: UseSubscribeOptions & { @@ -117,54 +121,30 @@ export function useGlobalUpsertSubscribe>( atom: PrimitiveAtom>; }, ) { - const setValue = useSetAtom(options.atom); - const httpAddress = useCurrentEnvironmentHttpAddress(); + const store = useStore(); const request = useSubscribeRequest(options.subscribe); - const [subscribe] = React.useState( - new SubscribeManager({ + React.useEffect(() => { + if (!request) return; + const session = createAtomSubscribeSession({ + store, request, - httpAddress, - upsert: { - key: options.upsertKey, - }, - sessionVariables: { - cluster: options?.clusterName, - }, - closeSocketOnComplete: options?.closeSocketOnComplete, + atom: options.atom, + upsertKey: options.upsertKey, select: options.select, - }), - ); - useAutomaticallyConnectSocket({ - target: subscribe, - subscribe, - request, - }); - - React.useEffect(() => { - const cleanup = subscribe.onChange(() => { - const snapshot = subscribe.getSnapshot(); - const current = getStore().get(options.atom); - if (current === snapshot) return; - - // Hold cached atom data through a fresh manager's empty pre-snapshot state. - const snapshotIsEmptyPreload = - !snapshot.snapshotComplete && !snapshot.data.length && !snapshot.error; - if (snapshotIsEmptyPreload && current.data.length) return; - - setValue(snapshot); + sessionVariables: { cluster: options.clusterName }, + closeSocketOnComplete: options.closeSocketOnComplete, }); - return cleanup; - }, [options.atom, setValue, subscribe]); - - return { - subscribe, - }; + return session.destroy; + }, [store, request, options]); } /** - * Executes a subscribe query and feeds the upsert-reduced state into a TanStack DB - * collection (via createSubscribeCollection). Mirrors useGlobalUpsertSubscribe but - * targets a collection + status atom instead of a single SubscribeState atom. + * Runs an app-session subscribe that feeds a TanStack DB collection, for as + * long as the calling component is mounted. Mirrors useGlobalUpsertSubscribe + * with createCollectionSubscribeSession owning the wiring. + * + * `options` must be referentially stable (declare it at module scope); a new + * object restarts the subscribe session. * * Note that the subscribe statement must have WITH (PROGRESS) and ENVELOPE UPSERT. */ @@ -175,50 +155,25 @@ export function useGlobalSubscribeCollection< options: UseSubscribeOptions & { upsertKey: UpsertKeyFunction; target: SubscribeCollection; + scopeAtom?: ScopeLoadableAtom; }, ) { - const httpAddress = useCurrentEnvironmentHttpAddress(); + const store = useStore(); const request = useSubscribeRequest(options.subscribe); - const target = options.target; - const [subscribe] = React.useState( - new SubscribeManager({ + React.useEffect(() => { + if (!request) return; + const session = createCollectionSubscribeSession({ + store, request, - httpAddress, - upsert: { - key: options.upsertKey, - }, - sessionVariables: { - cluster: options?.clusterName, - }, - closeSocketOnComplete: options?.closeSocketOnComplete, + target: options.target, + scopeAtom: options.scopeAtom, + upsertKey: options.upsertKey, select: options.select, - }), - ); - useAutomaticallyConnectSocket({ - target: subscribe, - subscribe, - request, - }); - - React.useEffect(() => { - // applySnapshot itself drops empty pre-snapshots when state already exists, - // so the manager can push every snapshot unconditionally. - const apply = () => target.applySnapshot(subscribe.getSnapshot()); - const cleanup = subscribe.onChange(apply); - apply(); - return cleanup; - }, [target, subscribe]); - - // Keep the collection actively syncing for the app session by holding a - // subscriber, so it doesn't pause/GC when no component is querying it. - React.useEffect(() => { - const subscription = target.collection.subscribeChanges(() => {}); - return () => subscription.unsubscribe(); - }, [target]); - - return { - subscribe, - }; + sessionVariables: { cluster: options.clusterName }, + closeSocketOnComplete: options.closeSocketOnComplete, + }); + return session.destroy; + }, [store, request, options]); } /** diff --git a/console/src/platform/object-explorer/allNamespacesCollection.ts b/console/src/platform/object-explorer/allNamespacesCollection.ts index c980e80839668..c91590620823c 100644 --- a/console/src/platform/object-explorer/allNamespacesCollection.ts +++ b/console/src/platform/object-explorer/allNamespacesCollection.ts @@ -9,7 +9,7 @@ import { useLiveQuery } from "@tanstack/react-db"; import { useAtomValue } from "jotai"; -import { useEffect, useMemo } from "react"; +import { useMemo } from "react"; import { buildSubscribeQuery } from "~/api/materialize/buildSubscribeQuery"; import { @@ -19,7 +19,7 @@ import { import { createSubscribeCollection } from "~/api/materialize/subscribeCollection"; import { SubscribeRow } from "~/api/materialize/SubscribeManager"; import { useGlobalSubscribeCollection } from "~/api/materialize/useSubscribe"; -import { useSyncEngineCacheScope } from "~/store/syncEngineCache"; +import { syncEngineCacheScopeLoadableAtom } from "~/store/syncEngineCache"; /** Subset of AllNamespaceItem stored per row, matching the existing atom's select. */ export type NamespaceItem = Pick< @@ -39,29 +39,23 @@ export const allNamespacesCollection = createSubscribeCollection( }, ); -export function useSubscribeToAllNamespacesCollection() { - const scope = useSyncEngineCacheScope(); - useEffect(() => { - if (scope) allNamespacesCollection.hydrate(scope); - }, [scope]); - - const subscribe = useMemo(() => { - return buildSubscribeQuery(buildAllNamespacesQuery(), { - upsertKey: ["schemaId", "databaseId"], - }); - }, []); +const ALL_NAMESPACES_SUBSCRIBE_OPTIONS = { + target: allNamespacesCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + subscribe: buildSubscribeQuery(buildAllNamespacesQuery(), { + upsertKey: ["schemaId", "databaseId"], + }), + select: (row: SubscribeRow): NamespaceItem => ({ + schemaId: row.data.schemaId, + schemaName: row.data.schemaName, + databaseId: row.data.databaseId, + databaseName: row.data.databaseName, + }), + upsertKey: (row: SubscribeRow) => namespaceKey(row.data), +}; - return useGlobalSubscribeCollection({ - target: allNamespacesCollection, - subscribe, - select: (row: SubscribeRow) => ({ - schemaId: row.data.schemaId, - schemaName: row.data.schemaName, - databaseId: row.data.databaseId, - databaseName: row.data.databaseName, - }), - upsertKey: (row: SubscribeRow) => namespaceKey(row.data), - }); +export function useSubscribeToAllNamespacesCollection() { + useGlobalSubscribeCollection(ALL_NAMESPACES_SUBSCRIBE_OPTIONS); } /** Returns the namespace items for the object explorer tree, from the collection. */ diff --git a/console/src/store/allClusters.ts b/console/src/store/allClusters.ts index ec84fbcfa2f0f..d300aebd380d4 100644 --- a/console/src/store/allClusters.ts +++ b/console/src/store/allClusters.ts @@ -14,7 +14,10 @@ import { buildClustersQuery, Cluster, } from "~/api/materialize/cluster/clusterList"; -import { SubscribeState } from "~/api/materialize/SubscribeManager"; +import { + SubscribeRow, + SubscribeState, +} from "~/api/materialize/SubscribeManager"; import { buildSubscribeQuery, useGlobalUpsertSubscribe, @@ -26,22 +29,18 @@ export const allClusters = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllClusters() { - const subscribe = React.useMemo(() => { - return buildSubscribeQuery( - buildClustersQuery({ queryOwnership: false, includeSystemObjects: true }), - { - upsertKey: "id", - }, - ); - }, []); +const ALL_CLUSTERS_SUBSCRIBE_OPTIONS = { + atom: allClusters, + subscribe: buildSubscribeQuery( + buildClustersQuery({ queryOwnership: false, includeSystemObjects: true }), + { upsertKey: "id" }, + ), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; - return useGlobalUpsertSubscribe({ - atom: allClusters, - subscribe, - select: (row) => row.data, - upsertKey: (row) => row.data.id, - }); +export function useSubscribeToAllClusters() { + useGlobalUpsertSubscribe(ALL_CLUSTERS_SUBSCRIBE_OPTIONS); } export function useAllClusters() { diff --git a/console/src/store/allObjects.ts b/console/src/store/allObjects.ts index 549e56365a804..6ce6e0a6d66ce 100644 --- a/console/src/store/allObjects.ts +++ b/console/src/store/allObjects.ts @@ -14,7 +14,10 @@ import { buildAllObjectsQuery, DatabaseObject, } from "~/api/materialize/objects"; -import { SubscribeState } from "~/api/materialize/SubscribeManager"; +import { + SubscribeRow, + SubscribeState, +} from "~/api/materialize/SubscribeManager"; import { buildSubscribeQuery, useGlobalUpsertSubscribe, @@ -26,17 +29,15 @@ export const allObjects = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllObjects() { - const subscribe = React.useMemo(() => { - return buildSubscribeQuery(buildAllObjectsQuery(), { upsertKey: "id" }); - }, []); +const ALL_OBJECTS_SUBSCRIBE_OPTIONS = { + atom: allObjects, + subscribe: buildSubscribeQuery(buildAllObjectsQuery(), { upsertKey: "id" }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; - return useGlobalUpsertSubscribe({ - atom: allObjects, - subscribe, - select: (row) => row.data, - upsertKey: (row) => row.data.id, - }); +export function useSubscribeToAllObjects() { + useGlobalUpsertSubscribe(ALL_OBJECTS_SUBSCRIBE_OPTIONS); } export function useAllObjects() { diff --git a/console/src/store/allObjectsCollection.ts b/console/src/store/allObjectsCollection.ts index 7f17569771a60..7467ca0ebc4f9 100644 --- a/console/src/store/allObjectsCollection.ts +++ b/console/src/store/allObjectsCollection.ts @@ -8,13 +8,14 @@ // by the Apache License, Version 2.0. import { useLiveQuery } from "@tanstack/react-db"; -import { useAtomValue } from "jotai"; +import { useAtomValue, useStore } from "jotai"; import React from "react"; import { DatabaseObject } from "~/api/materialize/objects"; import { createSubscribeCollection } from "~/api/materialize/subscribeCollection"; +import { createAtomFedCollectionSession } from "~/api/materialize/subscribeSession"; import { allObjects } from "~/store/allObjects"; -import { useSyncEngineCacheScope } from "~/store/syncEngineCache"; +import { syncEngineCacheScopeLoadableAtom } from "~/store/syncEngineCache"; /** * TanStack DB-backed view of the `allObjects` jotai atom, so consumers can run @@ -35,14 +36,16 @@ export const allObjectsCollection = createSubscribeCollection({ * tree is mounted, and sync re-seeds from the retained row set if it restarts. */ export function useSubscribeToAllObjectsCollection() { - const state = useAtomValue(allObjects); - const scope = useSyncEngineCacheScope(); + const store = useStore(); React.useEffect(() => { - if (scope) allObjectsCollection.hydrate(scope); - }, [scope]); - React.useEffect(() => { - allObjectsCollection.applySnapshot(state); - }, [state]); + const session = createAtomFedCollectionSession({ + store, + sourceAtom: allObjects, + target: allObjectsCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + }); + return session.destroy; + }, [store]); } /** diff --git a/console/src/store/allRoles.ts b/console/src/store/allRoles.ts index 40fe7971bb90a..d8cf3dc44ea8c 100644 --- a/console/src/store/allRoles.ts +++ b/console/src/store/allRoles.ts @@ -8,7 +8,7 @@ // by the Apache License, Version 2.0. import { atom, useAtomValue } from "jotai"; -import React, { useMemo } from "react"; +import React from "react"; import { buildSubscribeQuery } from "~/api/materialize/buildSubscribeQuery"; import { @@ -29,19 +29,17 @@ export const allRoles = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllRoles() { - const subscribe = useMemo(() => { - return buildSubscribeQuery(buildRolesListQuery(), { - upsertKey: ["roleName"], - }); - }, []); +const ALL_ROLES_SUBSCRIBE_OPTIONS = { + atom: allRoles, + subscribe: buildSubscribeQuery(buildRolesListQuery(), { + upsertKey: ["roleName"], + }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.roleName, +}; - return useGlobalUpsertSubscribe({ - atom: allRoles, - subscribe, - select: (row: SubscribeRow) => row.data, - upsertKey: (row: SubscribeRow) => row.data.roleName, - }); +export function useSubscribeToAllRoles() { + useGlobalUpsertSubscribe(ALL_ROLES_SUBSCRIBE_OPTIONS); } export function useAllRoles() { diff --git a/console/src/store/allSchemas.ts b/console/src/store/allSchemas.ts index b80c7de02fac5..6d57a89f2cc5d 100644 --- a/console/src/store/allSchemas.ts +++ b/console/src/store/allSchemas.ts @@ -15,7 +15,10 @@ import { buildAllSchemaListQuery, SchemaWithOptionalDatabase, } from "~/api/materialize/schemaList"; -import { SubscribeState } from "~/api/materialize/SubscribeManager"; +import { + SubscribeRow, + SubscribeState, +} from "~/api/materialize/SubscribeManager"; import { buildSubscribeQuery, useGlobalUpsertSubscribe, @@ -27,17 +30,17 @@ export const allSchemas = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllSchemas() { - const subscribe = React.useMemo(() => { - return buildSubscribeQuery(buildAllSchemaListQuery(), { upsertKey: "id" }); - }, []); +const ALL_SCHEMAS_SUBSCRIBE_OPTIONS = { + atom: allSchemas, + subscribe: buildSubscribeQuery(buildAllSchemaListQuery(), { + upsertKey: "id", + }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; - return useGlobalUpsertSubscribe({ - atom: allSchemas, - subscribe, - select: (row) => row.data, - upsertKey: (row) => row.data.id, - }); +export function useSubscribeToAllSchemas() { + useGlobalUpsertSubscribe(ALL_SCHEMAS_SUBSCRIBE_OPTIONS); } export function useAllSchemas(options?: { includeSystemSchemas?: boolean }) { diff --git a/console/src/store/syncEngineCache.ts b/console/src/store/syncEngineCache.ts index e098efa2d1758..acd211f9105e0 100644 --- a/console/src/store/syncEngineCache.ts +++ b/console/src/store/syncEngineCache.ts @@ -7,26 +7,41 @@ // the Business Source License, use of this software will be governed // by the Apache License, Version 2.0. -import { useAtomValue } from "jotai"; +import { atom } from "jotai"; +import { loadable } from "jotai/utils"; -import { useMaybeCurrentOrganizationId } from "~/api/auth"; +import { + fetchCurrentOrganization, + isOrganizationFetchEnabled, + queryKeys as authQueryKeys, +} from "~/api/auth"; +import { Organization } from "~/api/cloudGlobalApi"; +import { appConfigAtom } from "~/config/store"; +import { getQueryClient } from "~/queryClient"; import { currentRegionIdSyncAtom } from "~/store/environments"; /** * The auth/region scope for the sync-engine localStorage caches, as - * `organizationId|regionId`. Returns undefined until both are known, so callers - * defer seeding until the scope is settled and never share one tenant's cache - * with another on the shared `console.materialize.com` origin. + * `organizationId|regionId`. Resolves to undefined until both are known, so + * consumers defer cache seeding until the scope is settled and never share one + * tenant's cache with another on a shared origin. */ -export function useSyncEngineCacheScope(): string | undefined { - const maybeOrganizationId = useMaybeCurrentOrganizationId(); - const regionId = useAtomValue(currentRegionIdSyncAtom); +export const syncEngineCacheScopeAtom = atom(async (get) => { + const regionId = get(currentRegionIdSyncAtom); + if (!regionId) return undefined; - const organizationIdLoading = - maybeOrganizationId !== null && maybeOrganizationId.isLoading; - if (organizationIdLoading || !regionId) return undefined; - - const organizationId = - maybeOrganizationId !== null ? maybeOrganizationId.data : undefined; + const organizationId = isOrganizationFetchEnabled(get(appConfigAtom)) + ? ( + await getQueryClient().ensureQueryData({ + queryKey: authQueryKeys.currentOrganization(), + queryFn: fetchCurrentOrganization, + }) + ).id + : undefined; return `${organizationId ?? ""}|${regionId}`; -} +}); + +/** Synchronous view of the scope for store.sub consumers (subscribe sessions). */ +export const syncEngineCacheScopeLoadableAtom = loadable( + syncEngineCacheScopeAtom, +); From 008991d90af7e633787f57145c269d9a5420237b Mon Sep 17 00:00:00 2001 From: Materialize Date: Fri, 4 Sep 2026 00:00:00 +0000 Subject: [PATCH 8/8] LICENSE: update change date --- LICENSE | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE b/LICENSE index 5edc644cf0eed..e034c879dac64 100644 --- a/LICENSE +++ b/LICENSE @@ -13,7 +13,7 @@ Business Source License 1.1 Licensor: Materialize, Inc. -Licensed Work: Materialize Version 20260903 +Licensed Work: Materialize Version 20260904 The Licensed Work is © 2026 Materialize, Inc. Additional Use Grant: Within a single installation of the Licensed Work, @@ -39,7 +39,7 @@ Additional Use Grant: Within a single installation of the Licensed Work, whose definitions are controlled by such third parties. -Change Date: September 03, 2030 +Change Date: September 04, 2030 Change License: Apache License, Version 2.0