diff --git a/doc/user/content/get-started/quickstart.md b/doc/user/content/get-started/quickstart.md index 94456307c226e..c23a351839389 100644 --- a/doc/user/content/get-started/quickstart.md +++ b/doc/user/content/get-started/quickstart.md @@ -112,24 +112,42 @@ generator](/sql/create-source/load-generator/#auction) to create the source. For the [sample `Auction` load generator](/sql/create-source/load-generator/#auction), the quickstart uses - [`CREATE SOURCE`](/sql/create-source/) with the `FROM LOAD GENERATOR` clause + [`CREATE SOURCE` with the `FROM LOAD + GENERATOR` clause](/sql/create-source/load-generator/#creating-an-auction-load-generator) that works specifically with Materialize's sample data generators. The tutorial specifies that the generator should emit new data every 1s. ```mzsql CREATE SOURCE auction_house FROM LOAD GENERATOR AUCTION - (TICK INTERVAL '1s', AS OF 100000) - FOR ALL TABLES; + (TICK INTERVAL '1s', AS OF 100000); ``` - `CREATE SOURCE` can create **multiple** tables (referred to as `subsources` - in Materialize) when ingesting data from multiple upstream tables. For each - upstream table that is selected for ingestion, Materialize creates a - subsource. + The source connects to the data generator but does not ingest its relations + yet. To start ingesting data, you create a table from the source for each + relation you want to ingest. -1. Use the [`SHOW SOURCES`](/sql/show-sources/) command to see the results of - the previous step. +1. Use [`CREATE TABLE ... FROM + SOURCE`](/sql/create-source/load-generator/#creating-an-auction-load-generator) + in a [DDL transaction block](/sql/begin/#ddl-only-transactions) to create + **read-only** tables for the `auctions` and `bids` relations, the two + relations this quickstart uses: + + ```mzsql + BEGIN; + CREATE TABLE auctions FROM SOURCE auction_house (REFERENCE auctions); + CREATE TABLE bids FROM SOURCE auction_house (REFERENCE bids); + COMMIT; + ``` + + Tables created from a source are read-only; that is: + + - Only the source can write to the table; in this case, the load + generator. + + - Users can read from the table. + +1. Use the [`SHOW SOURCES`](/sql/show-sources/) command to see the new source: ```mzsql SHOW SOURCES; @@ -138,24 +156,24 @@ generator](/sql/create-source/load-generator/#auction) to create the source. The output should resemble the following: ```nofmt - | name | type | cluster | comment | - | ---------------------- | -------------- | ---------- | ------- | - | accounts | subsource | quickstart | | - | auction_house | load-generator | quickstart | | - | auction_house_progress | progress | null | | - | auctions | subsource | quickstart | | - | bids | subsource | quickstart | | - | organizations | subsource | quickstart | | - | users | subsource | quickstart | | + | name | type | cluster | comment | + | ------------- | -------------- | ---------- | ------- | + | auction_house | load-generator | quickstart | | ``` - A [`subsource`](/sql/show-subsources) is how Materialize refers to a table - that has the following properties: + Use the [`SHOW TABLES`](/sql/show-tables/) command to see the tables + created from the source: - - A subsource can only be written by the source; in this case, the - load-generator. + ```mzsql + SHOW TABLES; + ``` - - Users can read from subsources. + ```nofmt + | name | comment | + | -------- | ------- | + | auctions | | + | bids | | + ``` 1. Use the [`SELECT`](/sql/select) statement to query `auctions` and `bids`. @@ -364,9 +382,6 @@ creates: AND datediff('days', w2.bid_time, w1.bid_time) < 8; ``` - The `flip_activities` view can use the index created on `winning_bids` view - to provide up-to-date data. - To view a sample row in `flip_activities`, run the following [`SELECT`](/sql/select) command: @@ -374,6 +389,30 @@ creates: SELECT * FROM flip_activities LIMIT 10; ``` + The query may take a while to return, even though it uses the + `wins_by_item` index. The `flip_activities` view self-joins `winning_bids` + on `item` and the corresponding `buyer` and `seller` columns (`w1.buyer = + w2.seller AND w1.item = w2.item`), but the `wins_by_item` index is keyed + on the `item` column only. As a result, the join must first form every + pair of rows with the same `item` (the part supported by the index) before + evaluating the buyer/seller predicates on each pair to find the actual + matches. Because the sample data contains only a small number of distinct + item values, as `winning_bids` grows, the number of pairs per item grows + quadratically. + +1. Create indexes on the `winning_bids` join keys used by `flip_activities`. + + To avoid the aforementioned quadratic work, index the complete join keys so + that the join considers only rows that can actually match: + + ```mzsql + CREATE INDEX wins_by_item_seller ON winning_bids (item, seller); + CREATE INDEX wins_by_item_buyer ON winning_bids (item, buyer); + ``` + + Rerun the previous query on `flip_activities`. The query should return + faster. + 1. Use [`CREATE TABLE`](/sql/create-table) to create a `known_flippers` table that you can manually populate with known flippers. That is, assume that separate from your auction activities data, you receive independent data @@ -405,11 +444,14 @@ creates: {{< note >}} -Both the `flip_activities` and `flippers` views can use the index created on -`winning_bids` view to provide up-to-date data. Depending upon your query -patterns and usage, an existing index may be sufficient, such as in this -quickstart. In other use cases, creating an index only on the view(s) from which -you will serve results may be preferred. +Both the `flip_activities` and `flippers` views can use the indexes created on +the `winning_bids` view to provide up-to-date results. Index keys matter: the +`wins_by_item` index serves point lookups on `item`, while the +`wins_by_item_seller` and `wins_by_item_buyer` indexes serve the +`flip_activities` self-join on its join keys. Depending on your query patterns +and usage, indexing the upstream views may be sufficient, such as indexing +`winning_bids` in this quickstart. In other use cases, it may be preferable to +index only the views from which applications directly read results. {{}} @@ -529,8 +571,8 @@ data comes in, this step creates the following views for completed auctions: To clean up the quickstart environment: 1. Use the [`DROP SOURCE ... CASCADE`](/sql/drop-source/) command to drop - `auction_house` source and its dependent objects, including views and indexes - created on the `auction_house` subsources. + `auction_house` source and its dependent objects, including the tables + created from the source and the views and indexes created on them. ```mzsql DROP SOURCE auction_house CASCADE; diff --git a/doc/user/content/headless/agent-skills-table.md b/doc/user/content/headless/agent-skills-table.md index 702200edc3dbd..ef9aae2e735b5 100644 --- a/doc/user/content/headless/agent-skills-table.md +++ b/doc/user/content/headless/agent-skills-table.md @@ -1,10 +1,10 @@ --- headless: true --- -| Skill | What it provides | When to use | -|-------|------------------|-------------| -| `mcp-developer-analysis` | Exact catalog schemas, diagnostic workflows, remediation runbooks, and guardrails for known pitfalls (cluster-scoped queries, uint8 ID mismatches, etc.). | Operational introspection and troubleshooting via the `materialize-developer` server. Examples: *"why is my materialized view stale?"*, *"what can I optimize to save costs?"*, *"is my source healthy?"* | -| `materialize-docs` | Comprehensive Materialize documentation, including SQL syntax, idiomatic patterns, data ingestion, concepts, and best practices (400+ reference files). | Authoring view definitions, learning concepts, looking up patterns. Useful with either MCP server. Examples: *"show me how to deduplicate a stream"*, *"what's the idiomatic top-K pattern?"*, *"how do I create a Kafka source?"* | -| `materialize-dbt` | dbt-materialize adapter usage: materializations, profile configuration, index creation, blue/green deployments, and testing. | Managing Materialize pipelines with dbt. Examples: *"write a dbt model for a materialized view"*, *"how do I do a blue/green deployment with dbt?"* | -| `materialize-terraform-provider` | Provider configuration for Cloud and self-managed, navigation into the provider's auto-generated resource reference, cross-resource patterns, import workflows, and gotchas. | Managing Materialize resources declaratively with Terraform. Examples: *"create a Kafka source with Terraform"*, *"import my existing clusters into Terraform state"*, *"set up RBAC grants in Terraform"* | -| `materialize-terraform-self-managed` | Module layout and variables for deploying self-managed Materialize on AWS, Azure, and GCP: networking, Kubernetes, backend URL formats, instance sizing, upgrades, and gotchas. | Deploying or operating self-managed Materialize infrastructure with Terraform. Examples: *"deploy Materialize on EKS"*, *"what instance types should Materialize nodes use?"*, *"upgrade my self-managed Materialize"* | +| Skill | Description | +|-------|-------------| +| `mcp-developer-analysis` | Use for operational introspection and troubleshooting via the `materialize-developer` server. Covers exact catalog schemas, diagnostic workflows, remediation runbooks, and guardrails for known pitfalls (cluster-scoped queries, uint8 ID mismatches, etc.).

Examples: *"why is my materialized view stale?"*, *"what can I optimize to save costs?"*, *"is my source healthy?"* | +| `materialize-docs` | Use for authoring view definitions, learning concepts, and looking up patterns; useful with either MCP server. Covers comprehensive Materialize documentation, including SQL syntax, idiomatic patterns, data ingestion, concepts, and best practices (400+ reference files).

Examples: *"show me how to deduplicate a stream"*, *"what's the idiomatic top-K pattern?"*, *"how do I create a Kafka source?"* | +| `materialize-dbt` | Use for managing Materialize pipelines with dbt. Covers dbt-materialize adapter usage: materializations, profile configuration, index creation, blue/green deployments, and testing.

Examples: *"write a dbt model for a materialized view"*, *"how do I do a blue/green deployment with dbt?"* | +| `materialize-terraform-provider` | Use for managing Materialize resources declaratively with Terraform. Covers provider configuration for Cloud and self-managed, navigation into the provider's auto-generated resource reference, cross-resource patterns, import workflows, and gotchas.

Examples: *"create a Kafka source with Terraform"*, *"import my existing clusters into Terraform state"*, *"set up RBAC grants in Terraform"* | +| `materialize-terraform-self-managed` | Use for deploying or operating self-managed Materialize infrastructure with Terraform. Covers module layout and variables for deploying on AWS, Azure, and GCP: networking, Kubernetes, backend URL formats, instance sizing, upgrades, and gotchas.

Examples: *"deploy Materialize on EKS"*, *"what instance types should Materialize nodes use?"*, *"upgrade my self-managed Materialize"* | diff --git a/doc/user/content/integrations/_index.md b/doc/user/content/integrations/_index.md index 6d6cbaafad745..4254eb02ab3f0 100644 --- a/doc/user/content/integrations/_index.md +++ b/doc/user/content/integrations/_index.md @@ -25,20 +25,8 @@ menu: ## Agent skills and MCP servers -### Agent skills - -Materialize provides the following open-source [agent -skills](https://github.com/MaterializeInc/agent-skills) to help developers build -with Materialize. - -{{% include-headless "/headless/agent-skills-table" %}} - -### MCP servers - -Materialize providesthe following built-in Model Context Protocol (MCP) servers -that AI agents can use. - -{{% include-headless "/headless/mcp-servers-table" %}} +For Materialize's open-source agent skills and built-in Model Context Protocol +(MCP) servers, see [AI & agents](/integrations/mcp-server/). ## SQL clients/client libraries diff --git a/doc/user/content/integrations/coding-agent-skills.md b/doc/user/content/integrations/coding-agent-skills.md index 6bbfa1550f219..98bccd1ac1b2f 100644 --- a/doc/user/content/integrations/coding-agent-skills.md +++ b/doc/user/content/integrations/coding-agent-skills.md @@ -7,7 +7,16 @@ menu: weight: 7 --- -Coding agents like [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Codex](https://openai.com/index/codex/), [Cursor](https://www.cursor.com/), and others can work with Materialize using our open-source [agent skills](https://github.com/MaterializeInc/agent-skills). Once installed, these skills give your coding agent access to Materialize documentation and reference material so it can provide more accurate assistance when writing queries, setting up sources, creating materialized views, and more. +Coding agents like [Claude +Code](https://docs.anthropic.com/en/docs/claude-code), +[Codex](https://openai.com/index/codex/), [Cursor](https://www.cursor.com/), and +others can work with Materialize using the open-source [Materialize agent +skills](https://github.com/MaterializeInc/agent-skills). These skills follow the +[Agent Skills Open Standard](https://agentskills.io/home) and work with any +coding agent that supports the standard. Once installed, these skills give your +coding agent access to Materialize documentation and reference material so it +can provide more accurate assistance when writing queries, setting up sources, +creating materialized views, and more. ## Skills @@ -27,8 +36,6 @@ npx skills add MaterializeInc/agent-skills Once installed, you can update installed skills by running `npx skills update`. -The skills follow the [Agent Skills Open Standard](https://agentskills.io/home) and work with any coding agent that supports the standard. - ## Reduce permission prompts (Claude Code) Claude Code prompts before reading files outside your project. Since globally diff --git a/doc/user/content/integrations/mcp-server/_index.md b/doc/user/content/integrations/mcp-server/_index.md index 2ed7400313a36..e35c7d0f2ecb6 100644 --- a/doc/user/content/integrations/mcp-server/_index.md +++ b/doc/user/content/integrations/mcp-server/_index.md @@ -4,10 +4,9 @@ description: "This section contains guides for installing Materialize Agent skil disable_list: true menu: main: - parent: integrations - name: "MCP Server and skills" + name: "AI & agents" identifier: mcp-server - weight: 6 + weight: 48 aliases: - /integrations/mcp-server/llm/ - /integrations/llm/ diff --git a/misc/python/materialize/checks/all_checks/builtin_cluster_replication_factor.py b/misc/python/materialize/checks/all_checks/builtin_cluster_replication_factor.py new file mode 100644 index 0000000000000..95742b6b87482 --- /dev/null +++ b/misc/python/materialize/checks/all_checks/builtin_cluster_replication_factor.py @@ -0,0 +1,77 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. +from textwrap import dedent + +from materialize.checks.actions import Testdrive +from materialize.checks.checks import Check +from materialize.checks.executors import Executor +from materialize.mz_version import MzVersion + + +class BuiltinClusterReplicationFactor(Check): + """A builtin cluster's replication factor owns its replica set across restarts. + + `mz_support` is the interesting cluster. It runs nothing by default, and it is + the break-glass path a support engineer scales up by hand. A catalog open that + does not read the cluster's replication factor tears that replica down, leaving + the cluster reporting a factor it is not honoring. + """ + + def _can_run(self, e: Executor) -> bool: + # Only versions that reconcile the replica set against the cluster's own + # replication factor keep this replica across a restart. Confine the check + # to scenarios whose boots are all the current version: the upgrade + # scenarios restart on released binaries, and `manipulate` and `validate` + # both run on those. + return self.base_version >= MzVersion.parse_cargo() + + def manipulate(self) -> list[Testdrive]: + # `mz_support` owns its own cluster, so the ALTER has to run as that role. + # Its sessions default to `mz_catalog_server`, which has a replica, so a + # statement from this connection is safe to issue. + # + # A restart separates the two phases, so phase 2 asserting the replica is + # still there is the regression assertion: the replica set has to be + # rederived from the cluster's factor rather than from a fixed list. + return [ + Testdrive(dedent(s)) + for s in [ + """ + $ postgres-execute connection=postgres://mz_support:materialize@${testdrive.materialize-internal-sql-addr} + ALTER CLUSTER mz_support SET (REPLICATION FACTOR 1); + """, + """ + > SELECT c.replication_factor, count(r.id) + FROM mz_clusters c + LEFT JOIN mz_cluster_replicas r ON r.cluster_id = c.id + WHERE c.name = 'mz_support' + GROUP BY c.replication_factor + 1 1 + """, + ] + ] + + def validate(self) -> Testdrive: + return Testdrive(dedent(""" + > SELECT c.replication_factor, count(r.id) + FROM mz_clusters c + LEFT JOIN mz_cluster_replicas r ON r.cluster_id = c.id + WHERE c.name = 'mz_support' + GROUP BY c.replication_factor + 1 1 + + # Named by the managed-cluster rule, and owned by the cluster's owner + # rather than by mz_system. + > SELECT r.name, o.name + FROM mz_cluster_replicas r + JOIN mz_clusters c ON c.id = r.cluster_id + JOIN mz_roles o ON o.id = r.owner_id + WHERE c.name = 'mz_support' + r1 mz_support + """)) diff --git a/misc/python/materialize/checks/scenarios_upgrade.py b/misc/python/materialize/checks/scenarios_upgrade.py index 4f7022693ebfe..fff47cba79512 100644 --- a/misc/python/materialize/checks/scenarios_upgrade.py +++ b/misc/python/materialize/checks/scenarios_upgrade.py @@ -337,6 +337,13 @@ def actions(self) -> list[Action]: print( "Upgrade path: v26.17.1 -> initialize -> v26.18.0 -> manipulate#1 -> restart -> manipulate#2 -> v26.24.0 (v82_to_v83) -> current (v83_to_v84) -> restart -> validate" ) + # The factor-0 override below only applies to the boots that pass it, but + # it is durable: the cluster row is seeded once and a later boot honors the + # row rather than its own flag. So mz_system has no replica for the whole + # run, including the final unoverridden boots. Nothing here needs one, + # since the checks peek as `materialize` on `quickstart` and the mz_system + # connection only issues DDL. A check that peeked at a non-catalog object + # as `mz_system` would hang, and would need this override lifted. return [ StartMz( self, diff --git a/misc/python/materialize/mzcompose/services/materialized.py b/misc/python/materialize/mzcompose/services/materialized.py index 2378b7c07ece8..8dc37ff0f32cd 100644 --- a/misc/python/materialize/mzcompose/services/materialized.py +++ b/misc/python/materialize/mzcompose/services/materialized.py @@ -129,9 +129,17 @@ def __init__( if cluster_replica_size is None: cluster_replica_size = cluster_replica_size_map() if builtin_system_cluster_replication_factor is None: - builtin_system_cluster_replication_factor = default_replication_factor + # Deliberately not `default_replication_factor`. A builtin cluster + # honors its replication factor, so tracking the user default here + # would run two mz_system and two mz_probe replicas in every + # composition that asks for a replicated user cluster, at no benefit + # to what those tests actually exercise. One replica is all these + # clusters need, and is what the flags below are here to guarantee. + # A test that needs a different builtin factor, including 0, has to + # ask for it explicitly. + builtin_system_cluster_replication_factor = 1 if builtin_probe_cluster_replication_factor is None: - builtin_probe_cluster_replication_factor = default_replication_factor + builtin_probe_cluster_replication_factor = 1 environment = [ "MZ_NO_TELEMETRY=1", diff --git a/src/adapter/src/catalog/builtin_table_updates.rs b/src/adapter/src/catalog/builtin_table_updates.rs index ce1c40aa83108..7c100ef46ef30 100644 --- a/src/adapter/src/catalog/builtin_table_updates.rs +++ b/src/adapter/src/catalog/builtin_table_updates.rs @@ -19,12 +19,11 @@ use mz_catalog::builtin::{ MZ_AWS_PRIVATELINK_CONNECTIONS, MZ_BASE_TYPES, MZ_CLUSTER_REPLICA_SIZE_INTERNAL, MZ_CLUSTER_REPLICA_SIZES, MZ_COLUMNS, MZ_EGRESS_IPS, MZ_FUNCTIONS, MZ_HISTORY_RETENTION_STRATEGIES, MZ_ICEBERG_SINKS, MZ_INDEX_COLUMNS, MZ_KAFKA_CONNECTIONS, - MZ_KAFKA_SINKS, MZ_KAFKA_SOURCE_TABLES, MZ_LICENSE_KEYS, MZ_LIST_TYPES, MZ_MAP_TYPES, - MZ_MATERIALIZED_VIEW_REFRESH_STRATEGIES, MZ_MYSQL_SOURCE_TABLES, MZ_OBJECT_DEPENDENCIES, - MZ_OBJECT_GLOBAL_IDS, MZ_OPERATORS, MZ_POSTGRES_SOURCE_TABLES, MZ_PSEUDO_TYPES, - MZ_REPLACEMENTS, MZ_ROLE_AUTH, MZ_SESSIONS, MZ_SINKS, MZ_SOURCE_REFERENCES, - MZ_SQL_SERVER_SOURCE_TABLES, MZ_SSH_TUNNEL_CONNECTIONS, MZ_STORAGE_USAGE_BY_SHARD, - MZ_SUBSCRIPTIONS, MZ_TABLES, MZ_TYPE_PG_METADATA, MZ_TYPES, MZ_VIEWS, MZ_WEBHOOKS_SOURCES, + MZ_KAFKA_SINKS, MZ_LICENSE_KEYS, MZ_LIST_TYPES, MZ_MAP_TYPES, + MZ_MATERIALIZED_VIEW_REFRESH_STRATEGIES, MZ_OBJECT_DEPENDENCIES, MZ_OBJECT_GLOBAL_IDS, + MZ_OPERATORS, MZ_PSEUDO_TYPES, MZ_REPLACEMENTS, MZ_ROLE_AUTH, MZ_SESSIONS, MZ_SINKS, + MZ_SOURCE_REFERENCES, MZ_SSH_TUNNEL_CONNECTIONS, MZ_STORAGE_USAGE_BY_SHARD, MZ_SUBSCRIPTIONS, + MZ_TABLES, MZ_TYPE_PG_METADATA, MZ_TYPES, MZ_VIEWS, MZ_WEBHOOKS_SOURCES, }; use mz_catalog::config::AwsPrincipalContext; use mz_catalog::durable::SourceReferences; @@ -48,7 +47,7 @@ use mz_repr::role_id::RoleId; use mz_repr::{ CatalogItemId, Datum, Diff, GlobalId, ReprColumnType, Row, RowPacker, SqlScalarType, Timestamp, }; -use mz_sql::ast::{CreateIndexStatement, Statement, UnresolvedItemName}; +use mz_sql::ast::{CreateIndexStatement, Statement}; use mz_sql::catalog::{CatalogType, TypeCategory}; use mz_sql::func::FuncImplCatalogDetails; use mz_sql::names::SchemaSpecifier; @@ -60,7 +59,6 @@ use mz_storage_types::connections::aws::{AwsAuth, AwsConnection}; use mz_storage_types::connections::inline::ReferencedConnection; use mz_storage_types::connections::string_or_secret::StringOrSecret; use mz_storage_types::sinks::{IcebergSinkConnection, KafkaSinkConnection, StorageSinkConnection}; -use mz_storage_types::sources::SourceConnection; use smallvec::smallvec; // DO NOT add any more imports from `crate` outside of `crate::catalog`. @@ -174,177 +172,24 @@ impl CatalogState { let mut updates = match entry.item() { CatalogItem::Index(index) => self.pack_index_update(id, index, diff), CatalogItem::Table(table) => { - let mut updates = self - .pack_table_update(id, oid, schema_id, name, owner_id, privileges, diff, table); - - if let TableDataSource::DataSource { - desc: data_source, - timeline: _, - } = &table.data_source - { - updates.extend(match data_source { - DataSourceDesc::IngestionExport { - ingestion_id, - external_reference: UnresolvedItemName(external_reference), - details: _, - data_config: _, - } => { - let ingestion_entry = self - .get_entry(ingestion_id) - .source_desc() - .expect("primary source exists") - .expect("primary source is a source"); - - match ingestion_entry.connection.name() { - "postgres" => { - mz_ore::soft_assert_eq_no_log!(external_reference.len(), 3); - // The left-most qualification of Postgres - // tables is the database, but this - // information is redundant because each - // Postgres connection connects to only one - // database. - let schema_name = external_reference[1].as_str(); - let table_name = external_reference[2].as_str(); - - self.pack_postgres_source_tables_update( - id, - schema_name, - table_name, - diff, - ) - } - "mysql" => { - mz_ore::soft_assert_eq_no_log!(external_reference.len(), 2); - let schema_name = external_reference[0].as_str(); - let table_name = external_reference[1].as_str(); - - self.pack_mysql_source_tables_update( - id, - schema_name, - table_name, - diff, - ) - } - "sql-server" => { - mz_ore::soft_assert_eq_no_log!(external_reference.len(), 3); - // The left-most qualification of SQL Server tables is - // the database, but this information is redundant - // because each SQL Server connection connects to - // only one database. - let schema_name = external_reference[1].as_str(); - let table_name = external_reference[2].as_str(); - - self.pack_sql_server_source_table_update( - id, - schema_name, - table_name, - diff, - ) - } - // Load generator sources don't have any special - // updates. - "load-generator" => vec![], - "kafka" => { - mz_ore::soft_assert_eq_no_log!(external_reference.len(), 1); - let topic = external_reference[0].as_str(); - let envelope = data_source.envelope(); - let (key_format, value_format) = data_source.formats(); - - self.pack_kafka_source_tables_update( - id, - topic, - envelope, - key_format, - value_format, - diff, - ) - } - s => unreachable!("{s} sources do not have tables"), - } - } - DataSourceDesc::Ingestion { .. } - | DataSourceDesc::OldSyntaxIngestion { .. } - | DataSourceDesc::Introspection(_) - | DataSourceDesc::Progress - | DataSourceDesc::Webhook { .. } - | DataSourceDesc::Catalog => vec![], - }); - } - - updates + // Source-table metadata (mz_postgres/mysql/sql_server/kafka_source_tables) + // is now derived from the persisted create_sql by materialized + // views over mz_catalog_raw, so ingestion-export tables need no + // special packing here. + self.pack_table_update(id, oid, schema_id, name, owner_id, privileges, diff, table) } CatalogItem::Source(source) => { match &source.data_source { - DataSourceDesc::Ingestion { .. } - | DataSourceDesc::OldSyntaxIngestion { .. } => vec![], - DataSourceDesc::IngestionExport { - ingestion_id, - external_reference: UnresolvedItemName(external_reference), - details: _, - data_config: _, - } => { - let ingestion_entry = self - .get_entry(ingestion_id) - .source_desc() - .expect("primary source exists") - .expect("primary source is a source"); - - match ingestion_entry.connection.name() { - "postgres" => { - mz_ore::soft_assert_eq_no_log!(external_reference.len(), 3); - // The left-most qualification of Postgres - // tables is the database, but this - // information is redundant because each - // Postgres connection connects to only one - // database. - let schema_name = external_reference[1].as_str(); - let table_name = external_reference[2].as_str(); - - self.pack_postgres_source_tables_update( - id, - schema_name, - table_name, - diff, - ) - } - "mysql" => { - mz_ore::soft_assert_eq_no_log!(external_reference.len(), 2); - let schema_name = external_reference[0].as_str(); - let table_name = external_reference[1].as_str(); - - self.pack_mysql_source_tables_update( - id, - schema_name, - table_name, - diff, - ) - } - "sql-server" => { - mz_ore::soft_assert_eq_no_log!(external_reference.len(), 3); - // The left-most qualification of SQL Server tables is - // the database, but this information is redundant - // because each SQL Server connection connects to - // only one database. - let schema_name = external_reference[1].as_str(); - let table_name = external_reference[2].as_str(); - - self.pack_sql_server_source_table_update( - id, - schema_name, - table_name, - diff, - ) - } - // Load generator sources don't have any special - // updates. - "load-generator" => vec![], - s => unreachable!("{s} sources do not have subsources"), - } - } DataSourceDesc::Webhook { .. } => { vec![self.pack_webhook_source_update(id, diff)] } - DataSourceDesc::Introspection(_) + // Old-syntax subsource metadata (mz_postgres/mysql/sql_server_source_tables) + // is now derived from create_sql by materialized views over + // mz_catalog_raw, so ingestion exports need no special packing. + DataSourceDesc::Ingestion { .. } + | DataSourceDesc::OldSyntaxIngestion { .. } + | DataSourceDesc::IngestionExport { .. } + | DataSourceDesc::Introspection(_) | DataSourceDesc::Progress | DataSourceDesc::Catalog => vec![], } @@ -548,82 +393,6 @@ impl CatalogState { )] } - fn pack_postgres_source_tables_update( - &self, - id: CatalogItemId, - schema_name: &str, - table_name: &str, - diff: Diff, - ) -> Vec> { - vec![BuiltinTableUpdate::row( - &*MZ_POSTGRES_SOURCE_TABLES, - Row::pack_slice(&[ - Datum::String(&id.to_string()), - Datum::String(schema_name), - Datum::String(table_name), - ]), - diff, - )] - } - - fn pack_mysql_source_tables_update( - &self, - id: CatalogItemId, - schema_name: &str, - table_name: &str, - diff: Diff, - ) -> Vec> { - vec![BuiltinTableUpdate::row( - &*MZ_MYSQL_SOURCE_TABLES, - Row::pack_slice(&[ - Datum::String(&id.to_string()), - Datum::String(schema_name), - Datum::String(table_name), - ]), - diff, - )] - } - - fn pack_sql_server_source_table_update( - &self, - id: CatalogItemId, - schema_name: &str, - table_name: &str, - diff: Diff, - ) -> Vec> { - vec![BuiltinTableUpdate::row( - &*MZ_SQL_SERVER_SOURCE_TABLES, - Row::pack_slice(&[ - Datum::String(&id.to_string()), - Datum::String(schema_name), - Datum::String(table_name), - ]), - diff, - )] - } - - fn pack_kafka_source_tables_update( - &self, - id: CatalogItemId, - topic: &str, - envelope: Option<&str>, - key_format: Option<&str>, - value_format: Option<&str>, - diff: Diff, - ) -> Vec> { - vec![BuiltinTableUpdate::row( - &*MZ_KAFKA_SOURCE_TABLES, - Row::pack_slice(&[ - Datum::String(&id.to_string()), - Datum::String(topic), - Datum::from(envelope), - Datum::from(key_format), - Datum::from(value_format), - ]), - diff, - )] - } - fn pack_connection_update( &self, id: CatalogItemId, diff --git a/src/adapter/src/catalog/open.rs b/src/adapter/src/catalog/open.rs index 693515ccca026..c946b5ef5c87a 100644 --- a/src/adapter/src/catalog/open.rs +++ b/src/adapter/src/catalog/open.rs @@ -27,14 +27,17 @@ use mz_audit_log::{ use mz_auth::hash::scram256_hash; use mz_catalog::SYSTEM_CONN_ID; use mz_catalog::builtin::{ - BUILTIN_CLUSTER_REPLICAS, BUILTIN_CLUSTERS, BUILTIN_PREFIXES, BUILTIN_ROLES, BUILTINS, Builtin, - Fingerprint, MZ_CATALOG_RAW, RUNTIME_ALTERABLE_FINGERPRINT_SENTINEL, + BUILTIN_CLUSTERS, BUILTIN_PREFIXES, BUILTIN_ROLES, BUILTINS, Builtin, Fingerprint, + MZ_CATALOG_RAW, RUNTIME_ALTERABLE_FINGERPRINT_SENTINEL, }; use mz_catalog::config::StateConfig; use mz_catalog::durable::objects::{ SystemObjectDescription, SystemObjectMapping, SystemObjectUniqueIdentifier, }; -use mz_catalog::durable::{ClusterReplica, ClusterVariant, ClusterVariantManaged, Transaction}; +use mz_catalog::durable::{ + ClusterReplica, ClusterVariant, ClusterVariantManaged, ReplicaConfig, ReplicaLocation, + Transaction, managed_cluster_replica_name, +}; use mz_catalog::expr_cache::{ ExpressionCacheConfig, ExpressionCacheHandle, GlobalExpressions, LocalExpressions, }; @@ -210,7 +213,7 @@ impl Catalog { config.boot_ts, )?; add_new_remove_old_builtin_introspection_source_migration(&mut txn)?; - add_new_remove_old_builtin_cluster_replicas_migration( + reconcile_builtin_cluster_replicas( &mut txn, &builtin_bootstrap_cluster_config_map, config.boot_ts, @@ -1114,67 +1117,135 @@ fn add_new_remove_old_builtin_roles_migration( Ok(()) } -fn add_new_remove_old_builtin_cluster_replicas_migration( +/// Converges each builtin cluster's replica set on the cluster's own managed +/// config. +/// +/// A builtin cluster is normally managed, and a managed cluster's replicas are +/// derived state: exactly `replication_factor` replicas, named by +/// [`managed_cluster_replica_name`]. One can be altered to unmanaged, which leaves +/// no factor to derive from, and then its replica set is the operator's. +/// +/// Replicas this creates are shaped from the cluster's config. An existing replica +/// is matched by name alone and left untouched, so this converges cardinality and +/// names rather than shape. An internal replica is never derived state and is left +/// alone entirely. +/// +/// The bootstrap flags seed `replication_factor` and `size` when a cluster is +/// first created (see [`add_new_remove_old_builtin_clusters_migration`]) and are +/// deliberately not consulted here, so an `ALTER CLUSTER` against a builtin +/// cluster survives a restart. +/// +/// This runs at catalog open so the replicas a cluster's config calls for exist as +/// early as possible. The coordinator's bootstrap brings up only replicas already +/// recorded durably and runs before the cluster controller is spawned, and the +/// controller does not run at all while a deployment is read-only. +/// +/// The controller derives its target from the same cluster config, so it converges +/// on the same replica set rather than competing for it. It excludes system +/// clusters today, but nothing here depends on that staying true. +fn reconcile_builtin_cluster_replicas( txn: &mut Transaction<'_>, builtin_cluster_config_map: &BuiltinBootstrapClusterConfigMap, boot_ts: Timestamp, ) -> Result<(), AdapterError> { - let cluster_lookup: BTreeMap<_, _> = txn - .get_clusters() - .map(|cluster| (cluster.name.clone(), cluster.clone())) + let builtin_cluster_names: BTreeSet<&str> = BUILTIN_CLUSTERS + .iter() + .map(|cluster| cluster.name) .collect(); - let cluster_id_to_name: BTreeMap = cluster_lookup - .values() - .map(|cluster| (cluster.id, cluster.name.clone())) + // Replicas of a cluster that is no longer a builtin need no handling here: + // `Transaction::remove_clusters` cascades to a cluster's replicas, and the + // clusters migration has already run against this transaction. + let clusters: Vec<_> = txn + .get_clusters() + .filter(|cluster| { + cluster.id.is_system() && builtin_cluster_names.contains(cluster.name.as_str()) + }) .collect(); - let mut durable_replicas: BTreeMap> = txn - .get_cluster_replicas() - .filter(|replica| replica.replica_id.is_system()) - .fold(BTreeMap::new(), |mut acc, replica| { - acc.entry(replica.cluster_id) - .or_insert_with(BTreeMap::new) - .insert(replica.name.to_string(), replica); - acc - }); + let builtin_cluster_ids: BTreeSet = + clusters.iter().map(|cluster| cluster.id).collect(); - // Add new replicas. - for builtin_replica in BUILTIN_CLUSTER_REPLICAS { - let cluster = cluster_lookup - .get(builtin_replica.cluster_name) - .expect("builtin cluster replica references non-existent cluster"); - // `empty_map` is a hack to simplify the if statement below. - let mut empty_map: BTreeMap = BTreeMap::new(); - let replica_names = durable_replicas - .get_mut(&cluster.id) - .unwrap_or(&mut empty_map); - - let builtin_cluster_bootstrap_config = - builtin_cluster_config_map.get_config(builtin_replica.cluster_name)?; - if replica_names.remove(builtin_replica.name).is_none() - // NOTE(SangJunBak): We need to explicitly check the replication factor because - // BUILT_IN_CLUSTER_REPLICAS is constant throughout all deployments but the replication - // factor is configurable on bootstrap. - && builtin_cluster_bootstrap_config.replication_factor > 0 - { - let replica_size = match cluster.config.variant { - ClusterVariant::Managed(ClusterVariantManaged { ref size, .. }) => size.clone(), - ClusterVariant::Unmanaged => builtin_cluster_bootstrap_config.size.clone(), - }; + // Only the builtin clusters' replicas. An environment can have thousands of + // user replicas and none of them are in scope here. + // + // An internal replica is deliberately out of scope. `CREATE CLUSTER REPLICA ... + // INTERNAL` is allowed on a managed cluster, and its name is barred from + // matching the derived `r1..rN` pattern so it cannot collide with one. It is a + // break-glass replica an operator added rather than derived state, so reaping it + // would undo that. The `ALTER CLUSTER` path skips it for the same reason. + let mut replicas_by_cluster: BTreeMap> = + BTreeMap::new(); + for replica in txn.get_cluster_replicas().filter(|replica| { + builtin_cluster_ids.contains(&replica.cluster_id) + && !matches!( + replica.config.location, + ReplicaLocation::Managed { internal: true, .. } + ) + }) { + replicas_by_cluster + .entry(replica.cluster_id) + .or_default() + .insert(replica.name.clone(), replica); + } + + let mut to_drop: Vec<(String, ClusterReplica)> = Vec::new(); + + for cluster in clusters { + // An unmanaged cluster has no replication factor, so there is no target to + // converge on. Its replica set is whatever an operator made it, and it is + // not ours to reshape. + let ClusterVariant::Managed(managed) = &cluster.config.variant else { + continue; + }; + + // The bootstrap flags seed a cluster at creation and are never re-applied, + // which is what keeps an `ALTER CLUSTER` from being reverted on every + // restart. That leaves an operator who changed a flag on an existing + // deployment with no feedback, so say so. + let bootstrap_config = builtin_cluster_config_map.get_config(&cluster.name)?; + if bootstrap_config.replication_factor != managed.replication_factor { + warn!( + cluster = %cluster.name, + configured_replication_factor = managed.replication_factor, + bootstrap_replication_factor = bootstrap_config.replication_factor, + "bootstrap replication factor is not applied to an already-existing \ + builtin cluster. Use ALTER CLUSTER ... SET (REPLICATION FACTOR ...) \ + to change it", + ); + } + + // Reading the cluster's factor is what makes this compose with the other + // writers of a replica set. The refresh scheduler parks a scheduled cluster + // by writing its factor to 0, so converging on the factor honors that + // instead of resurrecting a replica the scheduler just dropped. + let mut surplus = replicas_by_cluster.remove(&cluster.id).unwrap_or_default(); + for index in 0..managed.replication_factor { + let replica_name = managed_cluster_replica_name(index); + if surplus.remove(&replica_name).is_some() { + continue; + } - let config = builtin_cluster_replica_config(replica_size.clone()); // Builtin replicas live on system clusters. This runs inside the - // catalog-open transaction with no coordinator, so allocating from - // the same transaction is single-source and safe. + // catalog-open transaction with no coordinator, so allocating from the + // same transaction is single-source and safe. let replica_id = txn.allocate_system_replica_id()?; txn.insert_cluster_replica_with_id( cluster.id, replica_id, - builtin_replica.name, - config, - MZ_SYSTEM_ROLE_ID, + &replica_name, + managed_replica_config(managed), + // The cluster's owner, not `mz_system`. `mz_support` and + // `mz_analytics` are owned by their own roles, and replica + // ownership is checked against the replica's own `owner_id`, so + // stamping `mz_system` here would stop those roles from altering + // a replica of a cluster they own. + cluster.owner_id, )?; + info!( + cluster = %cluster.name, replica = %replica_name, %replica_id, + "creating builtin cluster replica to match the cluster's replication factor" + ); let audit_id = txn.allocate_audit_log_id()?; txn.insert_audit_log_event(VersionedEvent::new( @@ -1185,8 +1256,8 @@ fn add_new_remove_old_builtin_cluster_replicas_migration( cluster_id: cluster.id.to_string(), cluster_name: cluster.name.clone(), replica_id: Some(replica_id.to_string()), - replica_name: builtin_replica.name.to_string(), - logical_size: replica_size, + replica_name, + logical_size: managed.size.clone(), billed_as: None, internal: false, reason: CreateOrDropClusterReplicaReasonV1::System, @@ -1196,21 +1267,32 @@ fn add_new_remove_old_builtin_cluster_replicas_migration( boot_ts.into(), )); } + + // Whatever the config does not call for is surplus: a replica left over from + // a higher replication factor, or a `-pending` replica belonging to a + // graceful reconfiguration that a restart interrupted. Dropping the latter + // here means `remove_pending_cluster_replicas_migration` never sees it, + // which matches the abort semantics it would have applied anyway. + to_drop.extend( + surplus + .into_values() + .map(|replica| (cluster.name.clone(), replica)), + ); } - // Remove old replicas. - let old_replicas: Vec<_> = durable_replicas - .values() - .flat_map(|replicas| replicas.values()) + // Batched: `Transaction::remove_cluster_replica` is linear in the total replica + // count, so it must not be called in a loop. + let drop_ids = to_drop + .iter() + .map(|(_cluster_name, replica)| replica.replica_id) .collect(); - let old_replica_ids = old_replicas.iter().map(|r| r.replica_id).collect(); - txn.remove_cluster_replicas(&old_replica_ids)?; + txn.remove_cluster_replicas(&drop_ids)?; - for replica in &old_replicas { - let cluster_name = cluster_id_to_name - .get(&replica.cluster_id) - .cloned() - .unwrap_or_else(|| "".to_string()); + for (cluster_name, replica) in to_drop { + info!( + cluster = %cluster_name, replica = %replica.name, replica_id = %replica.replica_id, + "dropping builtin cluster replica not called for by the cluster's replication factor" + ); let audit_id = txn.allocate_audit_log_id()?; txn.insert_audit_log_event(VersionedEvent::new( @@ -1221,7 +1303,7 @@ fn add_new_remove_old_builtin_cluster_replicas_migration( cluster_id: replica.cluster_id.to_string(), cluster_name, replica_id: Some(replica.replica_id.to_string()), - replica_name: replica.name.clone(), + replica_name: replica.name, reason: CreateOrDropClusterReplicaReasonV1::System, scheduling_policies: None, }), @@ -1233,6 +1315,47 @@ fn add_new_remove_old_builtin_cluster_replicas_migration( Ok(()) } +/// The replica config implied by a managed cluster's config. +/// +/// Every field a replica shares with its cluster is taken from the cluster. +/// `availability_zones` is the cluster's pool, which is what the provisioning +/// paths stamp onto a managed replica. +/// +/// Two things the `ALTER CLUSTER` path does that this does not. It normalizes +/// logging through the plan, which discards `log_logging` when no interval is +/// set, so for the odd cluster config that has debugging on and logging off the +/// two produce different records. And it validates the size against the replica +/// size map and the role's allowed sizes, which cannot be done from here because +/// the size map is not part of the durable catalog. A builtin cluster's size only +/// ever comes from a bootstrap flag or an `ALTER` that already validated it. +fn managed_replica_config(managed: &ClusterVariantManaged) -> ReplicaConfig { + // Exhaustive destructure (no `..`): a field added to the managed config is a + // compile error here until we decide whether a replica has to carry it. + let ClusterVariantManaged { + size, + availability_zones, + logging, + arrangement_compression, + replication_factor: _, + optimizer_feature_overrides: _, + schedule: _, + auto_scaling_strategy: _, + reconfiguration: _, + burst: _, + } = managed; + ReplicaConfig { + location: ReplicaLocation::Managed { + size: size.clone(), + availability_zones: availability_zones.clone(), + billed_as: None, + internal: false, + pending: false, + }, + logging: logging.clone(), + arrangement_compression: *arrangement_compression, + } +} + /// Roles can have default values for configuration parameters, e.g. you can set a Role default for /// the 'cluster' parameter. /// @@ -1338,22 +1461,6 @@ fn remove_pending_cluster_replicas_migration( Ok(()) } -pub(crate) fn builtin_cluster_replica_config( - replica_size: String, -) -> mz_catalog::durable::ReplicaConfig { - mz_catalog::durable::ReplicaConfig { - location: mz_catalog::durable::ReplicaLocation::Managed { - availability_zones: Vec::new(), - billed_as: None, - pending: false, - internal: false, - size: replica_size, - }, - logging: default_logging_config(), - arrangement_compression: false, - } -} - fn default_logging_config() -> ReplicaLogging { ReplicaLogging { log_logging: false, @@ -1430,3 +1537,64 @@ pub(crate) fn into_consolidatable_updates_startup( }) .collect() } + +#[cfg(test)] +mod tests { + use mz_catalog::durable::ClusterVariantManaged; + + use super::*; + + /// A replica the reconciler creates takes every shared field from its + /// cluster. Nothing here may fall back to a default, or a builtin cluster + /// ends up running a replica that disagrees with its own config. + #[mz_ore::test] + fn test_managed_replica_config_derives_every_shared_field() { + let managed = ClusterVariantManaged { + size: "somesize".into(), + availability_zones: vec!["az1".into(), "az2".into()], + logging: ReplicaLogging { + log_logging: true, + interval: Some(Duration::from_millis(10)), + }, + arrangement_compression: true, + replication_factor: 3, + optimizer_feature_overrides: Default::default(), + schedule: Default::default(), + auto_scaling_strategy: None, + reconfiguration: None, + burst: None, + }; + + let config = managed_replica_config(&managed); + + // Exhaustive destructures throughout, so a field added to either type is a + // compile error here rather than a silently unasserted one. + let ReplicaConfig { + location, + logging, + arrangement_compression, + } = config; + + assert_eq!(logging, managed.logging); + assert_eq!(arrangement_compression, managed.arrangement_compression); + match location { + ReplicaLocation::Managed { + size, + availability_zones, + billed_as, + internal, + pending, + } => { + assert_eq!(size, managed.size); + assert_eq!(availability_zones, managed.availability_zones); + // A builtin replica is neither manually managed nor part of an + // in-flight reconfiguration, and those three traits are what + // exclude a replica from controller ownership. + assert_eq!(billed_as, None); + assert!(!internal); + assert!(!pending); + } + ReplicaLocation::Unmanaged { .. } => panic!("expected a managed location"), + } + } +} diff --git a/src/adapter/src/catalog/open/builtin_schema_migration.rs b/src/adapter/src/catalog/open/builtin_schema_migration.rs index bda72b0eafd1f..77da28cefcd23 100644 --- a/src/adapter/src/catalog/open/builtin_schema_migration.rs +++ b/src/adapter/src/catalog/open/builtin_schema_migration.rs @@ -301,6 +301,33 @@ static MIGRATIONS: LazyLock> = LazyLock::new(|| { MZ_CATALOG_SCHEMA, "mz_kafka_sources", ), + // Converting the four mz_*_source_tables from builtin tables to + // materialized views changes their catalog fingerprint, so each needs an + // explicit replacement step. + MigrationStep::replacement( + "26.37.0-dev.0", + CatalogItemType::MaterializedView, + MZ_INTERNAL_SCHEMA, + "mz_postgres_source_tables", + ), + MigrationStep::replacement( + "26.37.0-dev.0", + CatalogItemType::MaterializedView, + MZ_INTERNAL_SCHEMA, + "mz_mysql_source_tables", + ), + MigrationStep::replacement( + "26.37.0-dev.0", + CatalogItemType::MaterializedView, + MZ_INTERNAL_SCHEMA, + "mz_sql_server_source_tables", + ), + MigrationStep::replacement( + "26.37.0-dev.0", + CatalogItemType::MaterializedView, + MZ_INTERNAL_SCHEMA, + "mz_kafka_source_tables", + ), ] }); diff --git a/src/adapter/src/coord/cluster_controller.rs b/src/adapter/src/coord/cluster_controller.rs index 1b4247bb574f0..be53782f19608 100644 --- a/src/adapter/src/coord/cluster_controller.rs +++ b/src/adapter/src/coord/cluster_controller.rs @@ -22,8 +22,9 @@ //! the gate off the task does not tick, so the legacy scheduling and graceful //! paths remain the sole writers of the replica set. With the gate on the //! controller owns the *user* managed-cluster replica set; the legacy entry -//! points no-op. (System/builtin clusters are never controller-owned. The -//! catalog's bootstrap migration owns their replicas.) +//! points no-op. (System/builtin clusters are excluded here. Their config-implied +//! replicas are materialized by `reconcile_builtin_cluster_replicas` at catalog +//! open, which derives the same target from the same config.) use std::collections::BTreeSet; use std::sync::Arc; @@ -58,8 +59,8 @@ use crate::error::AdapterError; #[derive(Debug)] pub enum ClusterControllerRequest { /// The ids of all *user* managed clusters the controller owns this tick. - /// System/builtin clusters are excluded. Their replica set is owned by the - /// catalog's bootstrap migration, not the controller. + /// System/builtin clusters are excluded. `reconcile_builtin_cluster_replicas` + /// has already materialized their config-implied replicas at catalog open. ManagedClusterIds { tx: oneshot::Sender> }, /// A consistent durable view of the given clusters and their replicas, plus /// the current time. @@ -277,17 +278,13 @@ impl Coordinator { self.catalog() .clusters() // Only *user* managed clusters. System/builtin clusters - // (mz_system, mz_catalog_server, …) are also managed, but - // their replica set is owned by the bootstrap migration - // (`add_new_remove_old_builtin_cluster_replicas_migration`), - // which holds exactly the `BUILTIN_CLUSTER_REPLICAS`-defined - // replicas regardless of the cluster's `replication_factor`. - // Letting the controller own them too would make two writers - // of one replica set: the baseline would, for example, add a - // replica to reach a builtin cluster's `replication_factor`, - // which the bootstrap migration then tears down on the next - // open. The legacy scheduler likewise only ever acted on user - // clusters. + // (mz_system, mz_catalog_server, …) are also managed, and + // `reconcile_builtin_cluster_replicas` materializes their + // config-implied replica set at catalog open, so those + // replicas exist before the controller could ever tick. + // Both derive the target from the cluster's + // `replication_factor`, so extending ownership here would + // converge rather than conflict. .filter(|c| c.is_managed() && c.id.is_user()) .map(|c| c.id) .collect() diff --git a/src/adapter/src/coord/sequencer/inner/cluster.rs b/src/adapter/src/coord/sequencer/inner/cluster.rs index 599dc2fe2d5c9..06232d8765f00 100644 --- a/src/adapter/src/coord/sequencer/inner/cluster.rs +++ b/src/adapter/src/coord/sequencer/inner/cluster.rs @@ -14,6 +14,7 @@ use itertools::Itertools; use maplit::btreeset; use mz_adapter_types::cluster_state::ReconfigurationAudit; use mz_catalog::builtin::BUILTINS; +use mz_catalog::durable::managed_cluster_replica_name; use mz_catalog::memory::objects::{ ClusterConfig, ClusterReplica, ClusterVariant, ClusterVariantManaged, ManagedReplicaConfigShape, ReconfigurationState, ReconfigurationStatus, ReconfigurationTarget, @@ -2566,10 +2567,6 @@ impl Coordinator { } } -fn managed_cluster_replica_name(index: u32) -> String { - format!("r{}", index + 1) -} - /// Which reconfiguration-target dimensions an `ALTER` left unset (`Unchanged`). /// Drives [`fold_reconfiguration_target`]. Logging is two sub-dimensions /// because `INTROSPECTION DEBUGGING` and `INTROSPECTION INTERVAL` are diff --git a/src/avro/src/decode.rs b/src/avro/src/decode.rs index 9039b32ce9ffc..74adcf9ad6f94 100644 --- a/src/avro/src/decode.rs +++ b/src/avro/src/decode.rs @@ -1048,6 +1048,54 @@ fn min_encoded_len_piece( } } +/// Bounds the object count an object-container-file block declares against the +/// payload that is supposed to hold those objects. +/// +/// The count comes straight off the wire, and `util::safe_len` alone caps it at +/// `MAX_ALLOCATION_BYTES` — a sensible ceiling for a byte length, an enormous one +/// for a count, and in no way related to the block it describes. `payload_len` is +/// the block's *decompressed* length. +/// +/// Which bound applies is decided the same way as for an array block, and for the +/// same reason (see [`min_encoded_len`]): the byte floor when the object schema has +/// a proven positive one, otherwise the node-weighted cap, because a zero-width +/// object — an empty record, or a record of only `null`/empty-record fields — +/// encodes to no bytes at all, so no payload length can constrain how many of them +/// a block may claim. +/// +/// Unlike an array block, the two bounds are *alternatives* rather than both being +/// applied. An array materializes all of its elements at once, so its cap is about +/// retained memory; a block's objects are yielded and dropped one at a time, so the +/// cap here is about work amplified out of a few bytes. Once the byte floor holds, +/// the work is linear in the file and needs no further ceiling — and capping nodes +/// as well would reject a legitimate large block of small records. +/// +/// Not charged against [`DECODE_NODES`]: that budget is per top-level datum and +/// each object in a block is its own datum. This is a standalone check on the +/// block header. +pub(crate) fn bound_block_object_count( + schema: SchemaNode, + count: usize, + payload_len: usize, +) -> Result<(), AvroError> { + let min_bytes = min_encoded_len(schema); + if min_bytes > 0 { + if count.saturating_mul(min_bytes) > payload_len { + return Err(AvroError::Decode(DecodeError::Custom(format!( + "Avro block object count {count} exceeds block payload ({payload_len} bytes)" + )))); + } + return Ok(()); + } + let nodes = count.saturating_mul(min_value_nodes(schema)); + if nodes > MAX_VALUE_NODES { + return Err(AvroError::Decode(DecodeError::Custom(format!( + "Avro block object count {count} exceeds limit {MAX_VALUE_NODES} decoded values" + )))); + } + Ok(()) +} + /// A *lower* bound on the number of `Value` nodes a single value of `schema` /// materializes into when decoded. /// diff --git a/src/avro/src/reader.rs b/src/avro/src/reader.rs index 44352e3da5f28..03a86a065ef07 100644 --- a/src/avro/src/reader.rs +++ b/src/avro/src/reader.rs @@ -29,7 +29,7 @@ use std::str::{FromStr, from_utf8}; use aws_lc_rs::digest; use serde_json::from_slice; -use crate::decode::{AvroRead, decode}; +use crate::decode::{AvroRead, bound_block_object_count, decode}; use crate::error::{DecodeError, Error as AvroError}; use crate::schema::{ FullName, NamedSchemaPiece, ParseSchemaError, RecordField, ResolvedDefaultValueField, @@ -259,6 +259,16 @@ impl Reader { } self.inner.read_exact(&mut self.buf[..n])?; + // Cut the buffer down to exactly this block's payload. The resize above + // only ever grows, so a block shorter than a previous one would otherwise + // leave that block's tail visible past its own payload, and everything + // downstream reads the buffer by its length: `read_next` slices + // `self.buf[self.buf_idx..]`, and `Codec::decompress` hands the whole + // buffer to the decompressor. A block whose declared object count outruns + // its own bytes would then decode stale bytes as values instead of hitting + // the end of the block. `truncate` keeps the allocation, so the buffer is + // still reused across blocks. + self.buf.truncate(n); self.buf_idx = 0; Ok(()) } @@ -295,6 +305,18 @@ impl Reader { // into the buffer. But this is fine, for now. self.header.codec.decompress(&mut self.buf)?; + // `safe_len` above bounds the object count only by + // `MAX_ALLOCATION_BYTES`, which says nothing about the block that + // is supposed to contain those objects: a zero-width schema + // encodes every object to no bytes, so a handful of wire bytes can + // claim hundreds of millions of them and the reader will decode + // every one. Bound it against the payload now that the payload is + // known, which is only here, since the declared byte size is the + // *compressed* size. + let count = self.messages_remaining; + let payload_len = self.buf.len(); + bound_block_object_count(self.schema().top_node(), count, payload_len)?; + Ok(()) } Err(e) => { @@ -957,6 +979,110 @@ mod tests { use super::*; + /// Assemble an object-container file: header (writer schema, `null` codec, + /// sync marker) followed by `blocks`. Each block is given as `(declared + /// count, declared byte size, payload)` so a test can lie about the framing + /// the way a corrupt or hostile file does. + fn ocf(schema_json: &str, blocks: &[(i64, i64, &[u8])]) -> Vec { + fn blob(bytes: &[u8], out: &mut Vec) { + util::zig_i64(bytes.len() as i64, out); + out.extend_from_slice(bytes); + } + + let marker = [7u8; 16]; + let mut out = b"Obj\x01".to_vec(); + util::zig_i64(2, &mut out); // metadata map: one block of two entries + blob(b"avro.schema", &mut out); + blob(schema_json.as_bytes(), &mut out); + blob(b"avro.codec", &mut out); + blob(b"null", &mut out); + util::zig_i64(0, &mut out); // end of metadata map + out.extend_from_slice(&marker); + for (count, size, payload) in blocks { + util::zig_i64(*count, &mut out); + util::zig_i64(*size, &mut out); + out.extend_from_slice(payload); + out.extend_from_slice(&marker); + } + out + } + + /// A record of only `null` fields: valid, and encodes to zero bytes. + const ZERO_WIDTH_SCHEMA: &str = + r#"{"type":"record","name":"R","fields":[{"name":"g","type":"null"}]}"#; + + #[mz_ore::test] + fn reader_does_not_decode_a_previous_blocks_bytes() { + // One buffer is reused across blocks, so a block shorter than its + // predecessor must not leave that predecessor's tail readable. Block 2 + // here declares two objects but carries only one, and its payload is + // shorter than block 1's, whose bytes are arranged so that what lands + // past block 2's payload would decode as a perfectly good `"x"`. The + // second decode has to run out of input instead. + // + // The block bound does not catch this: `string` has a one-byte floor and + // the block claims 2 objects in 3 bytes. + let long = &[0x0a, b'A', b'B', 0x02, b'x', b'C'][..]; // one 5-char string + let short = &[0x04, b'a', b'b'][..]; // one 2-char string + let file = ocf( + r#""string""#, + &[(1, long.len() as i64, long), (2, short.len() as i64, short)], + ); + + let items: Vec<_> = Reader::new(&file[..]).expect("OCF header parses").collect(); + assert_eq!(items.len(), 3, "expected two values then an error"); + assert_eq!( + items[1].as_ref().expect("block 2's real value decodes"), + &Value::String("ab".into()) + ); + assert_err!(&items[2]); + } + + #[mz_ore::test] + fn reader_rejects_zero_width_block_count_within_allocation_budget() { + // `safe_len` accepts any count up to `MAX_ALLOCATION_BYTES`, and a + // zero-width object consumes no input, so nothing but the block bound + // stops a 0-byte payload from claiming 100M objects and being believed. + // Regression for a reader_decode cargo-fuzz timeout. + let file = ocf(ZERO_WIDTH_SCHEMA, &[(100_000_000, 0, &[])]); + let err = Reader::new(&file[..]) + .expect("OCF header parses") + .find_map(|item| item.err()) + .expect("the oversized count must be rejected"); + assert!( + err.to_string().contains("exceeds limit"), + "unexpected error: {err}" + ); + } + + #[mz_ore::test] + fn reader_accepts_honest_zero_width_block() { + // The bound above must not reject a real zero-width block: the whole + // point of the node weighting is that a byte count cannot judge one. + let file = ocf(ZERO_WIDTH_SCHEMA, &[(3, 0, &[])]); + let values: Vec = Reader::new(&file[..]) + .expect("OCF header parses") + .collect::>() + .expect("an honest zero-width block must decode"); + assert_eq!(values.len(), 3); + } + + #[mz_ore::test] + fn reader_rejects_block_count_exceeding_payload() { + // A `string` occupies at least its one-byte length varint, so 100 of them + // cannot fit in three bytes however the payload is arranged. + let payload = &[0x04, b'a', b'b'][..]; + let file = ocf(r#""string""#, &[(100, payload.len() as i64, payload)]); + let err = Reader::new(&file[..]) + .expect("OCF header parses") + .find_map(|item| item.err()) + .expect("a count larger than the payload must be rejected"); + assert!( + err.to_string().contains("exceeds block payload"), + "unexpected error: {err}" + ); + } + #[mz_ore::test] fn reader_rejects_huge_block_object_count() { // A crafted object-container block with a huge object count and a diff --git a/src/catalog/src/builtin.rs b/src/catalog/src/builtin.rs index 931aac1423a17..2ec708348e946 100644 --- a/src/catalog/src/builtin.rs +++ b/src/catalog/src/builtin.rs @@ -66,7 +66,6 @@ use crate::durable::objects::SystemObjectDescription; use crate::memory::objects::DataSourceDesc; pub const BUILTIN_PREFIXES: &[&str] = &["mz_", "pg_", "external_"]; -const BUILTIN_CLUSTER_REPLICA_NAME: &str = "r1"; /// A sentinel used in place of a fingerprint that indicates that a builtin /// object is runtime alterable. Runtime alterable objects don't have meaningful @@ -696,14 +695,6 @@ pub struct BuiltinCluster { pub owner_id: &'static RoleId, } -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct BuiltinClusterReplica { - /// Name of the compute replica. - pub name: &'static str, - /// Name of the cluster that this replica belongs to. - pub cluster_name: &'static str, -} - /// Uniquely identifies the definition of a builtin object. pub trait Fingerprint { fn fingerprint(&self) -> String; @@ -906,11 +897,6 @@ pub const MZ_SYSTEM_CLUSTER: BuiltinCluster = BuiltinCluster { ], }; -pub const MZ_SYSTEM_CLUSTER_REPLICA: BuiltinClusterReplica = BuiltinClusterReplica { - name: BUILTIN_CLUSTER_REPLICA_NAME, - cluster_name: MZ_SYSTEM_CLUSTER.name, -}; - pub const MZ_CATALOG_SERVER_CLUSTER: BuiltinCluster = BuiltinCluster { name: "mz_catalog_server", owner_id: &MZ_SYSTEM_ROLE_ID, @@ -929,11 +915,6 @@ pub const MZ_CATALOG_SERVER_CLUSTER: BuiltinCluster = BuiltinCluster { ], }; -pub const MZ_CATALOG_SERVER_CLUSTER_REPLICA: BuiltinClusterReplica = BuiltinClusterReplica { - name: BUILTIN_CLUSTER_REPLICA_NAME, - cluster_name: MZ_CATALOG_SERVER_CLUSTER.name, -}; - pub const MZ_PROBE_CLUSTER: BuiltinCluster = BuiltinCluster { name: "mz_probe", owner_id: &MZ_SYSTEM_ROLE_ID, @@ -951,10 +932,6 @@ pub const MZ_PROBE_CLUSTER: BuiltinCluster = BuiltinCluster { rbac::owner_privilege(ObjectType::Cluster, MZ_SYSTEM_ROLE_ID), ], }; -pub const MZ_PROBE_CLUSTER_REPLICA: BuiltinClusterReplica = BuiltinClusterReplica { - name: BUILTIN_CLUSTER_REPLICA_NAME, - cluster_name: MZ_PROBE_CLUSTER.name, -}; pub const MZ_SUPPORT_CLUSTER: BuiltinCluster = BuiltinCluster { name: "mz_support", @@ -1144,10 +1121,10 @@ pub static BUILTINS_STATIC: LazyLock>> = LazyLock::ne // mz_sources is generated dynamically below with inlined builtin VALUES. Builtin::Table(&MZ_SOURCE_REFERENCES), Builtin::MaterializedView(&MZ_POSTGRES_SOURCES), - Builtin::Table(&MZ_POSTGRES_SOURCE_TABLES), - Builtin::Table(&MZ_MYSQL_SOURCE_TABLES), - Builtin::Table(&MZ_SQL_SERVER_SOURCE_TABLES), - Builtin::Table(&MZ_KAFKA_SOURCE_TABLES), + Builtin::MaterializedView(&MZ_POSTGRES_SOURCE_TABLES), + Builtin::MaterializedView(&MZ_MYSQL_SOURCE_TABLES), + Builtin::MaterializedView(&MZ_SQL_SERVER_SOURCE_TABLES), + Builtin::MaterializedView(&MZ_KAFKA_SOURCE_TABLES), Builtin::Table(&MZ_SINKS), Builtin::Table(&MZ_VIEWS), Builtin::Table(&MZ_TYPES), @@ -1576,11 +1553,6 @@ pub const BUILTIN_CLUSTERS: &[&BuiltinCluster] = &[ &MZ_SUPPORT_CLUSTER, &MZ_ANALYTICS_CLUSTER, ]; -pub const BUILTIN_CLUSTER_REPLICAS: &[&BuiltinClusterReplica] = &[ - &MZ_SYSTEM_CLUSTER_REPLICA, - &MZ_CATALOG_SERVER_CLUSTER_REPLICA, - &MZ_PROBE_CLUSTER_REPLICA, -]; #[allow(non_snake_case)] pub mod BUILTINS { diff --git a/src/catalog/src/builtin/mz_internal.rs b/src/catalog/src/builtin/mz_internal.rs index 6601c93c44db7..9ad99a5e417b1 100644 --- a/src/catalog/src/builtin/mz_internal.rs +++ b/src/catalog/src/builtin/mz_internal.rs @@ -116,164 +116,280 @@ WHERE }), } }); -pub static MZ_POSTGRES_SOURCE_TABLES: LazyLock = LazyLock::new(|| BuiltinTable { - name: "mz_postgres_source_tables", - schema: MZ_INTERNAL_SCHEMA, - oid: oid::TABLE_MZ_POSTGRES_SOURCE_TABLES_OID, - desc: RelationDesc::builder() - .with_column("id", SqlScalarType::String.nullable(false)) - .with_column("schema_name", SqlScalarType::String.nullable(false)) - .with_column("table_name", SqlScalarType::String.nullable(false)) - .finish(), - column_comments: BTreeMap::from_iter([ - ( - "id", - "The ID of the subsource or table. Corresponds to `mz_catalog.mz_sources.id` or `mz_catalog.mz_tables.id`.", - ), - ( - "schema_name", - "The schema of the upstream table being ingested.", - ), - ( - "table_name", - "The name of the upstream table being ingested.", - ), - ]), - is_retained_metrics_object: true, - access: vec![PUBLIC_SELECT], - ontology: Some(Ontology { - entity_name: "postgres_source_table", - description: "Postgres source table-level details", - links: &const { - [OntologyLink { - name: "describes_source_table", - target: "table", - properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), - }] - }, - column_semantic_types: &[("id", SemanticType::CatalogItemId)], - }), +// The three relational source-table views (postgres, mysql, sql-server) share +// the same shape. Each reads Item rows from `mz_catalog_raw`, pulls the parent +// source id and external reference out of the persisted `create_sql` via +// `parse_source_export_details`, and joins `mz_sources` to keep only exports +// whose parent is of the matching connection type. The external reference for +// postgres and sql-server is `[database, schema, table]`, so schema/table come +// from positions 1 and 2. MySQL references are `[schema, table]`, so positions +// 0 and 1. This is the same slicing the removed packers applied. +pub static MZ_POSTGRES_SOURCE_TABLES: LazyLock = LazyLock::new(|| { + BuiltinMaterializedView { + name: "mz_postgres_source_tables", + schema: MZ_INTERNAL_SCHEMA, + oid: oid::MV_MZ_POSTGRES_SOURCE_TABLES_OID, + desc: RelationDesc::builder() + .with_column("id", SqlScalarType::String.nullable(false)) + .with_column("schema_name", SqlScalarType::String.nullable(false)) + .with_column("table_name", SqlScalarType::String.nullable(false)) + .with_key(vec![0]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "id", + "The ID of the subsource or table. Corresponds to `mz_catalog.mz_sources.id` or `mz_catalog.mz_tables.id`.", + ), + ( + "schema_name", + "The schema of the upstream table being ingested.", + ), + ( + "table_name", + "The name of the upstream table being ingested.", + ), + ]), + sql: " +IN CLUSTER mz_catalog_server +WITH ( + ASSERT NOT NULL id, + ASSERT NOT NULL schema_name, + ASSERT NOT NULL table_name +) AS +SELECT + mz_internal.parse_catalog_id(r.data->'key'->'gid') AS id, + details->'external_reference'->>1 AS schema_name, + details->'external_reference'->>2 AS table_name +FROM + mz_internal.mz_catalog_raw r, + LATERAL ( + SELECT mz_internal.parse_source_export_details( + r.data->'value'->'definition'->'V1'->>'create_sql') + ) AS d(details) + JOIN mz_catalog.mz_sources s + ON s.id = details->>'source_id' AND s.type = 'postgres' +WHERE + r.data->>'kind' = 'Item' AND + details IS NOT NULL", + is_retained_metrics_object: true, + access: vec![PUBLIC_SELECT], + ontology: Some(Ontology { + entity_name: "postgres_source_table", + description: "Postgres source table-level details", + links: &const { + [OntologyLink { + name: "describes_source_table", + target: "table", + properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), + }] + }, + column_semantic_types: &[("id", SemanticType::CatalogItemId)], + }), + } }); -pub static MZ_MYSQL_SOURCE_TABLES: LazyLock = LazyLock::new(|| BuiltinTable { - name: "mz_mysql_source_tables", - schema: MZ_INTERNAL_SCHEMA, - oid: oid::TABLE_MZ_MYSQL_SOURCE_TABLES_OID, - desc: RelationDesc::builder() - .with_column("id", SqlScalarType::String.nullable(false)) - .with_column("schema_name", SqlScalarType::String.nullable(false)) - .with_column("table_name", SqlScalarType::String.nullable(false)) - .finish(), - column_comments: BTreeMap::from_iter([ - ( - "id", - "The ID of the subsource or table. Corresponds to `mz_catalog.mz_sources.id` or `mz_catalog.mz_tables.id`.", - ), - ( - "schema_name", - "The schema (or, database) of the upstream table being ingested.", - ), - ( - "table_name", - "The name of the upstream table being ingested.", - ), - ]), - is_retained_metrics_object: true, - access: vec![PUBLIC_SELECT], - ontology: Some(Ontology { - entity_name: "mysql_source_table", - description: "MySQL source table-level details", - links: &const { - [OntologyLink { - name: "describes_source_table", - target: "table", - properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), - }] - }, - column_semantic_types: &[("id", SemanticType::CatalogItemId)], - }), +pub static MZ_MYSQL_SOURCE_TABLES: LazyLock = LazyLock::new(|| { + BuiltinMaterializedView { + name: "mz_mysql_source_tables", + schema: MZ_INTERNAL_SCHEMA, + oid: oid::MV_MZ_MYSQL_SOURCE_TABLES_OID, + desc: RelationDesc::builder() + .with_column("id", SqlScalarType::String.nullable(false)) + .with_column("schema_name", SqlScalarType::String.nullable(false)) + .with_column("table_name", SqlScalarType::String.nullable(false)) + .with_key(vec![0]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "id", + "The ID of the subsource or table. Corresponds to `mz_catalog.mz_sources.id` or `mz_catalog.mz_tables.id`.", + ), + ( + "schema_name", + "The schema (or, database) of the upstream table being ingested.", + ), + ( + "table_name", + "The name of the upstream table being ingested.", + ), + ]), + sql: " +IN CLUSTER mz_catalog_server +WITH ( + ASSERT NOT NULL id, + ASSERT NOT NULL schema_name, + ASSERT NOT NULL table_name +) AS +SELECT + mz_internal.parse_catalog_id(r.data->'key'->'gid') AS id, + details->'external_reference'->>0 AS schema_name, + details->'external_reference'->>1 AS table_name +FROM + mz_internal.mz_catalog_raw r, + LATERAL ( + SELECT mz_internal.parse_source_export_details( + r.data->'value'->'definition'->'V1'->>'create_sql') + ) AS d(details) + JOIN mz_catalog.mz_sources s + ON s.id = details->>'source_id' AND s.type = 'mysql' +WHERE + r.data->>'kind' = 'Item' AND + details IS NOT NULL", + is_retained_metrics_object: true, + access: vec![PUBLIC_SELECT], + ontology: Some(Ontology { + entity_name: "mysql_source_table", + description: "MySQL source table-level details", + links: &const { + [OntologyLink { + name: "describes_source_table", + target: "table", + properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), + }] + }, + column_semantic_types: &[("id", SemanticType::CatalogItemId)], + }), + } }); -pub static MZ_SQL_SERVER_SOURCE_TABLES: LazyLock = LazyLock::new(|| BuiltinTable { - name: "mz_sql_server_source_tables", - schema: MZ_INTERNAL_SCHEMA, - oid: oid::TABLE_MZ_SQL_SERVER_SOURCE_TABLES_OID, - desc: RelationDesc::builder() - .with_column("id", SqlScalarType::String.nullable(false)) - .with_column("schema_name", SqlScalarType::String.nullable(false)) - .with_column("table_name", SqlScalarType::String.nullable(false)) - .finish(), - column_comments: BTreeMap::from_iter([ - ( - "id", - "The ID of the subsource or table. Corresponds to `mz_catalog.mz_sources.id` or `mz_catalog.mz_tables.id`.", - ), - ( - "schema_name", - "The schema of the upstream table being ingested.", - ), - ( - "table_name", - "The name of the upstream table being ingested.", - ), - ]), - is_retained_metrics_object: true, - access: vec![PUBLIC_SELECT], - ontology: Some(Ontology { - entity_name: "sql_server_source_table", - description: "SQL Server source table-level details", - links: &const { - [OntologyLink { - name: "describes_source_table", - target: "table", - properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), - }] - }, - column_semantic_types: &[("id", SemanticType::CatalogItemId)], - }), +pub static MZ_SQL_SERVER_SOURCE_TABLES: LazyLock = LazyLock::new(|| { + BuiltinMaterializedView { + name: "mz_sql_server_source_tables", + schema: MZ_INTERNAL_SCHEMA, + oid: oid::MV_MZ_SQL_SERVER_SOURCE_TABLES_OID, + desc: RelationDesc::builder() + .with_column("id", SqlScalarType::String.nullable(false)) + .with_column("schema_name", SqlScalarType::String.nullable(false)) + .with_column("table_name", SqlScalarType::String.nullable(false)) + .with_key(vec![0]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "id", + "The ID of the subsource or table. Corresponds to `mz_catalog.mz_sources.id` or `mz_catalog.mz_tables.id`.", + ), + ( + "schema_name", + "The schema of the upstream table being ingested.", + ), + ( + "table_name", + "The name of the upstream table being ingested.", + ), + ]), + sql: " +IN CLUSTER mz_catalog_server +WITH ( + ASSERT NOT NULL id, + ASSERT NOT NULL schema_name, + ASSERT NOT NULL table_name +) AS +SELECT + mz_internal.parse_catalog_id(r.data->'key'->'gid') AS id, + details->'external_reference'->>1 AS schema_name, + details->'external_reference'->>2 AS table_name +FROM + mz_internal.mz_catalog_raw r, + LATERAL ( + SELECT mz_internal.parse_source_export_details( + r.data->'value'->'definition'->'V1'->>'create_sql') + ) AS d(details) + JOIN mz_catalog.mz_sources s + ON s.id = details->>'source_id' AND s.type = 'sql-server' +WHERE + r.data->>'kind' = 'Item' AND + details IS NOT NULL", + is_retained_metrics_object: true, + access: vec![PUBLIC_SELECT], + ontology: Some(Ontology { + entity_name: "sql_server_source_table", + description: "SQL Server source table-level details", + links: &const { + [OntologyLink { + name: "describes_source_table", + target: "table", + properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), + }] + }, + column_semantic_types: &[("id", SemanticType::CatalogItemId)], + }), + } }); -pub static MZ_KAFKA_SOURCE_TABLES: LazyLock = LazyLock::new(|| BuiltinTable { - name: "mz_kafka_source_tables", - schema: MZ_INTERNAL_SCHEMA, - oid: oid::TABLE_MZ_KAFKA_SOURCE_TABLES_OID, - desc: RelationDesc::builder() - .with_column("id", SqlScalarType::String.nullable(false)) - .with_column("topic", SqlScalarType::String.nullable(false)) - .with_column("envelope_type", SqlScalarType::String.nullable(true)) - .with_column("key_format", SqlScalarType::String.nullable(true)) - .with_column("value_format", SqlScalarType::String.nullable(true)) - .finish(), - column_comments: BTreeMap::from_iter([ - ( - "id", - "The ID of the table. Corresponds to `mz_catalog.mz_tables.id`.", - ), - ("topic", "The topic being ingested."), - ( - "envelope_type", - "The envelope type: `none`, `upsert`, or `debezium`. `NULL` for other source types.", - ), - ( - "key_format", - "The format of the Kafka message key: `avro`, `csv`, `regex`, `bytes`, `json`, `text`, or `NULL`.", - ), - ( - "value_format", - "The format of the Kafka message value: `avro`, `csv`, `regex`, `bytes`, `json`, `text`. `NULL` for other source types.", - ), - ]), - is_retained_metrics_object: true, - access: vec![PUBLIC_SELECT], - ontology: Some(Ontology { - entity_name: "kafka_source_table", - description: "Kafka source table-level details", - links: &const { - [OntologyLink { - name: "describes_source_table", - target: "table", - properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), - }] - }, - column_semantic_types: &[("id", SemanticType::CatalogItemId)], - }), +pub static MZ_KAFKA_SOURCE_TABLES: LazyLock = LazyLock::new(|| { + BuiltinMaterializedView { + name: "mz_kafka_source_tables", + schema: MZ_INTERNAL_SCHEMA, + oid: oid::MV_MZ_KAFKA_SOURCE_TABLES_OID, + desc: RelationDesc::builder() + .with_column("id", SqlScalarType::String.nullable(false)) + .with_column("topic", SqlScalarType::String.nullable(false)) + .with_column("envelope_type", SqlScalarType::String.nullable(true)) + .with_column("key_format", SqlScalarType::String.nullable(true)) + .with_column("value_format", SqlScalarType::String.nullable(true)) + .with_key(vec![0]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "id", + "The ID of the table. Corresponds to `mz_catalog.mz_tables.id`.", + ), + ("topic", "The topic being ingested."), + ( + "envelope_type", + "The envelope type: `none`, `upsert`, or `debezium`. `NULL` for other source types.", + ), + ( + "key_format", + "The format of the Kafka message key: `avro`, `csv`, `regex`, `bytes`, `json`, `text`, or `NULL`.", + ), + ( + "value_format", + "The format of the Kafka message value: `avro`, `csv`, `regex`, `bytes`, `json`, `text`. `NULL` for other source types.", + ), + ]), + // Kafka exports are only ever created with the new + // `CREATE TABLE ... FROM SOURCE` syntax (kafka has no subsource path), + // so the topic sits at position 0 of the single-part external + // reference. `parse_source_export_details` resolves the envelope and + // key/value formats straight from the table's own `create_sql`, + // reproducing the runtime `DataSourceDesc::formats()`/`envelope()` the + // old packer read. + sql: " +IN CLUSTER mz_catalog_server +WITH ( + ASSERT NOT NULL id, + ASSERT NOT NULL topic +) AS +SELECT + mz_internal.parse_catalog_id(r.data->'key'->'gid') AS id, + details->'external_reference'->>0 AS topic, + details->>'envelope_type' AS envelope_type, + details->>'key_format' AS key_format, + details->>'value_format' AS value_format +FROM + mz_internal.mz_catalog_raw r, + LATERAL ( + SELECT mz_internal.parse_source_export_details( + r.data->'value'->'definition'->'V1'->>'create_sql') + ) AS d(details) + JOIN mz_catalog.mz_sources s + ON s.id = details->>'source_id' AND s.type = 'kafka' +WHERE + r.data->>'kind' = 'Item' AND + details IS NOT NULL", + is_retained_metrics_object: true, + access: vec![PUBLIC_SELECT], + ontology: Some(Ontology { + entity_name: "kafka_source_table", + description: "Kafka source table-level details", + links: &const { + [OntologyLink { + name: "describes_source_table", + target: "table", + properties: LinkProperties::fk("id", "id", Cardinality::OneToOne), + }] + }, + column_semantic_types: &[("id", SemanticType::CatalogItemId)], + }), + } }); pub static MZ_OBJECT_DEPENDENCIES: LazyLock = LazyLock::new(|| BuiltinTable { name: "mz_object_dependencies", diff --git a/src/catalog/src/durable.rs b/src/catalog/src/durable.rs index 2ad4462147682..32fb47d562874 100644 --- a/src/catalog/src/durable.rs +++ b/src/catalog/src/durable.rs @@ -37,7 +37,7 @@ pub use crate::durable::objects::{ NetworkPolicy, ReconfigurationState, ReconfigurationStatus, ReconfigurationTarget, ReplicaConfig, ReplicaLocation, ReplicaSystemConfiguration, Role, RoleAuth, Schema, SourceReference, SourceReferences, StorageCollectionMetadata, SystemConfiguration, - SystemObjectDescription, SystemObjectMapping, UnfinalizedShard, + SystemObjectDescription, SystemObjectMapping, UnfinalizedShard, managed_cluster_replica_name, }; pub use crate::durable::persist::shard_id; use crate::durable::persist::{Timestamp, UnopenedPersistCatalogState}; diff --git a/src/catalog/src/durable/objects.rs b/src/catalog/src/durable/objects.rs index 6f57ae51eb3b4..f7f981a43fccb 100644 --- a/src/catalog/src/durable/objects.rs +++ b/src/catalog/src/durable/objects.rs @@ -363,6 +363,22 @@ pub struct ClusterVariantManaged { pub burst: Option, } +/// The canonical name of the `index`-th (zero-based) replica of a managed +/// cluster. +/// +/// A managed cluster's replicas are derived from its `replication_factor`: for a +/// factor of N they are named `r1` through `rN`. +/// +/// `ALTER CLUSTER` computes the replicas it creates and drops by this rule, and +/// so does the catalog-open reconciler that materializes builtin replicas. The +/// two have to share it, or `ALTER` cannot find the replicas it means to change. +/// The cluster controller deliberately does not: its `ReplicaNameGen` picks names +/// that avoid the observed set, which is why `ALTER` against a controller-owned +/// cluster drops by observed id rather than by derived name. +pub fn managed_cluster_replica_name(index: u32) -> String { + format!("r{}", index + 1) +} + /// The latest graceful reconfiguration: the config shape the cluster is moving /// to or most recently moved toward, plus its deadline and terminal state. /// diff --git a/src/environmentd/src/http/mcp.rs b/src/environmentd/src/http/mcp.rs index 27d5ca293e998..4c8b69538c53b 100644 --- a/src/environmentd/src/http/mcp.rs +++ b/src/environmentd/src/http/mcp.rs @@ -475,7 +475,22 @@ async fn handle_mcp_request( |status: McpCallStatus| metrics.record_request(endpoint_label, &method_label, status); // Check the per-endpoint feature flag via a catalog snapshot, similar to frontend_peek.rs. - let catalog = client.client.catalog_snapshot("mcp").await; + // The configured `MCP_REQUEST_TIMEOUT` lives in the snapshot we are about + // to fetch, so bound this phase with the compiled-in default. Without it a + // stalled snapshot would hang the request past any configured timeout. + let catalog = match tokio::time::timeout( + *MCP_REQUEST_TIMEOUT.default(), + client.client.catalog_snapshot("mcp"), + ) + .await + { + Ok(catalog) => catalog, + Err(_elapsed) => { + warn!(endpoint = %endpoint_type, "MCP catalog snapshot timed out"); + record_request(McpCallStatus::Timeout); + return StatusCode::SERVICE_UNAVAILABLE.into_response(); + } + }; let dyncfgs = catalog.system_config().dyncfgs(); let enabled = match endpoint_type { McpEndpointType::Agent => ENABLE_MCP_AGENT.get(dyncfgs), diff --git a/src/environmentd/src/test_util.rs b/src/environmentd/src/test_util.rs index b29bd530d32c3..6706e2623f8b6 100644 --- a/src/environmentd/src/test_util.rs +++ b/src/environmentd/src/test_util.rs @@ -592,6 +592,15 @@ impl TestHarness { self } + pub fn with_builtin_support_cluster_replication_factor( + mut self, + builtin_support_cluster_replication_factor: u32, + ) -> Self { + self.builtin_support_cluster_config.replication_factor = + builtin_support_cluster_replication_factor; + self + } + pub fn with_builtin_catalog_server_cluster_replica_size( mut self, builtin_catalog_server_cluster_replica_size: String, diff --git a/src/environmentd/tests/bootstrap_builtin_clusters.rs b/src/environmentd/tests/bootstrap_builtin_clusters.rs index 3ca92611e7f12..27689048514d3 100644 --- a/src/environmentd/tests/bootstrap_builtin_clusters.rs +++ b/src/environmentd/tests/bootstrap_builtin_clusters.rs @@ -8,31 +8,320 @@ // by the Apache License, Version 2.0. //! Integration tests for builtin clusters on bootstrap. +//! +//! A builtin cluster is a managed cluster, so its `replication_factor` is the +//! single source of truth for its replica set. These tests pin that down from +//! both directions: the factor a deployment bootstraps with is realized, and a +//! factor an operator sets with `ALTER CLUSTER` survives a restart. -use mz_environmentd::test_util::{self}; +use mz_environmentd::test_util::{self, TestHarness, TestServerWithRuntime}; -// Test that a cluster with a replication factor of 0 should not create any replicas. +/// A builtin cluster's declared replication factor and how many replicas it +/// actually has. +/// +/// Always read over the public port, whose session cluster is `quickstart`, so a +/// test that drops `mz_system`'s replicas can still observe the result. +fn declared_and_actual(server: &TestServerWithRuntime, cluster: &str) -> (i32, i32) { + let mut client = server.connect(postgres::NoTls).unwrap(); + let row = client + .query_one( + "SELECT c.replication_factor::integer, + (SELECT COUNT(*) FROM mz_cluster_replicas r WHERE r.cluster_id = c.id)::integer + FROM mz_clusters c + WHERE c.name = $1", + &[&cluster], + ) + .unwrap(); + (row.get(0), row.get(1)) +} + +/// Runs `stmt` as `user` over the internal port, which is where the system roles +/// that own the builtin clusters can connect. +fn execute_as(server: &TestServerWithRuntime, user: &str, stmt: &str) { + let mut config = server.pg_config_internal(); + config.user(user); + let mut client = config.connect(postgres::NoTls).unwrap(); + // The `Sql` composition wrappers are async and only defined for + // tokio-postgres clients, and these statements are fixed test literals. + #[allow(clippy::disallowed_methods)] + client.batch_execute(stmt).unwrap(); +} + +/// A cluster's replica names, sorted. +/// +/// Stronger than a count when the point is which replicas survived, and usable on +/// an unmanaged cluster, where `replication_factor` is null. +fn replica_names(server: &TestServerWithRuntime, cluster: &str) -> Vec { + let mut client = server.connect(postgres::NoTls).unwrap(); + client + .query( + "SELECT r.name FROM mz_cluster_replicas r + JOIN mz_clusters c ON c.id = r.cluster_id + WHERE c.name = $1 ORDER BY r.name", + &[&cluster], + ) + .unwrap() + .iter() + .map(|row| row.get::<_, String>(0)) + .collect() +} + +// A cluster with a replication factor of 0 should not create any replicas. #[mz_ore::test] fn test_zero_replication_factor_no_replicas() { let server = test_util::TestHarness::default() .with_builtin_system_cluster_replication_factor(0) .start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (0, 0)); +} + +// A replication factor above one is realized, not truncated to a single replica. +// The bootstrap flags accept 0..=2, and the replica set is derived from the +// factor, so asking for two replicas has to produce two. +#[mz_ore::test] +fn test_replication_factor_above_one_is_realized() { + let server = test_util::TestHarness::default() + .with_builtin_system_cluster_replication_factor(2) + .start_blocking(); + + assert_eq!(declared_and_actual(&server, "mz_system"), (2, 2)); +} + +// Scaling a builtin cluster to zero replicas sticks across a restart. The +// bootstrap flag seeds the factor at creation and must not resurrect a replica +// the operator removed. +#[mz_ore::test] +fn test_alter_to_zero_replicas_survives_restart() { + let data_dir = tempfile::tempdir().unwrap(); + let harness = TestHarness::default().data_directory(data_dir.path()); + + { + let server = harness.clone().start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (1, 1)); + execute_as( + &server, + "mz_system", + "ALTER CLUSTER mz_system SET (REPLICATION FACTOR 0)", + ); + assert_eq!(declared_and_actual(&server, "mz_system"), (0, 0)); + } + + let server = harness.start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (0, 0)); +} + +// Scaling a builtin cluster above one replica sticks across a restart. +#[mz_ore::test] +fn test_alter_above_one_replica_survives_restart() { + let data_dir = tempfile::tempdir().unwrap(); + let harness = TestHarness::default().data_directory(data_dir.path()); + + { + let server = harness.clone().start_blocking(); + execute_as( + &server, + "mz_system", + "ALTER CLUSTER mz_system SET (REPLICATION FACTOR 2)", + ); + assert_eq!(declared_and_actual(&server, "mz_system"), (2, 2)); + } + + let server = harness.start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (2, 2)); +} + +// A bootstrap flag that disagrees with an existing cluster's row does not act on +// it. This is the path a helm deployment takes, since the chart defaults +// `defaultReplicationFactor.system` to 0: install at 0, then restart with the +// flag back at its default of 1. The row has to win, or the cluster ends up at +// factor 0 with a live replica, which wedges every later factor `ALTER` on the +// `(cluster_id, name)` uniqueness constraint. +#[mz_ore::test] +fn test_bootstrap_flag_does_not_act_on_existing_cluster() { + let data_dir = tempfile::tempdir().unwrap(); + let harness = TestHarness::default().data_directory(data_dir.path()); + + { + let server = harness + .clone() + .with_builtin_system_cluster_replication_factor(0) + .start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (0, 0)); + } + + // Restarting without the flag leaves it at its default of 1, which must not + // resurrect a replica. Each boot is scoped so the previous one is shut down + // before the next opens the catalog, otherwise the older instance is + // epoch-fenced while still running. + { + let server = harness.clone().start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (0, 0)); + } + + // Nor does raising it further. + let server = harness + .with_builtin_system_cluster_replication_factor(2) + .start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (0, 0)); +} + +// `mz_support` defaults to zero replicas and stays that way, but a support +// engineer who scales it up keeps the replica across a restart. This is the +// break-glass path: `mz_support` runs nothing until someone needs it. +#[mz_ore::test] +fn test_support_cluster_replica_survives_restart() { + let data_dir = tempfile::tempdir().unwrap(); + let harness = TestHarness::default().data_directory(data_dir.path()); + + { + let server = harness.clone().start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_support"), (0, 0)); + execute_as( + &server, + "mz_support", + "ALTER CLUSTER mz_support SET (REPLICATION FACTOR 1)", + ); + assert_eq!(declared_and_actual(&server, "mz_support"), (1, 1)); + } + + let server = harness.start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_support"), (1, 1)); +} + +// A replica the reconciler creates is owned by its cluster's owner, not by +// `mz_system`. `mz_support` and `mz_analytics` are owned by their own roles, and +// replica ownership is checked against the replica's own `owner_id`, so stamping +// `mz_system` here would lock those roles out of a replica of a cluster they own. +// +// Bootstrapping `mz_support` above zero also pins that its replication-factor flag +// reaches the replica set at all, which is only true because the flag seeds the +// cluster's factor and the factor is what the replica set is derived from. +#[mz_ore::test] +fn test_created_replica_is_owned_by_its_cluster_owner() { + let server = TestHarness::default() + .with_builtin_support_cluster_replication_factor(1) + .start_blocking(); + + assert_eq!(declared_and_actual(&server, "mz_support"), (1, 1)); + let mut client = server.connect(postgres::NoTls).unwrap(); - let system_cluster = client + let owner: String = client .query_one( - r#" - SELECT c.id, c.name, c.replication_factor::integer, COUNT(cr.id)::integer as replica_count - FROM mz_clusters c - LEFT JOIN mz_cluster_replicas cr ON c.id = cr.cluster_id - WHERE c.name = 'mz_system' - GROUP BY c.id, c.name, c.replication_factor"#, + "SELECT o.name FROM mz_cluster_replicas r + JOIN mz_clusters c ON c.id = r.cluster_id + JOIN mz_roles o ON o.id = r.owner_id + WHERE c.name = 'mz_support'", &[], ) - .unwrap(); + .unwrap() + .get(0); + assert_eq!(owner, "mz_support"); +} + +// A restart must not disturb a builtin cluster whose replica set already matches +// its factor. Two consecutive boots leave the same replica ids in place, so +// nothing is torn down and recreated behind the operator's back. +#[mz_ore::test] +fn test_restart_is_idempotent() { + let data_dir = tempfile::tempdir().unwrap(); + let harness = TestHarness::default().data_directory(data_dir.path()); + + let replica_ids = |server: &TestServerWithRuntime| -> Vec { + let mut client = server.connect(postgres::NoTls).unwrap(); + client + .query( + "SELECT r.id FROM mz_cluster_replicas r + JOIN mz_clusters c ON c.id = r.cluster_id + WHERE c.id LIKE 's%' ORDER BY r.id", + &[], + ) + .unwrap() + .iter() + .map(|row| row.get::<_, String>(0)) + .collect() + }; + + let before = { + let server = harness.clone().start_blocking(); + let before = replica_ids(&server); + assert!(!before.is_empty(), "expected some builtin replicas"); + before + }; + + let server = harness.start_blocking(); + assert_eq!(replica_ids(&server), before); +} + +// An internal replica on a builtin cluster survives a restart. `CREATE CLUSTER +// REPLICA ... INTERNAL` is allowed on a managed cluster, and its name is barred +// from matching the derived `r1..rN` pattern, so it coexists with the derived +// replicas instead of being one of them. It is the break-glass way to attach a +// replica to a builtin cluster, and `ALTER CLUSTER` already preserves it, so +// reconciling has to leave it alone too while still converging the derived set. +#[mz_ore::test] +fn test_internal_replica_survives_restart() { + let data_dir = tempfile::tempdir().unwrap(); + let harness = TestHarness::default().data_directory(data_dir.path()); + + { + let server = harness.clone().start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (1, 1)); + execute_as( + &server, + "mz_system", + "CREATE CLUSTER REPLICA mz_system.breakglass SIZE 'scale=1,workers=1', INTERNAL", + ); + // An internal replica is not derived from the factor, so it adds a replica + // without changing what the cluster declares. + assert_eq!(declared_and_actual(&server, "mz_system"), (1, 2)); + } + + let server = harness.start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_system"), (1, 2)); + assert_eq!( + replica_names(&server, "mz_system"), + vec!["breakglass".to_string(), "r1".to_string()] + ); +} + +// An unmanaged builtin cluster's replica set belongs to whoever made it. There is +// no replication factor to derive a target from, so reconciling has to skip the +// cluster rather than treat all of its replicas as surplus. +#[mz_ore::test] +fn test_unmanaged_builtin_cluster_is_left_alone() { + let data_dir = tempfile::tempdir().unwrap(); + let harness = TestHarness::default() + .with_builtin_support_cluster_replication_factor(1) + .data_directory(data_dir.path()); + + { + let server = harness.clone().start_blocking(); + assert_eq!(declared_and_actual(&server, "mz_support"), (1, 1)); + // The conversion adopts the existing `r1`. Adding a second replica then + // makes the set one that no replication factor would produce. + // + // As `mz_support`, not `mz_system`: altering a cluster requires being its + // owner, and `mz_support` owns this one. + execute_as( + &server, + "mz_support", + "ALTER CLUSTER mz_support SET (MANAGED = false)", + ); + execute_as( + &server, + "mz_support", + "CREATE CLUSTER REPLICA mz_support.extra SIZE 'scale=1,workers=1'", + ); + assert_eq!( + replica_names(&server, "mz_support"), + vec!["extra".to_string(), "r1".to_string()] + ); + } - let replication_factor: i32 = system_cluster.get(2); - let replica_count: i32 = system_cluster.get(3); - assert_eq!(replication_factor, 0); - assert_eq!(replica_count, 0); + let server = harness.start_blocking(); + assert_eq!( + replica_names(&server, "mz_support"), + vec!["extra".to_string(), "r1".to_string()] + ); } diff --git a/src/environmentd/tests/pgwire.rs b/src/environmentd/tests/pgwire.rs index 986ae76063026..fd6fcbb9886d5 100644 --- a/src/environmentd/tests/pgwire.rs +++ b/src/environmentd/tests/pgwire.rs @@ -1034,6 +1034,11 @@ fn test_pgtest_mz_stray_copy() { pg_test_inner(Path::new("../../test/pgtest-mz/stray-copy.pt"), true); } +#[mz_ore::test] +fn test_pgtest_mz_set_local() { + pg_test_inner(Path::new("../../test/pgtest-mz/set-local.pt"), true); +} + #[mz_ore::test] fn test_pgtest_mz_transactions() { pg_test_inner(Path::new("../../test/pgtest-mz/transactions.pt"), true); diff --git a/src/expr/src/scalar/func/impls/jsonb.rs b/src/expr/src/scalar/func/impls/jsonb.rs index 9195a9c3dee5e..1788c342e7a4f 100644 --- a/src/expr/src/scalar/func/impls/jsonb.rs +++ b/src/expr/src/scalar/func/impls/jsonb.rs @@ -18,8 +18,9 @@ use mz_repr::role_id::RoleId; use mz_repr::{ArrayRustType, Datum, Row, RowPacker, SqlColumnType, SqlScalarType, strconv}; use mz_sql_parser::ast::display::AstDisplay; use mz_sql_parser::ast::{ - AstInfo, Format, FormatSpecifier, KafkaSourceConfigOptionName, PgConfigOptionName, - RawClusterName, RawItemName, Value, WithOptionValue, + AstInfo, AvroSchema, CreateSubsourceOptionName, Format, FormatSpecifier, + KafkaSourceConfigOptionName, PgConfigOptionName, ProtobufSchema, RawClusterName, RawItemName, + SourceEnvelope, SourceErrorPolicy, Value, WithOptionValue, }; use prost::Message as _; use serde::{Deserialize, Serialize}; @@ -700,6 +701,194 @@ fn parse_kafka_source_details<'a>(a: &'a str) -> Result { Ok(jsonb) } +/// Extracts source-export (source table) metadata from a catalog `create_sql`. +/// +/// Returns, for a `CREATE TABLE ... FROM SOURCE` or a non-progress +/// `CREATE SUBSOURCE ... OF SOURCE ...` statement: +/// +/// ```json +/// { +/// "source_id": "", +/// "external_reference": ["part1", "part2", ...], +/// "envelope_type": , +/// "key_format": , +/// "value_format": +/// } +/// ``` +/// +/// `envelope_type`, `key_format`, and `value_format` are always null for a +/// `CREATE SUBSOURCE` (the postgres/mysql/sql-server exports that use the old +/// subsource syntax carry neither format nor envelope). They may also be null +/// for a `CREATE TABLE ... FROM SOURCE` that omits FORMAT/ENVELOPE. +/// +/// Returns jsonb `null` for progress subsources and for any statement that is +/// not a source export. The caller distinguishes the four source-table views +/// by joining `source_id` against `mz_sources` and filtering on the parent's +/// type, so this helper stays connection-type agnostic. +/// +/// Errors if the statement fails to parse, references an unresolved item name, +/// or is a non-progress subsource missing its OF SOURCE or EXTERNAL REFERENCE. +/// +/// The `key_format`/`value_format` derivation mirrors the runtime +/// `DataSourceDesc::formats()` that the removed `pack_kafka_source_tables_update` +/// packer read. A bare FORMAT only carries a key when it resolves to an +/// encoding that has one, which among bare formats is only Avro or Protobuf +/// read from a Confluent Schema Registry whose purified seed carries a key +/// schema. A KEY FORMAT ... VALUE FORMAT ... spec always carries both. +#[sqlfunc] +fn parse_source_export_details<'a>(a: &'a str) -> Result { + fn get_item_id(item: RawItemName) -> Result { + match item { + RawItemName::Id(id, _, _) => Ok(id), + RawItemName::Name(_) => Err("unresolved item name"), + } + } + + fn format_name(fmt: &Format) -> &'static str { + match fmt { + Format::Bytes => "bytes", + Format::Avro(_) => "avro", + Format::Protobuf(_) => "protobuf", + Format::Regex(_) => "regex", + Format::Csv { .. } => "csv", + Format::Json { .. } => "json", + Format::Text => "text", + } + } + + // A bare FORMAT resolves to an encoding with a key only for Avro or + // Protobuf read from a schema registry whose purified seed carries a key + // schema. Every other bare format is value-only. + fn bare_format_has_key(fmt: &Format) -> bool { + match fmt { + Format::Avro(AvroSchema::Csr { csr_connection }) => csr_connection + .seed + .as_ref() + .is_some_and(|seed| seed.key_schema.is_some()), + Format::Protobuf(ProtobufSchema::Csr { csr_connection }) => csr_connection + .seed + .as_ref() + .is_some_and(|seed| seed.key.is_some()), + _ => false, + } + } + + fn key_value_formats( + spec: &FormatSpecifier, + ) -> (Option<&'static str>, Option<&'static str>) { + match spec { + FormatSpecifier::KeyValue { key, value } => { + (Some(format_name(key)), Some(format_name(value))) + } + FormatSpecifier::Bare(fmt) => { + let value = Some(format_name(fmt)); + let key = bare_format_has_key(fmt).then(|| format_name(fmt)); + (key, value) + } + } + } + + fn envelope_name(envelope: &SourceEnvelope) -> &'static str { + match envelope { + SourceEnvelope::None => "none", + SourceEnvelope::Debezium => "debezium", + SourceEnvelope::Upsert { + value_decode_err_policy, + } => { + if value_decode_err_policy + .iter() + .any(|p| matches!(p, SourceErrorPolicy::Inline { .. })) + { + "upsert-value-err-inline" + } else { + "upsert" + } + } + SourceEnvelope::CdcV2 => "materialize", + } + } + + let parse = || -> Result { + let mut stmts = mz_sql_parser::parser::parse_statements(a) + .map_err(|e| format!("failed to parse create_sql: {e}"))?; + let stmt = match stmts.len() { + 1 => stmts.remove(0).ast, + n => return Err(format!("expected a single statement, found {n}")), + }; + + use mz_sql_parser::ast::Statement::{CreateSubsource, CreateTableFromSource}; + match stmt { + CreateTableFromSource(stmt) => { + let source_id = get_item_id(stmt.source)?; + let external_reference = stmt + .external_reference + .ok_or("missing external reference on CREATE TABLE FROM SOURCE")? + .0 + .into_iter() + .map(|ident| ident.into_string()) + .collect::>(); + + let envelope_type = stmt.envelope.as_ref().map(envelope_name); + let (key_format, value_format) = match &stmt.format { + Some(spec) => key_value_formats(spec), + None => (None, None), + }; + + Ok(json!({ + "source_id": source_id, + "external_reference": external_reference, + "envelope_type": envelope_type, + "key_format": key_format, + "value_format": value_format, + })) + } + CreateSubsource(stmt) => { + // Progress subsources track ingestion progress and are not + // source tables. They have no external reference. + let is_progress = stmt + .with_options + .iter() + .any(|o| matches!(o.name, CreateSubsourceOptionName::Progress)); + if is_progress { + return Ok(serde_json::Value::Null); + } + + let source_id = stmt + .of_source + .ok_or("non-progress CREATE SUBSOURCE without OF SOURCE") + .and_then(get_item_id)?; + + let external_reference = stmt + .with_options + .into_iter() + .find(|o| matches!(o.name, CreateSubsourceOptionName::ExternalReference)) + .and_then(|o| match o.value { + Some(WithOptionValue::UnresolvedItemName(name)) => Some(name), + _ => None, + }) + .ok_or("CREATE SUBSOURCE missing EXTERNAL REFERENCE option")? + .0 + .into_iter() + .map(|ident| ident.into_string()) + .collect::>(); + + Ok(json!({ + "source_id": source_id, + "external_reference": external_reference, + "envelope_type": serde_json::Value::Null, + "key_format": serde_json::Value::Null, + "value_format": serde_json::Value::Null, + })) + } + _ => Ok(serde_json::Value::Null), + } + }; + + let val = parse().map_err(|e| EvalError::InvalidCatalogJson(e.into()))?; + let jsonb = Jsonb::from_serde_json(val).expect("valid JSONB"); + Ok(jsonb) +} + #[cfg(test)] mod tests { use mz_repr::adt::jsonb::Jsonb; @@ -889,4 +1078,163 @@ mod tests { "wrong error variant/message" ); } + + // --- parse_source_export_details ----------------------------------------- + + fn table_from_source_sql(reference: &str, suffix: &str) -> String { + format!( + "CREATE TABLE \"materialize\".\"public\".\"tbl\" \ + FROM SOURCE [u1 AS \"materialize\".\"public\".\"src\"] \ + (REFERENCE = {reference}){suffix}" + ) + } + + #[mz_ore::test] + fn export_table_postgres_style_no_format() { + // Postgres/mysql/sql-server tables carry a multi-part external + // reference and no format or envelope. + let sql = table_from_source_sql("\"db\".\"public\".\"t\"", ""); + let out = super::parse_source_export_details(&sql).expect("ok"); + assert_eq!( + as_serde(out), + json!({ + "source_id": "u1", + "external_reference": ["db", "public", "t"], + "envelope_type": null, + "key_format": null, + "value_format": null, + }), + ); + } + + #[mz_ore::test] + fn export_table_kafka_bare_value_only() { + // A bare non-registry FORMAT is value-only: no key format. + let sql = table_from_source_sql("\"topic\"", " FORMAT TEXT ENVELOPE NONE"); + let out = super::parse_source_export_details(&sql).expect("ok"); + assert_eq!( + as_serde(out), + json!({ + "source_id": "u1", + "external_reference": ["topic"], + "envelope_type": "none", + "key_format": null, + "value_format": "text", + }), + ); + } + + #[mz_ore::test] + fn export_table_kafka_key_value_format() { + let sql = table_from_source_sql( + "\"topic\"", + " KEY FORMAT TEXT VALUE FORMAT TEXT ENVELOPE NONE", + ); + let out = super::parse_source_export_details(&sql).expect("ok"); + assert_eq!( + as_serde(out), + json!({ + "source_id": "u1", + "external_reference": ["topic"], + "envelope_type": "none", + "key_format": "text", + "value_format": "text", + }), + ); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn export_table_kafka_bare_avro_seed_with_key() { + // A bare Avro CSR format whose seed carries a key schema resolves to + // an encoding with a key, so key_format mirrors value_format. This is + // the upsert/debezium path. + let sql = table_from_source_sql( + "\"topic\"", + " FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY \ + CONNECTION [u5 AS \"materialize\".\"public\".\"csr\"] \ + SEED KEY SCHEMA 'k' VALUE SCHEMA 'v' ENVELOPE UPSERT", + ); + let out = super::parse_source_export_details(&sql).expect("ok"); + assert_eq!( + as_serde(out), + json!({ + "source_id": "u1", + "external_reference": ["topic"], + "envelope_type": "upsert", + "key_format": "avro", + "value_format": "avro", + }), + ); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn export_table_kafka_bare_avro_seed_without_key() { + // A bare Avro CSR seed with only a value schema is value-only. + let sql = table_from_source_sql( + "\"topic\"", + " FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY \ + CONNECTION [u5 AS \"materialize\".\"public\".\"csr\"] \ + SEED VALUE SCHEMA 'v' ENVELOPE NONE", + ); + let out = super::parse_source_export_details(&sql).expect("ok"); + assert_eq!( + as_serde(out), + json!({ + "source_id": "u1", + "external_reference": ["topic"], + "envelope_type": "none", + "key_format": null, + "value_format": "avro", + }), + ); + } + + #[mz_ore::test] + fn export_subsource_non_progress() { + // Old-syntax subsource: external reference lives in a WITH option, and + // there is never a format or envelope. + let sql = "CREATE SUBSOURCE \"materialize\".\"public\".\"sub\" (id int4) \ + OF SOURCE [u1 AS \"materialize\".\"public\".\"src\"] \ + WITH (EXTERNAL REFERENCE = \"db\".\"public\".\"t\")"; + let out = super::parse_source_export_details(sql).expect("ok"); + assert_eq!( + as_serde(out), + json!({ + "source_id": "u1", + "external_reference": ["db", "public", "t"], + "envelope_type": null, + "key_format": null, + "value_format": null, + }), + ); + } + + #[mz_ore::test] + fn export_progress_subsource_returns_null_jsonb() { + let sql = "CREATE SUBSOURCE \"materialize\".\"public\".\"progress\" (id int4) \ + WITH (PROGRESS)"; + let out = super::parse_source_export_details(sql).expect("ok"); + assert_eq!(as_serde(out), serde_json::Value::Null); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn export_non_source_export_returns_null_jsonb() { + let sql = "CREATE VIEW v AS SELECT 1"; + let out = super::parse_source_export_details(sql).expect("ok"); + assert_eq!(as_serde(out), serde_json::Value::Null); + } + + #[mz_ore::test] + fn export_unresolved_source_name_errors() { + let sql = "CREATE TABLE \"materialize\".\"public\".\"tbl\" \ + FROM SOURCE src (REFERENCE = \"topic\")"; + let err = super::parse_source_export_details(sql).unwrap_err(); + assert!( + matches!(err, EvalError::InvalidCatalogJson(msg) if msg.contains("unresolved item name")), + "wrong error variant/message" + ); + } } diff --git a/src/expr/src/scalar/func/unary.rs b/src/expr/src/scalar/func/unary.rs index 26ea95368ebe3..9723c257be76a 100644 --- a/src/expr/src/scalar/func/unary.rs +++ b/src/expr/src/scalar/func/unary.rs @@ -470,6 +470,7 @@ derive_unary!( ParseCatalogPrivileges, ParseKafkaSourceDetails, ParsePostgresSourceDetails, + ParseSourceExportDetails, RedactSql, RoundFloat32, RoundFloat64, diff --git a/src/persist-client/src/internal/encoding.rs b/src/persist-client/src/internal/encoding.rs index 6664f4ac39ab1..0005b7b642111 100644 --- a/src/persist-client/src/internal/encoding.rs +++ b/src/persist-client/src/internal/encoding.rs @@ -1729,9 +1729,16 @@ pub struct LazyPartStats { impl Debug for LazyPartStats { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { - f.debug_tuple("LazyPartStats") - .field(&self.decode()) - .finish() + let mut f = f.debug_tuple("LazyPartStats"); + // These bytes come from blob and are never validated on the way in, so + // `Debug` runs on malformed encodings: `Spine::validate` formats the + // whole spine into the message it rejects an untrusted rollup with, and + // `Trace::unflatten` returns that as a decode error. Rendering the + // failure keeps that rejection path a hard error instead of a panic. + match self.try_decode() { + Ok(stats) => f.field(&stats).finish(), + Err(err) => f.field(&format_args!("")).finish(), + } } } @@ -1748,11 +1755,26 @@ impl LazyPartStats { /// /// This does not cache the returned value, it decodes each time it's /// called. + /// + /// Panics if the encoded bytes are malformed. Only call this where the value + /// is known to have come from `Self::encode` rather than straight off blob. pub fn decode(&self) -> PartStats { - let key = self.key.decode().expect("valid proto"); - PartStats { - key: key.into_rust().expect("valid stats"), - } + self.try_decode().expect("valid stats") + } + + /// Like [Self::decode], but surfaces a malformed encoding as an error. + /// + /// The bytes are stored undecoded (see the [RustType] impl), so a corrupted + /// or crafted blob reaches here intact. Anything running on state that has + /// not been validated yet must use this. + pub fn try_decode(&self) -> Result { + let key = self + .key + .decode() + .map_err(|err| TryFromProtoError::InvalidPersistState(err.to_string()))?; + Ok(PartStats { + key: key.into_rust()?, + }) } } @@ -2630,4 +2652,18 @@ mod tests { testcase("28.0.0", "26.1000.1", Err(())); testcase("28.0.0", "27.0.0", Ok(())); } + + /// `LazyPartStats`'s bytes are stored undecoded, so `Debug` runs on + /// encodings that came straight off blob and may be malformed. It must render + /// the failure: `Spine::validate` formats the whole spine into the message + /// with which `Trace::unflatten` rejects an untrusted rollup, so a panic here + /// turns that rejection into an unrecoverable shard. + #[mz_ore::test] + fn lazy_part_stats_debug_does_not_panic_on_garbage() { + // Tag 0 is never a legal protobuf tag. + let stats = LazyPartStats::from_proto(Bytes::from_static(&[0x00, 0xff])) + .expect("stats bytes are stored undecoded"); + assert_err!(stats.try_decode()); + assert!(format!("{stats:?}").contains("undecodable")); + } } diff --git a/src/persist-client/src/internal/state_diff.rs b/src/persist-client/src/internal/state_diff.rs index 3a4a67c684e1e..96985586a979e 100644 --- a/src/persist-client/src/internal/state_diff.rs +++ b/src/persist-client/src/internal/state_diff.rs @@ -1267,7 +1267,24 @@ impl ProtoStateFieldDiffs { )); } - let expected_data_bytes = usize::cast_from(self.data_lens.iter().copied().sum::()); + // NOTE: A crafted diff can declare lengths whose sum exceeds `u64::MAX`. + // Overflow checks are off in release/optimized builds, so an unchecked + // sum would wrap to a small value, match `data_bytes.len()`, and let the + // diff past validation. `ProtoStateFieldDiffsIter` would then slice + // `data_bytes` far out of range and panic. + let Some(expected_data_bytes) = self + .data_lens + .iter() + .copied() + .try_fold(0u64, |acc, len| acc.checked_add(len)) + .and_then(|sum| usize::try_from(sum).ok()) + else { + return Err(format!( + "data_lens sum overflows, got {} lens over {} data bytes", + self.data_lens.len(), + self.data_bytes.len() + )); + }; if expected_data_bytes != self.data_bytes.len() { return Err(format!( "expected {} data bytes got {}", @@ -1381,6 +1398,37 @@ mod tests { ); } + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // unsupported operation: can't call foreign function on OS `linux` + fn proto_state_diff_data_lens_overflow_is_error() { + // `data_lens` entries that sum past `u64::MAX` are the one shape that can + // get *past* `validate()`: with overflow checks off the sum wraps to 0, + // matching an empty `data_bytes`, and the iterator then slices + // `0..u64::MAX` out of an empty slice. + use crate::internal::state::ProtoStateDiff; + use mz_proto::ProtoType; + + let proto = ProtoStateDiff { + applier_version: "0.1.2".into(), + seqno_from: 0, + seqno_to: 1, + walltime_ms: 0, + latest_rollup_key: "rollup".into(), + field_diffs: Some(ProtoStateFieldDiffs { + fields: vec![ProtoStateField::Hostname.into()], + // Insert expects two data slices: the key and the new value. + diff_types: vec![ProtoStateFieldDiffType::Insert.into()], + data_lens: vec![u64::MAX, 1], + data_bytes: Bytes::new(), + }), + }; + let result: Result, _> = proto.into_rust(); + assert!( + result.is_err(), + "data_lens that overflow must be a decode error" + ); + } + /// Model a situation where a "leader" is constantly making changes to its state, and a "follower" /// is applying those changes as diffs. #[mz_ore::test] diff --git a/src/persist-client/src/internal/trace.rs b/src/persist-client/src/internal/trace.rs index 790e96b443393..9832581742deb 100644 --- a/src/persist-client/src/internal/trace.rs +++ b/src/persist-client/src/internal/trace.rs @@ -1581,12 +1581,17 @@ impl FuelingMerge { /// /// If `fuel` is non-zero after the call, the merging is complete and one /// should call `done` to extract the merged results. - // TODO(benesch): rewrite to avoid usage of `as`. - #[allow(clippy::as_conversions)] fn work(&mut self, _: &[SpineBatch], fuel: &mut isize) { - let used = std::cmp::min(*fuel as usize, self.remaining_work); + // A negative `fuel` means a caller already overspent, so there is + // nothing to spend here. Reading it as a `usize` instead would wrap to a + // huge budget, spend `remaining_work` against it, and then underflow the + // subtraction below. + let available = usize::try_from(*fuel).unwrap_or(0); + let used = std::cmp::min(available, self.remaining_work); self.remaining_work = self.remaining_work.saturating_sub(used); - *fuel -= used as isize; + // `used <= available <= isize::MAX`, so neither conversion nor the + // subtraction can overflow. + *fuel -= isize::try_from(used).expect("used is bounded by fuel"); } /// Extracts merged results. @@ -1922,14 +1927,21 @@ impl Spine { // batches. We could try and make this be less, or be scaled to merges // based on their deficit at time of instantiation. For now, we remain // conservative. - let mut fuel = 8 << batch_index; - // Scale up by the effort parameter, which is calibrated to one as the - // minimum amount of effort. - fuel *= self.effort; - // Convert to an `isize` so we can observe any fuel shortfall. - // TODO(benesch): avoid dangerous usage of `as`. - #[allow(clippy::as_conversions)] - let fuel = fuel as isize; + // + // `batch_index` is derived from a batch's `len`, which for a trace + // reconstructed from an untrusted blob can be large enough that + // `8 << batch_index` overflows. Saturate at `isize::MAX` rather than + // wrapping: fuel is a budget, so more of it only completes merges sooner, + // whereas a wrapped value can land negative and starve them. The result + // is an `isize` so a fuel shortfall stays observable. + let fuel = u32::try_from(batch_index) + .ok() + .and_then(|shift| 8usize.checked_shl(shift)) + // Scale up by the effort parameter, which is calibrated to one as the + // minimum amount of effort. + .and_then(|fuel| fuel.checked_mul(self.effort)) + .and_then(|fuel| isize::try_from(fuel).ok()) + .unwrap_or(isize::MAX); // Step 1. Apply fuel to each in-progress merge. // @@ -2749,4 +2761,33 @@ pub(crate) mod tests { assert_eq!(sb_too_big.parts.len(), 3); assert_eq!(sb_too_big.len(), 30); } + + /// Inserting a batch whose `len` is large enough to saturate the fuel + /// computation must not disturb an in-progress merge's accounting. + /// + /// `introduce_batch` derives its fuel from `8 << batch_index`, where + /// `batch_index` is `len.next_power_of_two().trailing_zeros()`. A `len` near + /// `2^60` pushes that past what an `isize` holds, and `Trace::unflatten` + /// accepts such a `len` from an untrusted blob: its `MAX_TOTAL_LEN` guard + /// only caps the total at `usize::MAX >> 3`. + #[mz_ore::test] + fn spine_fuel_isize_overflow() { + let mut trace = Trace::::default(); + let mut push = |lower, upper, key: &str, len| { + trace.push_batch_no_merge_reqs(crate::internal::state::tests::hollow::( + lower, + upper, + &[key], + len, + )); + }; + // Two same-size batches fill a level and begin a merge whose + // `remaining_work` (the sum of their lens) far exceeds the `8 << 0` fuel + // that a subsequent len-1 batch delivers, so the merge is still in + // progress when the saturating batch arrives. + push(0, 1, "a", 100); + push(1, 2, "b", 100); + push(2, 3, "c", 1); + push(3, 4, "d", 1 << 60); + } } diff --git a/src/pgrepr-consts/src/oid.rs b/src/pgrepr-consts/src/oid.rs index 7a7332b388cf0..293bf64e90dc6 100644 --- a/src/pgrepr-consts/src/oid.rs +++ b/src/pgrepr-consts/src/oid.rs @@ -817,3 +817,8 @@ pub const MV_MZ_CLUSTER_AUTO_SCALING_STRATEGIES_OID: u32 = 17103; pub const INDEX_MZ_CLUSTER_AUTO_SCALING_STRATEGIES_IND_OID: u32 = 17104; pub const FUNC_PARSE_POSTGRES_SOURCE_DETAILS_OID: u32 = 17105; pub const FUNC_PARSE_KAFKA_SOURCE_DETAILS_OID: u32 = 17106; +pub const MV_MZ_POSTGRES_SOURCE_TABLES_OID: u32 = 17107; +pub const MV_MZ_MYSQL_SOURCE_TABLES_OID: u32 = 17108; +pub const MV_MZ_SQL_SERVER_SOURCE_TABLES_OID: u32 = 17109; +pub const MV_MZ_KAFKA_SOURCE_TABLES_OID: u32 = 17110; +pub const FUNC_PARSE_SOURCE_EXPORT_DETAILS_OID: u32 = 17111; diff --git a/src/pgtest/src/lib.rs b/src/pgtest/src/lib.rs index f9bfe409fe37c..c73fe810298f4 100644 --- a/src/pgtest/src/lib.rs +++ b/src/pgtest/src/lib.rs @@ -43,6 +43,9 @@ //! The default is `CMS` (code, message, severity). For example: `until //! err_field_typs=SC` would return the severity and code fields in any //! ErrorResponse message. +//! - `parameter_status` names the parameters whose `ParameterStatus` messages +//! are recorded. All others are skipped. For example `until +//! parameter_status=application_name`. //! //! For example, to execute a simple prepared statement: //! ```pgtest @@ -135,7 +138,12 @@ impl PgConn { Message::AuthenticationOk => {} _ => bail!("expected AuthenticationOk"), }; - conn.until(vec!["ReadyForQuery"], vec!['C', 'S', 'M'], BTreeSet::new())?; + conn.until( + vec!["ReadyForQuery"], + vec!['C', 'S', 'M'], + BTreeSet::new(), + &BTreeSet::new(), + )?; Ok(conn) } @@ -150,6 +158,7 @@ impl PgConn { until: Vec<&str>, err_field_typs: Vec, ignore: BTreeSet, + parameter_status: &BTreeSet, ) -> anyhow::Result> { let mut msgs = Vec::with_capacity(until.len()); for expect in until { @@ -296,7 +305,19 @@ impl PgConn { parameters: body.parameters().collect().unwrap(), })?, ), - Message::ParameterStatus(_) => continue, + Message::ParameterStatus(body) => { + let name = body.name()?; + if !parameter_status.contains(name) { + continue; + } + ( + "ParameterStatus", + serde_json::to_string(&ParameterStatus { + name: name.to_string(), + value: body.value()?.to_string(), + })?, + ) + } Message::NoData => ("NoData", "".to_string()), Message::EmptyQueryResponse => ("EmptyQueryResponse", "".to_string()), _ => ("UNKNOWN", format!("'{}'", ch)), @@ -414,9 +435,10 @@ impl PgTest { until: Vec<&str>, err_field_typs: Vec, ignore: BTreeSet, + parameter_status: &BTreeSet, ) -> anyhow::Result> { let conn = self.get_conn(conn, options)?; - conn.until(until, err_field_typs, ignore) + conn.until(until, err_field_typs, ignore, parameter_status) } } @@ -458,6 +480,12 @@ pub struct CommandComplete { pub tag: String, } +#[derive(Serialize)] +pub struct ParameterStatus { + pub name: String, + pub value: String, +} + #[derive(Serialize)] pub struct ErrorResponse { pub fields: Vec, @@ -618,14 +646,26 @@ pub fn run_test(tf: &mut datadriven::TestFile, addr: String, user: String, timeo ignore.insert(v); } } + let parameter_status: BTreeSet = args + .remove("parameter_status") + .unwrap_or_default() + .into_iter() + .collect(); if !args.is_empty() { panic!("extra until arguments: {:?}", args); } format!( "{}\n", - pgt.until(conn, options, lines.collect(), err_field_typs, ignore) - .unwrap() - .join("\n") + pgt.until( + conn, + options, + lines.collect(), + err_field_typs, + ignore, + ¶meter_status, + ) + .unwrap() + .join("\n") ) } _ => panic!("unknown directive {}", tc.input), diff --git a/src/pgwire/src/protocol.rs b/src/pgwire/src/protocol.rs index dbbc47bdf38a7..ce39b6cbb9261 100644 --- a/src/pgwire/src/protocol.rs +++ b/src/pgwire/src/protocol.rs @@ -1453,15 +1453,53 @@ where } /// End a transaction and report to the user if an error occurred. + /// + /// The parameters this changes must be announced, exactly as an explicit + /// `COMMIT`/`ROLLBACK` announces them. Otherwise a `SET LOCAL` outside an + /// explicit transaction announces its new value and never its revert, and a + /// client that caches parameters keeps the reverted value. #[instrument(level = "debug")] async fn end_transaction(&mut self, action: EndTransactionAction) -> Result<(), io::Error> { self.txn_needs_commit = false; - let resp = self.adapter_client.end_transaction(action).await; - if let Err(err) = resp { - self.send(BackendMessage::ErrorResponse( - err.into_response(Severity::Error), - )) - .await?; + match self.adapter_client.end_transaction(action).await { + Ok( + ExecuteResponse::TransactionCommitted { params } + | ExecuteResponse::TransactionRolledBack { params }, + ) => { + self.send_parameter_statuses(params).await?; + } + Ok(_) => {} + Err(err) => { + self.send(BackendMessage::ErrorResponse( + err.into_response(Severity::Error), + )) + .await?; + } + } + Ok(()) + } + + /// Announces changed parameters, restricted to those the client is told + /// about at startup. + #[instrument(level = "debug")] + async fn send_parameter_statuses( + &mut self, + params: BTreeMap<&'static str, String>, + ) -> Result<(), io::Error> { + let notify_set: mz_ore::collections::HashSet = self + .adapter_client + .session() + .vars() + .notify_set() + .map(|v| v.name().to_string()) + .collect(); + + for (name, value) in params + .into_iter() + .filter(|(name, _value)| notify_set.contains(*name)) + { + self.send(BackendMessage::ParameterStatus(name, value)) + .await?; } Ok(()) } @@ -2377,22 +2415,7 @@ where } ExecuteResponse::TransactionCommitted { params } | ExecuteResponse::TransactionRolledBack { params } => { - let notify_set: mz_ore::collections::HashSet = self - .adapter_client - .session() - .vars() - .notify_set() - .map(|v| v.name().to_string()) - .collect(); - - // Only report on parameters that are in the notify set. - for (name, value) in params - .into_iter() - .filter(|(name, _v)| notify_set.contains(*name)) - { - let msg = BackendMessage::ParameterStatus(name, value); - self.send(msg).await?; - } + self.send_parameter_statuses(params).await?; command_complete!() } diff --git a/src/sql/src/func.rs b/src/sql/src/func.rs index 002ff5c84559f..6d9296a847b92 100644 --- a/src/sql/src/func.rs +++ b/src/sql/src/func.rs @@ -5315,6 +5315,11 @@ pub static MZ_INTERNAL_BUILTINS: LazyLock> = LazyLo func::ParsePostgresSourceDetails, ) => Jsonb, oid::FUNC_PARSE_POSTGRES_SOURCE_DETAILS_OID; }, + "parse_source_export_details" => Scalar { + params!(String) => UnaryFunc::ParseSourceExportDetails( + func::ParseSourceExportDetails, + ) => Jsonb, oid::FUNC_PARSE_SOURCE_EXPORT_DETAILS_OID; + }, "redact_sql" => Scalar { params!(String) => UnaryFunc::RedactSql(func::RedactSql) => String, oid::FUNC_REDACT_SQL_OID; diff --git a/test/catalog-migration/incident-1007/mzcompose.py b/test/catalog-migration/incident-1007/mzcompose.py index 206b2d6917c2b..15423aabced5f 100644 --- a/test/catalog-migration/incident-1007/mzcompose.py +++ b/test/catalog-migration/incident-1007/mzcompose.py @@ -84,10 +84,15 @@ def _mz(image: str | None) -> Materialized: external_metadata_store=True, external_blob_store=True, sanity_restart=False, - # Set the default replication factor since - # one of the heuristics of a "non-cloud" environment is that the - # replication factor is 0 for mz_system. + # The buggy `v80_to_v81` migration decides an environment is cloud by + # checking whether `mz_system`'s replication factor is above 0, so the + # self-managed path this test reproduces needs that factor at 0. Passed + # explicitly: the builtin factors do not track + # `default_replication_factor`, since a builtin cluster honors its own + # factor and most tests want one replica there regardless of what they ask + # for on the user cluster. default_replication_factor=0, + builtin_system_cluster_replication_factor=0, # MZ_SOFT_ASSERTIONS=1 turns # soft asserts into panics, killing environmentd mid-upgrade # before v83->v84 ever runs. Demote them to log-only so the upgrade diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py index 476085025a6d3..c02e246a7af7b 100644 --- a/test/cluster-spec-sheet/mzcompose.py +++ b/test/cluster-spec-sheet/mzcompose.py @@ -22,8 +22,9 @@ from abc import ABC, abstractmethod from collections.abc import Callable, Iterable from dataclasses import dataclass +from functools import partial from textwrap import dedent -from typing import Any, TextIO +from typing import Any, LiteralString, TextIO import matplotlib.pyplot as plt import pandas as pd @@ -34,7 +35,6 @@ from materialize import MZ_ROOT, buildkite from materialize.mz_env_util import print_environment_id from materialize.mz_version import MzVersion -from materialize.mzcompose import _wait_for_pg from materialize.mzcompose.composition import ( Composition, WorkflowArgumentParser, @@ -3237,9 +3237,11 @@ def measure(self, runner: ScenarioRunner, point: ScalePoint) -> None: class EnvdCpuSweep(Scenario): """Sweep environmentd's CPU allocation at a fixed compute cluster size. - Wraps a `ClusterScalingScenario` workload. On Cloud, we reset envd's - CPU allocation to the default in ``teardown`` regardless of outcome to - avoid accidentally burning credits.""" + Wraps a `ClusterScalingScenario` workload. On Cloud, each scale point + recreates the region, so every point measures a freshly set up + environment. We reset envd's CPU allocation to the default in + ``teardown`` regardless of outcome to avoid accidentally burning + credits.""" # (So far, I haven't seen a difference between 16 and 32 in manual # testing in cloud. When we start seeing a difference, consider @@ -3283,6 +3285,10 @@ def apply(self, runner: ScenarioRunner, point: ScalePoint) -> bool: assert point.envd_cpus is not None assert self._fixed_replica_size is not None reconfigure_envd_cpus(runner.target, point.envd_cpus, runner) + # On Cloud the CPU change recreates the region, which drops the state that + # `prepare` set up. Re-preparing is cheap, and harmless on Docker, where the + # state survives the restart. + runner.connection.retryable(lambda: self.prepare(runner)) fixed_size = self._fixed_replica_size def recreate() -> None: @@ -3301,20 +3307,10 @@ def measure(self, runner: ScenarioRunner, point: ScalePoint) -> None: def teardown(self, runner: ScenarioRunner) -> None: if isinstance(runner.target, CloudTarget): print("--- Resetting Cloud environmentd CPUs to the default") - target = runner.target - version_args = ( - ["--version", target.version] if target.version is not None else [] - ) - target.composition.run( - "mz", - "region", - "enable", - "--environmentd-cpu-allocation", - "2", - *enable_extra_args(target), - *version_args, - rm=True, - ) + # Recreating rather than enabling on top of the existing region keeps the + # region out of the read-only limbo that a 0dt rollover would put it in, so + # whatever runs after this scenario finds a usable region. + cloud_recreate_region_with_envd_cpus(runner.target, 2) class EnvdObjectsSweep(Scenario): @@ -3369,9 +3365,8 @@ def teardown(self, runner: ScenarioRunner) -> None: self._workload.teardown(runner) -# TODO: We should factor out the below -# `disable_region`, `cloud_disable_enable_and_wait`, `reconfigure_envd_cpus`, `wait_for_envd` -# functions into a separate module. (Similar `disable_region` functions also occur in other tests.) +# TODO: We should factor the region helpers below out into a separate module. +# (Similar `disable_region` functions also occur in other tests.) def disable_region(composition: Composition, hard: bool) -> None: print("Shutting down region ...") @@ -3385,32 +3380,35 @@ def disable_region(composition: Composition, hard: bool) -> None: pass -# Enabling a region whose catalog already exists is a 0dt deployment: the new envd -# boots read-only and only promotes (and starts serving) once the caught-up check -# passes. Since #37255 that check includes a stability soak with a production -# default of 10 minutes, during which a soft-disabled region serves nothing. Keep -# the soak out of the critical path for test regions, like mzcompose does for -# Docker-based tests. -_STABILITY_SOAK_OVERRIDE = [ - "--environmentd-extra-arg=--system-parameter-default=with_0dt_caught_up_check_stability_period=0s", -] - - -def enable_extra_args(target: "CloudTarget") -> list[str]: +def enable_region(target: "CloudTarget", envd_cpus: int | None = None) -> None: """ - Extra `mz region enable` args, applied to staging only. + Run `mz region enable`, pinning environmentd's CPU allocation when one is given. - Production Cloud forbids callers from injecting environmentd args and rejects - `--environmentd-extra-arg` with a 403 Forbidden, so the stability-soak override - cannot be used there. + Enabling a region whose catalog still exists is a 0dt deployment: the new envd boots + read-only and only promotes, and starts serving, once the caught-up check passes. That + check includes a stability soak whose production default is ten minutes, during which a + soft-disabled region serves nothing at all. We turn the soak off for test regions, like + mzcompose does for Docker-based tests. """ - return _STABILITY_SOAK_OVERRIDE if target.is_staging else [] + args = [] + if envd_cpus is not None: + args += ["--environmentd-cpu-allocation", str(envd_cpus)] -def cloud_disable_enable_and_wait( - target: "BenchTarget", - environmentd_cpu_allocation: int | None = None, -) -> None: + if target.is_staging: + # Production Cloud forbids callers from injecting environmentd args and rejects + # `--environmentd-extra-arg` with a 403 Forbidden, so this is staging-only. + args += [ + "--environmentd-extra-arg=--system-parameter-default=with_0dt_caught_up_check_stability_period=0s" + ] + + if target.version is not None: + args += ["--version", target.version] + + target.composition.run("mz", "region", "enable", *args, rm=True) + + +def cloud_disable_enable_and_wait(target: "BenchTarget") -> None: """ Soft-disable and then enable the Cloud region, then wait for environmentd readiness. @@ -3424,9 +3422,6 @@ def cloud_disable_enable_and_wait( envd were still serving, both `mz region enable`'s readiness check and our `wait_for_envd` could pass against the old envd. Waiting for the old envd to stop serving first makes a later successful connection prove that the new envd is up. - - When `environmentd_cpu_allocation` is provided, it is passed to `mz region enable` via - `--environmentd-cpu-allocation` to reconfigure environmentd's CPU allocation. """ assert isinstance(target, CloudTarget) @@ -3443,27 +3438,7 @@ def cloud_disable_enable_and_wait( if host is not None: wait_for_envd_down(target, host) - version_args = ( - ["--version", target.version] - if isinstance(target, CloudTarget) and target.version is not None - else [] - ) - - if environmentd_cpu_allocation is None: - target.composition.run( - "mz", "region", "enable", *enable_extra_args(target), *version_args, rm=True - ) - else: - target.composition.run( - "mz", - "region", - "enable", - "--environmentd-cpu-allocation", - str(environmentd_cpu_allocation), - *enable_extra_args(target), - *version_args, - rm=True, - ) + enable_region(target) time.sleep(10) @@ -3471,6 +3446,61 @@ def cloud_disable_enable_and_wait( wait_for_envd(target) +def cloud_recreate_region_with_envd_cpus( + target: "CloudTarget", envd_cpus: int, attempts: int = 3 +) -> None: + """ + Recreate the Cloud region from scratch, with the given environmentd CPU allocation. + + All state in the region is dropped, so the caller has to set up whatever the scenario + needs again. + + We hard-disable rather than soft-disable on purpose. Enabling a region that still has + durable state is a 0dt rollout, and nothing serves SQL while we wait for the new envd to + catch up and promote, because the soft disable already took the old envd away. A hard + disable leaves no predecessor generation to catch up with, so the new envd promotes as + soon as it has booted, in about a minute. + """ + for attempt in range(1, attempts + 1): + disable_region(target.composition, hard=True) + try: + enable_region(target, envd_cpus=envd_cpus) + break + except UIError as e: + # A sweep does a dozen of these calls and the staging Cloud API returns the + # occasional 502, which `mz region enable` does not retry on its own. We + # disable again before retrying, so that a half-finished enable can't leave + # the region running with the previous allocation. + if attempt == attempts: + raise + print( + f"WARNING: 'mz region enable' failed (attempt {attempt}/{attempts}): {e}" + ) + time.sleep(30) + + assert "materialize.cloud" in target.composition.cloud_hostname() + wait_for_envd(target) + assert_region_is_empty(target) + + +def assert_region_is_empty(target: "CloudTarget") -> None: + """ + Assert that the region we can talk to has no user tables. + + Proves that we reached the freshly created region rather than an environmentd that + outlived the recreate: a leftover envd still has the scenario's tables. Neither + `mz region enable` (which only checks that something answers on the SQL port) nor a + successful query on its own can tell the two apart. + """ + host = target.composition.cloud_hostname() + rows = cloud_sql(target, host, "SELECT count(*) FROM mz_tables WHERE id LIKE 'u%'") + + assert rows, "could not count user tables in recreated region" + assert ( + rows[0][0] == 0 + ), f"recreated region still has {rows[0][0]} user table(s), we are talking to a stale environmentd" + + def reconfigure_envd_cpus( target: "BenchTarget", envd_cpus: int, runner: ScenarioRunner ) -> None: @@ -3479,8 +3509,9 @@ def reconfigure_envd_cpus( - Docker target: recreate the local `materialized` container with a CPU limit equal to envd_cpus, wait for SQL readiness, and force the benchmark connection to reconnect. - - Cloud target: soft-disable/enable the region with the desired envd CPU allocation, wait for - SQL readiness, and force the benchmark connection to reconnect. + - Cloud target: recreate the region with the desired envd CPU allocation, wait for SQL + readiness, and force the benchmark connection to reconnect. The recreate drops all + state in the region, see `cloud_recreate_region_with_envd_cpus`. """ if isinstance(target, DockerTarget): # For Docker target: restart `materialized` with a CPU limit equal to envd_cpus. @@ -3511,9 +3542,10 @@ def reconfigure_envd_cpus( raise UIError(f"failed to apply Docker CPU override for environmentd: {e}") else: # Cloud target: reconfigure environmentd CPUs via `mz region`. + assert isinstance(target, CloudTarget) try: print(f"--- Reconfiguring Cloud environmentd CPUs to {envd_cpus}") - cloud_disable_enable_and_wait(target, environmentd_cpu_allocation=envd_cpus) + cloud_recreate_region_with_envd_cpus(target, envd_cpus) except Exception as e: raise UIError( f"failed to apply Cloud CPU override for environmentd via 'mz region': {e}" @@ -3526,89 +3558,129 @@ def reconfigure_envd_cpus( pass -def wait_for_envd(target: "BenchTarget", timeout_secs: int = 300) -> None: +def cloud_sql( + target: "CloudTarget", host: str, query: LiteralString +) -> list[tuple[Any, ...]]: """ - Wait until the environmentd SQL endpoint is ready. + Run `query` against the environmentd at `host` on a fresh connection and return its rows. - - Cloud: uses cloud hostname:6875, sslmode=require, and prefers the per-run - app password if available; falls back to MZ_CLI_APP_PASSWORD. - - Docker: probes SQL readiness via Composition.sql_query + Takes an explicit host rather than looking it up, because a recreated region gets a new + hostname and callers that probe for the *old* envd need to keep talking to the old one. + """ + conn = psycopg.connect( + host=host, + port=6875, + user=target.username, + # Prefer the newly created app password when present, fall back to the CLI one. + password=target.new_app_password or target.app_password or "", + dbname="materialize", + sslmode="require", + connect_timeout=10, + ) + try: + with conn.cursor() as cur: + cur.execute(query) + return cur.fetchall() + finally: + conn.close() + + +def _err_detail(err: Exception | None) -> str: + return f": {err}" if err is not None else "" + + +def _await_probe( + probe: Callable[[], object], + what: str, + *, + until_ok: bool, + timeout_secs: int, + sustain_probes: int = 3, +) -> float: + """ + Poll `probe` until it has either succeeded or failed, per `until_ok`, `sustain_probes` + times in a row, and return how long that took. Raises `UIError` on timeout. + + A single probe decides nothing in either direction. While a region rolls over, a + connection can land on the generation that is about to go away, and a single failure can + be a transient network error against an envd that is still serving fine. + + `what` completes the sentence "waiting for ...", e.g. "environmentd to serve SQL". + """ + start = time.time() + deadline = start + timeout_secs + last_report = start + streak = 0 + last_err: Exception | None = None + + while time.time() < deadline: + try: + probe() + ok = True + last_err = None + except Exception as e: + last_err = e + ok = False + + streak = streak + 1 if ok == until_ok else 0 + if streak >= sustain_probes: + return time.time() - start + + now = time.time() + if now - last_report >= 30: + print( + f" still waiting for {what} ({now - start:.0f}s){_err_detail(last_err)}" + ) + last_report = now + time.sleep(2) + + raise UIError( + f"timed out after {timeout_secs}s waiting for {what}{_err_detail(last_err)}" + ) + + +def wait_for_envd(target: "BenchTarget", timeout_secs: int = 600) -> None: + """ + Wait until environmentd serves SQL, and report how long that took. + + The elapsed time is worth printing: a region that only comes back after minutes is + promoting out of read-only mode, which none of the callers here expect. """ if isinstance(target, CloudTarget): host = target.composition.cloud_hostname() - user = target.username - # Prefer the newly created app password when present; fall back to the CLI password. - password = target.new_app_password or target.app_password or "" - sslmode = "require" - print( - f"Waiting for cloud environmentd at {host}:6875 to come up with username {user} ..." - ) - _wait_for_pg( - host=host, - user=user, - password=password, - port=6875, - query="SELECT 1", - expected=[(1,)], - timeout_secs=timeout_secs, - dbname="materialize", - sslmode=sslmode, - ) + who = f"cloud environmentd at {host}:6875" + probe: Callable[[], object] = partial(cloud_sql, target, host, "SELECT 1") else: - # Docker target: use the composition helper to query the service via the - # host-mapped port on 127.0.0.1; the container hostname "materialized" - # is not resolvable from the host network when using psycopg directly. - print("Waiting for local environmentd (docker) at materialized:6875 ...") - deadline = time.time() + timeout_secs - last_err: Exception | None = None - while time.time() < deadline: - try: - target.composition.sql_query("SELECT 1", service="materialized") - return - except Exception as e: - last_err = e - time.sleep(1) - raise UIError( - f"materialized did not accept SQL connections within {timeout_secs}s after restart: {last_err}" + # The container hostname "materialized" does not resolve from the host network, so + # we go through the composition instead of connecting with psycopg. + who = "local environmentd (docker)" + probe = partial( + target.composition.sql_query, "SELECT 1", service="materialized" ) + print(f"Waiting for {who} to come up ...") + elapsed = _await_probe( + probe, f"{who} to serve SQL", until_ok=True, timeout_secs=timeout_secs + ) + print(f"{who} is serving SQL, after {elapsed:.0f}s") + def wait_for_envd_down( target: "CloudTarget", host: str, timeout_secs: int = 600 ) -> None: """ - Wait until the environmentd SQL endpoint at `host` stops accepting connections. + Wait until the environmentd at `host` stops serving SQL. - Used after a soft `mz region disable` to observe that the teardown has reached the - data plane before the region is enabled again. + Used after a soft `mz region disable` to observe that the teardown has reached the data + plane before the region is enabled again. """ - print(f"Waiting for cloud environmentd at {host}:6875 to go down ...") - password = target.new_app_password or target.app_password or "" - deadline = time.time() + timeout_secs - consecutive_failures = 0 - while time.time() < deadline: - try: - conn = psycopg.connect( - host=host, - port=6875, - user=target.username, - password=password, - dbname="materialize", - sslmode="require", - connect_timeout=10, - ) - conn.close() - consecutive_failures = 0 - except psycopg.Error: - # A single failed attempt could be a transient network error against a - # live envd. Only treat repeated failures as the teardown having landed. - consecutive_failures += 1 - if consecutive_failures >= 3: - return - time.sleep(1) - raise UIError( - f"environmentd at {host}:6875 still accepting SQL connections " - f"{timeout_secs}s after region disable" + who = f"cloud environmentd at {host}:6875" + print(f"Waiting for {who} to go down ...") + _await_probe( + partial(cloud_sql, target, host, "SELECT 1"), + f"{who} to stop serving SQL", + until_ok=False, + timeout_secs=timeout_secs, ) diff --git a/test/pgtest-mz/set-local.pt b/test/pgtest-mz/set-local.pt new file mode 100644 index 0000000000000..0c549b2932941 --- /dev/null +++ b/test/pgtest-mz/set-local.pt @@ -0,0 +1,75 @@ +# Unlike Postgres, which refuses `SET LOCAL` outside a transaction block, we +# apply it for the duration of the implicit transaction and revert it at commit. +# A client that caches `ParameterStatus` (pgjdbc, psycopg) has to hear about the +# revert too, otherwise it keeps serving a value the server has discarded. + +# A session value to revert to. Promoting a non-local SET from staged to session +# does not change the value, so that commit announces nothing. +send +Query {"query": "SET application_name = 'session_value'"} +---- + +until parameter_status=application_name +ReadyForQuery +---- +ParameterStatus {"name":"application_name","value":"session_value"} +CommandComplete {"tag":"SET"} +ReadyForQuery {"status":"I"} + +# Simple protocol: committed at the end of the Query message. +send +Query {"query": "SET LOCAL application_name = 'simple_local'"} +Query {"query": "SHOW application_name"} +---- + +until parameter_status=application_name +ReadyForQuery +ReadyForQuery +---- +ParameterStatus {"name":"application_name","value":"simple_local"} +CommandComplete {"tag":"SET"} +ParameterStatus {"name":"application_name","value":"session_value"} +ReadyForQuery {"status":"I"} +RowDescription {"fields":[{"name":"application_name"}]} +DataRow {"fields":["session_value"]} +CommandComplete {"tag":"SELECT 1"} +ReadyForQuery {"status":"I"} + +# Extended protocol: committed at Sync. +send +Parse {"query": "SET LOCAL application_name = 'extended_local'"} +Bind +Execute +Sync +---- + +until parameter_status=application_name +ReadyForQuery +---- +ParseComplete +BindComplete +ParameterStatus {"name":"application_name","value":"extended_local"} +CommandComplete {"tag":"SET"} +ParameterStatus {"name":"application_name","value":"session_value"} +ReadyForQuery {"status":"I"} + +# Explicit COMMIT already announced the revert; guard the shared path. +send +Query {"query": "BEGIN"} +Query {"query": "SET LOCAL application_name = 'txn_local'"} +Query {"query": "COMMIT"} +---- + +until parameter_status=application_name +ReadyForQuery +ReadyForQuery +ReadyForQuery +---- +CommandComplete {"tag":"BEGIN"} +ReadyForQuery {"status":"T"} +ParameterStatus {"name":"application_name","value":"txn_local"} +CommandComplete {"tag":"SET"} +ReadyForQuery {"status":"T"} +ParameterStatus {"name":"application_name","value":"session_value"} +CommandComplete {"tag":"COMMIT"} +ReadyForQuery {"status":"I"} diff --git a/test/sqllogictest/catalog_server_explain.slt b/test/sqllogictest/catalog_server_explain.slt index 738f4d7a96074..e11092f9d5c18 100644 --- a/test/sqllogictest/catalog_server_explain.slt +++ b/test/sqllogictest/catalog_server_explain.slt @@ -4907,7 +4907,7 @@ mz_catalog.mz_materialized_views: Project: #4, #0, #5, #1, #6, #2, #9, #8, #3, #7 Map: {=r/s1, s1=r/s1}, "s1" →Arrange (#1{schema_name}, #2{name}) - →Constant (29 rows) + →Constant (33 rows) →Arrange (empty key) (#0{schema_name}) (#0{schema_name}, #1{name}) →Fused with Child Map/Filter/Project Project: #4, #3, #5 @@ -5303,6 +5303,102 @@ Target cluster: mz_catalog_server EOF +query T multiline +EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_kafka_source_tables"; +---- +mz_internal.mz_kafka_source_tables: + →Delta Join [%0:mz_catalog_raw[#0] » %1[#0] » %2:mz_sources[#0{id}]] [%1[#0] » %2:mz_sources[#0{id}] » %0:mz_catalog_raw[#0]] [%2:mz_sources[#0{id}] » %1[(#1{details} ->> "source_id")] » %0:mz_catalog_raw[#0]] + path %0: + after %1: + Project: #2, #1, #3..=#6 + Map: ((#2{details} -> "external_reference") ->> 0), (#2{details} ->> "envelope_type"), (#2{details} ->> "key_format"), (#2{details} ->> "value_format") + after %2: + Project: #2..=#6 + Filter: (#10{type} = "kafka") + path %1: + after %2: + Project: #1, #3..=#6 + Filter: (#10{type} = "kafka") + after %0: + Project: #5, #1..=#4 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + path %2: + after %1: + Project: #1, #3..=#6 + Map: ((#2{details} -> "external_reference") ->> 0), (#2{details} ->> "envelope_type"), (#2{details} ->> "key_format"), (#2{details} ->> "value_format") + after %0: + Project: #5, #1..=#4 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + →Arrange (#0) + →Read mz_internal.mz_catalog_raw + →Arrange (#0) ((#1{details} ->> "source_id")) + →Map/Filter/Project + Project: #0, #1 + Map: parse_source_export_details(((((#0{data} -> "value") -> "definition") -> "V1") ->> "create_sql")) + →Distinct GroupAggregate + →Fused with Child Map/Filter/Project + Filter: ((parse_source_export_details(((((#0 -> "value") -> "definition") -> "V1") ->> "create_sql")) ->> "source_id")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arranged mz_catalog.mz_sources + +Source mz_internal.mz_catalog_raw + filter=(("Item" = (#0{data} ->> "kind"))) + +Used Indexes: + - mz_catalog.mz_sources_ind (delta join lookup) + +Target cluster: mz_catalog_server + +EOF + +query T multiline +EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_mysql_source_tables"; +---- +mz_internal.mz_mysql_source_tables: + →Delta Join [%0:mz_catalog_raw[#0] » %1[#0] » %2:mz_sources[#0{id}]] [%1[#0] » %2:mz_sources[#0{id}] » %0:mz_catalog_raw[#0]] [%2:mz_sources[#0{id}] » %1[(#1{details} ->> "source_id")] » %0:mz_catalog_raw[#0]] + path %0: + after %1: + Project: #2, #1, #4, #5 + Map: (#2{details} -> "external_reference"), (#3 ->> 0), (#3 ->> 1) + after %2: + Project: #2..=#4 + Filter: (#8{type} = "mysql") + path %1: + after %2: + Project: #1, #3, #4 + Filter: (#8{type} = "mysql") + after %0: + Project: #3, #1, #2 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + path %2: + after %1: + Project: #1, #4, #5 + Map: (#2{details} -> "external_reference"), (#3 ->> 0), (#3 ->> 1) + after %0: + Project: #3, #1, #2 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + →Arrange (#0) + →Read mz_internal.mz_catalog_raw + →Arrange (#0) ((#1{details} ->> "source_id")) + →Map/Filter/Project + Project: #0, #1 + Map: parse_source_export_details(((((#0{data} -> "value") -> "definition") -> "V1") ->> "create_sql")) + →Distinct GroupAggregate + →Fused with Child Map/Filter/Project + Filter: ((parse_source_export_details(((((#0 -> "value") -> "definition") -> "V1") ->> "create_sql")) ->> "source_id")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arranged mz_catalog.mz_sources + +Source mz_internal.mz_catalog_raw + filter=(("Item" = (#0{data} ->> "kind"))) + +Used Indexes: + - mz_catalog.mz_sources_ind (delta join lookup) + +Target cluster: mz_catalog_server + +EOF + query T multiline EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_network_policies"; ---- @@ -5365,6 +5461,54 @@ Target cluster: mz_catalog_server EOF +query T multiline +EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_postgres_source_tables"; +---- +mz_internal.mz_postgres_source_tables: + →Delta Join [%0:mz_catalog_raw[#0] » %1[#0] » %2:mz_sources[#0{id}]] [%1[#0] » %2:mz_sources[#0{id}] » %0:mz_catalog_raw[#0]] [%2:mz_sources[#0{id}] » %1[(#1{details} ->> "source_id")] » %0:mz_catalog_raw[#0]] + path %0: + after %1: + Project: #2, #1, #4, #5 + Map: (#2{details} -> "external_reference"), (#3 ->> 1), (#3 ->> 2) + after %2: + Project: #2..=#4 + Filter: (#8{type} = "postgres") + path %1: + after %2: + Project: #1, #3, #4 + Filter: (#8{type} = "postgres") + after %0: + Project: #3, #1, #2 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + path %2: + after %1: + Project: #1, #4, #5 + Map: (#2{details} -> "external_reference"), (#3 ->> 1), (#3 ->> 2) + after %0: + Project: #3, #1, #2 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + →Arrange (#0) + →Read mz_internal.mz_catalog_raw + →Arrange (#0) ((#1{details} ->> "source_id")) + →Map/Filter/Project + Project: #0, #1 + Map: parse_source_export_details(((((#0{data} -> "value") -> "definition") -> "V1") ->> "create_sql")) + →Distinct GroupAggregate + →Fused with Child Map/Filter/Project + Filter: ((parse_source_export_details(((((#0 -> "value") -> "definition") -> "V1") ->> "create_sql")) ->> "source_id")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arranged mz_catalog.mz_sources + +Source mz_internal.mz_catalog_raw + filter=(("Item" = (#0{data} ->> "kind"))) + +Used Indexes: + - mz_catalog.mz_sources_ind (delta join lookup) + +Target cluster: mz_catalog_server + +EOF + query T multiline EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_postgres_sources"; ---- @@ -5395,6 +5539,54 @@ Target cluster: mz_catalog_server EOF +query T multiline +EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_sql_server_source_tables"; +---- +mz_internal.mz_sql_server_source_tables: + →Delta Join [%0:mz_catalog_raw[#0] » %1[#0] » %2:mz_sources[#0{id}]] [%1[#0] » %2:mz_sources[#0{id}] » %0:mz_catalog_raw[#0]] [%2:mz_sources[#0{id}] » %1[(#1{details} ->> "source_id")] » %0:mz_catalog_raw[#0]] + path %0: + after %1: + Project: #2, #1, #4, #5 + Map: (#2{details} -> "external_reference"), (#3 ->> 1), (#3 ->> 2) + after %2: + Project: #2..=#4 + Filter: (#8{type} = "sql-server") + path %1: + after %2: + Project: #1, #3, #4 + Filter: (#8{type} = "sql-server") + after %0: + Project: #3, #1, #2 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + path %2: + after %1: + Project: #1, #4, #5 + Map: (#2{details} -> "external_reference"), (#3 ->> 1), (#3 ->> 2) + after %0: + Project: #3, #1, #2 + Map: parse_catalog_id(((#0{data} -> "key") -> "gid")) + →Arrange (#0) + →Read mz_internal.mz_catalog_raw + →Arrange (#0) ((#1{details} ->> "source_id")) + →Map/Filter/Project + Project: #0, #1 + Map: parse_source_export_details(((((#0{data} -> "value") -> "definition") -> "V1") ->> "create_sql")) + →Distinct GroupAggregate + →Fused with Child Map/Filter/Project + Filter: ((parse_source_export_details(((((#0 -> "value") -> "definition") -> "V1") ->> "create_sql")) ->> "source_id")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arranged mz_catalog.mz_sources + +Source mz_internal.mz_catalog_raw + filter=(("Item" = (#0{data} ->> "kind"))) + +Used Indexes: + - mz_catalog.mz_sources_ind (delta join lookup) + +Target cluster: mz_catalog_server + +EOF + query T multiline EXPLAIN SELECT * FROM "information_schema"."applicable_roles"; ---- @@ -7650,7 +7842,7 @@ query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_builtin_materialized_views"; ---- Explained Query (fast path): - →Constant (29 rows) + →Constant (33 rows) Target cluster: mz_catalog_server diff --git a/test/sqllogictest/information_schema_tables.slt b/test/sqllogictest/information_schema_tables.slt index 10c48392eea69..9faa98ae3cf76 100644 --- a/test/sqllogictest/information_schema_tables.slt +++ b/test/sqllogictest/information_schema_tables.slt @@ -422,7 +422,7 @@ MATERIALIZED VIEW materialize mz_internal mz_kafka_source_tables -BASE TABLE +MATERIALIZED VIEW materialize mz_internal mz_license_keys @@ -454,7 +454,7 @@ VIEW materialize mz_internal mz_mysql_source_tables -BASE TABLE +MATERIALIZED VIEW materialize mz_internal mz_network_policies @@ -542,7 +542,7 @@ MATERIALIZED VIEW materialize mz_internal mz_postgres_source_tables -BASE TABLE +MATERIALIZED VIEW materialize mz_internal mz_postgres_sources @@ -762,7 +762,7 @@ VIEW materialize mz_internal mz_sql_server_source_tables -BASE TABLE +MATERIALIZED VIEW materialize mz_internal mz_sql_text diff --git a/test/sqllogictest/mz_catalog_server_index_accounting.slt b/test/sqllogictest/mz_catalog_server_index_accounting.slt index 0259167bc76bf..53ef49be9947b 100644 --- a/test/sqllogictest/mz_catalog_server_index_accounting.slt +++ b/test/sqllogictest/mz_catalog_server_index_accounting.slt @@ -37,107 +37,107 @@ mz_arrangement_heap_capacity_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangemen mz_arrangement_heap_size_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_heap_size_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_heap_size_raw"␠("operator_id",␠"worker_id") mz_arrangement_records_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_records_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_records_raw"␠("operator_id",␠"worker_id") mz_arrangement_sharing_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_sharing_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_sharing_raw"␠("operator_id",␠"worker_id") -mz_cluster_auto_scaling_strategies_ind CREATE␠INDEX␠"mz_cluster_auto_scaling_strategies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s513␠AS␠"mz_internal"."mz_cluster_auto_scaling_strategies"]␠("cluster_id") -mz_cluster_deployment_lineage_ind CREATE␠INDEX␠"mz_cluster_deployment_lineage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s760␠AS␠"mz_internal"."mz_cluster_deployment_lineage"]␠("cluster_id") +mz_cluster_auto_scaling_strategies_ind CREATE␠INDEX␠"mz_cluster_auto_scaling_strategies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s514␠AS␠"mz_internal"."mz_cluster_auto_scaling_strategies"]␠("cluster_id") +mz_cluster_deployment_lineage_ind CREATE␠INDEX␠"mz_cluster_deployment_lineage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s761␠AS␠"mz_internal"."mz_cluster_deployment_lineage"]␠("cluster_id") mz_cluster_prometheus_metrics_s2_primary_idx CREATE␠INDEX␠"mz_cluster_prometheus_metrics_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_cluster_prometheus_metrics"␠("process_id",␠"metric_name",␠"labels") -mz_cluster_reconfigurations_ind CREATE␠INDEX␠"mz_cluster_reconfigurations_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s512␠AS␠"mz_internal"."mz_cluster_reconfigurations"]␠("cluster_id") -mz_cluster_replica_frontiers_ind CREATE␠INDEX␠"mz_cluster_replica_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s754␠AS␠"mz_catalog"."mz_cluster_replica_frontiers"]␠("object_id") -mz_cluster_replica_history_ind CREATE␠INDEX␠"mz_cluster_replica_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s610␠AS␠"mz_internal"."mz_cluster_replica_history"]␠("dropped_at") -mz_cluster_replica_metrics_history_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s518␠AS␠"mz_internal"."mz_cluster_replica_metrics_history"]␠("replica_id") -mz_cluster_replica_metrics_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s519␠AS␠"mz_internal"."mz_cluster_replica_metrics"]␠("replica_id") -mz_cluster_replica_name_history_ind CREATE␠INDEX␠"mz_cluster_replica_name_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s611␠AS␠"mz_internal"."mz_cluster_replica_name_history"]␠("id") -mz_cluster_replica_size_internal_ind CREATE␠INDEX␠"mz_cluster_replica_size_internal_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s508␠AS␠"mz_internal"."mz_cluster_replica_size_internal"]␠("size") -mz_cluster_replica_sizes_ind CREATE␠INDEX␠"mz_cluster_replica_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s507␠AS␠"mz_catalog"."mz_cluster_replica_sizes"]␠("size") -mz_cluster_replica_status_history_ind CREATE␠INDEX␠"mz_cluster_replica_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s520␠AS␠"mz_internal"."mz_cluster_replica_status_history"]␠("replica_id") -mz_cluster_replica_statuses_ind CREATE␠INDEX␠"mz_cluster_replica_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s521␠AS␠"mz_internal"."mz_cluster_replica_statuses"]␠("replica_id") -mz_cluster_replicas_ind CREATE␠INDEX␠"mz_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s517␠AS␠"mz_catalog"."mz_cluster_replicas"]␠("id") -mz_clusters_ind CREATE␠INDEX␠"mz_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s509␠AS␠"mz_catalog"."mz_clusters"]␠("id") -mz_columns_ind CREATE␠INDEX␠"mz_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s480␠AS␠"mz_catalog"."mz_columns"]␠("name") -mz_comments_ind CREATE␠INDEX␠"mz_comments_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s536␠AS␠"mz_internal"."mz_comments"]␠("id") +mz_cluster_reconfigurations_ind CREATE␠INDEX␠"mz_cluster_reconfigurations_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s513␠AS␠"mz_internal"."mz_cluster_reconfigurations"]␠("cluster_id") +mz_cluster_replica_frontiers_ind CREATE␠INDEX␠"mz_cluster_replica_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s755␠AS␠"mz_catalog"."mz_cluster_replica_frontiers"]␠("object_id") +mz_cluster_replica_history_ind CREATE␠INDEX␠"mz_cluster_replica_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s611␠AS␠"mz_internal"."mz_cluster_replica_history"]␠("dropped_at") +mz_cluster_replica_metrics_history_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s519␠AS␠"mz_internal"."mz_cluster_replica_metrics_history"]␠("replica_id") +mz_cluster_replica_metrics_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s520␠AS␠"mz_internal"."mz_cluster_replica_metrics"]␠("replica_id") +mz_cluster_replica_name_history_ind CREATE␠INDEX␠"mz_cluster_replica_name_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s612␠AS␠"mz_internal"."mz_cluster_replica_name_history"]␠("id") +mz_cluster_replica_size_internal_ind CREATE␠INDEX␠"mz_cluster_replica_size_internal_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s509␠AS␠"mz_internal"."mz_cluster_replica_size_internal"]␠("size") +mz_cluster_replica_sizes_ind CREATE␠INDEX␠"mz_cluster_replica_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s508␠AS␠"mz_catalog"."mz_cluster_replica_sizes"]␠("size") +mz_cluster_replica_status_history_ind CREATE␠INDEX␠"mz_cluster_replica_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s521␠AS␠"mz_internal"."mz_cluster_replica_status_history"]␠("replica_id") +mz_cluster_replica_statuses_ind CREATE␠INDEX␠"mz_cluster_replica_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s522␠AS␠"mz_internal"."mz_cluster_replica_statuses"]␠("replica_id") +mz_cluster_replicas_ind CREATE␠INDEX␠"mz_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s518␠AS␠"mz_catalog"."mz_cluster_replicas"]␠("id") +mz_clusters_ind CREATE␠INDEX␠"mz_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s510␠AS␠"mz_catalog"."mz_clusters"]␠("id") +mz_columns_ind CREATE␠INDEX␠"mz_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s481␠AS␠"mz_catalog"."mz_columns"]␠("name") +mz_comments_ind CREATE␠INDEX␠"mz_comments_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s537␠AS␠"mz_internal"."mz_comments"]␠("id") mz_compute_dataflow_global_ids_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_dataflow_global_ids_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_dataflow_global_ids_per_worker"␠("id",␠"worker_id",␠"global_id") -mz_compute_dependencies_ind CREATE␠INDEX␠"mz_compute_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s736␠AS␠"mz_internal"."mz_compute_dependencies"]␠("dependency_id") +mz_compute_dependencies_ind CREATE␠INDEX␠"mz_compute_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s737␠AS␠"mz_internal"."mz_compute_dependencies"]␠("dependency_id") mz_compute_error_counts_raw_s2_primary_idx CREATE␠INDEX␠"mz_compute_error_counts_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_error_counts_raw"␠("export_id",␠"worker_id") mz_compute_exports_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_exports_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_exports_per_worker"␠("export_id",␠"worker_id") mz_compute_frontiers_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_frontiers_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_frontiers_per_worker"␠("export_id",␠"worker_id") -mz_compute_hydration_times_ind CREATE␠INDEX␠"mz_compute_hydration_times_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s745␠AS␠"mz_internal"."mz_compute_hydration_times"]␠("replica_id") +mz_compute_hydration_times_ind CREATE␠INDEX␠"mz_compute_hydration_times_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s746␠AS␠"mz_internal"."mz_compute_hydration_times"]␠("replica_id") mz_compute_hydration_times_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_hydration_times_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_hydration_times_per_worker"␠("export_id",␠"worker_id") mz_compute_import_frontiers_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_import_frontiers_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_import_frontiers_per_worker"␠("export_id",␠"import_id",␠"worker_id") mz_compute_lir_mapping_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_lir_mapping_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_lir_mapping_per_worker"␠("global_id",␠"lir_id",␠"worker_id") mz_compute_operator_durations_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_compute_operator_durations_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_operator_durations_histogram_raw"␠("id",␠"worker_id",␠"duration_ns") mz_compute_operator_hydration_statuses_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_operator_hydration_statuses_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_operator_hydration_statuses_per_worker"␠("export_id",␠"lir_id",␠"worker_id") -mz_connections_ind CREATE␠INDEX␠"mz_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s515␠AS␠"mz_catalog"."mz_connections"]␠("schema_id") -mz_console_cluster_utilization_overview_24h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_24h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s741␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_24h"]␠("cluster_id") -mz_console_cluster_utilization_overview_3h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_3h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s740␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_3h"]␠("cluster_id") -mz_console_cluster_utilization_overview_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s739␠AS␠"mz_internal"."mz_console_cluster_utilization_overview"]␠("cluster_id") -mz_databases_ind CREATE␠INDEX␠"mz_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s478␠AS␠"mz_catalog"."mz_databases"]␠("name") +mz_connections_ind CREATE␠INDEX␠"mz_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s516␠AS␠"mz_catalog"."mz_connections"]␠("schema_id") +mz_console_cluster_utilization_overview_24h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_24h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s742␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_24h"]␠("cluster_id") +mz_console_cluster_utilization_overview_3h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_3h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s741␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_3h"]␠("cluster_id") +mz_console_cluster_utilization_overview_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s740␠AS␠"mz_internal"."mz_console_cluster_utilization_overview"]␠("cluster_id") +mz_databases_ind CREATE␠INDEX␠"mz_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s479␠AS␠"mz_catalog"."mz_databases"]␠("name") mz_dataflow_addresses_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_addresses_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_addresses_per_worker"␠("id",␠"worker_id") mz_dataflow_channels_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_channels_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_channels_per_worker"␠("id",␠"worker_id") mz_dataflow_operator_reachability_raw_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_operator_reachability_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_operator_reachability_raw"␠("id",␠"worker_id",␠"source",␠"port",␠"update_type",␠"time") mz_dataflow_operators_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_operators_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_operators_per_worker"␠("id",␠"worker_id") -mz_frontiers_ind CREATE␠INDEX␠"mz_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s727␠AS␠"mz_internal"."mz_frontiers"]␠("object_id") -mz_hydration_statuses_ind CREATE␠INDEX␠"mz_hydration_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s756␠AS␠"mz_internal"."mz_hydration_statuses"]␠("object_id",␠"replica_id") -mz_indexes_ind CREATE␠INDEX␠"mz_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s481␠AS␠"mz_catalog"."mz_indexes"]␠("id") -mz_kafka_sources_ind CREATE␠INDEX␠"mz_kafka_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s475␠AS␠"mz_catalog"."mz_kafka_sources"]␠("id") -mz_materialized_views_ind CREATE␠INDEX␠"mz_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s539␠AS␠"mz_catalog"."mz_materialized_views"]␠("id") +mz_frontiers_ind CREATE␠INDEX␠"mz_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s728␠AS␠"mz_internal"."mz_frontiers"]␠("object_id") +mz_hydration_statuses_ind CREATE␠INDEX␠"mz_hydration_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s757␠AS␠"mz_internal"."mz_hydration_statuses"]␠("object_id",␠"replica_id") +mz_indexes_ind CREATE␠INDEX␠"mz_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s482␠AS␠"mz_catalog"."mz_indexes"]␠("id") +mz_kafka_sources_ind CREATE␠INDEX␠"mz_kafka_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s476␠AS␠"mz_catalog"."mz_kafka_sources"]␠("id") +mz_materialized_views_ind CREATE␠INDEX␠"mz_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s540␠AS␠"mz_catalog"."mz_materialized_views"]␠("id") mz_message_batch_counts_received_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_batch_counts_received_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_batch_counts_received_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_batch_counts_sent_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_batch_counts_sent_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_batch_counts_sent_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_counts_received_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_counts_received_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_counts_received_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_counts_sent_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_counts_sent_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_counts_sent_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") -mz_notices_ind CREATE␠INDEX␠"mz_notices_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s830␠AS␠"mz_internal"."mz_notices"]␠("id") -mz_object_arrangement_size_history_object_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_object_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s748␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("object_id") -mz_object_arrangement_size_history_ts_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_ts_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s748␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("collection_timestamp") -mz_object_arrangement_sizes_ind CREATE␠INDEX␠"mz_object_arrangement_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s746␠AS␠"mz_internal"."mz_object_arrangement_sizes"]␠("replica_id") -mz_object_dependencies_ind CREATE␠INDEX␠"mz_object_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s476␠AS␠"mz_internal"."mz_object_dependencies"]␠("object_id") -mz_object_history_ind CREATE␠INDEX␠"mz_object_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s550␠AS␠"mz_internal"."mz_object_history"]␠("id") -mz_object_lifetimes_ind CREATE␠INDEX␠"mz_object_lifetimes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s551␠AS␠"mz_internal"."mz_object_lifetimes"]␠("id") -mz_object_transitive_dependencies_ind CREATE␠INDEX␠"mz_object_transitive_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s567␠AS␠"mz_internal"."mz_object_transitive_dependencies"]␠("object_id") -mz_objects_ind CREATE␠INDEX␠"mz_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s547␠AS␠"mz_catalog"."mz_objects"]␠("schema_id") +mz_notices_ind CREATE␠INDEX␠"mz_notices_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s831␠AS␠"mz_internal"."mz_notices"]␠("id") +mz_object_arrangement_size_history_object_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_object_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s749␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("object_id") +mz_object_arrangement_size_history_ts_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_ts_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s749␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("collection_timestamp") +mz_object_arrangement_sizes_ind CREATE␠INDEX␠"mz_object_arrangement_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s747␠AS␠"mz_internal"."mz_object_arrangement_sizes"]␠("replica_id") +mz_object_dependencies_ind CREATE␠INDEX␠"mz_object_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s477␠AS␠"mz_internal"."mz_object_dependencies"]␠("object_id") +mz_object_history_ind CREATE␠INDEX␠"mz_object_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s551␠AS␠"mz_internal"."mz_object_history"]␠("id") +mz_object_lifetimes_ind CREATE␠INDEX␠"mz_object_lifetimes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s552␠AS␠"mz_internal"."mz_object_lifetimes"]␠("id") +mz_object_transitive_dependencies_ind CREATE␠INDEX␠"mz_object_transitive_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s568␠AS␠"mz_internal"."mz_object_transitive_dependencies"]␠("object_id") +mz_objects_ind CREATE␠INDEX␠"mz_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s548␠AS␠"mz_catalog"."mz_objects"]␠("schema_id") mz_peek_durations_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_peek_durations_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_peek_durations_histogram_raw"␠("worker_id",␠"type",␠"duration_ns") -mz_recent_activity_log_thinned_ind CREATE␠INDEX␠"mz_recent_activity_log_thinned_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s711␠AS␠"mz_internal"."mz_recent_activity_log_thinned"]␠("sql_hash") -mz_recent_sql_text_ind CREATE␠INDEX␠"mz_recent_sql_text_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s707␠AS␠"mz_internal"."mz_recent_sql_text"]␠("sql_hash") -mz_recent_storage_usage_ind CREATE␠INDEX␠"mz_recent_storage_usage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s823␠AS␠"mz_catalog"."mz_recent_storage_usage"]␠("object_id") -mz_roles_ind CREATE␠INDEX␠"mz_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s499␠AS␠"mz_catalog"."mz_roles"]␠("id") +mz_recent_activity_log_thinned_ind CREATE␠INDEX␠"mz_recent_activity_log_thinned_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s712␠AS␠"mz_internal"."mz_recent_activity_log_thinned"]␠("sql_hash") +mz_recent_sql_text_ind CREATE␠INDEX␠"mz_recent_sql_text_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s708␠AS␠"mz_internal"."mz_recent_sql_text"]␠("sql_hash") +mz_recent_storage_usage_ind CREATE␠INDEX␠"mz_recent_storage_usage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s824␠AS␠"mz_catalog"."mz_recent_storage_usage"]␠("object_id") +mz_roles_ind CREATE␠INDEX␠"mz_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s500␠AS␠"mz_catalog"."mz_roles"]␠("id") mz_scheduling_elapsed_raw_s2_primary_idx CREATE␠INDEX␠"mz_scheduling_elapsed_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_scheduling_elapsed_raw"␠("id",␠"worker_id") mz_scheduling_parks_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_scheduling_parks_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_scheduling_parks_histogram_raw"␠("worker_id",␠"slept_for_ns",␠"requested_ns") -mz_schemas_ind CREATE␠INDEX␠"mz_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s479␠AS␠"mz_catalog"."mz_schemas"]␠("database_id") -mz_secrets_ind CREATE␠INDEX␠"mz_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s514␠AS␠"mz_catalog"."mz_secrets"]␠("name") -mz_show_all_objects_ind CREATE␠INDEX␠"mz_show_all_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s595␠AS␠"mz_internal"."mz_show_all_objects"]␠("schema_id") -mz_show_cluster_replicas_ind CREATE␠INDEX␠"mz_show_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s758␠AS␠"mz_internal"."mz_show_cluster_replicas"]␠("cluster") -mz_show_clusters_ind CREATE␠INDEX␠"mz_show_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s597␠AS␠"mz_internal"."mz_show_clusters"]␠("name") -mz_show_columns_ind CREATE␠INDEX␠"mz_show_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s596␠AS␠"mz_internal"."mz_show_columns"]␠("id") -mz_show_connections_ind CREATE␠INDEX␠"mz_show_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s605␠AS␠"mz_internal"."mz_show_connections"]␠("schema_id") -mz_show_databases_ind CREATE␠INDEX␠"mz_show_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s599␠AS␠"mz_internal"."mz_show_databases"]␠("name") -mz_show_indexes_ind CREATE␠INDEX␠"mz_show_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s609␠AS␠"mz_internal"."mz_show_indexes"]␠("schema_id") -mz_show_materialized_views_ind CREATE␠INDEX␠"mz_show_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s608␠AS␠"mz_internal"."mz_show_materialized_views"]␠("schema_id") -mz_show_roles_ind CREATE␠INDEX␠"mz_show_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s604␠AS␠"mz_internal"."mz_show_roles"]␠("name") -mz_show_schemas_ind CREATE␠INDEX␠"mz_show_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s600␠AS␠"mz_internal"."mz_show_schemas"]␠("database_id") -mz_show_secrets_ind CREATE␠INDEX␠"mz_show_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s598␠AS␠"mz_internal"."mz_show_secrets"]␠("schema_id") -mz_show_sinks_ind CREATE␠INDEX␠"mz_show_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s607␠AS␠"mz_internal"."mz_show_sinks"]␠("schema_id") -mz_show_sources_ind CREATE␠INDEX␠"mz_show_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s606␠AS␠"mz_internal"."mz_show_sources"]␠("schema_id") -mz_show_tables_ind CREATE␠INDEX␠"mz_show_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s601␠AS␠"mz_internal"."mz_show_tables"]␠("schema_id") -mz_show_types_ind CREATE␠INDEX␠"mz_show_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s603␠AS␠"mz_internal"."mz_show_types"]␠("schema_id") -mz_show_views_ind CREATE␠INDEX␠"mz_show_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s602␠AS␠"mz_internal"."mz_show_views"]␠("schema_id") -mz_sink_statistics_ind CREATE␠INDEX␠"mz_sink_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s724␠AS␠"mz_internal"."mz_sink_statistics"]␠("id",␠"replica_id") -mz_sink_status_history_ind CREATE␠INDEX␠"mz_sink_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s696␠AS␠"mz_internal"."mz_sink_status_history"]␠("sink_id") -mz_sink_statuses_ind CREATE␠INDEX␠"mz_sink_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s697␠AS␠"mz_internal"."mz_sink_statuses"]␠("id") -mz_sinks_ind CREATE␠INDEX␠"mz_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s491␠AS␠"mz_catalog"."mz_sinks"]␠("id") -mz_source_statistics_ind CREATE␠INDEX␠"mz_source_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s722␠AS␠"mz_internal"."mz_source_statistics"]␠("id",␠"replica_id") -mz_source_statistics_with_history_ind CREATE␠INDEX␠"mz_source_statistics_with_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s720␠AS␠"mz_internal"."mz_source_statistics_with_history"]␠("id",␠"replica_id") -mz_source_status_history_ind CREATE␠INDEX␠"mz_source_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s698␠AS␠"mz_internal"."mz_source_status_history"]␠("source_id") -mz_source_statuses_ind CREATE␠INDEX␠"mz_source_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s715␠AS␠"mz_internal"."mz_source_statuses"]␠("id") -mz_sources_ind CREATE␠INDEX␠"mz_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s484␠AS␠"mz_catalog"."mz_sources"]␠("id") -mz_tables_ind CREATE␠INDEX␠"mz_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s483␠AS␠"mz_catalog"."mz_tables"]␠("schema_id") -mz_types_ind CREATE␠INDEX␠"mz_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s493␠AS␠"mz_catalog"."mz_types"]␠("schema_id") -mz_views_ind CREATE␠INDEX␠"mz_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s492␠AS␠"mz_catalog"."mz_views"]␠("schema_id") -mz_wallclock_global_lag_recent_history_ind CREATE␠INDEX␠"mz_wallclock_global_lag_recent_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s731␠AS␠"mz_internal"."mz_wallclock_global_lag_recent_history"]␠("object_id") -mz_webhook_sources_ind CREATE␠INDEX␠"mz_webhook_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s537␠AS␠"mz_internal"."mz_webhook_sources"]␠("id") -pg_attrdef_all_databases_ind CREATE␠INDEX␠"pg_attrdef_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s636␠AS␠"mz_internal"."pg_attrdef_all_databases"]␠("oid",␠"adrelid",␠"adnum",␠"adbin",␠"adsrc") -pg_attribute_all_databases_ind CREATE␠INDEX␠"pg_attribute_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s629␠AS␠"mz_internal"."pg_attribute_all_databases"]␠("attrelid",␠"attname",␠"atttypid",␠"attlen",␠"attnum",␠"atttypmod",␠"attnotnull",␠"atthasdef",␠"attidentity",␠"attgenerated",␠"attisdropped",␠"attcollation",␠"database_name",␠"pg_type_database_name") -pg_authid_core_ind CREATE␠INDEX␠"pg_authid_core_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s646␠AS␠"mz_internal"."pg_authid_core"]␠("rolname") -pg_class_all_databases_ind CREATE␠INDEX␠"pg_class_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s617␠AS␠"mz_internal"."pg_class_all_databases"]␠("relname") -pg_description_all_databases_ind CREATE␠INDEX␠"pg_description_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s626␠AS␠"mz_internal"."pg_description_all_databases"]␠("objoid",␠"classoid",␠"objsubid",␠"description",␠"oid_database_name",␠"class_database_name") -pg_namespace_all_databases_ind CREATE␠INDEX␠"pg_namespace_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s614␠AS␠"mz_internal"."pg_namespace_all_databases"]␠("nspname") -pg_type_all_databases_ind CREATE␠INDEX␠"pg_type_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s623␠AS␠"mz_internal"."pg_type_all_databases"]␠("oid") +mz_schemas_ind CREATE␠INDEX␠"mz_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s480␠AS␠"mz_catalog"."mz_schemas"]␠("database_id") +mz_secrets_ind CREATE␠INDEX␠"mz_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s515␠AS␠"mz_catalog"."mz_secrets"]␠("name") +mz_show_all_objects_ind CREATE␠INDEX␠"mz_show_all_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s596␠AS␠"mz_internal"."mz_show_all_objects"]␠("schema_id") +mz_show_cluster_replicas_ind CREATE␠INDEX␠"mz_show_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s759␠AS␠"mz_internal"."mz_show_cluster_replicas"]␠("cluster") +mz_show_clusters_ind CREATE␠INDEX␠"mz_show_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s598␠AS␠"mz_internal"."mz_show_clusters"]␠("name") +mz_show_columns_ind CREATE␠INDEX␠"mz_show_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s597␠AS␠"mz_internal"."mz_show_columns"]␠("id") +mz_show_connections_ind CREATE␠INDEX␠"mz_show_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s606␠AS␠"mz_internal"."mz_show_connections"]␠("schema_id") +mz_show_databases_ind CREATE␠INDEX␠"mz_show_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s600␠AS␠"mz_internal"."mz_show_databases"]␠("name") +mz_show_indexes_ind CREATE␠INDEX␠"mz_show_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s610␠AS␠"mz_internal"."mz_show_indexes"]␠("schema_id") +mz_show_materialized_views_ind CREATE␠INDEX␠"mz_show_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s609␠AS␠"mz_internal"."mz_show_materialized_views"]␠("schema_id") +mz_show_roles_ind CREATE␠INDEX␠"mz_show_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s605␠AS␠"mz_internal"."mz_show_roles"]␠("name") +mz_show_schemas_ind CREATE␠INDEX␠"mz_show_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s601␠AS␠"mz_internal"."mz_show_schemas"]␠("database_id") +mz_show_secrets_ind CREATE␠INDEX␠"mz_show_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s599␠AS␠"mz_internal"."mz_show_secrets"]␠("schema_id") +mz_show_sinks_ind CREATE␠INDEX␠"mz_show_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s608␠AS␠"mz_internal"."mz_show_sinks"]␠("schema_id") +mz_show_sources_ind CREATE␠INDEX␠"mz_show_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s607␠AS␠"mz_internal"."mz_show_sources"]␠("schema_id") +mz_show_tables_ind CREATE␠INDEX␠"mz_show_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s602␠AS␠"mz_internal"."mz_show_tables"]␠("schema_id") +mz_show_types_ind CREATE␠INDEX␠"mz_show_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s604␠AS␠"mz_internal"."mz_show_types"]␠("schema_id") +mz_show_views_ind CREATE␠INDEX␠"mz_show_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s603␠AS␠"mz_internal"."mz_show_views"]␠("schema_id") +mz_sink_statistics_ind CREATE␠INDEX␠"mz_sink_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s725␠AS␠"mz_internal"."mz_sink_statistics"]␠("id",␠"replica_id") +mz_sink_status_history_ind CREATE␠INDEX␠"mz_sink_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s697␠AS␠"mz_internal"."mz_sink_status_history"]␠("sink_id") +mz_sink_statuses_ind CREATE␠INDEX␠"mz_sink_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s698␠AS␠"mz_internal"."mz_sink_statuses"]␠("id") +mz_sinks_ind CREATE␠INDEX␠"mz_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s492␠AS␠"mz_catalog"."mz_sinks"]␠("id") +mz_source_statistics_ind CREATE␠INDEX␠"mz_source_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s723␠AS␠"mz_internal"."mz_source_statistics"]␠("id",␠"replica_id") +mz_source_statistics_with_history_ind CREATE␠INDEX␠"mz_source_statistics_with_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s721␠AS␠"mz_internal"."mz_source_statistics_with_history"]␠("id",␠"replica_id") +mz_source_status_history_ind CREATE␠INDEX␠"mz_source_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s699␠AS␠"mz_internal"."mz_source_status_history"]␠("source_id") +mz_source_statuses_ind CREATE␠INDEX␠"mz_source_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s716␠AS␠"mz_internal"."mz_source_statuses"]␠("id") +mz_sources_ind CREATE␠INDEX␠"mz_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s485␠AS␠"mz_catalog"."mz_sources"]␠("id") +mz_tables_ind CREATE␠INDEX␠"mz_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s484␠AS␠"mz_catalog"."mz_tables"]␠("schema_id") +mz_types_ind CREATE␠INDEX␠"mz_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s494␠AS␠"mz_catalog"."mz_types"]␠("schema_id") +mz_views_ind CREATE␠INDEX␠"mz_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s493␠AS␠"mz_catalog"."mz_views"]␠("schema_id") +mz_wallclock_global_lag_recent_history_ind CREATE␠INDEX␠"mz_wallclock_global_lag_recent_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s732␠AS␠"mz_internal"."mz_wallclock_global_lag_recent_history"]␠("object_id") +mz_webhook_sources_ind CREATE␠INDEX␠"mz_webhook_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s538␠AS␠"mz_internal"."mz_webhook_sources"]␠("id") +pg_attrdef_all_databases_ind CREATE␠INDEX␠"pg_attrdef_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s637␠AS␠"mz_internal"."pg_attrdef_all_databases"]␠("oid",␠"adrelid",␠"adnum",␠"adbin",␠"adsrc") +pg_attribute_all_databases_ind CREATE␠INDEX␠"pg_attribute_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s630␠AS␠"mz_internal"."pg_attribute_all_databases"]␠("attrelid",␠"attname",␠"atttypid",␠"attlen",␠"attnum",␠"atttypmod",␠"attnotnull",␠"atthasdef",␠"attidentity",␠"attgenerated",␠"attisdropped",␠"attcollation",␠"database_name",␠"pg_type_database_name") +pg_authid_core_ind CREATE␠INDEX␠"pg_authid_core_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s647␠AS␠"mz_internal"."pg_authid_core"]␠("rolname") +pg_class_all_databases_ind CREATE␠INDEX␠"pg_class_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s618␠AS␠"mz_internal"."pg_class_all_databases"]␠("relname") +pg_description_all_databases_ind CREATE␠INDEX␠"pg_description_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s627␠AS␠"mz_internal"."pg_description_all_databases"]␠("objoid",␠"classoid",␠"objsubid",␠"description",␠"oid_database_name",␠"class_database_name") +pg_namespace_all_databases_ind CREATE␠INDEX␠"pg_namespace_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s615␠AS␠"mz_internal"."pg_namespace_all_databases"]␠("nspname") +pg_type_all_databases_ind CREATE␠INDEX␠"pg_type_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s624␠AS␠"mz_internal"."pg_type_all_databases"]␠("oid") # Record all transitive dependencies (tables, sources, views, mvs) of indexes on # the mz_catalog_server cluster. diff --git a/test/sqllogictest/mz_kafka_source_tables.slt b/test/sqllogictest/mz_kafka_source_tables.slt new file mode 100644 index 0000000000000..2bdc3acd5f3ae --- /dev/null +++ b/test/sqllogictest/mz_kafka_source_tables.slt @@ -0,0 +1,68 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Behavioural tests for mz_internal.mz_kafka_source_tables. The view is a +# BuiltinMaterializedView backed by SQL over mz_internal.mz_catalog_raw: see +# MZ_KAFKA_SOURCE_TABLES in src/catalog/src/builtin/mz_internal.rs and the +# parse_source_export_details helper in +# src/expr/src/scalar/func/impls/jsonb.rs. Rows with real upstream data, and +# the envelope/format resolution, are exercised by test/testdrive/mz-sources.td; +# this file pins the column shape and the parent-type filtering that need no +# live Kafka. + +mode cockroach + +reset-server + +# --- Schema ------------------------------------------------------------------- + +# Lock down: 5 columns, canonical order, types, and nullability. +query TTT +SELECT c.name, c.type, c.nullable::text +FROM mz_catalog.mz_columns c +JOIN mz_catalog.mz_objects o ON c.id = o.id +JOIN mz_catalog.mz_schemas s ON o.schema_id = s.id +WHERE o.name = 'mz_kafka_source_tables' AND s.name = 'mz_internal' +ORDER BY c.position +---- +id text false +topic text false +envelope_type text true +key_format text true +value_format text true + +# --- Empty on a fresh env ----------------------------------------------------- + +query I +SELECT count(*) FROM mz_internal.mz_kafka_source_tables +---- +0 + +# --- Non-kafka exports are excluded ------------------------------------------- + +statement ok +CREATE SOURCE lg_src IN CLUSTER quickstart FROM LOAD GENERATOR AUCTION FOR ALL TABLES + +query I +SELECT count(*) FROM mz_internal.mz_kafka_source_tables +---- +0 + +# --- FK-like invariant -------------------------------------------------------- + +query I +SELECT count(*) FROM mz_internal.mz_kafka_source_tables kst +LEFT JOIN mz_sources s ON kst.id = s.id +LEFT JOIN mz_tables t ON kst.id = t.id +WHERE s.id IS NULL AND t.id IS NULL +---- +0 + +statement ok +DROP SOURCE lg_src CASCADE diff --git a/test/sqllogictest/mz_mysql_source_tables.slt b/test/sqllogictest/mz_mysql_source_tables.slt new file mode 100644 index 0000000000000..ec599e474b64b --- /dev/null +++ b/test/sqllogictest/mz_mysql_source_tables.slt @@ -0,0 +1,65 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Behavioural tests for mz_internal.mz_mysql_source_tables. The view is a +# BuiltinMaterializedView backed by SQL over mz_internal.mz_catalog_raw: see +# MZ_MYSQL_SOURCE_TABLES in src/catalog/src/builtin/mz_internal.rs and the +# parse_source_export_details helper in +# src/expr/src/scalar/func/impls/jsonb.rs. Rows with real upstream data are +# exercised by test/mysql-cdc; this file pins the column shape and the +# parent-type filtering that need no live MySQL. + +mode cockroach + +reset-server + +# --- Schema ------------------------------------------------------------------- + +# Lock down: 3 columns, canonical order, types, and nullability. +query TTT +SELECT c.name, c.type, c.nullable::text +FROM mz_catalog.mz_columns c +JOIN mz_catalog.mz_objects o ON c.id = o.id +JOIN mz_catalog.mz_schemas s ON o.schema_id = s.id +WHERE o.name = 'mz_mysql_source_tables' AND s.name = 'mz_internal' +ORDER BY c.position +---- +id text false +schema_name text false +table_name text false + +# --- Empty on a fresh env ----------------------------------------------------- + +query I +SELECT count(*) FROM mz_internal.mz_mysql_source_tables +---- +0 + +# --- Non-mysql exports are excluded ------------------------------------------- + +statement ok +CREATE SOURCE lg_src IN CLUSTER quickstart FROM LOAD GENERATOR AUCTION FOR ALL TABLES + +query I +SELECT count(*) FROM mz_internal.mz_mysql_source_tables +---- +0 + +# --- FK-like invariant -------------------------------------------------------- + +query I +SELECT count(*) FROM mz_internal.mz_mysql_source_tables mst +LEFT JOIN mz_sources s ON mst.id = s.id +LEFT JOIN mz_tables t ON mst.id = t.id +WHERE s.id IS NULL AND t.id IS NULL +---- +0 + +statement ok +DROP SOURCE lg_src CASCADE diff --git a/test/sqllogictest/mz_postgres_source_tables.slt b/test/sqllogictest/mz_postgres_source_tables.slt new file mode 100644 index 0000000000000..9655a5e8207b9 --- /dev/null +++ b/test/sqllogictest/mz_postgres_source_tables.slt @@ -0,0 +1,71 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Behavioural tests for mz_internal.mz_postgres_source_tables. The view is a +# BuiltinMaterializedView backed by SQL over mz_internal.mz_catalog_raw: see +# MZ_POSTGRES_SOURCE_TABLES in src/catalog/src/builtin/mz_internal.rs and the +# parse_source_export_details helper in +# src/expr/src/scalar/func/impls/jsonb.rs. Rows with real upstream data are +# exercised by test/pg-cdc; this file pins the column shape and the +# parent-type filtering that need no live Postgres. + +mode cockroach + +reset-server + +# --- Schema ------------------------------------------------------------------- + +# Lock down: 3 columns, canonical order, types, and nullability. +query TTT +SELECT c.name, c.type, c.nullable::text +FROM mz_catalog.mz_columns c +JOIN mz_catalog.mz_objects o ON c.id = o.id +JOIN mz_catalog.mz_schemas s ON o.schema_id = s.id +WHERE o.name = 'mz_postgres_source_tables' AND s.name = 'mz_internal' +ORDER BY c.position +---- +id text false +schema_name text false +table_name text false + +# --- Empty on a fresh env ----------------------------------------------------- + +# Fresh envs have no upstream Postgres, so this view must be empty. +query I +SELECT count(*) FROM mz_internal.mz_postgres_source_tables +---- +0 + +# --- Non-postgres exports are excluded ---------------------------------------- + +# A LOAD GENERATOR ... FOR ALL TABLES source creates subsources whose +# create_sql parse_source_export_details does read, but the parent-type filter +# must keep them out of the postgres view. +statement ok +CREATE SOURCE lg_src IN CLUSTER quickstart FROM LOAD GENERATOR AUCTION FOR ALL TABLES + +query I +SELECT count(*) FROM mz_internal.mz_postgres_source_tables +---- +0 + +# --- FK-like invariant -------------------------------------------------------- + +# Every row must join back to a source or table. With no rows this is trivially +# satisfied, but the query also guards the join columns' types. +query I +SELECT count(*) FROM mz_internal.mz_postgres_source_tables pst +LEFT JOIN mz_sources s ON pst.id = s.id +LEFT JOIN mz_tables t ON pst.id = t.id +WHERE s.id IS NULL AND t.id IS NULL +---- +0 + +statement ok +DROP SOURCE lg_src CASCADE diff --git a/test/sqllogictest/mz_sql_server_source_tables.slt b/test/sqllogictest/mz_sql_server_source_tables.slt new file mode 100644 index 0000000000000..db079af1c322b --- /dev/null +++ b/test/sqllogictest/mz_sql_server_source_tables.slt @@ -0,0 +1,65 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Behavioural tests for mz_internal.mz_sql_server_source_tables. The view is a +# BuiltinMaterializedView backed by SQL over mz_internal.mz_catalog_raw: see +# MZ_SQL_SERVER_SOURCE_TABLES in src/catalog/src/builtin/mz_internal.rs and the +# parse_source_export_details helper in +# src/expr/src/scalar/func/impls/jsonb.rs. Rows with real upstream data are +# exercised by test/sql-server-cdc; this file pins the column shape and the +# parent-type filtering that need no live SQL Server. + +mode cockroach + +reset-server + +# --- Schema ------------------------------------------------------------------- + +# Lock down: 3 columns, canonical order, types, and nullability. +query TTT +SELECT c.name, c.type, c.nullable::text +FROM mz_catalog.mz_columns c +JOIN mz_catalog.mz_objects o ON c.id = o.id +JOIN mz_catalog.mz_schemas s ON o.schema_id = s.id +WHERE o.name = 'mz_sql_server_source_tables' AND s.name = 'mz_internal' +ORDER BY c.position +---- +id text false +schema_name text false +table_name text false + +# --- Empty on a fresh env ----------------------------------------------------- + +query I +SELECT count(*) FROM mz_internal.mz_sql_server_source_tables +---- +0 + +# --- Non-sql-server exports are excluded -------------------------------------- + +statement ok +CREATE SOURCE lg_src IN CLUSTER quickstart FROM LOAD GENERATOR AUCTION FOR ALL TABLES + +query I +SELECT count(*) FROM mz_internal.mz_sql_server_source_tables +---- +0 + +# --- FK-like invariant -------------------------------------------------------- + +query I +SELECT count(*) FROM mz_internal.mz_sql_server_source_tables sst +LEFT JOIN mz_sources s ON sst.id = s.id +LEFT JOIN mz_tables t ON sst.id = t.id +WHERE s.id IS NULL AND t.id IS NULL +---- +0 + +statement ok +DROP SOURCE lg_src CASCADE diff --git a/test/sqllogictest/oid.slt b/test/sqllogictest/oid.slt index cc839b61ee796..8b4f2ce860925 100644 --- a/test/sqllogictest/oid.slt +++ b/test/sqllogictest/oid.slt @@ -1096,8 +1096,6 @@ SELECT oid, name FROM mz_objects WHERE id LIKE 's%' AND oid < 20000 ORDER BY oid 16945 mz_role_parameters 16946 mz_materialized_view_refresh_strategies 16947 mz_cluster_schedules -16948 mz_postgres_source_tables -16949 mz_mysql_source_tables 16950 mz_kafka_sources_ind 16951 mz_webhook_sources_ind 16952 mz_history_retention_strategies @@ -1167,7 +1165,6 @@ SELECT oid, name FROM mz_objects WHERE id LIKE 's%' AND oid < 20000 ORDER BY oid 17022 mz_wallclock_global_lag_history 17023 mz_wallclock_global_lag_recent_history 17024 mz_wallclock_global_lag_recent_history_ind -17025 mz_kafka_source_tables 17030 mz_objects_id_namespace_types 17031 mz_object_history 17032 mz_object_history_ind @@ -1191,7 +1188,6 @@ SELECT oid, name FROM mz_objects WHERE id LIKE 's%' AND oid < 20000 ORDER BY oid 17054 mz_wallclock_global_lag 17055 mz_wallclock_global_lag_histogram_raw 17056 mz_wallclock_global_lag_histogram -17057 mz_sql_server_source_tables 17058 mz_license_keys 17059 mz_role_auth 17060 mz_iceberg_sinks @@ -1240,3 +1236,8 @@ SELECT oid, name FROM mz_objects WHERE id LIKE 's%' AND oid < 20000 ORDER BY oid 17104 mz_cluster_auto_scaling_strategies_ind 17105 parse_postgres_source_details 17106 parse_kafka_source_details +17107 mz_postgres_source_tables +17108 mz_mysql_source_tables +17109 mz_sql_server_source_tables +17110 mz_kafka_source_tables +17111 parse_source_export_details diff --git a/test/testdrive/catalog.td b/test/testdrive/catalog.td index c5d92e7064f46..ab904914157af 100644 --- a/test/testdrive/catalog.td +++ b/test/testdrive/catalog.td @@ -606,19 +606,15 @@ mz_aggregates "" mz_aws_connections "" mz_cluster_replica_size_internal "" mz_history_retention_strategies "" -mz_kafka_source_tables "" mz_license_keys "" mz_materialized_view_refresh_strategies "" -mz_mysql_source_tables "" mz_object_arrangement_size_history "" mz_object_dependencies "" mz_object_global_ids "" mz_optimizer_notices "" -mz_postgres_source_tables "" mz_replacements "" mz_sessions "" mz_source_references "" -mz_sql_server_source_tables "" mz_storage_usage_by_shard "" mz_subscriptions "" mz_type_pg_metadata "" @@ -728,12 +724,16 @@ mz_cluster_system_parameters mz_catalog_server "" mz_cluster_workload_classes mz_catalog_server "" mz_comments mz_catalog_server "" mz_internal_cluster_replicas mz_catalog_server "" +mz_kafka_source_tables mz_catalog_server "" +mz_mysql_source_tables mz_catalog_server "" mz_network_policies mz_catalog_server "" mz_network_policy_rules mz_catalog_server "" mz_overridden_system_parameters mz_catalog_server "" mz_pending_cluster_replicas mz_catalog_server "" +mz_postgres_source_tables mz_catalog_server "" mz_postgres_sources mz_catalog_server "" mz_replica_system_parameters mz_catalog_server "" +mz_sql_server_source_tables mz_catalog_server "" # Check default sources, tables, and views in mz_introspection. @@ -830,7 +830,7 @@ test_table "" # `SHOW TABLES` and `mz_tables` should agree. > SELECT COUNT(*) FROM mz_tables WHERE id LIKE 's%' -42 +38 # There is one entry in mz_indexes for each field_number/expression of the index. > SELECT COUNT(id) FROM mz_indexes WHERE id LIKE 's%'