[pull] main from MaterializeInc:main

doc/user/content/get-started/quickstart.md

@@ -112,24 +112,42 @@ generator](/sql/create-source/load-generator/#auction) to create the source.
 
    For the [sample `Auction` load
    generator](/sql/create-source/load-generator/#auction), the quickstart uses
-   [`CREATE SOURCE`](/sql/create-source/) with the `FROM LOAD GENERATOR` clause
+   [`CREATE SOURCE` with the `FROM LOAD
+   GENERATOR` clause](/sql/create-source/load-generator/#creating-an-auction-load-generator)
    that works specifically with Materialize's sample data generators. The
    tutorial specifies that the generator should emit new data every 1s.
 
     ```mzsql
     CREATE SOURCE auction_house
     FROM LOAD GENERATOR AUCTION
-    (TICK INTERVAL '1s', AS OF 100000)
-    FOR ALL TABLES;
+    (TICK INTERVAL '1s', AS OF 100000);
     ```
 
-    `CREATE SOURCE` can create **multiple** tables (referred to as `subsources`
-    in Materialize) when ingesting data from multiple upstream tables. For each
-    upstream table that is selected for ingestion, Materialize creates a
-    subsource.
+    The source connects to the data generator but does not ingest its relations
+    yet. To start ingesting data, you create a table from the source for each
+    relation you want to ingest.
 
-1. Use the [`SHOW SOURCES`](/sql/show-sources/) command to see the results of
-   the previous step.
+1. Use [`CREATE TABLE ... FROM
+   SOURCE`](/sql/create-source/load-generator/#creating-an-auction-load-generator)
+   in a [DDL transaction block](/sql/begin/#ddl-only-transactions) to create
+   **read-only** tables for the `auctions` and `bids` relations, the two
+   relations this quickstart uses:
+
+    ```mzsql
+    BEGIN;
+    CREATE TABLE auctions FROM SOURCE auction_house (REFERENCE auctions);
+    CREATE TABLE bids FROM SOURCE auction_house (REFERENCE bids);
+    COMMIT;
+    ```
+
+    Tables created from a source are read-only; that is:
+
+    - Only the source can write to the table; in this case, the load
+      generator.
+
+    - Users can read from the table.
+
+1. Use the [`SHOW SOURCES`](/sql/show-sources/) command to see the new source:
 
     ```mzsql
     SHOW SOURCES;
@@ -138,24 +156,24 @@ generator](/sql/create-source/load-generator/#auction) to create the source.
     The output should resemble the following:
 
     ```nofmt
-    | name                   | type           | cluster    | comment |
-    | ---------------------- | -------------- | ---------- | ------- |
-    | accounts               | subsource      | quickstart |         |
-    | auction_house          | load-generator | quickstart |         |
-    | auction_house_progress | progress       | null       |         |
-    | auctions               | subsource      | quickstart |         |
-    | bids                   | subsource      | quickstart |         |
-    | organizations          | subsource      | quickstart |         |
-    | users                  | subsource      | quickstart |         |
+    | name          | type           | cluster    | comment |
+    | ------------- | -------------- | ---------- | ------- |
+    | auction_house | load-generator | quickstart |         |
     ```
 
-    A [`subsource`](/sql/show-subsources) is how Materialize refers to a table
-    that has the following properties:
+    Use the [`SHOW TABLES`](/sql/show-tables/) command to see the tables
+    created from the source:
 
-    - A subsource can only be written by the source; in this case, the
-      load-generator.
+    ```mzsql
+    SHOW TABLES;
+    ```
 
-    - Users can read from subsources.
+    ```nofmt
+    | name     | comment |
+    | -------- | ------- |
+    | auctions |         |
+    | bids     |         |
+    ```
 
 1. Use the [`SELECT`](/sql/select) statement to query `auctions` and `bids`.
 
@@ -364,16 +382,37 @@ creates:
        AND datediff('days', w2.bid_time, w1.bid_time) < 8;
     ```
 
-    The `flip_activities` view can use the index created on `winning_bids` view
-    to provide up-to-date data.
-
     To view a sample row in `flip_activities`, run the following
     [`SELECT`](/sql/select) command:
 
     ```mzsql
     SELECT * FROM flip_activities LIMIT 10;
     ```
 
+    The query may take a while to return, even though it uses the
+    `wins_by_item` index. The `flip_activities` view self-joins `winning_bids`
+    on `item` and the corresponding `buyer` and `seller` columns (`w1.buyer =
+    w2.seller AND w1.item = w2.item`), but the `wins_by_item` index is keyed
+    on the `item` column only. As a result, the join must first form every
+    pair of rows with the same `item` (the part supported by the index) before
+    evaluating the buyer/seller predicates on each pair to find the actual
+    matches. Because the sample data contains only a small number of distinct
+    item values, as `winning_bids` grows, the number of pairs per item grows
+    quadratically.
+
+1. Create indexes on the `winning_bids` join keys used by `flip_activities`.
+
+    To avoid the aforementioned quadratic work, index the complete join keys so
+    that the join considers only rows that can actually match:
+
+    ```mzsql
+    CREATE INDEX wins_by_item_seller ON winning_bids (item, seller);
+    CREATE INDEX wins_by_item_buyer ON winning_bids (item, buyer);
+    ```
+
+    Rerun the previous query on `flip_activities`. The query should return
+    faster.
+
 1. Use [`CREATE TABLE`](/sql/create-table) to create a `known_flippers` table
    that you can manually populate with known flippers. That is, assume that
    separate from your auction activities data, you receive independent data
@@ -405,11 +444,14 @@ creates:
 
 {{< note >}}
 
-Both the `flip_activities` and `flippers` views can use the index created on
-`winning_bids` view to provide up-to-date data. Depending upon your query
-patterns and usage, an existing index may be sufficient, such as in this
-quickstart. In other use cases, creating an index only on the view(s) from which
-you will serve results may be preferred.
+Both the `flip_activities` and `flippers` views can use the indexes created on
+the `winning_bids` view to provide up-to-date results. Index keys matter: the
+`wins_by_item` index serves point lookups on `item`, while the
+`wins_by_item_seller` and `wins_by_item_buyer` indexes serve the
+`flip_activities` self-join on its join keys. Depending on your query patterns
+and usage, indexing the upstream views may be sufficient, such as indexing
+`winning_bids` in this quickstart. In other use cases, it may be preferable to
+index only the views from which applications directly read results.
 
 {{</ note >}}
 
@@ -529,8 +571,8 @@ data comes in, this step creates the following views for completed auctions:
 To clean up the quickstart environment:
 
 1. Use the [`DROP SOURCE ... CASCADE`](/sql/drop-source/) command to drop
-   `auction_house` source and its dependent objects, including views and indexes
-   created on the `auction_house` subsources.
+   `auction_house` source and its dependent objects, including the tables
+   created from the source and the views and indexes created on them.
 
    ```mzsql
    DROP SOURCE auction_house CASCADE;

doc/user/content/headless/agent-skills-table.md

@@ -1,10 +1,10 @@
 ---
 headless: true
 ---
-| Skill | What it provides | When to use |
-|-------|------------------|-------------|
-| `mcp-developer-analysis` | Exact catalog schemas, diagnostic workflows, remediation runbooks, and guardrails for known pitfalls (cluster-scoped queries, uint8 ID mismatches, etc.). | Operational introspection and troubleshooting via the `materialize-developer` server. Examples: *"why is my materialized view stale?"*, *"what can I optimize to save costs?"*, *"is my source healthy?"* |
-| `materialize-docs` | Comprehensive Materialize documentation, including SQL syntax, idiomatic patterns, data ingestion, concepts, and best practices (400+ reference files). | Authoring view definitions, learning concepts, looking up patterns. Useful with either MCP server. Examples: *"show me how to deduplicate a stream"*, *"what's the idiomatic top-K pattern?"*, *"how do I create a Kafka source?"* |
-| `materialize-dbt` | dbt-materialize adapter usage: materializations, profile configuration, index creation, blue/green deployments, and testing. | Managing Materialize pipelines with dbt. Examples: *"write a dbt model for a materialized view"*, *"how do I do a blue/green deployment with dbt?"* |
-| `materialize-terraform-provider` | Provider configuration for Cloud and self-managed, navigation into the provider's auto-generated resource reference, cross-resource patterns, import workflows, and gotchas. | Managing Materialize resources declaratively with Terraform. Examples: *"create a Kafka source with Terraform"*, *"import my existing clusters into Terraform state"*, *"set up RBAC grants in Terraform"* |
-| `materialize-terraform-self-managed` | Module layout and variables for deploying self-managed Materialize on AWS, Azure, and GCP: networking, Kubernetes, backend URL formats, instance sizing, upgrades, and gotchas. | Deploying or operating self-managed Materialize infrastructure with Terraform. Examples: *"deploy Materialize on EKS"*, *"what instance types should Materialize nodes use?"*, *"upgrade my self-managed Materialize"* |
+| Skill | Description |
+|-------|-------------|
+| `mcp-developer-analysis` | Use for operational introspection and troubleshooting via the `materialize-developer` server. Covers exact catalog schemas, diagnostic workflows, remediation runbooks, and guardrails for known pitfalls (cluster-scoped queries, uint8 ID mismatches, etc.).<br><br>Examples: *"why is my materialized view stale?"*, *"what can I optimize to save costs?"*, *"is my source healthy?"* |
+| `materialize-docs` | Use for authoring view definitions, learning concepts, and looking up patterns; useful with either MCP server. Covers comprehensive Materialize documentation, including SQL syntax, idiomatic patterns, data ingestion, concepts, and best practices (400+ reference files).<br><br>Examples: *"show me how to deduplicate a stream"*, *"what's the idiomatic top-K pattern?"*, *"how do I create a Kafka source?"* |
+| `materialize-dbt` | Use for managing Materialize pipelines with dbt. Covers dbt-materialize adapter usage: materializations, profile configuration, index creation, blue/green deployments, and testing.<br><br>Examples: *"write a dbt model for a materialized view"*, *"how do I do a blue/green deployment with dbt?"* |
+| `materialize-terraform-provider` | Use for managing Materialize resources declaratively with Terraform. Covers provider configuration for Cloud and self-managed, navigation into the provider's auto-generated resource reference, cross-resource patterns, import workflows, and gotchas.<br><br>Examples: *"create a Kafka source with Terraform"*, *"import my existing clusters into Terraform state"*, *"set up RBAC grants in Terraform"* |
+| `materialize-terraform-self-managed` | Use for deploying or operating self-managed Materialize infrastructure with Terraform. Covers module layout and variables for deploying on AWS, Azure, and GCP: networking, Kubernetes, backend URL formats, instance sizing, upgrades, and gotchas.<br><br>Examples: *"deploy Materialize on EKS"*, *"what instance types should Materialize nodes use?"*, *"upgrade my self-managed Materialize"* |

doc/user/content/integrations/_index.md

@@ -25,20 +25,8 @@ menu:
 
 ## Agent skills and MCP servers
 
-### Agent skills
-
-Materialize provides the following open-source [agent
-skills](https://github.com/MaterializeInc/agent-skills) to help developers build
-with Materialize.
-
-{{% include-headless "/headless/agent-skills-table" %}}
-
-### MCP servers
-
-Materialize providesthe following built-in Model Context Protocol (MCP) servers
-that AI agents can use.
-
-{{% include-headless "/headless/mcp-servers-table" %}}
+For Materialize's open-source agent skills and built-in Model Context Protocol
+(MCP) servers, see [AI & agents](/integrations/mcp-server/).
 
 ## SQL clients/client libraries
 

doc/user/content/integrations/coding-agent-skills.md

@@ -7,7 +7,16 @@ menu:
     weight: 7
 ---
 
-Coding agents like [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Codex](https://openai.com/index/codex/), [Cursor](https://www.cursor.com/), and others can work with Materialize using our open-source [agent skills](https://github.com/MaterializeInc/agent-skills). Once installed, these skills give your coding agent access to Materialize documentation and reference material so it can provide more accurate assistance when writing queries, setting up sources, creating materialized views, and more.
+Coding agents like [Claude
+Code](https://docs.anthropic.com/en/docs/claude-code),
+[Codex](https://openai.com/index/codex/), [Cursor](https://www.cursor.com/), and
+others can work with Materialize using the open-source [Materialize agent
+skills](https://github.com/MaterializeInc/agent-skills). These skills follow the
+[Agent Skills Open Standard](https://agentskills.io/home) and work with any
+coding agent that supports the standard. Once installed, these skills give your
+coding agent access to Materialize documentation and reference material so it
+can provide more accurate assistance when writing queries, setting up sources,
+creating materialized views, and more.
 
 ## Skills
 
@@ -27,8 +36,6 @@ npx skills add MaterializeInc/agent-skills
 
 Once installed, you can update installed skills by running `npx skills update`.
 
-The skills follow the [Agent Skills Open Standard](https://agentskills.io/home) and work with any coding agent that supports the standard.
-
 ## Reduce permission prompts (Claude Code)
 
 Claude Code prompts before reading files outside your project. Since globally

doc/user/content/integrations/mcp-server/_index.md

@@ -4,10 +4,9 @@ description: "This section contains guides for installing Materialize Agent skil
 disable_list: true
 menu:
   main:
-    parent: integrations
-    name: "MCP Server and skills"
+    name: "AI & agents"
     identifier: mcp-server
-    weight: 6
+    weight: 48
 aliases:
   - /integrations/mcp-server/llm/
   - /integrations/llm/

misc/python/materialize/checks/all_checks/builtin_cluster_replication_factor.py

@@ -0,0 +1,77 @@
+# Copyright Materialize, Inc. and contributors. All rights reserved.
+#
+# Use of this software is governed by the Business Source License
+# included in the LICENSE file at the root of this repository.
+#
+# As of the Change Date specified in that file, in accordance with
+# the Business Source License, use of this software will be governed
+# by the Apache License, Version 2.0.
+from textwrap import dedent
+
+from materialize.checks.actions import Testdrive
+from materialize.checks.checks import Check
+from materialize.checks.executors import Executor
+from materialize.mz_version import MzVersion
+
+
+class BuiltinClusterReplicationFactor(Check):
+    """A builtin cluster's replication factor owns its replica set across restarts.
+
+    `mz_support` is the interesting cluster. It runs nothing by default, and it is
+    the break-glass path a support engineer scales up by hand. A catalog open that
+    does not read the cluster's replication factor tears that replica down, leaving
+    the cluster reporting a factor it is not honoring.
+    """
+
+    def _can_run(self, e: Executor) -> bool:
+        # Only versions that reconcile the replica set against the cluster's own
+        # replication factor keep this replica across a restart. Confine the check
+        # to scenarios whose boots are all the current version: the upgrade
+        # scenarios restart on released binaries, and `manipulate` and `validate`
+        # both run on those.
+        return self.base_version >= MzVersion.parse_cargo()
+
+    def manipulate(self) -> list[Testdrive]:
+        # `mz_support` owns its own cluster, so the ALTER has to run as that role.
+        # Its sessions default to `mz_catalog_server`, which has a replica, so a
+        # statement from this connection is safe to issue.
+        #
+        # A restart separates the two phases, so phase 2 asserting the replica is
+        # still there is the regression assertion: the replica set has to be
+        # rederived from the cluster's factor rather than from a fixed list.
+        return [
+            Testdrive(dedent(s))
+            for s in [
+                """
+                $ postgres-execute connection=postgres://mz_support:materialize@${testdrive.materialize-internal-sql-addr}
+                ALTER CLUSTER mz_support SET (REPLICATION FACTOR 1);
+                """,
+                """
+                > SELECT c.replication_factor, count(r.id)
+                  FROM mz_clusters c
+                  LEFT JOIN mz_cluster_replicas r ON r.cluster_id = c.id
+                  WHERE c.name = 'mz_support'
+                  GROUP BY c.replication_factor
+                1 1
+                """,
+            ]
+        ]
+
+    def validate(self) -> Testdrive:
+        return Testdrive(dedent("""
+            > SELECT c.replication_factor, count(r.id)
+              FROM mz_clusters c
+              LEFT JOIN mz_cluster_replicas r ON r.cluster_id = c.id
+              WHERE c.name = 'mz_support'
+              GROUP BY c.replication_factor
+            1 1
+
+            # Named by the managed-cluster rule, and owned by the cluster's owner
+            # rather than by mz_system.
+            > SELECT r.name, o.name
+              FROM mz_cluster_replicas r
+              JOIN mz_clusters c ON c.id = r.cluster_id
+              JOIN mz_roles o ON o.id = r.owner_id
+              WHERE c.name = 'mz_support'
+            r1 mz_support
+            """))

misc/python/materialize/checks/scenarios_upgrade.py

@@ -337,6 +337,13 @@ def actions(self) -> list[Action]:
         print(
             "Upgrade path: v26.17.1 -> initialize -> v26.18.0 -> manipulate#1 -> restart -> manipulate#2 -> v26.24.0 (v82_to_v83) -> current (v83_to_v84) -> restart -> validate"
         )
+        # The factor-0 override below only applies to the boots that pass it, but
+        # it is durable: the cluster row is seeded once and a later boot honors the
+        # row rather than its own flag. So mz_system has no replica for the whole
+        # run, including the final unoverridden boots. Nothing here needs one,
+        # since the checks peek as `materialize` on `quickstart` and the mz_system
+        # connection only issues DDL. A check that peeked at a non-catalog object
+        # as `mz_system` would hang, and would need this override lifted.
         return [
             StartMz(
                 self,

misc/python/materialize/mzcompose/services/materialized.py

@@ -129,9 +129,17 @@ def __init__(
         if cluster_replica_size is None:
             cluster_replica_size = cluster_replica_size_map()
         if builtin_system_cluster_replication_factor is None:
-            builtin_system_cluster_replication_factor = default_replication_factor
+            # Deliberately not `default_replication_factor`. A builtin cluster
+            # honors its replication factor, so tracking the user default here
+            # would run two mz_system and two mz_probe replicas in every
+            # composition that asks for a replicated user cluster, at no benefit
+            # to what those tests actually exercise. One replica is all these
+            # clusters need, and is what the flags below are here to guarantee.
+            # A test that needs a different builtin factor, including 0, has to
+            # ask for it explicitly.
+            builtin_system_cluster_replication_factor = 1
         if builtin_probe_cluster_replication_factor is None:
-            builtin_probe_cluster_replication_factor = default_replication_factor
+            builtin_probe_cluster_replication_factor = 1
 
         environment = [
             "MZ_NO_TELEMETRY=1",