Repository navigation
[pull] main from MaterializeInc:main - #1353
Merged
Merged
Conversation
…38629) ### Motivation It is not obvious from the docs that a cluster moves through distinct lifecycle states, nor how to tell whether it is making progress (vs. stuck) in each one. This came out of a Slack discussion on cluster lifecycle and freshness/"total lag": https://materializeinc.slack.com/archives/CU7ELJ6E9/p1788365831558719?thread_ts=1788365056.126659&cid=CU7ELJ6E9 ### Description Adds a **Lifecycle of a cluster** section to the [clusters concept page](https://materialize.com/docs/concepts/clusters/). It briefly describes the states a cluster's replicas move through and, for each state, recommends how to monitor progress: - **Provisioning** — replicas are scheduled and brought online. Monitor via `mz_cluster_replica_statuses` and `mz_clusters`. - **Hydrating** — replicas reconstruct in-memory state from Materialize's storage. Monitor via `mz_hydration_statuses` (and `mz_compute_hydration_statuses` for indexes/materialized views). - **Catching up** — the cluster works through the accumulated backlog and total lag comes down. Monitor via `mz_wallclock_global_lag_recent_history` and `mz_materialization_lag`. - **Steady state** — lag holds low and roughly constant. Monitor via the same lag relation to catch regressions. A note also points to the snapshotting step sources go through on first start. ### Verification Docs-only change. Verified all system-catalog and concept-page links resolve against the repo, and ran `bin/format-docs` (no changes). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015919agXAtbTfF1iHNpGQoj --- _Generated by [Claude Code](https://claude.ai/code/session_015919agXAtbTfF1iHNpGQoj)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Release notes for the v26.40.0 train. Dates in RC snapshots are provisional until the release ships. One commit is added per snapshot; the final snapshot sets the real dates. **Applied so far: `rc.1`, `rc.2`, `rc.3`, `final` — the train is complete.** `final` adds no entries: `v26.40.0` is exactly one commit ahead of `v26.40.0-rc.3`, and that commit is the version bump. Nothing shipped between the last RC and the release tag, so the increment is genuinely empty rather than a gap; the release's 19 published entries all live in the `rc.1`, `rc.2`, and `rc.3` increments. **Dates are now real, not provisional — please confirm them.** Cloud **2026-09-02**, Self-Managed **2026-09-03**. No dates were supplied, so these come from the `v26.40.0` tag's commit date (2026-09-02T17:06:24Z) and `+1 day`. Two checks support them: v26.39.0's tag date was exactly its published Cloud date, and the `+1` Self-Managed offset matches v26.39.0, v26.37.0, v26.35.0, and v26.33.0. They supersede the provisional pair (Cloud 2026-09-03 / Self-Managed 2026-09-04) that all three RC snapshots carried, anchored on the `rc.1` tag. The release shipped **one day earlier** than the cadence estimate — the direction `rc.1` predicted, since v26.37.0, v26.38.0, and v26.39.0 each did the same. **Merge ordering — this PR currently conflicts with `main`, by design of how the branch was cut.** It was branched from a `main` that contained none of the v26.38.0, v26.38.1, or v26.39.0 sections, so `## v26.40.0` sits directly above `## v26.38.2` here. Since then #38204 (v26.38.0) and #38383 (v26.39.0) have merged, and `main` now opens with `## v26.39.0` — hence the conflict. #38365 (v26.38.1) is **still open**, so resolving the conflict now would only reintroduce it when that PR merges. The resolution is mechanical and belongs to whichever of these merges last: `## v26.40.0` must end up **above** all of those sections, and its operator-compatibility row at the **top** of `rows:`. Left unresolved deliberately rather than by oversight. **For the release owner — decisions worth a look before merge.** - **RESOLVED: #38556 was never picked, and `rc.3`'s narrowed bug fix is the correct final statement.** `rc.3` published #38558, the second of two [MZC-59](https://linear.app/materializeinc/issue/MZC-59) commits, but not #38556 ("return 202 for notifications so MCP SDK clients complete the handshake"), and deliberately narrowed its entry to the transport-level fact that build delivers, naming no client as fixed. The open question was whether #38556 would land in an `rc.4` or in `final`. **It did not:** no `v26.40.0-rc.4` tag exists, the final tag adds only the version bump, and #38556 is still not an ancestor of it. So no supersession and no merged bullet — the entry stands as written. **[MZC-59](https://linear.app/materializeinc/issue/MZC-59) remains open for SDK-based MCP clients such as Codex in v26.40.0** and should carry to a later train. - **RESOLVED: #38462 is in the release**, which retires a caveat `rc.3` could not settle at the time. [agent-skills#55](MaterializeInc/agent-skills#55) flagged that its claim about `mz_dataflow_arrangement_sizes` on pre-v26.40 servers *assumed* #38462 would ship in v26.40, when no v26.40 tag yet existed. The tag now exists and #38462 is an ancestor of it, so the `mz-optimize-memory` entry needs no change. #38462 itself stays omitted — `rc.1` classified it Borderline as instruction text with no server behavior change, and seeing the whole release does not disturb that call. - **The Iceberg options this train publishes are undocumented at the tag.** `grep -rn "ACCESS DELEGATION\|OAUTH2 SERVER URL" doc/user/content/` returns nothing, though `doc/user/content/sql/create-connection.md` is where they belong. The feature section is published anyway because the SQL surface is reachable, unflagged, and tested against a real catalog — but if the docs are not landing in this release, hold the section for the release that documents them. - **#38480 is a privilege tightening that can break working deployments on upgrade.** `CREATE TABLE ... FROM SOURCE` now requires `SELECT` on the source and `USAGE` on its schema. It is published as a bug fix; you may want it called out more prominently than the Bug Fixes list, since affected admins have to grant `SELECT` before upgrading. - **`rc.2`'s recovery path is deliberately not in the release note.** #38545 fixes the root cause of a production incident in which `environmentd` entered a deterministic crash loop. A deployment that applied a replacement on v26.39.x or earlier keeps the stale cache entries until it upgrades, and one already crash-looping needs an `enable_expression_cache=false` boot to come up at all. Consider whether the upgrade guide or a support note should carry that. - **Nine flag-gated items were held as Borderline in `rc.1`** pending a call on whether their flags are on in production: `CREATE METRIC SINK` (`enable_metric_sink`, `false` at the tag), object hydration history (`hydration_history_collection_interval`, `Duration::ZERO`, which disables collection), hedged persist blob gets (`persist_blob_hedged_get_enabled`, `false`), three upsert/batcher spill optimizations (`enable_column_paged_batcher` / `enable_upsert_chunked_stash` / `enable_upsert_paged_spill`, all `false`), the columnar merge batcher (`enable_columnar_merge_batcher`, `false`), the peek row-iteration limit (`enable_compute_peek_row_iteration_limit`, `false`), parallel MySQL snapshots (`mysql_source_snapshot_parallelism`, `false`), and Console cluster-list usage metrics (`usage-metrics-in-cluster-list-CNS121`, off). No published entry in any snapshot is behind an off-by-default flag. - **#38399 is omitted as a cross-train duplicate.** It appears in the v26.40.0 tracking issue as its `main` merge, but v26.39.0 `rc.2` already published it (#38383), so restating it under v26.40.0 would show the reader the same fix twice in `_index.md`. Override if you would rather it appear here. - **Cosmetic, your call:** the **Agent Skills** list reads `mz-optimize-memory` then `mcp-developer-analysis` — newest increment first, an artifact of each snapshot prepending into the existing list. Both entries are correct; flagging only in case you prefer them reordered before merge. - **Per-snapshot inclusion:** `rc.1` 23 of 86 PRs (26.7% — a property of a small single-week denominator carrying two penetration-test security fixes, a GA graduation, and two multi-PR capabilities, not a loosened bar); `rc.2` 1 of 1; `rc.3` 2 of 4; `final` 0 of 3. - **The `final` Step 5 verification pass ran in its widened form** over all four snapshot directories and the assembled section on this branch, since the increment had no entries of its own: **0 cut, 0 rewritten, 0 merged.** All 19 published entries stand as reviewed, and no RC-time Borderline call flipped when re-read with the full release in view. Snapshot drafts (with PR links) in mz-skills: - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.1/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.1/omitted.md) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.2/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.2/omitted.md) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.3/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.3/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/final/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 17 — [review borderline calls](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.1/omitted.md#borderline), rc.2: 0 — [none](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.2/omitted.md#borderline), rc.3: 0 — [none](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/rc.3/omitted.md#borderline), final: 0 — [none](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.40.0/final/omitted.md#borderline) --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Pranshu Maheshwari <maheshwarip@users.noreply.github.com> Co-authored-by: Pranshu Maheshwari <pranshu.maheshwari@materialize.com>
#38610) ### Motivation Fixes [CNS-38](https://linear.app/materializeinc/issue/CNS-38) Instead of simply displaying "Retry" after a disconnection, add an error message. ### Description Investigation revealed that we can mitigate the disconnection problem at its source. This UX improvement is a first step. <img width="821" height="526" alt="Screenshot 2026-09-01 at 4 19 47 PM" src="https://github.com/user-attachments/assets/6ded122f-41ef-448a-80d9-ab8bb40f0c8b" /> ### Verification New test case asserting CONNECTION_CLOSED mid-command sets both interrupted and the error. This event had no coverage at all before. Assert that an interrupted command renders the message and an enabled Retry button, and that a normal command renders neither. Manually tested with a non-trivial bit of chicanery.
Addresses [SS-414](https://linear.app/materializeinc/issue/SS-414/sentry-panic-itertools-zip-eq-reached-end-of-one-iterator-before-the) by adding schema validation to `load_or_create_table` in `iceberg.rs` that checks that the schema materialize has built for this sink matches the schema of the existing iceberg table, if one exists already (in the case of cluster/sink restart, 0dt upgrade, etc). This avoids the panic caused during parquet writing when there is a schema mismatch. Adds the repro test case from [SS-414](https://linear.app/materializeinc/issue/SS-414) with a small tweak to verify that the change works as intended
I've tried my best to explain each change in the commit messages, but I've also shared some nicely formatted explanations that I used to understand the changes myself: - Why we don't resolve an array type to both its array type (e.g. _int4) and its element type (int4) anymore (commit 1): https://claude.ai/code/artifact/f9d39cf2-eff9-4a80-b5e0-c712183a24bd - How we were able to recreate the dependency graph via a SQL function (commits >= 2): https://claude.ai/code/artifact/f3b59447-9924-49d6-8085-1db9dff69048 . Ignore the diff harness comment. You might notice some code is different, but the motivation/high level approach is the same. ### Motivation Partially does sql-499 ### Verification - More SLTs for testing the ported builtin more in depth - Unit tests for the parser changes - I created a harness that diffs builtins for all system objects, and user objects created from Dennis' workload replay. It's in draft #38489 but used it and things were 1:1
## Motivation Switching regions in the console left pages fed by the app-wide SUBSCRIBEs (cluster list, object explorer) showing the previous region's catalog until a full page refresh, and the sync-engine localStorage cache could be poisoned with another region's rows. Fixing this properly also motivated restructuring how the subscribe stack is wired, and documenting the resulting architecture. Fixes: [CNS-157 ](https://linear.app/materializeinc/issue/CNS-157/fix-sync-engine-reseeding-when-changing-regions) Fixes: [CNS-155](https://linear.app/materializeinc/issue/CNS-155/clusters-do-not-update-when-switching-regions) ## The commits (one concern each; the branch squash-merges) 1. **Sync-engine cache scope fixes** (`subscribeCollection.ts`): a scope change drops in-memory rows and any pending persist before re-seeding, so one tenant's catalog is never written under another's cache key, and a reconnect's empty pre-snapshot clears a stale error so the UI falls back to loading during backoff. (Both flagged by the QA review on #38609.) 2. **Region-aware connection manager** (`WebsocketConnectionManager.ts`): tracks the address and region of the most recent connection attempt. Reconnects when the region's address changes (including mid-handshake), and tears down a paused socket only when it points at a region the user has left — keyed on region so disabled regions (whose address never updates) are covered, while a same-region health blip leaves a working socket alone (an intentional disconnect fires no close callback, so unconditional teardown would silently replace the SQL Shell's session). 3. **Subscribe sessions** (`subscribeSession.ts` + hooks + stores): the wiring that had accumulated as one React effect per concern moves into plain session objects that subscribe to the jotai store directly, the pattern the connection manager already used. Sessions reset their manager and sink on region change and hydrate collections from the scoped cache via a new cache-scope atom. Hooks keep a single lifecycle effect, cannot suspend, and take referentially stable module-scope options. The allObjects-to-collection bridge runs off a store subscription instead of re-renders. 4. **Documentation** (`doc/design/20260902_sync_engine.md`, `doc/architecture.md`, `AGENTS.md`): a design doc explaining the sync engine end to end (feed, sessions, both sinks, the collection bridge and its scoped instant-load cache, region-switch data flow, boundaries, adoption recipes), plus updates to the data-layer sections of the architecture doc and the agent guidance so they describe the server-state split: React Query for polled/one-shot data, the sync engine for live catalog data. ## Tips for reviewer Every behavioral claim has a regression test that was verified to fail against the code it fixes: cache scope and error-clear tests in `subscribeCollection.test.ts`, the region round-trip / same-region-blip / disabled-region / mid-handshake tests in `WebsocketConnectionManager.test.ts`, and headless session tests in `useSubscribe.test.tsx`. Known unrelated failure: `PlanDetails.test.tsx` (billing) fails identically on current main — it started failing on 2026-09-03 with no billing changes here, so it looks date-dependent and is not introduced by this branch. ## Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [x] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. - [ ] If this PR evolves [an existing $T ⇔ Proto$T mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )