Repository navigation
[pull] main from MaterializeInc:main - #1370
Merged
Merged
Conversation
Today's implementation of sources requires that all snapshot data is ingested before CDC events start getting processed. Two places that are impacted by this behavior: 1. Adding a new table/subsource to a source causes all tables to wait for the newly added table to complete snapshotting. 2. When multiple tables/subsources snapshot, all of them must complete together. This prevents smaller tables that finished snapshot from becoming available for read and extends failure window, which causes all tables to repeat the snapshot. The goal was to avoid making changes to persist, if possible. The major changes are in the storage persist sink, where data from the snapshot operator and replication operators will now arrive concurrently. Data needs to be offloaded to blob storage to prevent excessive memory usage, and append to the shard must maintain correctness, ensuring that the snapshot and CDC data are appended and races with other replicas are properly handled. The design relies on a leading window for batches that is larger than the source tick interval, allowing multiple timestamps to be written per batch. Eventually collapsing to the tick interval, once the snapshot is complete and the frontier is no longer pinned. This proposal also splits up exports into separate outputs and capabilities. In the current implementation, all exports move in lockstep. The changes allow each export to move independently, and this has knock on effects for how information about the sources and how `offset_committed` is recorded. Claude combed through the console code to see where assumptions where being made, and those are described in their own section. Besides the storage persist sink and `offset_committed changes`, the other changes are source specific. OLTP sources will be updated independently, starting with Postgres. Kafka and upsert are harder to solve, and I couldn't think of a good way to make this work for them in a reasonable time frame. Kafka sources/sinks create independent clients per worker, which is excessive, and this design would require that to double. There are cases where this can already cause problems with upstream systems, especially with larger cluster sizes. Upsert v2 wouldn't be able to emit any data from the merge batcher to persist due to the pinned frontier, causing data to backup in the upsert operator. Ok to defer for now per discussion with @maheshwarip and @ohbadiah.
…56) (#38528) [SQL-555](https://linear.app/materializeinc/issue/SQL-555) landed the machinery to install curated metric sinks on every replica, but `CURATED` was empty, so nothing ran at boot. Solution: Add the two v1 definitions: `mz_metric_arrangement_sizes` (three gauge families) and `mz_metric_dataflow_errors` (one gauge). Both read the raw `..._raw` logs, not the derived views, which re-aggregate expensively. The sizes sink labels each series with the exported object's `GlobalId`, not the timely dataflow id, which churns across re-renders. A dataflow with no export, like a subscribe, falls back to an `unattributable` bucket so its bytes still count. The errors sink reads the raw log rather than `mz_compute_error_counts` because that view joins in `mz_compute_dependencies`, a storage-managed source that `ensure_reads_only_logs` rejects: reading it would couple the sink's frontier to envd. Elapsed stays out of v1, its log churns continuously and cost 14% of a 2-worker replica even raw-sourced. `curated_definitions_plan` only called `plan_source`, so a definition that broke the log-only contract still passed the test but crash-looped the coordinator at boot. It now runs `ensure_reads_only_logs` too. Testing: - Unit tests plan each definition and run the log-only gate. - metric-sink.td gains a curated section and a multi-cluster, multi-replica section. - metric-sink.td covers both the attributable and `unattributable` id paths. - The MetricSink platform check now asserts the curated sinks survive a boot, which covers restart, upgrade, and zero-downtime. --------- Co-authored-by: Moritz Hoffmann <antiguru@gmail.com>
**Context:** DOC ON TYPE x and DOC ON COLUMN x.c resolve a name that may denote a type or a relation, and both spellings persist in the sink's create_sql. A relation in these positions prints as [id AS name], but resolution suppressed ids for types, so a type persisted as a bare qualified name. A bare name hides the sink's dependency on the type from create_sql reference extraction: mz_object_dependencies files the bare name under named_relations, whose join excludes Type rows, so the edge silently drops out of the view even though the in-memory dependency graph (which blocks DROP TYPE) still records it. The id form would also keep the reference valid if renames of types were ever added; today no rename strands a bare name, since ALTER TYPE has no RENAME and ALTER SCHEMA RENAME rewrites bare names too. Force ids in the two places that construct DOC ON references: the name resolver's DOC ON folds, and sink purification, which injects DOC ON options for every commented item the sink references. An AST migration rewrites existing catalogs to match. The migration binds bare names to types only, since a type may share (schema, name) with a secret, connection, sink, or function, and resolves builtin types through the system object mappings. ### Motivation Part of sql-499 ### Verification Added platform check to test the upgrade and modified a td to assert it. The platform check creates a secret with the type's name first, so the migration must bind the type rather than the older same-named item.
Ship IANA tzdata 2026c by pointing the workspace at a MaterializeInc fork of chrono-tz. ### Motivation Upstream chrono-tz stops at 0.10.4 with tzdata 2025b and its maintainer is winding the project down, so newer IANA data will not arrive from crates.io. Under 2025b, Materialize starts giving wrong answers about the present on 2026-09-20, when Morocco moves to permanent +00, and again on 2026-11-01, when Alberta and British Columbia would appear to fall back but no longer do. ### The fork [MaterializeInc/chrono-tz](https://github.com/MaterializeInc/chrono-tz) is forked at the v0.10.4 tag. Its `mz_changes` branch carries exactly two commits: bump the `tz` submodule to the IANA 2026c tag, and regenerate the crate's prebuilt data with its own codegen test. No code changes, same API, same phf 0.12, so the existing `deny.toml` exception and every call site are untouched. Upstream `main` has unreleased semver-breaking changes, which is why the fork is cut from the tag rather than from `main`. The `[patch.crates-io]` entry pins the rev. chrono-tz-build and parse-zoneinfo are path dependencies inside that repo, so they follow automatically. ### Data changes 2025b to 2026c Transition-level diff of every zone the prebuilt data touches: | Zones | Change | Past instants affected | |---|---|---| | Africa/Casablanca, Africa/El_Aaiun | Permanent +00 from 2026-09-20 | none | | America/Edmonton, America/Yellowknife, Canada/Mountain | Permanent -06, no 2026-11-01 fall-back, abbreviation CST | none | | America/Vancouver, Canada/Pacific | Permanent -07, no 2026-11-01 fall-back, abbreviation MST | none | | Europe/Chisinau, Europe/Tiraspol | EU transition instants since 2022, one hour later than before | nine one-hour windows, 2022-03-27 to 2026-03-29 | | America/Tijuana and links | DST observed 1953 and 1961 to 1975 | 32 transitions, all before 1976 | The zone directory is unchanged, so no zone appears, disappears, or renames. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
### Motivation Follow-up to #38744. On that PR, a reviewer noted that regularly querying the unindexed `mz_wallclock_global_lag_history` can bog down `mz_catalog_server`, which is risky because the catalog server also backs the Console and `SHOW`/catalog queries across the whole environment. Slack thread: https://materializeinc.slack.com/archives/CPNFV9CVB/p1789065229460309?thread_ts=1789064976.189309&cid=CPNFV9CVB ### Description Adds guidance to the "Monitor historical freshness" section of `doc/user/content/transform-data/monitor-freshness.md` to run recurring historical freshness queries from a separate cluster instead of the default `mz_catalog_server`. Because these queries read only system catalog relations, they auto-route to `mz_catalog_server` regardless of the session's active cluster, so the note also instructs disabling `auto_route_catalog_queries` so the query actually runs on the dedicated cluster. ### Verification - Docs-only prose change; `bin/format-docs` runs clean. - Validated the documented flow against a local Materialize emulator (v26.41.0): `CREATE CLUSTER ... (SIZE = '25cc')`, `SET cluster = ...`, `SET auto_route_catalog_queries = false`, and both freshness queries all run without error. Confirmed via the statement activity log that with auto-routing left on the query executes on `mz_catalog_server`, and only with `auto_route_catalog_queries = false` does it execute on the separate cluster. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Vihj1JSR7cd8CW3SKhquoD --- _Generated by [Claude Code](https://claude.ai/code/session_01Vihj1JSR7cd8CW3SKhquoD)_ Co-authored-by: Claude <noreply@anthropic.com>
… connection (#38732) Adds an option to the `CREATE CONNECTION ... TO ICEBERG CATALOG` statement to specify a storage provider for the catalog, and then consumes this option when building the opendal storage interface to create the correct one. Unblocks vended credential support for GCS and ADLS backed iceberg catalogs
…logs (#38733) Updates cargo.toml to point at the new revision of our iceberg fork, containing the custom credential loader support for GCS. Adds logic to opendal storage interface construction to build custom credential loaders for GCS when the access delegation option is specified, and removes the restriction on using `ACCESS DELEGATION` when creating a GCP biglake iceberg catalog connection.
…terd pods (#38457) ## Why The operator sets no priority class on environmentd or clusterd pods. Higher priority pods evict them when a node fills up. One eviction aborted a deploy mid-hydration. ## What Users set `environmentd.priorityClassName` and `clusterd.priorityClassName` to apply priority classes to those pods. Both values default to unset, which renders no argument and changes no pod spec. The clusterd argument requires environmentd v26.40 or newer. This matches how `clusterd.affinity` and `clusterd.tolerations` are already gated. Older environmentd versions exit on unrecognized arguments. We exclude balancerd and console because they hold no state and run multiple replicas. ## Before you set it * The values do not reach running pods until a user requests a rollout. * Kubernetes caps a user-defined class at 1000000000 and `system-cluster-critical` is 2000000000, so these pods remain vulnerable to critical preemptors. * Prefer a class with `preemptionPolicy: Never` to prevent new pods from evicting serving pods during the rollout. ## Testing Helm unit tests use `notMatchRegex` for negative assertions. The `notContains` function compares whole list elements, so it never fails against a flag rendered as `--flag=value`. The pre-existing `schedulerName` assertion has the same defect and is fixed here. An mzcompose test asserts the resolved integer priority to prove Kubernetes admitted the class. Open: I omitted the Chart.yaml version bump because a script manages it and PR #38406 omitted it, but which rule is live? --------- Co-authored-by: Jesse Luehrs <doy@materialize.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )