Repository navigation
[pull] main from MaterializeInc:main - #1329
Merged
Merged
Conversation
### Motivation There's a little O(depth*prefixes) algorithm going on. I think it can kind of be viewed as O(probes^2). Anyways, I did a little testing locally that confirmed it would be fast even up to 5k iterations over 5k prefixes and told folks I'd implement a cap. I just realized I never actually set that hard 5k cap. ### Description Implement a hard cap on probes to address some light concerns we had with the way we select prefixes. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
### Motivation Materialize exposes current hydration state, but it disappears when a dataflow or replica restarts. A user can tell whether an object is hydrated now, not how long the last hydration took or whether today's is unusually slow. This is the design for recording completed compute-object hydration episodes in a durable table. It lands first so the implementation PRs stacked on it can be reviewed against it. ### Description The document focuses on the parts that are not obvious from the implementation: * Why the collector's subscribe reads the table it writes, which makes collection idempotent across concurrent environmentd processes. * Why the collector aggregates every worker row visible at the OCC read timestamp, requires each visible row to have a non-null `hydrated_at`, and takes the latest finish. Replica restart resets the introspection collection as a unit. Per-process logging clocks can place a worker row beyond the sampled timestamp, and the design explicitly accepts that race rather than depending on configured worker counts. * Why the cutoff and anti-join apply after aggregation. Applying either before it would drop incomplete worker rows and make the completeness check trivially pass. * Why the sweep schedule is stable across restarts but offset from the full environment id, avoiding a fleet-wide burst on every interval boundary, including between regions and ordinals of one organization. * Why durability is best effort, how successive bounded retention batches drain a fixed-cutoff backlog, and how schema evolution preserves rows while shard replacement deliberately restarts the history. * Why the replica-targeted anti-join cannot use a catalog-server index, and why its recurring history import and arrangement cost is accepted. * What sampling a retracting current-state collection cannot record, and what the append-only lifecycle log proposed in #38403 would add later. Scope is limited to successful hydration of indexes and materialized views. Failed episodes, replica-wide episodes, resource peaks, and storage objects remain out of scope because the required signals are not available on `main` today. ### Verification Documentation only. Closes: [SQL-632](https://linear.app/materializeinc/issue/SQL-632)
Stacked on #38344. ### Motivation Adds the durable surface that hydration episodes are recorded into. The collector arrives in the next PR, so this rung can be reviewed as a catalog change. Design doc: [`20260817_durable_object_hydration_history.md`](https://github.com/MaterializeInc/materialize/blob/aljoscha/hydration-01-design/doc/developer/design/20260817_durable_object_hydration_history.md) ### Description Adds `mz_internal.mz_object_hydration_history`, with no index. The table starts in `mz_internal` to mark it unstable while its shape and semantics settle. An episode is identified operationally by `(object_id, replica_id, installed_at)`. `object_id` holds the dataflow id reported by the replica, with `SemanticType::GlobalId`, so consumers resolve it through `mz_internal.mz_object_global_ids` rather than joining `mz_objects` directly. `installed_at` is replica-stamped and stable across environmentd restarts. The identity is not declared as a relation key. A key is an optimizer contract, while this best-effort sampler enforces uniqueness through its anti-join. A duplicate must remain visible rather than letting the optimizer silently elide it. There is no index initially. The collector anti-joins by history identity, but its subscribe runs on the selected user replica and cannot use an index arranged on the catalog server. Such an index would pin the whole table without removing the collector's recurring import and arrangement cost. Adding one later also changes the `mz_indexes` fingerprint, so it requires a `MigrationStep::replacement` for `mz_indexes` pinned to the then-current development version. Durability is best effort. The table is exempt from the bootstrap reset and from forced **replacement**, because either would discard sampled history that cannot be rebuilt. Schema evolution is allowed and keeps the shard and rows. A replacement step naming this table trips an assert in `validate_migration_steps`; giving the exemption up is meant to be deliberate and release-noted. A focused policy test pins evolution participation and replacement exclusion. The bootstrap-reset filter is converted from name/schema comparisons to a set of resolved ids, which is clearer now that several history tables are exempt. The ontology link for `replica_id` uses `SemanticType::ReplicaId`, matching both the column and its target. User documentation states that 30 days is the retention default, disabling collection pauses retention, and dropped dataflow IDs may no longer resolve through `mz_object_global_ids`. ### Verification Catalog goldens cover builtin counts, OIDs, index accounting, `information_schema`, and the autogenerated `mz_internal` relation spec. The catalog migration tests exercise the migration path. This release will add the `mz_internal.mz_object_hydration_history` table. Ref: [SQL-644](https://linear.app/materializeinc/issue/SQL-644)
Stacked on #38346. ### Motivation Populates `mz_internal.mz_object_hydration_history`, which the preceding rung adds but leaves empty. Design doc: [`20260817_durable_object_hydration_history.md`](https://github.com/MaterializeInc/materialize/blob/aljoscha/hydration-01-design/doc/developer/design/20260817_durable_object_hydration_history.md) ### Description A single-flight sweep visits one user replica per interval, skipping replicas with introspection disabled and managed replicas whose rolled-up status is not `Online`. It installs an internal subscribe, aggregates the replica's worker rows per dataflow, anti-joins against history, and writes missing rows through timestamped OCC read-then-write. Retention retracts successive bounded batches on the catalog server even when collection from the user replica fails. The main correctness boundaries are: * **Idempotence across environmentd processes.** The subscribe reads its own target table. Concurrent collectors may compute the same candidate, but the oracle selects a write timestamp, the subscribe frontier certifies completeness below it, and a loser recomputes after observing the winner's row. * **Replica-scoped worker aggregation.** Installation adds one row per worker with a nullable `hydrated_at`, and the replica owns and resets the collection as a unit. `HAVING count(*) = count(hydrated_at)` waits for every row visible at the OCC read timestamp, and `max(hydrated_at)` normally includes the persist sink's elected worker, whose stamp covers the snapshot write. Per-process logging clocks can place an ahead worker beyond the sampled timestamp. The collector explicitly accepts that race instead of depending on configured worker counts. * **One row per dataflow.** The table stores the dataflow id under `object_id`; `mz_object_global_ids` is used only to filter to user indexes and materialized views. This preserves separate episodes when one catalog item has several live dataflows. * **Fleet-safe scheduling.** Fires use a stable SHA-256 seed over the full environment id, so a fleet-wide dyncfg does not synchronize environments, including regions and ordinals in one organization, on the same absolute boundary. * **Bounded retention catch-up.** The `LIMIT` is inside a derived table because top-level `RowSetFinishing` is not part of the OCC write selection. Each sweep retracts one bounded batch, so collection gets another turn while later sweeps drain the finite fixed-cutoff backlog without exceeding `max_result_size`. * **Background isolation.** Background mutations do not take the process-wide OCC semaphore, because their replica-targeted subscribe may take arbitrarily long to hydrate. The sweep is single-flight, so this adds at most one concurrent read-then-write. Its task is coordinator-owned and abort-on-drop, while fallible coordinator calls handle concurrent shutdown. The entry point requires an actual system-table target and system-only dependencies, separating it from session DML. * **Operational visibility.** Internal metrics report collection and retention outcomes, affected rows, complete sweep duration, and full retention batches without cluster, replica, or object labels. Repeated full batches indicate that retention may not be keeping up with the configured sweep interval. The read-then-write path gains explicit background-caller contracts for replica targeting, system-table writes, system-read dependency validation, ownership, cancellation, exact-timestamp writes, and trivial top-level row-set finishing. Background subscribes are coordinator-owned, write no `mz_subscriptions` row, and count against an internal gauge. Dropped coordinator responses unwind as errors, including a background compute-client lookup racing shutdown. Known limitations: a process whose logging clock is ahead can place its hydration row beyond the OCC read timestamp, so the durable finish can precede the latest worker's finish. Separately, the oracle chooses the write timestamp and the replica-targeted subscribe frontier certifies the read. If the replica's clock trails environmentd by more than an introspection interval, the frontier can certify each target only after the oracle has advanced past it. The resulting conflict loop ends at the sweep timeout or retry budget. It records nothing and logs a diagnostic naming the likely frontier. ### Verification * `test/testdrive/hydration-status.td` covers timestamp ordering, multiple replicas, idempotence, replica removal, and retention. * `test/restart/mzcompose.py` verifies one pre-restart episode survives unchanged, one fresh episode is collected, neither is duplicated, and a materialized view on a two-worker replica records the all-worker maximum rather than worker 0. * Unit tests cover replica rotation, all-worker completeness, filter placement, full-environment scheduling offsets, and forced migration policy. Collection defaults off in production and is runtime configurable. The rollout plan is CI on merge, then staging, then production, one week apart. This release will record completed hydration of indexes and materialized views in `mz_internal.mz_object_hydration_history` when collection is enabled. Closes: [SQL-644](https://linear.app/materializeinc/issue/SQL-644)
## Motivation Sources on the maintained-objects (Objects) page were described in replica-hydration terms, which misrepresents them three ways: * The status column showed `Not Hydrated` / `Hydrating` / `Running` buckets derived from `mz_hydration_statuses`. For a source, `hydrated` only tracks whether `rehydration_latency` is set, so a healthy source could read `Not Hydrated` (Slack: #console, Aug 19 thread on what "Not Hydrated" means). * A source could show `99%` snapshot progress next to `1s` freshness because completion was judged by the staged/known ratio, whose denominator is an estimate that can sit below 100% forever. * Webhook sources have no hydration rows at all, so they showed no status. Separately, the source detail panel gave no way to see the ingest cluster's resources while diagnosing a lagging source. ## Changes One commit per concern, reviewable in order: 1. **Source ingestion status on the Objects page.** The status column, filter, and detail-panel badge show the source's ingestion status (`snapshotting`, `running`, `paused`, `stalled`, ...) via `ConnectorStatusPill`, matching the Sources page. Non-source objects keep the hydration buckets. The filter derives from the same function the cell renders, so it can never offer a value the cell doesn't show. Status rides on the existing hydration-aggregate subscribe: the aggregate now FULL JOINs `mz_source_statuses` (so webhook sources get a row) and carries `bool_and(snapshot_committed)` from `mz_source_statistics`. 2. **Snapshot display keys off `snapshot_committed`.** The diagnostics card uses the authoritative boolean instead of the staged/known ratio, and now renders only when it has something diagnostic to say: a status error, or an in-progress snapshot (with a per-source-type note that counts are estimates, shared with the Sources page via `snapshotEstimateNote`). Steady-state lifecycle facts (rehydration time, messages received) already live on the source details page, so the card no longer duplicates them. 3. **Source cluster metrics on the Freshness tab.** The detail panel shows the ingest cluster's replica memory/CPU at the point selected on the freshness graph, including replicas dropped in the window (reuses the critical-path cluster metrics via a new shared `ClusterReplicaMetrics` component; falls back to each replica's latest bucket when the anchor bucket predates a replica swap). 4. **Feed hardening.** Subsources and progress collections are excluded from the aggregate (the UI hides them, and they multiply the feed's cardinality several times over), and the `snapshot_committed` aggregate is restricted to statistics rows from live replicas (keeping null-`replica_id` rows, which is how webhook sources report), so a dropped replica's stale row cannot pin a committed source back to snapshotting. ## Tests Extended `hydrationAggregate.test.sql.ts` for the FULL JOIN, `bool_and(snapshot_committed)`, subsource/progress exclusion, and live-replica semantics, and `MaintainedObjects.test.tsx` for the source status column and filter. Closes [CNS-97](https://linear.app/materializeinc/issue/CNS-97) Closes [CNS-82](https://linear.app/materializeinc/issue/CNS-82) Closes [CNS-85](https://linear.app/materializeinc/issue/CNS-85) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…on_root (#38432) ### Motivation Review feedback on #38199: the `publish_via_partition_root = true` warning on the PostgreSQL partitioned tables page listed *adding* a partition as one of the operations that surfaces problems. That isn't right. Partitions created with `CREATE TABLE ... PARTITION OF` are always empty, so they move no data into the root table and cause no divergence. Truncating a partition, which does remove root-table data, was missing from the list. ### Description Replaces "added, attached, or detached" with "attached, detached, or truncated" in the warning. All three of those are DDL that adds or removes root-table data without emitting CDC events, which is precisely the failure mode the warning describes. ### Verification Docs-only prose change; no code or test surface. The paragraph rewraps to stay within the file's line width. --- _Generated by [Claude Code](https://claude.ai/code/session_01HBUT5iCf2hkNY2hEfruLTp)_ Co-authored-by: Claude <noreply@anthropic.com>
…ides (#38492) ### Motivation Fixes SS-439 The MySQL snapshot feature-benchmark scenarios have raised `RELATIVE_THRESHOLD` values to absorb the wallclock changes from #38094. We don't want to leave the higher thresholds around indefinitely because they won't catch regressions. ### Changes Remove bumped thresholds and prevent comparison against older versions at the tighter threshold. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of [SS-358](https://linear.app/materializeinc/issue/SS-358). Adding docs for the new MySQL parallel snapshotting feature, some more context on the load we place on upstream databases, and some high-level debugging tips for triaging obvious database issues. The table-splitting feature ships with `mysql_source_snapshot_parallelism` off by default, so it is documented as private preview: a banner on the MySQL snapshot parallelism page and inline preview markers where the splitting is mentioned elsewhere. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016AgSuGEvoVT9TFHAgq6FCi --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )