Repository navigation
[pull] main from MaterializeInc:main - #1348
Merged
Merged
Conversation
) ### Motivation `EXPLAIN TIMESTAMP AS DOT FOR <query>` panics the parser (`unreachable!()` at `src/sql-parser/src/parser.rs`), and since the parse path is not wrapped in `catch_unwind` and the panic hook calls `process::abort()`, it crashes environmentd. Post-authentication, so any role that can run SQL can take the environment down with one statement. Verified on the v26.39.0 emulator (SIGABRT, exit 134). Found while reviewing the `EXPLAIN TIMESTAMP` docs page. Closes: [SQL-665](https://linear.app/materializeinc/issue/SQL-665) ### Description `parse_explain_timestamp` accepts `DOT` in `parse_one_of_keywords(&[TEXT, JSON, DOT])` but had no `Some(DOT)` arm, so `AS DOT` fell through to `_ => unreachable!()`. Added `Some(DOT) => Some(ExplainFormat::Dot)`, matching `parse_explain_plan`. The statement now parses and reaches the sequencer's existing `AdapterError::Unsupported("EXPLAIN TIMESTAMP AS DOT")` (`src/adapter/src/coord/sequencer/inner/explain_timestamp.rs`) instead of panicking. ### Verification Added a `parse-statement` regression case for `EXPLAIN TIMESTAMP AS DOT FOR SELECT 1` to `src/sql-parser/tests/testdata/explain`. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
CI derives its toolchain from the `rust-version` field in the root `Cargo.toml`, which pins 1.97.1. Warnings that newer toolchains introduce therefore go unreported until someone builds locally. This change clears everything that Rust 1.98 reports, plus the deprecations that the current nightly adds on top. Rust 1.98 extends `clippy::extra_unused_lifetimes` to higher-ranked binders in where-clauses, which flags three `for<'a>` binders whose lifetime never appears in the bound they introduce. Removing the binder leaves the bounds unchanged, because the parenthesized `Fn` sugar already introduces a fresh lifetime for each elided reference. The same toolchain also reports two unused imports and three `clippy::useless_format` calls. `parser_err!` calls `to_string` on a single-expression argument, so passing the literal directly still produces an owned `String`. Nightly additionally deprecates the legacy numeric constants and constructors. `max_value` and `min_value` become the `MAX` and `MIN` associated constants, and `std::usize::MAX` becomes `usize::MAX`. Dropping the `use std::i64` import in the Avro decoder is what silences the two warnings on `i64::MAX` and `i64::MIN`, because that import made the path resolve to the deprecated module instead of the primitive type. Two groups of nightly warnings are left alone. The `fetch_update` to `try_update` rename is too recent to use while CI builds with 1.97.1, and the `overflow evaluating the requirement` notices come from the new trait solver hitting its recursion limit rather than from a defect in this code. Note that this change does not close the gap that hid these warnings. CI keeps building with 1.97.1, so the next lint a newer toolchain adds will go unreported the same way. Bumping `rust-version` rebuilds the CI builder image and raises the declared minimum supported version, which belongs in its own change. ### Release notes No user-visible changes. 🤖 Posted by [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
### Motivation Materialize records successful hydration for individual indexes and materialized views, but it does not retain a replica-level history. This makes it difficult to answer when a replica finished hydrating, how many maintained compute exports participated, or what process resource high-water marks were visible when the episode was recorded. ### Description This change adds `mz_internal.mz_replica_hydration_history` and extends the existing durable hydration-history sweep to populate it. A replica hydration episode is a connected set of overlapping export intervals. Each live non-transient compute export contributes an interval from its earliest worker installation to its latest worker hydration, and counts as hydrated once every visible worker has reported its finish. A gap between intervals means that the replica was fully hydrated before the next export was installed, so the next interval starts a new episode. An export that has not hydrated keeps its episode in progress: whether it is slow or permanently stuck is unobservable, so an in-progress episode is recorded when it completes and does not block earlier, disconnected completed episodes. Transient exports are excluded because the collection query creates one itself and its intervals would otherwise show up as spurious episodes in later snapshots. The object-level collector remains limited to user indexes and materialized views. Each row records replica and cluster IDs, episode boundaries, the number of compute exports in the episode, status, and the largest resource high-water marks reported by any replica process. Memory uses cgroup `memory_peak`. Disk uses sampled `statvfs` `fs_used_peak` when a scratch filesystem is present, with cgroup `swap_peak` as the fallback. Managed resource limits apply independently to each process, so the maximum across processes is relevant. Summing process peaks could combine values that did not occur simultaneously. Resource observations have process-lifetime scope through collection. They are associated with an episode but are not bounded by its timestamps. Work after hydration can raise them, and a later episode can retain an earlier process peak. Collection is snapshot sampling rather than an event log. Each sweep records only the latest completed episode visible in its snapshot. Intermediate episodes can be missed, and exports retracted before collection leave no evidence. Only successful episodes are currently recorded. Process-local clock skew can merge intervals that did not overlap in real time. The collector reuses the replica-targeted timestamped read-then-write path. Exact-timestamp OCC and an anti-join make concurrent collection idempotent. A monotonic history guard prevents a later snapshot from recording an episode that precedes or overlaps retained history, and still admits an in-progress episode once it completes, because its start necessarily lies after every recorded finish. Retention is bounded and eventually consistent across concurrent collectors. The table survives normal `environmentd` and replica restarts and is protected from forced replacement migrations. Collection remains disabled by default through `hydration_history_collection_interval`. Coordinator shutdown explicitly aborts and waits for an active sweep before releasing timestamp-oracle workers, which preserves dependency ordering for timestamp-oracle clients cached by the sweep. ### Tests The hydration testdrive coverage exercises the production collector. It verifies that a system-only replica episode includes every visible non-transient export while excluding the collector's transient subscribe. It also checks single-export episode boundaries, compares persisted memory and disk values with independently published resource metrics, and verifies that a completed episode is recorded while a later, disconnected export is still hydrating. The restart workflow verifies that retained rows survive `environmentd` and replica restarts, restarts record fresh episode identities ordered after all pre-restart history, and repeated sweeps do not duplicate an identity. It re-reads the pre-restart episode set immediately before the restart, so its assertions do not depend on collection timing. Unit coverage checks episode query construction, latest-episode selection, process-count gating, replica rotation, task cancellation, and migration policy. Catalog expectations cover the relation, comments, OID, ontology metadata, and system catalog plans. ### Release note This release will add `mz_internal.mz_replica_hydration_history`, an internal durable history of successful replica hydration episodes and their observed process resource high-water marks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )