Repository navigation
[pull] main from MaterializeInc:main - #1305
Merged
Merged
Conversation
…ce (#38071) Every test in the adapter today needs a running Materialize, so the plain Python pieces like the config parsers, the connection options string and the relation types have no cheap coverage and small mistakes in them only show up through a full integration run. This adds a `tests/unit` directory for those, which runs in well under a second and needs no server. Test plan: 25 new tests covering `_build_options_string`, the index, refresh interval and partition by config parsers, and the Materialize relation types.
```
warning: use of deprecated method `std::sync::atomic::Atomic::<usize>::fetch_update`: renamed to `try_update` for consistency
--> src/timely-util/src/column_pager/policy.rs:136:18
|
136 | .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |cur| {
| ^^^^^^^^^^^^
|
```
## Summary
Automated daily refresh of `doc/developer/generated/` to reflect source
code changes.
This PR was generated by the `update-generated-docs` workflow running
`.claude/commands/update-docs.md`.
## Scope
Only files under `doc/developer/generated/` are modified.
## Review checklist
- [ ] No unrelated rewording or restructuring of existing text
- [ ] No changelog language ("now", "added", "previously", etc.)
- [ ] Changes correspond to actual source code changes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Moritz Hoffmann <antiguru@gmail.com>
Main currently fails `bin/fmt --check` on rustfmt diffs in the pushdown soundness fuzz target introduced by #37979, which turns the lint-and-rustfmt job red for every PR based on the current tip. This runs `bin/fmt` and commits the result, no functional changes.
) Follow-up to #38074, which disabled `ENABLE_SYNC_MV_SINK` by default after incident 1175. The sync MV sink's `write` Tokio task performed correction-buffer maintenance inline. Inserts merge chains spanning the whole buffer and consolidation sorts it, neither with an await point in between, so a pass over a large buffer occupied a Tokio worker thread for its entire duration and stopped it polling everything else scheduled there. Persist lease heartbeats were among the casualties: readers lost their leases and the process halted with `batch fetcher could not fetch batch part`. The async sink never hit this, because its operators run on the Timely thread. Under memory pressure the pass is not merely slow. `CorrectionV2` pages its chunks out through the column pager, and consolidation calls `Chunk::column` on every chunk it merges, so the chunks come back one blocking read at a time from inside the sorts and merges. The thread is blocked in the kernel rather than computing. That rules out fixing this by yielding: there is no program point between the faults at which a yield could be placed, and a cooperative fix is only ever as strong as the worst un-yielded path. Running the work on a blocking thread lets the OS preempt it, which holds regardless of what the buffer does. `Correction::updates_before` is split into `consolidate_before`, which does the expensive part, and `consolidated_updates_before`, which reads the result back. `updates_before` becomes the composition of the two, so the async sink and the benchmarks are unaffected. The `CorrectionV2` `since` guard moves from `updates_before` into `consolidate_before`; `consolidate_at_since` always passes an `upper` beyond the `since`, so it is unaffected. The write task runs the correction work on a blocking thread and then feeds the updates to the persist batch builder from the async task, where the loop awaits at every part flush and so stays bounded. The cost is a thread hop per command, one or two per Timely activation. Note that `persist-client`'s `IsolatedRuntime` documents a preference against `spawn_blocking` for CPU-bound work, on thread-count and shutdown grounds. It is used here deliberately: the failure mode is a thread blocked in the kernel, and OS preemption is what addresses that. This does not re-flip the `ENABLE_SYNC_MV_SINK` default, which stays `false` pending gradual rollout via LaunchDarkly. mzcompose continues to set the parameter to `true`, so CI exercises the path. Adds `write::tests::correction_work_does_not_stall_the_runtime`, which stalls every worker thread of a two-worker runtime inside the correction path via a new `mv_sink_correction` failpoint and asserts an unrelated task keeps getting polled. Moving the `apply_batch` body back out of `spawn_blocking` fails it. The failpoint also drives the end-to-end lease-expiry scenario when set through the `FAILPOINTS` environment variable on a clusterd process; the `failpoints` session variable only reaches environmentd, never a cluster. ### Release notes This release will not include user-visible changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
### Motivation The two-runtime read-isolation work (#37770) runs two compute runtimes in a single clusterd process, sharing one metrics registry. Without per-runtime labels the two runtimes register the same metric series and collide. This PR adds the role concept and the metric labeling on its own, so that the two-runtime PR is left with only the runtime wiring. ### Description Introduce `ComputeRuntimeRole` (`Solo`, `Maintenance`, `Interactive`) in `mz_compute::server` and thread a role through `serve` into `ComputeMetrics::register_with`. Each named role stamps a distinct `role` const label on its metrics, so two runtimes sharing one process registry register distinct series rather than colliding. The whole-registry `workload_class` postprocessor is gated on the globals-owning role so it is installed exactly once. `Solo` is the single-runtime default and is behaviorally identical to compute before a second runtime existed: it emits no `role` label, so exact-match dashboards and alerts are byte-unchanged, and it owns the process globals as the sole runtime always has. Every call site passes `Solo` here. Two unit tests assert the two halves of the contract: `Solo` metrics carry no `role` label, and the two named roles carry distinct labels so they coexist on one registry. ### Context Prerequisite for #37770, which flips the call sites to `Maintenance` and `Interactive` and adds the runtimes that use them. Independent of the other #37770 prerequisites (#37880, #37881). ### Verification `cargo check -p mz-compute -p mz-clusterd` passes with no new warnings. The two new `metrics::tests` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
### Motivation The [temporal filter pushdown section](<https://materialize.com/docs/transform-data/patterns/temporal-filters/#temporal-filter-pushdown>) tells readers that filters matching the patterns shown on that page are pushed down to the storage layer, and that this substantially improves ad-hoc query latency and cuts the time to start serving results after a materialized view is created or its cluster restarts. As of v26.34 that does not hold for most temporal filters, so the section promises behavior users do not get. Requested by the docs discussion on filter pushdown; removing it for now is the agreed short-term step while the longer-term question of how much of filter pushdown to support is decided separately. ### Description Wrap the section in `{{< hide >}}` rather than deleting it, so the text stays in the source while the optimization is reworked. This follows the existing use of that shortcode in `reference/system-catalog/mz_catalog.md`. The one non-obvious part is the anchor stub. Nineteen links across the docs point at `#temporal-filter-pushdown`, including six webhook guides that recommend `try_parse_monotonic_iso8601_timestamp` specifically for this reason, `explain-plan.md`, `explain-filter-pushdown.md`, `functions/pushdown.md`, `troubleshooting.md`, and two published release notes that should not be rewritten. Hiding the heading removes the anchor those links target, which would leave all nineteen dangling and fail `htmltest`. An explicit `<a id="temporal-filter-pushdown" name="temporal-filter-pushdown">` stub is left outside the hidden block so the links still resolve to this page. Raw HTML is passed through: `doc/user/config.toml` sets `markup.goldmark.renderer.unsafe = true`, and the same pattern is already used in `headless/partition-by-option-description.md`. Not addressed here, and worth a decision separately: `/sql/functions/pushdown/`, `/sql/explain-filter-pushdown/`, the pushdown discussion on the partitioning guide, and the webhook guides' recommendations all still describe the optimization. This PR covers only the section that was called out. Docs-only, no release note. ### Verification Docs-only change. `hugo`/`htmltest` are not available in this environment, so verified by hand: `{{< hide >}}` renders nothing (`{{if false}}` in `doc/user/layouts/shortcodes/hide.html`), the shortcode already wraps a section containing a heading elsewhere in the docs, and the anchor stub sits outside the hidden block so it still renders. The `lint-docs` CI step will run `hugo` and `htmltest` over the result and is the real check on the nineteen inbound links. 🤖 Generated with [Claude Code](<https://claude.com/claude-code>) Co-authored-by: Claude <noreply@anthropic.com>
Follow-up to #37680, sorry for not catching this earlier Test run: https://buildkite.com/materialize/nightly/builds/18128
…38018) Problem: `CREATE METRIC SINK` only wrote a catalog item. It never optimized or shipped a dataflow, so nothing actually ran. On top of that a metric sink export had neither a trace nor a `sink_write_frontier`, so every frontier report logged `collection without write frontier`, the controller never saw progress, and the since of the sink's input stayed pinned. Solution: `CREATE METRIC SINK` now optimizes and ships a dataflow, `DROP` tears it down, and bootstrap re-renders every metric sink so one survives a restart. The assembly optimizer starts from the `GlobalId` of the collection to export, like `CREATE INDEX`, pushes the row-wise shaping into MIR, and exports a single `MetricSink` sink. Sequencing is staged so optimization runs off the coordinator thread. Bootstrap reuses the global expression cache, like materialized views do. A cached plan embeds a transient id for the shaped view, and reusing that id across a boot is safe because build ids are dataflow-local on the worker and never registered in the controller's instance-global collections. For progress, the operator already folds the combined ok+err input frontier, so we hand that to the `ComputeState`. Every worker reports it, not just the one that owns the registry collector. The controller meets the per-worker frontiers, so a worker stuck at the minimum would otherwise hold the input back. Rendering create-item notices and shipping a dataflow under a read hold move into the shared `render_create_item_notices` and `ship_new_dataflow` helpers, with `CREATE INDEX` moved onto both. Testing: testdrive drives a sink's series into the replica's Prometheus registry and back out through `mz_cluster_prometheus_metrics`, then checks that `DROP METRIC SINK` and `DROP VIEW ... CASCADE` retract them. The restart platform check now probes for the re-rendered dataflow, not just the re-parsed catalog item. Introspection is per-replica and the checks default to two replicas, so it targets one. An optimizer unit test pins the assembled dataflow to exactly one `MetricSink` export over the shaped view. Co-Authored-By: Moritz Hoffmann <antiguru@gmail.com> --------- Co-authored-by: Moritz Hoffmann <antiguru@gmail.com>
### Motivation Implements the [Improved Connect Modal](https://linear.app/materializeinc/project/improved-connect-modal-e31564018004) Linear project ([CNS-124](https://linear.app/materializeinc/issue/CNS-124), with [CNS-125](https://linear.app/materializeinc/issue/CNS-125), [CNS-126](https://linear.app/materializeinc/issue/CNS-126), [CNS-127](https://linear.app/materializeinc/issue/CNS-127), [CNS-128](https://linear.app/materializeinc/issue/CNS-128), [CNS-129](https://linear.app/materializeinc/issue/CNS-129)): make it easier for customers to get connection instructions through the console. Follows the design prototype at https://connect-modal-eight.vercel.app/. ### What changes The connect modal becomes a right-side drawer ("Connect to Materialize") with three method tabs: * **MCP Server** (default): one numbered step per concern: pick a server (Developer for troubleshooting and observability, Agent for querying data products), select a client, acquire a credential (token flow only), install the client-specific config, then an Authenticate step in the OAuth flow showing the client's sign-in command where one exists (`claude /mcp`, `codex mcp login <server>`) or its sign-in behavior otherwise. The client dropdown covers Claude Code, Cursor, Claude Desktop, VS Code, Windsurf, Codex CLI, Gemini CLI, and a generic option. Each config format was verified against the client's docs: CLI command for Claude Code, `mcpServers` JSON for Cursor, `servers` JSON with `"type": "http"` for VS Code, `serverUrl` JSON for Windsurf, `httpUrl` JSON for Gemini CLI, TOML for Codex, and for Claude Desktop a custom-connector URL (OAuth) or an `mcp-remote` proxy config (token flow). Cursor and VS Code additionally get one-click install buttons via their `cursor://` and `vscode:mcp/install` URL handlers. OAuth is the default flow where the environment supports it (existing 26.30.0 gate), with a "Use a token instead" toggle. In the token flow, cloud sessions generate an app password that is substituted into the command directly, masked in the display. Self-managed OIDC sessions get the token derived from their ID token in a secret box, and password-auth sessions get the Base64 command. Install links are only generated for the OAuth flow so tokens never end up in URLs. * **External tools**: copyable Host, Port, Database, and User rows plus an inline create-password flow (name it, create it, copy it once). The created password is substituted into the DBeaver, dbt, and .env snippets below, masked in the display while copying yields the real value. * **Terminal**: the psql one-liner, preserving the existing user encoding, sslmode, and cluster option behavior. The three previous modal variants (cloud `ConnectModal`, `OidcConnectModal`, `PasswordConnectModal`) and the unreferenced `McpConnectModal` are deleted and replaced by one `ConnectDrawer` driven by `appConfig.mode`, OIDC availability, and the environment gate. Self-managed OIDC surfaces the ID token as the password. Pure snippet and deep link builders live in `connectOptions.ts`. The App Passwords page per-row Connect button opens the same drawer pinned to the service account user, with password creation hidden. Client logos are raw SVGs in `icon/` compiled through the svgr pipeline and exported from `~/icons`, like the rest of the icon set. Incorporates the design feedback round: per-concern step layout, a dedicated Authenticate step, no separate copy-token box for generated passwords, service account and agent access notes removed, connection detail rows no longer overflow the drawer, and the password naming input is explicit about naming rather than entering a password ("Create new password", empty input with a placeholder and helper text). Also fixes OAuth discovery for MCP clients pointed at the console host: the self-managed nginx config and the dev server now proxy `/.well-known/oauth-protected-resource` to environmentd. Previously clients received the console SPA instead of the RFC 9728 metadata and browser sign-in never started. Verified end to end against a local OIDC deployment (Auth0): resource validation and discovery pass, sign-in completes subject to IdP tenant config. ## Screenshots: - In the OIDC mode, with token: <img width="1300" height="1562" alt="image" src="https://github.com/user-attachments/assets/f2062c4c-f59a-45a6-a5d0-408f08083a14" /> - In the OIDC mode, oauth flow: <img width="1276" height="1494" alt="image" src="https://github.com/user-attachments/assets/46933d90-9040-42e6-9eb5-45a54c0d89c5" /> - For password auth: <img width="1296" height="1490" alt="image" src="https://github.com/user-attachments/assets/61451e2b-bf73-4301-9277-946b9e523a87" /> ### Testing * New unit tests for all snippet builders and deep links (`connectOptions.test.ts`) and for drawer behavior across tabs, client switching, server switching, and app password creation with MSW (`ConnectDrawer.test.tsx`). * The dormant e2e helper `connectRegionPostgres` keeps working: the `connect-menu-button` test id, the "External tools" button name, and the Host, Port, and Database aria-labels are all preserved. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… file (#38178) Fixes [CLO-190](https://linear.app/materializeinc/issue/CLO-190/sgfleet-token-expiration-issue-with-azure-object-storage). Seems to work on an Azure env I spun up. ### Motivation A customer environment on AKS lost persist auth into Azure Blob Storage after running for a while. Root cause: `azure_identity` 0.21's default credential chain resolves AKS workload identity via `WorkloadIdentityCredential`, which reads the projected service account token from `AZURE_FEDERATED_TOKEN_FILE` **once at construction** and holds the contents for the life of the process. Kubernetes rotates that projected token, so once the last cached AAD access token expired, every refresh presented a stale client assertion to AAD and failed, permanently locking long-running processes out of persist blob storage. ### Changes * Add a `RefreshingWorkloadIdentityCredential` in `src/persist/src/azure.rs` that re-reads the projected token file on every AAD access token refresh and exchanges it via the public `azure_identity::federated_credentials_flow::perform`. Resulting access tokens are cached per scope until near expiry, mirroring `azure_identity`'s internal token cache semantics (including its 20s expiry buffer), with refresh under a write lock to avoid stampeding AAD. * Use it in `AzureBlobConfig::new` when `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, and `AZURE_FEDERATED_TOKEN_FILE` are all present (and `AZURE_FEDERATED_TOKEN` is not, that static-token path keeps its upstream precedence). Otherwise fall back to `create_default_credential()`, whose remaining credential types (e.g. IMDS managed identity) refresh correctly. * The SAS-token-in-URL path is unchanged. The stale TODO about refreshing credentials is replaced with a NOTE documenting the remaining limitation: a SAS token passed via the URL is static and must outlive the process. * Unit test covers token-file re-read on refresh, cache reuse before expiry, whitespace trimming, and `clear_cache`, without contacting AAD (the exchange is injected). ### Out of scope Migrating off the archived unofficial `azure_storage`/`azure_identity` 0.21 crates to the official Azure SDK for Rust is the long-term fix for this class of issue, but is a much larger change and not needed to resolve this incident class. ### Tips for reviewer Compare with the vendored `azure_identity` 0.21 sources: `WorkloadIdentityCredential::create` reads the token file once (`token_credentials/workload_identity_credentials.rs`), and it enters the default chain via `EnvironmentCredential` (`token_credentials/environment_credentials.rs`). ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
https://linear.app/materializeinc/issue/QAR-41/migrate-existing-issues-from-github-database-issues-linear We could merge this in case someone else wants to use it. I keep using it regularly so might be worth it.
### Motivation Implements the compute half of [Improved Hydration Visibility](https://linear.app/materializeinc/project/improved-hydration-visibility-f8cb205ddd89/overview), per the design in #38238 (`doc/developer/design/20260817_compute_hydration_timestamps.md`). That design PR is doc-only; this is the code. `mz_introspection.mz_compute_hydration_times_per_worker` reports a single duration, and only for objects that finished hydrating. There are no absolute timestamps, so nothing downstream can roll per-object rows up into a replica or cluster episode. In-flight hydration is opaque, because a NULL `time_ns` is indistinguishable from waiting on inputs, queued behind `HYDRATION_CONCURRENCY`, or never installed. And the duration measures the wrong interval: `ExportState::created_at` is stamped at `CreateDataflow`, before the `StartSignal` token is dropped by `Schedule`, so it spans input-frontier waiting and hydration-slot queueing along with the actual work. ### Description Adds three replica-stamped wallclock timestamps to the compute hydration log: | Column | Stamped when | | --- | --- | | `installed_at` | the `Export` event, from `CreateDataflow` | | `started_at` | the dataflow is unsuspended | | `hydrated_at` | the output frontier passes the as-of | `hydrated_at - started_at` is hydration time as users mean it; `started_at - installed_at` is the queueing interval. Three decisions carry the diff. **The replica stamps, not the compute controller.** Controller state is rebuilt on every environmentd restart, so its timestamps would reset on a restart that never disturbed the replica, and `ReplicaState::epoch` resets too, so it cannot serve as an episode key. Compute log event times are already Unix-epoch anchored and `Instant`-monotone, so `installed_at` is `NOT NULL`, stamped at inception, and stable across reconnection and reconciliation. **The event time is stamped, not the rounded update timestamp.** The demux has the event `Duration` per event and separately computes `ts()`, which rounds up to the next logging interval. Using the event time means the rounding costs visibility of very short episodes rather than accuracy of recorded ones. Values are rounded to microseconds (`timestamptz`'s maximum resolution) with `round_to_precision`, which is monotone and so cannot reorder two instants. **The relation is widened in place, keeping its name and OID.** The design called for renaming the log and giving the old name to a compatibility view projecting the original three columns, so that nothing observed a shape change. Review pushed back on carrying the old shape around, and it turns out to buy nothing: every consumer of `mz_compute_hydration_times_per_worker` is in this repo and selects named columns (the `ComputeHydrationTimes` subscribe in `coord/introspection.rs`, and one cluster test), the console reads `mz_hydration_statuses` and `mz_internal.mz_compute_hydration_times` rather than the per-worker relation, and the relation is `RELATION_SPEC_UNDOCUMENTED`, so it carries no documented contract. Widening in place therefore changes nothing a consumer can observe, and it avoids a compatibility view, a new OID, and two `MigrationStep::replacement` entries: `make_mz_sources` and `make_mz_indexes` inline each log's oid, schema and name into their generated SQL, all of which a rename changes and none of which the widening touches. `builtin.rs`, `oid.rs` and `builtin_schema_migration.rs` are untouched as a result. `time_ns` is also retained rather than derived from the timestamps, so its values do not change and no cross-worker arithmetic is introduced. The two carry distinct jobs: `time_ns` is the authoritative per-worker duration, the timestamps carry episode identity. Two cases need care to keep `installed_at <= started_at <= hydrated_at` total: - Log collections are marked scheduled implicitly by the controller and never receive a `Schedule`, so `initialize_logging` emits the start event itself. Without it they would sit permanently in the illegal state `(installed_at, NULL, hydrated_at)` on every replica. - `StartSignal` gates only imports, so an import-free dataflow can hydrate before its `Schedule` lands. Its operators were never suspended, so hydration really began when the dataflow was installed, and the demux backfills `started_at` from `installed_at`: the queueing interval reads as zero and the whole elapsed time as hydration work, which is what happened. This also keeps `hydrated_at - started_at` in agreement with `time_ns`, which measures the installed-to-hydrated interval. `handle_schedule` looks the collection up before logging, so a `Schedule` arriving after a drop is silent rather than an error. The demux guards on `started_at` already being set, making the event idempotent under reconciliation, where `Schedule` is re-sent for retained dataflows. ### Verification New `test/testdrive/hydration-timestamps.td`: - log collections report all three timestamps - a hydrated object reports all three, ordered, alongside a non-NULL `time_ns` - `installed_at` lands in the recent past, i.e. it is a wallclock instant and not an offset from an arbitrary origin - `time_ns` agrees with `hydrated_at - installed_at`, within a deliberately loose tolerance: the two measure the same interval by different means, one from an `Instant` and one from microsecond-rounded logging event times, so the check catches a wrong interval or a units error without being flaky about rounding - an object whose inputs are unavailable (input maintained by a `REPLICATION FACTOR 0` cluster) reports `installed_at` with `started_at` NULL, and acquires both once the input becomes available - the ordering invariant holds under repeated creation of import-free dataflows, without retries - an object queued behind `compute_hydration_concurrency = 1` reports `installed_at` with `started_at` NULL, and the object holding the slot reports `started_at` with `hydrated_at` NULL New `workflow_test_hydration_timestamps` in `test/cluster/mzcompose.py`, against an unorchestrated replica so environmentd and clusterd restart independently: timestamps are byte-identical across an environmentd restart, and every `installed_at` is fresh after a clusterd restart. The behaviour was also exercised directly against a local build: the relation reports six columns under its original name, log collections carry all three stamps, no row violates `installed_at <= started_at <= hydrated_at`, `installed_at` falls in the recent past, `time_ns` agrees with the timestamps, a never-converging dataflow holds the hydration slot with `started_at` set and `hydrated_at` NULL, and an object queued behind it reports `installed_at` with `started_at` NULL. The most important tests are the compatibility ones, and they were left alone rather than adjusted to fit: `test/testdrive/hydration-status.td`, the blue-green tests, and `workflow_test_subscribe_hydration_status` all assert on relations this change does not touch. Only one golden moves: the three added column rows in `mz_catalog_server_index_accounting.slt`. `oid.slt`, `information_schema_tables.slt`, `cluster.slt`, `cockroach/srfs.slt`, `catalog_server_explain.slt`, the autogenerated `mz_introspection.slt`, `catalog.td` and `indexes.td` were all regenerated and came back unchanged, since the relation keeps its name and OID. Out of scope, per the design: the per-replica `mz_compute_hydration_timestamps` rollup and durable episode history are follow-up work. --- _Generated by [Claude Code](https://claude.ai/code/session_01XMmAAJVxVNW2VnzWa4gskq)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
… on main (#38342) This increases cache reusage for Rust compilation, especially when one build occurs on Test pipeline and the other in Nightly, which are symlinked to the same path in our Buildkite setup --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
### Motivation Design doc for the compute half of the [Improved Hydration Visibility](https://linear.app/materializeinc/project/improved-hydration-visibility-f8cb205ddd89/overview) project. The PRD asks for per-object and per-replica hydration history: when hydration started, when it finished, and whether it finished at all. Compute observes none of that today. `mz_introspection.mz_compute_hydration_times_per_worker` reports a single duration, and only for objects that finished hydrating. There are no absolute timestamps, so nothing downstream can roll per-object rows up into a replica or cluster episode, and a replica has no notion of being globally hydrated, so that rollup is the only way to get one. In-flight hydration is opaque, because a NULL duration is indistinguishable from waiting on inputs, queued behind `HYDRATION_CONCURRENCY`, or never installed. And the duration itself measures the wrong interval: `ExportState::created_at` is stamped at `CreateDataflow`, before the `StartSignal` token is dropped by `Schedule`, so it spans input-frontier waiting and hydration-slot queueing along with the actual work. Ref: [SQL-632](https://linear.app/materializeinc/issue/SQL-632) Doc only, no code. ### Description Add three replica-stamped wallclock timestamps to the compute hydration log: `installed_at` (the `Export` event), `started_at` (the dataflow is unsuspended), and `hydrated_at` (the output frontier passes the as-of). `hydrated_at - started_at` is hydration time as users mean it, and `started_at - installed_at` is the queueing interval. **Scope is deliberately narrow.** This widens the log relation and the per-worker introspection relation only. The per-replica storage-managed relation and its cross-worker rollup are follow-up work, and the doc enumerates the questions that belong to it: which aggregate suits which column, how to avoid publishing internally contradictory rolled-up rows, cross-worker clock anchors, object-set scoping, and freshness gating. Persisting episodes durably is the adapter half. Two decisions carry the design. **The replica stamps, not the compute controller.** The controller would be less code, since it evaluates a near-identical predicate and already has an append-only introspection path for `WallclockLagHistory`. But its state is rebuilt on every environmentd restart, so its timestamps would reset on a restart that never disturbed the replica, and `ReplicaState::epoch` resets too, so it cannot serve as an episode key. Compute log event times are already Unix-epoch anchored and `Instant`-monotone, so a replica-stamped `installed_at` is `NOT NULL`, stamped at inception, and stable across reconnection and reconciliation, which makes it a usable idempotency key for whatever writes history down. **`time_ns` is retained rather than replaced.** That makes the existing per-worker relation an exact projection of the new one, so the introspection subscribe, `mz_compute_hydration_times`, `mz_compute_hydration_statuses`, `arrangement_sizes_snapshot` and the console query are all untouched. Deriving `time_ns` from the timestamps would have dropped to microsecond precision, since `timestamptz` caps there, and deriving it after aggregation would additionally have absorbed cross-worker install and clock-anchor skew. The two columns get distinct documented jobs: `time_ns` is the authoritative per-worker duration, the timestamps carry episode identity. The design is therefore purely additive. It adds columns and a relation, and changes nothing that anything currently reads. Correctness details worth calling out, all found in review: - Stamp the **event time**, not the rounded update timestamp. The interval rounding then costs visibility of very short episodes rather than accuracy of recorded ones. - Emit the start event from `initialize_logging` as well, or log collections sit permanently in an illegal `(installed_at, NULL, hydrated_at)` state, since the controller marks them scheduled implicitly and never sends `Schedule`. - Backfill `started_at` when hydration arrives without it. `StartSignal` gates only imports, so an import-free dataflow can hydrate before its `Schedule` lands, and the interceptor's own comment anticipates this. This keeps `installed_at <= started_at <= hydrated_at` total. Limits recorded rather than solved: an episode whose transitions fall inside one consolidation window presents only its final state, and one that dies before reaching persist leaves no record, which is unavoidable in the limit. `REFRESH` materialized views report a refresh interval rather than hydration work, because the reported output frontier is the meet of write and compute frontier. Crash detection is explicitly not compute's job, since a replica cannot report its own death and replica lifecycle is already in `mz_cluster_replica_status_history` to join against. ### Verification Doc only, so nothing to test here. The implementation section lists touch points and the coverage the change will need, including cases for a queue-gated object, an object waiting on inputs, an import-free dataflow, log collections, and both restart paths. The most important tests are compatibility ones: `mz_compute_hydration_times_per_worker`, `mz_compute_hydration_times` and `mz_compute_hydration_statuses` returning identical values before and after. The existing assertions in `test/testdrive/hydration-status.td` and the blue-green tests are called out as the contract, to be left alone rather than adjusted to fit. The doc was reviewed by two agents, one fact-checking every code reference and one attacking the design. A "Settled during review" section records what was decided and why, so it does not get relitigated. Open questions are empty for this design and all belong to the follow-up rollup. --------- Co-authored-by: Claude <noreply@anthropic.com>
This PR adds recommendations for storage configuration of the consensus DB in the three cloud providers. The required IOPS are informed from the benchmark results in #37153 whereas the required disk size is derived by back of the envelope calculations of the average shard state. The disk size results of the benchmark are not representative since the benchmark was run with purposefully empty shards. Closes [CPU-202](https://linear.app/materializeinc/issue/CPU-202) Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )