Repository navigation
[pull] main from MaterializeInc:main - #1403
Merged
Merged
Conversation
…n level (#39060) ### Motivation `OrchestratorWorker::handle_command` retries every failed `EnsureService`, `DropService` and `ListServices` call forever (`Retry::default()`, backoff starting at 125ms and capped at 10s), and logged every failed attempt with `tracing::error!`, which Sentry records as an event. A Kubernetes API error that clears on retry therefore reaches Sentry even though the command succeeds a moment later. [DATABASE-BACKEND-503](https://materializeinc.sentry.io/issues/7746503161/) ([CLO-288](https://linear.app/materializeinc/issue/CLO-288)) is one such case: on two staging environments running v26.43.0-rc.1 and rc.2, environmentd's service account was refused `list` on StatefulSets at startup, `statefulsets.apps is forbidden`. Each environment produced three events within the same second and none after, which matches three failed attempts followed by a success at about 875ms. The Cloud team attributes it to RBAC that had not yet propagated. ### Description - The first `RETRY_WARN_ATTEMPTS` (5) attempts of a failed worker command log at warn level, and later attempts at error level. With the existing backoff, the first error-level attempt comes about 4s after the first failure, so a persistent failure still reaches Sentry within seconds. - Both levels carry the attempt number and the time since the first attempt as structured fields. The message text is unchanged, so existing Sentry grouping is unaffected. - The retry behavior itself is unchanged. No tests were added. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ry (#39033) Adds a type override for iceberg sinks so that UUIDs are written as text instead of fixed binarys, as Unity Catalog doesn't have support for fixed types.
…as (#39041) ### Motivation Storage replicas hardcode an arrangement exertion proportionality of 1337 in `mz_controller::clusters`, while compute replicas read theirs from the `arrangement_exert_proportionality` dyncfg, defaulting to 16. With the storage and compute runtimes unified on one replica, there is no reason for the two kinds of arrangement to run different maintenance policies. The proportionality halves per layer below the largest batch, so 1337 lets the maintenance policy request consolidation of any non-empty layer within about ten levels of the largest one. Because the policy is consulted once per operator activation, every idle scheduling turn can then lift a freshly published small batch into the largest batch on its own, and the largest batch is rewritten once per published batch instead of once per doubling. On a skewed 105-source upsert hydration of about 407 GB of state (8 workers, 400cc), the synchronous v2 stash took 56 to 83 minutes at 1337 and 44 minutes at 16, level with the RocksDB-backed v1 on time and CPU. ### Description Storage's Timely config now reads `arrangement_exert_proportionality`, the same replica-scoped dyncfg compute uses, so both run at 16 by default and move together when it is changed. The value is read once per replica provisioning, as before. Zero still disables optional maintenance. No new dyncfg is introduced. Release note: None.
…#38848) Converts mz_object_global_ids to a builtin materialized view. Read commit messages for more details ### Motivation Closes [SQL-700](https://linear.app/materializeinc/issue/SQL-700) ### Description Part of the builtin migration work. The migration is pretty straight forward. ### Verification - Created a new SLT to test the SQL itself. - Diffed this relation using a customer's workload replay against main and this branch.
…9034) ### Motivation Swap metrics shipped in v26.43.0 on `mz_cluster_replica_metrics`, but the Console's cluster utilization graphs don't read that relation. They read the indexed `mz_console_cluster_utilization_overview*` views, which have no swap column, so the data can't reach the UI. ### Change Three columns added to both view bodies, and so to all three indexed views: ``` ram_percent = MAX(memory_bytes) / memory_bytes_per_process swap_percent = MAX(swap_bytes) / memory_bytes_per_process heap_limit_percent = MAX(heap_limit) / memory_bytes_per_process ``` All three are denominated in a single process's RAM allocation so they share one axis: `ram_percent` is the bar, `swap_percent` stacks on top of it, and `heap_limit_percent` marks where RAM + swap runs out, above 1.0. A replica that is spilling draws above the 100% RAM line — the state the existing columns cannot express. ### MAX rather than SUM These aggregate across processes with `MAX`, joining `heap_percent` rather than `cpu_percent`/`memory_percent`/`disk_percent`, which are `SUM`. A replica dies when any *one* process is OOM-killed, not when its average crosses a limit. Four processes at 63/6/6/6 GiB against a 64 GiB per-process limit is 32% under SUM and 98% under MAX; the replica is killed either way. On a chart with a RAM limit line drawn on it, SUM would show a comfortable bar for a replica that is actually crossing that limit. Swap is what happens *at* that limit, so the reasoning that already made `heap_percent` a worst-process metric applies here too. At `scale=1` the two are identical, so this only changes the multi-process sizes. ### Cost on mz_catalog_server Peak RAM and peak swap are taken as **independent** `MAX`es. On a skewed multi-process replica they can describe different processes, making the stacked total an upper bound rather than an observed state. Selecting them row-consistently needs an argmax over processes (`DISTINCT ON (occurred_at, replica_id) ORDER BY heap_bytes DESC`, joined back). I implemented and measured that: **+6 operators and +3 arrangements in each of the three indexed views**. Since idle catalog_server cost tracks operator count rather than data volume, that is a real standing charge on a shared cluster. The independent form folds into the hierarchical aggregate already present in each body — `Aggregations: max, max` becomes `max, max, max, max, max` — leaving counts identical to before: | View | operators | arrangements | |---|---|---| | `_overview` | 80 → 80 | 20 → 20 | | `_overview_24h` | 80 → 80 | 20 → 20 | | `_overview_3h` | 33 → 33 | 9 → 9 | If the exact-stack semantics turn out to matter in practice, the argmax version is a contained change with a known price. ### Notes for reviewers - Values are fractions in `0..1`, matching the other `_percent` columns in these views, which the Console scales client-side. This differs from `mz_cluster_replica_utilization`, where `swap_percent` is scaled in the view itself. - No builtin schema migration entry is needed — these are `BuiltinView`s, not persist-backed collections. - `swap_bytes` is null on replica sizes with swap disabled and on the process orchestrator, so `swap_percent` is null there rather than zero. `heap_limit` has its own null path via the `/proc/meminfo` fallback in `collect_heap_limit()`. - Unlike `heap_percent`, these have no SUM fallback when `heap_limit` is null; they go null instead, which seemed more honest for a chart with a limit line on it. ### Follow-up A Console PR consumes these as a stacked memory/swap chart, plus a cluster-list column. It needs version-gated query builders, since the three builders in `replicaUtilizationHistory.ts` currently have none and would break on environments predating these columns. ### Tests `test_smoketest_all_builtins` covers this — it plans every builtin and asserts the declared `RelationDesc` matches the SQL's output columns. Regenerated `catalog_server_explain.slt` and `mz_catalog_server_index_accounting.slt`. ### Release notes No user-facing release note. These views are `mz_internal` and marked `RELATION_SPEC_UNDOCUMENTED`; they exist for the Console. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
### Motivation https://materializeinc.slack.com/archives/C08AENAP691/p1790010546649459 ### Description Clarifies the LIR versioning policy, in comments and messages. This will have to change when LIR pinning is deployed. ### Verification N/A
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )