Repository navigation
[pull] main from MaterializeInc:main - #1321
Merged
Merged
Conversation
### Motivation https://linear.app/materializeinc/issue/CNS-121/add-memory-swap-and-cpu-metrics-in-the-clusters-list (Hydration Status not yet implemented) I had to significantly modify Cluster Table structure by adding child rows for replicas, as the metrics apply to replicas. ### Description Behind the Feature flag `usage-metrics-in-cluster-list-CNS121`, the clusters list nests each cluster's replicas beneath it, with per-replica size, status, CPU, Memory, and Disk usage. - **The flag picks a whole table component, not a set of columns.** This resulted in temporary code duplication, judged to be less annoying than interleaving feature flag checks throughout. Retiring the flag means deleting `ClusterTable.tsx` and one conditional statement. - **Shared cells moved out to `clusterTableCells.tsx`** to prevent circular dependencies. - **When sorting, a cluster row ranks by the maximum across its replicas.** By default, sorting was only reordering replicas within each cluster. - **Last status change stops ranking clusters by `latestStatusUpdate`,** which comes from the replica status *history* and counts replicas since dropped. - **`sortDescFirst` is pinned on all three columns.** TanStack otherwise infers direction from the first row's value type. Now that the first row (cluster) is now more of a grouping header, the first click wasn't giving meaningful results. - **Metrics columns compare numerically.** The shared `nullsLast` collates `.` as a separator, ranking 12.48 above 12.5 — which unrounded figures hit constantly. - **Search feature** when a cluster name matches search criteria, that cluster's replica rows are displayed along with the cluster row. When a replica name matches, its parent cluster row is displayed but its sibling clusters are hidden. **Unflagged Changes:** (the flag only picks a table component): - `UniversalTable`'s group-row treatment changed. The caret was changed to a button for accessibility & keyboard navigation reasons and moved to a column for alignment purposes. `AccountSpendBreakdown` table on Billing page was adapted to match and renders slightly differently as a result. - Container HTML elements were removed from data cells in order to manage row heights dependably. - `useLatestOfflineReplica` is re-keyed from cluster id to replica id, so out-of-memory warnings attach to the replica that was killed. The flat table (when flag = false) rolls its replicas up to keep the cluster-level warning. - `buildClustersQuery` always fetches per-replica metrics regardless of feature flag. ### Verification 830 tests pass; `yarn typecheck` and `yarn lint` clean. (`PlanDetails.test.tsx` fails on `main` too and is unrelated.) New coverage in `ClustersList.test.tsx`,`ClusterTable.test.tsx`, `UniversalTable.test.tsx`, and `useLatestOfflineReplica.test.ts`. SQL tests pass. Tested UI locally. Will confirm on integration environment. **Billing Page Layout:** <img width="1108" height="524" alt="Screenshot 2026-08-17 at 12 49 46 PM" src="https://github.com/user-attachments/assets/5ad14e10-ec1d-4d75-9110-18b4bff6097a" /> **Cluster Table (Flag = true)** <img width="1238" height="1139" alt="Screenshot 2026-08-17 at 12 50 03 PM" src="https://github.com/user-attachments/assets/a9bd02f7-3ef3-4303-8c4c-28a5634654de" /> **Cluster Table (Flag = false)** <img width="1240" height="683" alt="Screenshot 2026-08-17 at 12 50 37 PM" src="https://github.com/user-attachments/assets/f3426e96-81b3-4a13-a93f-475c3251e99d" /> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Problem: A metric sink was invisible, `mz_objects` did not carry it, and there was no way to read back the SQL that created it. Solution: `SHOW METRIC SINKS [FROM <schema>] [IN CLUSTER <c>] [LIKE ...]` lists name, `from`, and cluster. `SHOW [REDACTED] CREATE METRIC SINK` replays `create_sql`. Both require `enable_metric_sink`. `mz_internal.mz_metric_sinks` exposes what the catalog knows about a sink, shaped like `mz_catalog.mz_sinks`, and joins into `mz_objects` as type `metric-sink`. It is a materialized view derived from `mz_catalog_raw`. It carries `owner_id`, so a sink's owner is now visible in the catalog and its ownership checks are testable. The relation carries only the columns something reads. `create_sql` and `redacted_create_sql` have no reader, and `SHOW CREATE` is how you get the SQL back. Testing: - `metric_sink.slt` covers discovery and access control: the `mz_metric_sinks` row resolving to its `FROM` relation, cluster, schema, and owner, `mz_objects` and `mz_show_all_objects` membership, the three `SHOW METRIC SINKS` filters, the `SHOW CREATE` round-trip, the audit rows, and privileges. - The restart platform check reads `SHOW METRIC SINKS` instead of probing with a `CREATE` expected to fail. --------- Co-authored-by: Moritz Hoffmann <antiguru@gmail.com>
…38406) https://linear.app/materializeinc/issue/CNS-136/helm-chart-stop-hard-disabling-statement-logging-expose-sample-rate-as ## Problem The operator chart passed `--disable-statement-logging` unconditionally, so orchestratord always emitted `--system-parameter-default=statement_logging_max_sample_rate=0` and query history was permanently empty in self-managed installs. There was no way to turn it back on short of forking the chart. ## Solution Replace the boolean orchestratord flag with `--statement-logging-max-sample-rate=<f64>` (`Option<f64>`, unset means no override), surfaced as the `operator.args.statementLoggingMaxSampleRate` chart value, defaulting to `0.99` to match Materialize Cloud. `0` still fully disables statement logging, and `null` inherits environmentd's default. The sample rate does not bound what statement logging writes, `statement_logging_target_data_rate` does. That is exposed as `operator.args.statementLoggingTargetDataRate`, defaulting to unset so environmentd's 2071 B/s applies. Since environmentd already defaults to Cloud's values for the target data rate (2071) and max data credit (52428800), a default install lands on Cloud's exact configuration. Both flags validate at parse time: the sample rate to `[0, 1]`, which environmentd otherwise rejects by refusing to open its catalog, and the data rate to non-zero, since the token bucket refills at that rate and 0 throttles everything forever. ## Testing `helm unittest misc/helm-charts/operator` (43 tests) covers the default, a custom rate, `0`, and unset for both values, plus a values-file fixture pinning the `int64` coercion that large byte rates need. `cargo clippy`/`test -p mz-orchestratord`, `bin/fmt`, `check-helm-docs.sh` and `check-copyright.sh` pass. Both validators verified against the built binary. No cluster needed. ## For the reviewer - **The chart's `enableInternalStatementLogging: true` default was inert while the max rate was pinned at 0, and now becomes live.** Internal statements share the single `Arc<ThrottlingState>` token bucket with user statements, so they do not raise total bytes above the cap, they consume the same budget. Console polling and `mz_system` traffic therefore compete with the user's own queries for the 2071 B/s, and throttled statements are dropped silently. Left as-is to keep this PR to one decision, but worth measuring. - **Query history is never truncated** for these five collections ([database-issues#7666](MaterializeInc/database-issues#7666)), so it accumulates for the lifetime of the environment. Pre-existing and identical to Cloud, which runs the same rates. - The new arguments change the environmentd StatefulSet spec, so existing environments flip to `UpToDate=False` / `WaitingForApproval` after the operator upgrade, and the fix only lands for them once a rollout is requested. - The flag rename is safe: nothing outside this chart set `--disable-statement-logging`. It does break a pinned-old-`operator.image.tag` + new-chart combination, as any generation-affecting flag change would. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
#38407) Linear: https://linear.app/materializeinc/issue/CNS-137/console-un-gate-the-query-history-view-in-self-managed-builds ## Problem The console's query history view was hard-gated to Cloud: self-managed users hitting `/query-history` were silently redirected home, and the Monitoring > Query History nav item was hidden. ## Solution Remove both gates. The feature code already supports self-managed (`QueryHistoryRoutes` has a working self-managed branch that simply leaves the default user filter unset). With the nav item unconditional, the `canViewQueryHistory` plumbing through `getNavItems` was dead, so it is gone too. ## Testing `ci/test/console/test.sh` / `ci/test/console/lint.sh` equivalents locally (`yarn lint`, `yarn typecheck`, vitest for the touched dirs). One new test, verified to fail without the change: `AuthenticatedRoutes.test.tsx` asserts `/regions/:slug/query-history` renders the page in a self-managed deployment instead of redirecting home. The nav entry is now an unconditional literal in a static list, so it gets no test. ## RBAC finding (requested by the issue) A self-managed role without `mz_monitor` gets the existing `UnauthorizedState`, not a blank page or error boundary. The privilege probe hardcodes builtin OIDs (16748 `mz_recent_activity_log`, 16750 `mz_statement_lifecycle_history`), both of which still match `src/pgrepr-consts/src/oid.rs`, and `isAuthorizedSelector` requires `SELECT` on `mz_statement_lifecycle_history`, which needs `mz_monitor`. That path is already covered by an existing `QueryHistoryList` test and is deployment-mode agnostic. ## For the reviewer - The `UnauthorizedState` doc link points at the Cloud access-control docs (`/docs/security/cloud/access-control/manage-roles/`). Pre-existing, but self-managed users can now reach it. Left alone to avoid colliding with [CNS-138](https://linear.app/materializeinc/issue/CNS-138), which owns `QueryHistoryList.tsx`. - Merge order: this must land after [CNS-136](https://linear.app/materializeinc/issue/CNS-136) (Helm chart stops hard-disabling statement logging). Nothing here depends on it at build time, but until that ships, a self-managed console user reaching this view sees an empty list because sampling is off.
…ogging sampling is off (#38408) https://linear.app/materializeinc/issue/CNS-138/console-distinct-empty-state-when-the-statement-logging-sample-rate-is ## Problem Query history is backed by statement logging, which an operator can turn off by setting the sample rate to `0`. When that happens the console showed the generic filter-oriented "No results found." state, so the user could not tell sampling was off from their filters being too narrow. ## Solution * New API module `console/src/api/materialize/query-history/statementLoggingMaxSampleRate.ts` reads `SHOW statement_logging_max_sample_rate`, following the existing `maxReplicasPerCluster.ts` pattern. The effective rate is `min(session statement_logging_sample_rate, system statement_logging_max_sample_rate)`, so the cap alone detects a hard opt-out. * `QueryHistoryList` renders a distinct `SamplingDisabledState` when the result set is empty and the cap is `0`, naming both ways to raise it: `ALTER SYSTEM SET statement_logging_max_sample_rate`, and the Materialize operator's Helm chart values for self-managed. The filter-oriented state is unchanged for a non-zero rate. The new query is gated on `enabled: isEmpty`, so it never runs on the path that renders rows. ## Testing Snapshot test for the compiled query, plus two cases in `QueryHistoryList.test.tsx` covering both empty-state branches. While adding them I found `DEFAULT_FETCH_QUERY_LIST_HANDLER` never matched anything: the schema defaults `dateRange` to a window ending at `new Date()`, so its separate `queryHistoryListSchema.parse(...)` produced different filters than the `PARSED_DEFAULT_SCHEMA_VALUES` the component renders with. The handler now reuses the same parsed value. ## Notes for the reviewer * Rebased onto main now that #38407 ([CNS-137](https://linear.app/materializeinc/issue/CNS-137)) has landed. This branch previously carried a cherry-pick of @leedqin's #36533 un-gate; main has that change already, so the duplicate commit was dropped in the rebase. #36533 is now redundant and can be closed. * The Helm chart value the copy alludes to comes from #38406 ([CNS-136](https://linear.app/materializeinc/issue/CNS-136)). The message deliberately names no specific chart key, so it stays accurate regardless of what that PR settles on. * The copy renders in both deployment modes and mentions Helm only in a self-managed-qualified sentence, rather than branching on `AppConfigSwitch`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Comment-only cleanup of `mz_ore::pool` and `mz_timely_util::columnar`. No code changes: every edit is to a doc comment, inline comment, test doc comment, or assert message. An audit of these modules found the same facts documented in up to five places, doc comments that narrate their function bodies or their callees' contracts, and comments that restate the line below them. This PR applies the resulting rules: * Every fact keeps exactly one owning comment (the decision point for reasoning, the mechanism for mechanism docs, the public setter or constant for config semantics); other sites now point at the owner or say nothing. Examples: the copy-out read contract is stated once in `mz_ore::pool`, the write-behind tradeoff once on `Pool::set_eager_backing`, the steal-ledger settlement once at its `try_update`, the compression-floor migration rationale once on `survive_merge`. * Multi-paragraph docs that re-argued decisions documented elsewhere were condensed (`enforce_or_defer_compressed_cap`, `insert_with`, `survive_merge`, the `Merger` impl doc, the pool module doc's RSS-identity paragraph). * Deleted comments that restate the adjacent code or an assert message, ASCII section-divider banners in test modules, speculative notes about hypothetical future backends, and unverifiable performance color. * Test doc comments that restated the test's name were removed; docs remain where they carry non-obvious setup or multi-phase protocols. Safety comments, lock-ordering rules, ledger invariants, kernel-behavior gotchas (pagemap vs. mincore, khugepaged re-collapse, macOS `MADV_DONTNEED`), and magic-constant justifications are untouched. No tests were added or modified beyond their doc comments and a few shortened assert messages. Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )