Repository navigation
[pull] main from MaterializeInc:main - #1215
Merged
Merged
Conversation
### Motivation `COLUMN_PAGED_BATCHER_SWAP_PAGEOUT` lacked the `.scoped(ParameterScope::Replica)` annotation its three siblings carry (`enable_column_paged_batcher`, `enable_column_paged_batcher_spill`, `column_paged_batcher_lz4`), so it defaulted to `ParameterScope::Environment`. The sync loop never built a replica context for it, so per-replica or per-size-family LaunchDarkly rules could not bind and every replica resolved the environment-wide value. This blocked isolating the eager `MADV_PAGEOUT` layer on a single replica: in the pager hydration variant ladder a `paged_pageout` replica meant to add only the pageout layer silently equalled the `paged_lz4` rung. Fixes CPU-177. ### Description Add `.scoped(ParameterScope::Replica)` to match the sibling flags. ### Verification Compiles clean (`cargo check`, `cargo clippy`). The LaunchDarkly segment and flag rule for the pageout rung are already in place and will bind once a binary carrying the scope is deployed to staging. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Remove debug prints accidentally introduced in 7986bb0. * Keep CI_MZ_SYSTEM_PARAMETER_DEFAULT across MzRestart, Mz0dtDeploy and BackupAndRestore instead of silently reverting the flags under test on the first restart. The parsing now lives in one place. * Make --seed reproducible: pick free object names in sorted order (set iteration depends on PYTHONHASHSEED) and replace the unseeded np.random.zipf pad sizes with random.paretovariate. * Restore the 50% chance for upstream tables without a primary key. has_pk was always True, which made all ShiftForward/ShiftBackward actions permanent no-ops, so UPDATEs never flowed through any CDC source. Debezium sources now only pick tables with a primary key since ENVELOPE DEBEZIUM requires a key. * Let KillClusterd alternatively require TableExists so it can fire in CopyToFromS3, which creates no views. * Trim unexecuted actions' capabilities using the instances recorded at generation time. provides() may return fresh instances and removal compares by identity, so *IsRunning capabilities were never removed. * Join KafkaInsertParallel background threads at the end of the test and fail on their errors instead of dropping them silently. The threads also target the current mz_service now. * Pass mz_service in CreateDebeziumSource, it otherwise routes through balancerd's potentially stale resolver. * Re-point balancerd at the new leader after Mz0dtDeploy, its static resolver otherwise keeps routing to the stopped old generation. * parse_timedelta: support seconds, reject unparseable input. * Only bootstrap Polaris in scenarios that can create Iceberg sinks. * Remove mid-test BackupAndRestore from UserTables, sink state is not recorded across a backup&restore cycle (database-issues#9589). * Remove unused actions (the *Stop actions for external services and MzStartParameterized) and the write-only State.iceberg_* fields. * Add DropTable/DropView actions, guarded so that no depended-upon object is dropped, and wire them into UserTables and KafkaSources. User objects were never dropped before. Test run: https://buildkite.com/materialize/nightly/builds/17089 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…billing page (#37286) > [!IMPORTANT] > **Mature enough to merge into `main`, and intentionally open to change in Phase 5.** > > This is Phase 1 of the Orb account-hierarchy console work. It is purely **additive**: it introduces the new "Spend by account & cluster" section and leaves the existing spend total, daily chart, and "Spend between…" breakdown **untouched**, so it cannot regress today's page and is trivially revertable (the console redeploys in ~5 min and this section is not load-bearing). > > Please review for **correctness and safety-to-merge, not feature-completeness.** A few details are deliberately deferred to **Phase 5** and tracked as follow-ups below — friendly account names (SAS-141/142) and the full per-account Compute/Storage drilldown (blocked on the breakdown endpoint carrying usage quantities and replica sizes). Landing this now unblocks the staged rollout (Phases 3–4). ### Motivation Phase 1 of Orb account-hierarchy support in the console (Linear: SAS-128). Today the billing page shows only an org-wide spend total. With sub-accounts, a parent org needs to see how spend breaks down across its own and its child accounts, per cluster, bucketed by day — driven by `GET /api/costs/breakdown/daily`. ### Description - Adds a **"Spend by account & cluster"** section to Usage → Overview, sourced entirely from `GET /api/costs/breakdown/daily` (`useDailyCostsBreakdown` in `queries.ts`). The old aggregate `GET /api/costs/breakdown` endpoint and its frontend usage (`getCostsBreakdown`/`useCostsBreakdown`/the old `AccountClusterBreakdown` component) have been removed — this is bucketed-by-UTC-day from the start, not a follow-on aggregate. - **`AccountSpendBreakdown`**: a stacked-by-account daily bar chart (visx `BarStack`, one accent color per account, per-day tooltip) above a total-spend figure. A parent org sees every child account stacked in the same chart; a child or standalone org sees a single-account stack. - **`AccountSpendLedger`** (own full-width grid row, SAS-154): every account as an expandable row — share of total, a trend sparkline, and total cost — expanding to region-qualified per-cluster costs (`aws/us-east-1 / quickstart.r1`, `aws/us-east-1 / Storage`). Biggest spender first. A final Total row sums every account. All accounts are visible at once; there's no chip-switcher or select-to-drill-down. - **`PlanDetails`**: a separate box alongside the chart shows the account's plan info, credit balance, and a "Last 30 days" per-account breakdown, independent of the chart's selected time range. - Each account currently renders its Orb **external-customer UUID**; friendly names are a deliberate follow-up (backend SAS-141, frontend SAS-142). - **Shared `LedgerTable`/`LedgerGroup`/`LedgerCell`/`LedgerCaret`** component (`LedgerTable.tsx`) replaces duplicated cell/caret/Safari-subgrid-collapse styling previously spread across `SpendBreakdown.tsx` and this PR's `AccountSpendBreakdown.tsx` — a reviewer suggestion, landed as a net -185 lines across the two table files despite the new shared component. Also picks up keyboard (`Enter`/`Space`) + `aria-expanded` support on both tables' expand/collapse rows, which neither had before. - **Type generation:** the breakdown path/schemas are spliced into the committed `global-api` types rather than a full `yarn gen:api` re-sync, because the console's committed spec has drifted from cloud (regions / self-managed-subscription, tracked separately as SAS-161). On a clean rebase once the cloud spec lands, `yarn gen:api` reproduces these. ### Verification - **Component tests:** billing test suite, 43 tests across 7 files (`AccountSpendBreakdown`/`UsagePage`/`PlanDetails`/`dailyBreakdown`/`SpendBreakdown`/`DailyUsageChart`/`UsageRoutes`/`BillingPage`) — covers parent/child/standalone rendering, region-qualified cluster rows, storage/egress rows, error and empty states, and the account share-of-total column. Typecheck + lint clean. - **Manual, end-to-end:** ran this console locally against a personal stack (`rjimeno.dev`) wired to the cloud SAS-136 backend, logged in to a root/leaf Orb hierarchy (`+mock-root@` / `+mock-leaf@`). Confirmed parent + child account groups and region-qualified cluster/storage rows from live Orb data. Linear: SAS-128 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ks (#37694) Update the user documentation with AWS Glue schema registry for Kafka Sink. - Document schema-name and compatibility-level options and default values - List Confluent to Glue compatibility level mapping - Document set-if-unset compatibility behavior shared with Confluent Schema Registry - Update IAM permissions in `CREATE CONNECTION` to include sink This will be available in the v26.34 release --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…37550) Updates Materialize documentation with instructions on configuring Azure SQL as a source. The functionality has been merged, and is expected to GA 7/22. There are some minor differences in role setup, and Azure requires connection encryption. Because of those differences, some of the sections are inlined (SSH tunnel still requires that SSL Mode is set for the Azure connection, and connection policy cannot redirect).
It's been quite noisy. Alternative would be for someone else to look at it every day. @antiguru Would you want to do that?
…upport async operations) (#37759) We want to write rows to S3 as soon as we read them from the batch instead of queuing them in memory for the next operator (`write_data_files`). That means the body of our `for_each_diff_pair` must support async operations. This PR refactors `for_each_diff_pair` into an iterator so we can do that (in the stacked PR).
### Motivation Closes [CPU-173](https://linear.app/materializeinc/issue/CPU-173) Arrangement dictionary compression was a purely global feature flag captured at replica creation. This makes it configurable per cluster and per replica, modeled after the introspection options. ### Description Adds the `EXPERIMENTAL ARRANGEMENT COMPRESSION` option to `CREATE CLUSTER`, `CREATE CLUSTER REPLICA`, and `ALTER CLUSTER SET/RESET`, stored durably in the catalog (version bump to v90). The value is threaded through the cluster-controller shape model, so changing it reprovisions controller-owned replicas like a size or introspection change does. `enable_arrangement_dictionary_compression_alpha` now only gates the feature: a replica honors its configured value only while the flag is enabled, and the gate is evaluated at replica-creation time. The configuration is always accepted and stored regardless of the flag, so a cluster can be pre-configured and start honoring it once the flag is enabled on new or restarted replicas. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Dennis Felsing <dennis@felsing.org>
## Motivation The group commit path did its timestamp-oracle round trips inline on the coordinator loop: `peek_write_ts` and `write_ts` when allocating the write timestamp, and `read_ts` when downgrading read holds afterwards. The keepalive commit fires every `default_timestamp_interval` (1s) regardless of load, so under a slow oracle backend the loop spent most of its time blocked on these round trips and stalled every other session, including pure reads that never touch the oracle. Under an injected 500ms oracle latency with 8 concurrent INSERT workers, a victim `SELECT 1` goes from multiple seconds to p50 6ms / p95 9ms with this change. Fixes SQL-447 ## The design **A single group committer task owns all txns-shard writes.** The loop only does the cheap, state-touching work: `stage_group_commit` drains pending writes, validates and acquires write locks, builds the append batch, and hands a command to the committer. The committer owns the write protocol, per attempt: allocate the write timestamp, advance the catalog shard upper (keeping the catalog readable at the read timestamp the oracle moves to, and re-checking leadership before user data is written, see #28216), issue the txns-shard write, and apply the write to the oracle only after success. Finalization that needs coordinator state (statement-logging timestamps, retiring client responses, downgrading read holds) goes back to the loop via `Message::GroupCommitApplied`. **DDL's table registration and forgetting flow through the same queue** as commands, which makes the committer the single in-process writer to the txns shard. That single-sender ordering is load-bearing: a staged INSERT for a table always reaches the table-write worker before a later DROP TABLE's forget (DROP takes no write locks, so nothing else orders the two), and in-process timestamp conflicts on the txns shard are impossible by construction. An `InvalidUppers` conflict from the worker therefore means exactly one thing: another process wrote the txns shard. The committer retries at a fresh oracle timestamp, which is the posture we want for future concurrent `environmentd` instances. The worker returns the conflict instead of panicking, rolls back its bookkeeping so retries are clean, and opens its own table write handles from registration metadata (registration consumes handles, so retries need fresh ones). DROP TABLE's txns forget moves off the loop in the process. The bootstrap-only `register_table_collections` remains the one direct registration path, safe because nothing else runs during bootstrap. **The catalog shard gets a matching write-conflict protocol, enforced by the compare-and-append itself** rather than by checks before it (per the guidance in `doc/developer/guide-adapter.md`). The strict/tolerant `advance_upper` duality collapses into one to-at-least primitive, and both it and `commit_transaction` classify an upper mismatch by counting raw updates applied by the conflict sync: zero means pure empty progress (e.g. the committer bumping the upper with a timestamp that an on-loop catalog transaction overtook), which is rebased over and retried, while content from another writer surfaces as the graceful `CatalogOutOfSync` error instead of the previous panic paths. ID allocation and DDL transactions tolerate overtaken commit timestamps the same way. Raw updates rather than memory updates matter because some kinds (ID allocators) never surface as memory updates but still invalidate a rebase. **Throttle behavior.** While waiting in the wall-clock throttle the committer merges later queued group commits into the current batch, which bounds queue growth under a slow oracle and lets an arriving internal system write flip the throttle bypass. A queued register or forget also lifts the throttle, since a DDL statement is blocked on the loop behind it and DDL was never throttled when it ran on the loop. Merging stops at such commands to preserve queue order. ## Testing - New tests in `src/catalog/tests/read-write.rs` cover the to-at-least advance semantics, the commit rebase over an overtaken timestamp, and the compare-and-append conflict classification against a concurrent empty-progress writer (injected through a raw persist handle on the catalog shard). - A deterministic unit test in `appends.rs` drives the committer's `write_to_txns` through an injected txns-shard conflict against a real debug catalog: retry at a fresh, strictly larger oracle timestamp, durable catalog upper advancement, and exactly one oracle write application, for the successful attempt. (In-process txns conflicts are impossible by construction, so the conflict is injected through a mock table write handle.) - A regression test drops the server with INSERTs in flight, which used to abort the process via un-retired client responses in queues (a destructor panic). Shutdown posture: no graceful completion of in-flight responses, a drop backstop on `ExecuteContext` retires stranded clients with an error. - Manual validation: injected 500ms oracle latency with concurrent inserters and DDL (latency numbers above), and a fast-oracle stress with three inserters and three concurrent CREATE/DROP TABLE loops, with read-your-writes and cross-session reads intact and no panics. That stress exercised the catalog shard's empty-progress rebase (DDL commits racing the committer's upper advancement), not txns-shard conflicts, which are covered by the deterministic test above. - The single-writer invariant and the catalog conflict protocol are recorded in `doc/developer/guide-adapter.md`. ## Tips for reviewer The interesting files are `src/adapter/src/coord/appends.rs` (the committer task and its protocol), `src/catalog/src/durable/persist.rs` (`advance_upper` and `commit_transaction`), and `src/storage-controller/src/persist_handles.rs` (worker-side handle opening and conflict rollback). ## Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.
Adds support for autoscaling policies to mz-deploy under two commands: - Apply can now create clusters with auto scaling policies and alter policies when they change - Stage will copy the autoscaling policy of the cluster it is copying Most of the changes are added test coverage. ### Motivation Why does this change exist? Link to a GitHub issue, design doc, Slack thread, or explain the problem in a sentence or two. A reviewer who has no context should understand *why* after reading this section. If this implements or addresses an existing issue, it's enough to link to that: Closes <issue> Fixes <bug> etc. ### Description What does this PR actually do? Focus on the approach and any non-obvious decisions. The diff shows the code --- use this space to explain what the diff *can't* tell a reviewer. ### Verification How do you know this change is correct? Describe new or existing automated tests, or manual steps you took. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )