Repository navigation
[pull] main from MaterializeInc:main - #1266
Merged
Merged
Conversation
Mostly parallel-workload followups to #37590 and issues having been closed or moved to Linear from my sweep yesterday. Based on https://buildkite.com/materialize/nightly/builds/17774
…SQL-497) (#37725) Converts the four connection-detail builtin tables to materialized views over mz_catalog_raw, so they are derived from each connection's persisted create_sql instead of being written by a Rust packer on every catalog change on the leader environmentd (the [SQL-118](https://linear.app/materializeinc/issue/SQL-118) / multi-envd motivation). - mz_kafka_connections - mz_ssh_tunnel_connections - mz_aws_connections - mz_aws_privatelink_connections Do this in two parts: 1. kafka + ssh — adds the parse_connection_details SQL helper and flips both tables to MVs. The kafka progress-topic default is reconstructed in SQL to match KafkaConnection::progress_topic. 2. aws + aws-privatelink — extends the helper with an aws branch and adds three plan-time fold functions (mz_aws_account_id, mz_aws_external_id_prefix, mz_aws_connection_role_arn) for the values that come from environment context rather than create_sql. The aws view reconstructs the assume-role principal, external id, and trust policy. The privatelink view reconstructs the principal ARN and keeps WHERE principal IS NOT NULL, so it emits no row when the AWS principal context is absent, matching the packer it replaces. RBAC note. The three new fold functions are deliberately not gated by restrict_to_user_objects, exactly like mz_environment_id. Testing: - parse_connection_details unit tests cover every connection kind and branch plus the null cases. - New lockdown SLTs pin the column shape, nullability, and wrong-type exclusion for each view. The mz_analytics builtin connection doubles as the SQL/Rust parity check for the aws assume-role columns. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bump k8s-openapi feature to Kubernetes 1.34. Bump kind to v0.32.0 and kubectl to v1.34.10 in the CI builder image and dev tooling. Bump all kind cluster configs to `kindest/node:v1.34.8` nodes (the pre-built 1.34 image for kind v0.32.0). Fix a race in our minio setup, which shows up more consistently in Kubernetes 1.34, where the PVC could end up on a different node and minio pods would never become ready. [DB-143](https://linear.app/materializeinc/issue/DB-143) Kubernetes 1.34 is the oldest currently supported version. Newer Kubernetes versions may not support legacy APIs over time, so better to stay current. This will require a similar bump in the cloud repo when bumping the submodule. ### Motivation Keep up to date, on supported platforms. ### Verification Nightly pipelines that exercise the bumped kind/kubectl/node images and should be run on this PR: - `cloudtest`, `cloudtest-slow`, `cloudtest-upgrade`: run kind via the rebuilt ci-builder image. - `orchestratord-*`: run in the `test/orchestratord` kind cluster, now on 1.34.8 nodes. - `k8s-node-recovery-*`: use `misc/kind/cluster-node-recovery-test.yaml`, now on 1.34.8 nodes. These are the steps the minio fix targets. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Dennis Felsing <dennis@felsing.org>
Adds a nightly CI step that measures console performance against replayed production workloads. For each captured workload (sharded across 3 parallel jobs, `--skip-large`, 1% initial data), the `console-scalability` workflow: 1. Boots the workload and waits for hydration. 2. Keeps the captured queries and ingestions running as background load. 3. Runs the Playwright scalability suite (page loads, per-query timing audits, multi-tab stress) against the console bundled in the materialized image. 4. Replays the freshly dumped console query set with k6, ramping 0→100 virtual users against the SQL HTTP endpoint. 5. Uploads per-workload timing reports, query dumps, and k6 summaries as build artifacts. The runs are metrics-only while thresholds are determined from nightly trends: test failures and k6 threshold breaches are logged but do not fail the step. A red step means infrastructure breakage. Supporting changes: * The workload-replay executor's `test()` gains an optional `during_continuous` callback, so the continuous load phase can last exactly as long as the tests. * mzcompose allows the `ipc` service config key (Chromium needs more than the 64MB `/dev/shm` docker default). * The k6 script's labels are refreshed for the current console query set (`clusters.list` and `replicaUtilizationHistory` are SUBSCRIBE-fed now and no longer appear as HTTP queries). Validated end-to-end in [nightly build 17658](https://buildkite.com/materialize/nightly/builds/17658) (single-workload version): 13/14 Playwright tests passed under load, k6 completed the full ramp with 0% request failures, and all artifacts uploaded. The two k6 p95 outliers it found (clusterFreshness 3.8s, materializationLag 4.9s at 100 VUs) match the known production bottleneck ranking. ### Motivation Gives us recurring visibility into console performance under load. Part of [CNS-122](https://linear.app/materializeinc/issue/CNS-122/add-scalable-testing-in-the-nightly-builds). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- Adds build version to segment - Extracts json building function such that we can test ### Motivation Followup to retro for incident-1165 --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…38027) ## Summary - Fix `plan_alter_network_policy` to use `normalize::ident_ref()` instead of `name.to_string()` - Fix `fold_network_policy_name` to extract raw identifier string from `RawNetworkPolicyName` - Fix `resolve_network_policy` helper (used by ALTER OWNER) with the same pattern ## Test plan - Added regression tests for: - `ALTER NETWORK POLICY "hyphenated-name"` - `GRANT/REVOKE USAGE ON NETWORK POLICY "hyphenated-name"` - `ALTER NETWORK POLICY "hyphenated-name" OWNER TO role` Fixes https://linear.app/materializeinc/issue/CLO-143/alter-network-policy-cannot-resolve-quoted-hyphenated-name 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )