Repository navigation
[pull] main from MaterializeInc:main - #1332
Merged
Merged
Conversation
Updates the support docs to clarify wording, remove outdated information about retiring Slack/Teams access, and add a troubleshooting tip for users who can’t see Matty. --------- Co-authored-by: kay-kim <kay.kim@materialize.com>
### Motivation The upsert v2 (continual feedback) stash moves from its previous representation onto `ChunkBatcher`/`ChunkSpine`: stashed updates live in columnar chunks that participate in the buffer pool, and the feedback arrangement drains through `UnloadChunk` one chunk at a time instead of materializing whole batches. With `enable_upsert_paged_spill` on, stash and arrangement state past the residency budget spills as compressed, budget-accounted extents rather than raw kernel-swapped heap. Benchmarked extensively on AWS (self-managed EKS, MSK source, swap-enabled nodes), August 2026 campaign: * Hydration under pressure, 25cc replica: at 4.5x state-to-memory, v2-spill hydrates in 707 s vs 1187 s for v1 (RocksDB mem-env). At 9x: 1607 s vs 4607 s (v1 goes superlinear, v2 stays near-linear). At 13.5x, v1 and v2-resident crash-loop at the pod swap ceiling and never complete; v2-spill finishes in 2386 s with matching counts. * Swap traffic: v2-spill moves roughly half the swap bytes of v1 at every depth measured. * Steady-state ingest freshness under the same pressure: statistically identical across v1, v2-resident, and v2-spill. * Unpressured regimes: v2-resident tracks v1 within ~10% everywhere; spill costs appear only at a 50cc saturation edge and vanish one size up (or under the two chunk patches below this PR in the stack). The new representation is flagged: `enable_upsert_chunked_stash` (off in production, on and randomized in CI) selects between the `ChunkBatcher` representation above and the previous paged-columnar-merge-batcher + `ValRowSpine` representation, which stays the production default while the chunked flavor earns trust. The operator loop is shared; a small `UpsertStashArm` trait carries the flavor-specific pieces (stash batcher, feedback spine, flush, drain), and the arms are resolved from the config set once at operator construction, so a dataflow keeps its flavor for life. Spilling in either flavor remains gated by `enable_upsert_paged_spill`. ### Tips for reviewer * Stacked on #38253 and #38254 (the two chunk patches the campaign produced); the interesting upsert logic is `upsert_continual_feedback_v2.rs`. * `enable_upsert_paged_spill` is replica-scoped and composes with compute's spill gate as an OR on the process-wide pool. * The unit-test harness runs every scenario under both stash flavors and asserts they produce identical output, so the paged arm is exercised by the same tests as the chunked one. ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). - [x] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.
#38523) The Terraform guide documents the username environment variable as `MZ_USER`, but the provider only reads `MZ_USERNAME`, so following the table has no effect. Also adds a warning that setting `MZ_HOST` does not only supply a value, it makes the provider talk to self-managed Materialize, which surprised a customer whose leftover `MZ_HOST` silently redirected their Materialize Cloud project and failed with an invalid password error.
<!-- ccr-slack-attribution --> _Requested by **Dov Alperin** · [Slack thread](https://materializeinc.slack.com/archives/C04DJT084KA/p1785168342982679?thread_ts=1785168342.982679&cid=C04DJT084KA)_ ### Motivation `adbc_driver_postgresql` cannot connect to Materialize. Before running any user query it resolves the type catalog, and that query filters on `pg_type.typsend`, which we do not have: ``` ProgrammingError: WHERE clause error: column "typsend" does not exist (SQLSTATE XX000) ``` Reported by a user who wants to pull result sets straight into DuckDB without going through S3. ### Description **Before:** `adbc.dbapi.connect()` fails. **After:** it connects, and every column arrives as its real Arrow type, so an Arrow table can be handed to DuckDB directly. Two things were broken. 1. **`pg_type.typsend` was missing entirely.** Added, with each type's real PostgreSQL OID. 2. **`typreceive` rendered as a number.** The driver keys its Postgres to Arrow type map on that column's *text* value (`int4recv`, `boolrecv`). We sent `2406`, so every type silently fell back to opaque binary. `regproc` now renders the resolved function name in text output, matching PostgreSQL. Binary output is unchanged, since PostgreSQL sends a bare OID there too. The OID to name table lives in `mz-pgrepr-consts`, which has no dependencies, so no encoder signatures change and pgwire, `COPY`, and the Kafka text sink all pick it up for free. `test_regproc_names_match_builtin_functions` recomputes the table from `BUILTINS::funcs()` and fails printing the corrected version, so it cannot drift. This also fixes a latent bug: `typreceive::TEXT != 'array_recv'` was a tautology, so that filter returned 68 rows where a correct catalog returns 34. ### Verification New `test/adbc` suite, CI step `arrow-adbc-driver`: connect, scalar type resolution, arrays, nulls, a 5000 row fetch, and an Arrow to DuckDB handoff. Green. `test_compare_builtins_postgres` independently checks every new `typsend` OID against a live PostgreSQL. CI was fully green on the pre-merge head `778fbfa0e3`, 133/133, including `slt-4` once the auto-generated region of `test/sqllogictest/catalog_server_explain.slt` was regenerated. The branch has since been merged with `main` twice, so the current head is awaiting a fresh run. ### Notes for review - `mz_type_pg_metadata` gains a column, so it needs a `MigrationStep`. This uses `replacement`, not `evolution`: only `replacement` populates `replaced_items`, which drives the read-only backfill that lets `pg_type_all_databases_ind` hydrate. `evolution` failed `0dt-upgrade-smoke-test`. - That step is pinned to `26.37.0-dev.0`, repinned from `26.36.0-dev.0` when `main` bumped the workspace dev version. A step pinned to a superseded dev version is skipped on upgrade, and the fingerprint check then panics at catalog open. - `typsend` itself still renders in decimal, because we register no `*send` functions. It does not affect ADBC, which only compares it against `0`. - The HTTP API and JSON sinks still render these columns as numbers, because that module publishes them as 4 byte integers and changing one without the other would break existing sinks. - Overloaded names render schema-qualified (`pg_catalog.abs`), matching both PostgreSQL's `regprocout` and our own `regproc::text` cast. Those renderings do not parse back, which is also true of PostgreSQL. - The regproc `NAMES` table now has 627 entries, gaining `mz_internal.parse_source_export_details` after `main` added that builtin function. The drift test is what forced the update, which is the point of having it. --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )