Repository navigation
[pull] main from MaterializeInc:main - #1333
Merged
Merged
Conversation
…on (#38471) ### Motivation The Iceberg sink needs features from our `iceberg-rust` fork that are not on the currently pinned revision: catalog-vended storage credentials, the per-prefix FileIO credential routing they rely on, and OAuth2 token expiry handling. This is the dependency move on its own, so the behavioral changes above it review against a fixed base. ### Description Points the three `[patch.crates-io]` entries at `24e8ca7eb572b9383fbe85d5ada9a0a81cf1ffa8` on the fork's `vended-creds` branch, and raises the `iceberg`, `arrow`, and `parquet` requirements in `[workspace.dependencies]` to match what that revision links. The version requirements have to move with the patch: Cargo silently drops a patch whose version does not satisfy the requirement into `[[patch.unused]]` and builds against crates.io instead, with no error. Two API changes come with the revision: - `iceberg-storage-opendal` replaces `AwsCredentialLoad` with reqsign's `ProvideCredential`, and drops `configured_scheme` from `OpenDalStorageFactory::S3`. `AwsSdkCredentialLoader` is ported to the new trait, and now converts the AWS SDK's credential expiry into reqsign's `Timestamp` rather than discarding it. reqsign reads a `None` expiry as "valid forever", which would leave OpenDAL signing with stale assume-role credentials instead of asking for fresh ones. - `parquet` renames `set_max_row_group_size` to `set_max_row_group_row_count`, which takes an `Option`. `deny.toml` gains duplicate-version skips for the crates where the new dependency tree diverges from the rest of the workspace, chiefly `duckdb` holding arrow 57 while the workspace moves to 58. No new licenses appear, so `about.toml` is unchanged. ### Verification `cargo check` passes across the workspace. `cargo deny check bans` and `cargo deny check licenses` are both clean with no unmatched skips.
…38472) ### Motivation Unity Catalog serves its OAuth2 token endpoint at a path unrelated to the catalog URI, so the Iceberg REST client's default of `<uri>/v1/oauth/tokens` cannot reach it. Without a way to name the token endpoint separately, a Unity Catalog connection cannot authenticate at all. ### Description Adds an `OAUTH2 SERVER URL` option to Iceberg catalog connections, forwarded to the catalog client as the `oauth2-server-uri` property. The property is spelled `uri` because that is the name the Iceberg REST clients agree on, even though the SQL option reads `URL`. Leaving the option unset keeps the client's existing default, so existing connections are unaffected. ### Verification Parser round-trip coverage in `src/sql-parser/tests/testdata/ddl`, and a new `test/iceberg/oauth2-server-url.td` exercising the option through the mzcompose Iceberg suite. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
### Motivation Iceberg tables can set `write.data.path` to place data files somewhere other than the default `<table location>/data`, which catalogs such as Unity Catalog rely on. We ignored the property and always wrote to the default location, so files landed outside where the table metadata said they should. ### Description Reads `write.data.path` from the table metadata's properties and uses it as the base location for data files, falling back to the default layout when the property is absent. ### Verification Covered by the existing Iceberg sink mzcompose suite, which exercises tables whose data path is set by the catalog. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…38474) ### Motivation Catalogs such as Unity Catalog and Polaris can mint temporary, table-scoped storage credentials instead of handing out their own long-lived ones, which is how a sink writes to storage it holds no static keys for. Requesting them requires the `X-Iceberg-Access-Delegation` header on catalog requests. ### Description Adds an `ACCESS DELEGATION = 'vended-credentials'` option to REST Iceberg catalog connections. When set, Materialize sends the header as a `header.*` catalog property, which the client applies to every REST request, so it rides along on `loadTable` and `createTable` alike. The credentials the catalog returns are wired into the per-table FileIO. The option is opt-in rather than always-on deliberately. Requesting delegation is not free of consequence: a catalog that gates it behind privileges the principal lacks rejects the whole request rather than falling back to its configured storage credentials, so a connection that would otherwise work would start failing. ### Verification New `test/iceberg/vended-credentials.td` and an mzcompose workflow that runs it against Polaris with MinIO storage, plus parser coverage for the new option.
### Motivation Catalog-vended credentials expire, typically within the hour, and nothing refreshed them. `loadTable` hands their access keys to the FileIO once and OpenDAL keeps signing with them until they stop working, at which point the sink fails and only recovers by restarting the dataflow. Any sink running longer than one credential lifetime hits this. ### Description Adds `VendedCredentialLoader`, a `ProvideCredential` implementation that re-fetches from the catalog's `loadCredentials` endpoint and hands the result to OpenDAL's S3 credential chain. The loader caches with its own deadline rather than relying on reqsign's cache. OpenDAL rebuilds its `Operator` for every file operation, so the `Signer` that holds reqsign's cached credential never survives a single call; without an internal cache this would be one catalog round trip per parquet write and per metadata read. The deadline comes from `s3.session-token-expires-at-ms` where the catalog reports one, refreshing ahead of expiry, and from a short fixed interval where it does not. That interval is a constant for now, with a TODO to make it a dyncfg once we know what real catalogs report. On a 401 or 403 the loader invalidates the catalog token so the next attempt mints a fresh one, since nothing else on the storage path re-mints it. Two supporting changes: - Materialize now constructs the OAuth2 provider itself instead of passing a `credential` catalog property, so a single token object serves both catalog requests and credential refreshes. The two cannot coexist: the catalog client rejects a custom authenticator combined with that property. - `connect` takes the table the handle will be used against, because the credentials endpoint is table-scoped. Callers that only prove reachability (connection validation, sink purification) pass `None`. Installing a loader hands it sole responsibility for S3 credentials, since OpenDAL replaces its entire provider chain and discards the static keys parsed from the vended properties. It is therefore installed only when the connection asked for delegation and a table is known; every other case keeps the existing static-property path. ### Verification `cargo check` and `cargo clippy` are clean. Not yet exercised against a live catalog: the refresh path needs a sink run long enough to cross a credential lifetime, and which branch of the deadline logic applies depends on whether the catalog reports an expiry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )