From e76f5d2056e6e3e377f197982e5a1dce4ce9dbd6 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:10:15 +0200 Subject: [PATCH 1/2] docs(iceberg): document the forge-cli catalog follow-up fixes Companion to forge-cli fix/iceberg-catalog-followups (b87fd3a7), which closes the gaps the post-merge inspection of #707 found. Marked "(forge-cli fix/iceberg-catalog-followups, unreleased)". - Quoted CLI output regenerated from b87fd3a7: findings keep their square brackets (exposes[orders], [error]) and print on one line. - The streaming runners check a hand-written sink config too, for a build that declares sink.format: iceberg, and read the build they run. - GCP: an Iceberg expose a streaming sink writes must name its catalog. - catalog: bigquery on Kafka Connect warns (published sink 1.9.2). - Catalog-move guard: Snowflake EXTERNAL VOLUMEs, the Glue-table evidence rule, the iceberg_catalog_move_probe_skipped WARNING, and both events on the troubleshooting page. - The Snowflake module still emits a native snowflake_table whatever the catalog; fluid diff refuses an unknown catalog value. --- docs/advanced/production-troubleshooting.md | 2 + docs/advanced/source-aligned-acquisition.md | 136 +++++++++++++------- docs/cli/apply.md | 33 ++++- docs/cli/diff.md | 9 +- docs/cli/generate-iac.md | 12 +- docs/cli/validate.md | 30 +++-- docs/providers/aws.md | 6 +- docs/providers/gcp.md | 21 ++- docs/providers/snowflake.md | 27 +++- 9 files changed, 193 insertions(+), 83 deletions(-) diff --git a/docs/advanced/production-troubleshooting.md b/docs/advanced/production-troubleshooting.md index 1ff3225..83d16e8 100644 --- a/docs/advanced/production-troubleshooting.md +++ b/docs/advanced/production-troubleshooting.md @@ -107,6 +107,8 @@ Remote OpenTofu state is keyed per contract and provider. See [Environment varia | `state_migration_unverified` | After the copy, the new key holds different resources than the old one. The old object is untouched | Compare the two states before re-running | | `state_migration_probe_failed` | The old key could not be read | Check access to the bucket and the key named in the message | | `opentofu_region_moved` | State holds this contract's resources in a region other than the one the bindings now name. Applying would create them again and leave the originals unmanaged | If they should stay, set the binding's `location.region` to the region the error names. If they should move, empty and remove them there first (`tofu destroy` in the state directory named, with `AWS_REGION` set to the old region), then apply again | +| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) and forge-cli fix/iceberg-catalog-followups)* An Iceberg expose's `location.catalog` names a catalog an earlier release did not honour, and state still holds what that release created for it: a Glue database and table on AWS, or an EXTERNAL VOLUME on Snowflake. The plan would destroy them, and destroying a Glue database deletes every table in it | Run the `tofu -chdir= state rm
` commands the error prints, which change nothing in the cloud, then apply again. Do not pass `--allow-data-loss`. If the table belongs in Glue or in Snowflake's own catalog, remove `location.catalog` instead. Steps: [Upgrading an AWS contract that names another catalog](./source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog), [Upgrading a Snowflake contract that names another catalog](./source-aligned-acquisition.md#upgrading-a-snowflake-contract-that-names-another-catalog) | +| `iceberg_catalog_move_probe_skipped` (WARNING) | *(unreleased, forge-cli fix/iceberg-catalog-followups)* The catalog-move guard could not read the state, or its check failed, so the apply went on without it. The `reason` field says why | If the plan then destroys Glue resources or an EXTERNAL VOLUME of the exposes the warning names, stop: release them with `tofu state rm` and apply again, rather than passing `--allow-data-loss` to the data-loss gate | ## Contract load and overlay errors diff --git a/docs/advanced/source-aligned-acquisition.md b/docs/advanced/source-aligned-acquisition.md index 73c1459..82c341d 100644 --- a/docs/advanced/source-aligned-acquisition.md +++ b/docs/advanced/source-aligned-acquisition.md @@ -218,10 +218,10 @@ For the DuckDB engine, each connection runs in DuckDB's own sandbox ([DuckDB san ## Iceberg catalogs (`location.catalog`) ::: warning Not in a release yet -This section describes forge-cli [PR #707](https://github.com/Agenticstiger/forge-cli/pull/707). No release includes it yet. forge-cli 0.19.0 and earlier behave as described in [On 0.19.0 and earlier](#on-0-19-0-and-earlier), at the end of this section. +This section describes forge-cli [PR #707](https://github.com/Agenticstiger/forge-cli/pull/707) and the follow-up fixes in forge-cli fix/iceberg-catalog-followups, marked *(forge-cli fix/iceberg-catalog-followups, unreleased)*. No release includes either yet. forge-cli 0.19.0 and earlier behave as described in [On 0.19.0 and earlier](#on-0-19-0-and-earlier), at the end of this section. ::: -An Iceberg expose names the catalog that owns its table in `binding.location.catalog`. The streaming sinks (Kafka Connect and Debezium Server), dbt's `catalogs.yml`, the AWS, Snowflake and Confluent modules, `fluid policy compile`, `fluid diff`, `fluid test` and `fluid validate` read the value through one table in forge-cli (`fluid_build/providers/_iceberg_catalog.py`), so they agree on which catalog holds the table. +An Iceberg expose names the catalog that owns its table in `binding.location.catalog`. The streaming sinks (Kafka Connect and Debezium Server), dbt's `catalogs.yml`, the AWS and Confluent modules, `fluid policy compile`, `fluid diff`, `fluid test` and `fluid validate` read the value through one table in forge-cli (`fluid_build/providers/_iceberg_catalog.py`), so they agree on which catalog holds the table. The Snowflake module reads it for the Iceberg prerequisites only: the EXTERNAL VOLUME and the Glue catalog integration follow the table. It still emits a `snowflake_database`, `snowflake_schema` and `snowflake_table` for the expose's `location.database`, `location.schema` and `location.table`, whatever the catalog. ### Example: stream into Lakekeeper @@ -284,8 +284,7 @@ Validation completed in 0.001s Validation Errors: ================== - 1. iceberg sink (build 'stream_orders'): lakekeeper catalog requires -binding.location.uri + 1. iceberg sink (build 'stream_orders'): lakekeeper catalog requires binding.location.uri ``` These are the table and catalog keys of the sink config the Kafka Connect runner derives from it: @@ -317,9 +316,15 @@ The one resource is `aws_s3_bucket.bronze_orders_stream_acme_lake`. Without `buc ### How a value is read - **Spelling.** Case, surrounding whitespace, and `-` against `_` are folded, so `Lakekeeper` is `lakekeeper` and `SNOWFLAKE_MANAGED` is `snowflake-managed`. Two spellings are aliases: `iceberg-rest` (or `iceberg_rest`) is `rest`, and `snowflake` is `snowflake-managed`. -- **No value.** An Iceberg expose with no `location.catalog` gets its platform's default: `glue` on `platform: aws`, `snowflake-managed` on `platform: snowflake`, and `rest` on any other platform. On `platform: gcp` the default means two things: a streaming sink writes through a REST catalog, while dbt-bigquery and the GCP module create a BigLake table. Name the catalog on GCP so the sink and dbt agree. +- **No value.** An Iceberg expose with no `location.catalog` gets its platform's default: `glue` on `platform: aws`, `snowflake-managed` on `platform: snowflake`, and `rest` on any other platform. On `platform: gcp` the default means two things: a streaming sink writes through a REST catalog, while dbt-bigquery and the GCP module create a BigLake table. *(forge-cli fix/iceberg-catalog-followups, unreleased)* So a `platform: gcp` Iceberg expose that a streaming sink writes must name its catalog. Without one, `fluid validate` fails, and so does the Kafka Connect or embedded Debezium Server run before it creates anything. Set `catalog: bigquery`, or the REST kind your catalog is. #707 alone accepted it. An expose that no streaming sink writes may still leave the catalog out. The error, for a Kafka Connect build `stream_events` writing a GCP expose with no catalog: + + ```text + 1. iceberg sink (build 'stream_events'): the GCP Iceberg expose sets no binding.location.catalog, so it is read two ways: the sink would write through a REST catalog (the 'gcp' platform default) while dbt-bigquery and the GCP IaC, which read only binding.location.catalog, create a BigLake metastore table. Set binding.location.catalog: bigquery, or the REST kind your catalog is (e.g. rest, lakekeeper) + ``` + + Without `uri` and `warehouse`, the same build also draws the `rest` errors for them. When a `sink.catalog` or a hand-written sink config sets the catalog, the parentheses name it instead of the platform default. - **`sink.catalog`.** A streaming build may set `properties.sink.catalog`, but it must name the expose's catalog, because dbt and the modules read only the expose. Spellings are compared after folding. Leave it unset and the sink uses the expose's catalog. -- **Any other value** is refused by `fluid validate` on every Iceberg expose, with the accepted spellings. A `platform: confluent` expose has its own rule, [below](#other-commands). `fluid apply` on AWS refuses the value too, as `unknown-iceberg-catalog`, because whether a Glue table exists depends on it. +- **Any other value** is refused by `fluid validate` on every Iceberg expose, with the accepted spellings. A `platform: confluent` expose has its own rule, [below](#other-commands). `fluid apply` on AWS refuses the value too, as `unknown-iceberg-catalog`, because whether a Glue table exists depends on it. *(forge-cli fix/iceberg-catalog-followups, unreleased)* So does the native AWS planner that [`fluid diff`](../cli/diff.md) runs; on #707 alone it planned the S3 buckets and no Glue table. A typo on an expose that also declares `governance.lakeFormation` draws only the two errors below, not a Lake Formation refusal that names the typo as a catalog. ```text ❌ Invalid FLUID contract (2 error(s)) (schema v0.7.5) @@ -327,25 +332,14 @@ Validation completed in 0.001s Validation Errors: ================== - 1. iceberg sink (build 'stream_orders'): unknown Iceberg catalog 'lakekeper' in -binding.location.catalog; use one of: bigquery, dynamodb, glue, hadoop, hive, -jdbc, lakekeeper, nessie, polaris, rest, snowflake-managed, unity, iceberg-rest, -snowflake - 2. expose 'orders': binding.location.catalog 'lakekeper' is not a catalog kind -FLUID knows. Use one of: bigquery, dynamodb, glue, hadoop, hive, jdbc, -lakekeeper, nessie, polaris, rest, snowflake-managed, unity, iceberg-rest, -snowflake. Each emitter guesses differently for an unknown value (the streaming -sink writes over Iceberg REST, dbt and the Snowflake IaC treat it as -Snowflake-managed), so the table would be written to two different catalogs. + 1. iceberg sink (build 'stream_orders'): unknown Iceberg catalog 'lakekeper' in binding.location.catalog; use one of: bigquery, dynamodb, glue, hadoop, hive, jdbc, lakekeeper, nessie, polaris, rest, snowflake-managed, unity, iceberg-rest, snowflake + 2. expose 'orders': binding.location.catalog 'lakekeper' is not a catalog kind FLUID knows. Use one of: bigquery, dynamodb, glue, hadoop, hive, jdbc, lakekeeper, nessie, polaris, rest, snowflake-managed, unity, iceberg-rest, snowflake. Each emitter guesses differently for an unknown value (the streaming sink writes over Iceberg REST, dbt and the Snowflake IaC treat it as Snowflake-managed), so the table would be written to two different catalogs. ``` A `sink.catalog` that names another catalog than the expose: ```text - 1. iceberg sink (build 'stream_orders'): sink.catalog 'glue' disagrees with the -expose's catalog 'lakekeeper' (binding.location.catalog); the sink would write -through glue while dbt and the IaC read lakekeeper. Set -binding.location.catalog: glue and drop sink.catalog + 1. iceberg sink (build 'stream_orders'): sink.catalog 'glue' disagrees with the expose's catalog 'lakekeeper' (binding.location.catalog); the sink would write through glue while dbt and the IaC read lakekeeper. Set binding.location.catalog: glue and drop sink.catalog ``` ### What each kind produces @@ -358,7 +352,7 @@ binding.location.catalog: glue and drop sink.catalog | `polaris` | `type=rest` | `iceberg_rest` | nothing; validate warns | the bucket only | `uri`, `warehouse` | | `unity` | `type=rest` | `iceberg_rest` | nothing; validate warns | the bucket only | `uri`, `warehouse` | | `nessie` | `type=nessie` | `iceberg_rest` | nothing; validate warns | the bucket only | `uri`, `warehouse` | -| `bigquery` | `type=bigquery` | `iceberg_rest` | nothing; validate warns | the bucket only | nothing | +| `bigquery` | `type=bigquery` | `iceberg_rest` | nothing; validate warns | the bucket only | nothing (a Kafka Connect build warns) | | `hive` | `type=hive` | left out, with a warning | nothing; validate errors | the bucket only | nothing | | `jdbc` | `type=jdbc` | left out, with a warning | nothing; validate errors | the bucket only | `uri` | | `hadoop` | `type=hadoop` | left out, with a warning | nothing; validate errors | the bucket only | `warehouse` | @@ -367,10 +361,20 @@ binding.location.catalog: glue and drop sink.catalog How to read the columns: -- **Sink catalog selector.** Kafka Connect takes the key with the `iceberg.catalog.` prefix (`iceberg.catalog.type`, `iceberg.catalog.catalog-impl`), and Debezium Server with `debezium.sink.iceberg.`. The `type` values are catalog types Apache Iceberg defines. Iceberg has no `lakekeeper`, `polaris` or `unity` type, so those catalogs are reached over Iceberg REST, and no `dynamodb` type, so DynamoDB is selected by class. `type=bigquery` needs an Iceberg runtime of 1.10 or later. For `nessie` the sink uses Iceberg's Nessie client, which the stock Apache Iceberg Kafka Connect runtime does not bundle, so `fluid validate` warns on a `kafka-connect` build. -- **dbt `catalogs.yml` on Snowflake** and **Snowflake module** apply to `platform: snowflake`. Apart from `glue`, Snowflake reaches the `iceberg_rest` kinds through a catalog integration that authenticates with a secret. The module is credential-free, so it creates none, and `fluid validate` warns (an error under `--strict`). Snowflake has no catalog integration for `hive`, `jdbc`, `hadoop` or `dynamodb`, so an expose naming one fails `fluid validate`. The prerequisites each emitted object needs are in [Iceberg tables via dbt](../providers/snowflake.md#iceberg-tables-via-dbt-since-0-13-1). +- **Sink catalog selector.** Kafka Connect takes the key with the `iceberg.catalog.` prefix (`iceberg.catalog.type`, `iceberg.catalog.catalog-impl`), and Debezium Server with `debezium.sink.iceberg.`. The `type` values are catalog types Apache Iceberg defines. Iceberg has no `lakekeeper`, `polaris` or `unity` type, so those catalogs are reached over Iceberg REST, and no `dynamodb` type, so DynamoDB is selected by class. `type=bigquery` needs an Iceberg runtime of 1.10 or later. For `nessie` the sink uses Iceberg's Nessie client, which the stock Apache Iceberg Kafka Connect runtime does not bundle, so `fluid validate` warns on a `kafka-connect` build. *(forge-cli fix/iceberg-catalog-followups, unreleased)* A `kafka-connect` build that sends `type=bigquery` draws a warning too, because the published Apache Iceberg Kafka Connect sink predates it: + + ```text + 1. iceberg sink (build 'stream_events'): the sink config sets iceberg.catalog.type=bigquery, which the published Apache Iceberg Kafka Connect sink (1.9.2 on Confluent Hub) cannot load: Iceberg's CatalogUtil gains the bigquery type in 1.10. Run a sink built from Iceberg >= 1.10, or the connector fails at start + ``` + + Both warnings follow the catalog type that reaches the worker, after `iceberg_catalog_overrides` and a hand-written `sink_connector_config` are merged in. A hand-written config that sets `iceberg.catalog.type: rest` for a `bigquery` or `nessie` expose draws neither. +- **dbt `catalogs.yml` on Snowflake** and **Snowflake module** apply to `platform: snowflake`. The **Snowflake module** column lists the Iceberg prerequisite the module creates; for every kind it also emits the database, schema and `snowflake_table` the binding's `location` names. Apart from `glue`, Snowflake reaches the `iceberg_rest` kinds through a catalog integration that authenticates with a secret. The module is credential-free, so it creates none, and `fluid validate` warns (an error under `--strict`). Snowflake has no catalog integration for `hive`, `jdbc`, `hadoop` or `dynamodb`, so an expose naming one fails `fluid validate`. The prerequisites each emitted object needs are in [Iceberg tables via dbt](../providers/snowflake.md#iceberg-tables-via-dbt-since-0-13-1). - **AWS module** applies to `platform: aws`: the bucket is the one the binding names. See [On AWS](#on-aws-a-table-in-another-catalog). -- **A streaming sink needs** the listed `binding.location` keys when a Kafka Connect build, or a Debezium Server build in `embedded` mode, writes the expose. Debezium in `bring-your-own` or `managed` mode creates only the source connector, so these checks do not apply to it. The Kafka Connect runner and the embedded Debezium Server runner run the same checks before they create anything, so a contract `fluid validate` refuses also fails its run. +- **A streaming sink needs** the listed `binding.location` keys when a Kafka Connect build, or a Debezium Server build in `embedded` mode, writes the expose. Debezium in `bring-your-own` or `managed` mode creates only the source connector, so these checks do not apply to it. The Kafka Connect runner and the embedded Debezium Server runner run the same checks before they create anything, so a contract `fluid validate` refuses also fails its run, for these builds: + - *(forge-cli fix/iceberg-catalog-followups, unreleased)* a build that declares `sink.format: iceberg`, whether the runner derives the sink config or pushes a hand-written one: `sink_connector_config` on Kafka Connect, `server.sink.config` on an embedded Debezium Server build whose `server.sink.type` is `iceberg` (the default). On #707 alone the runners checked only a config they derived, so a hand-written config that `fluid validate` refuses was still deployed; + - an embedded Debezium Server build that derives its sink: `server.sink.type: iceberg` (the default) with no hand-written `server.sink.config`, or with `server.sink.iceberg_sink_enabled: true`. + + *(forge-cli fix/iceberg-catalog-followups, unreleased)* In a contract with several builds, each of these two runners reads its properties from the build it runs, the build the checks read. They used to read the first build's. ### Kafka Connect: `catalog-impl` or `type`, never both @@ -395,12 +399,7 @@ The derived config now carries one selector, as the Debezium Server config alrea `iceberg_catalog_overrides` and a hand-written `sink_connector_config` are merged over the derived config, so an override that sets the other selector key would bring the failure back. `fluid validate` refuses it: ```text - 1. iceberg sink (build 'stream_orders'): the sink config would carry both -iceberg.catalog.type (from iceberg_catalog_overrides) and -iceberg.catalog.catalog-impl (from the derived glue catalog config); Iceberg's -CatalogUtil refuses a catalog with both type and catalog-impl set, so the sink -never starts. Keep one, and change catalogs with binding.location.catalog rather -than an override + 1. iceberg sink (build 'stream_orders'): the sink config would carry both iceberg.catalog.type (from iceberg_catalog_overrides) and iceberg.catalog.catalog-impl (from the derived glue catalog config); Iceberg's CatalogUtil refuses a catalog with both type and catalog-impl set, so the sink never starts. Keep one, and change catalogs with binding.location.catalog rather than an override ``` To move a sink to another catalog, change `binding.location.catalog`, not the selector keys. @@ -414,43 +413,44 @@ An Iceberg expose on `platform: aws` whose `location.catalog` is anything other - **Cannot carry Lake Formation governance.** `governance.lakeFormation`, `policy.authz.columnRestrictions` and `policy.authz.rowFilters` are refused by catalog name at `fluid validate` and at apply (`lake-formation-needs-glue-catalog`, `column-restriction-unenforceable`, `row-filter-unenforceable`), instead of being written against a Glue table that does not exist: ```text - 1. exposes declares governance.lakeFormation, but its Iceberg table lives in - the 'lakekeeper' catalog (location.catalog), not in AWS Glue. Lake Formation - governs only Glue Data Catalog tables, so its grants, LF-tags and filters would - have no table to name and would not be applied. Govern access in the lakekeeper - catalog itself and remove governance.lakeFormation from this binding. Remove - location.catalog (or set it to 'glue') to keep the table in the Glue catalog - under Lake Formation. + 1. exposes[orders] declares governance.lakeFormation, but its Iceberg table lives in the 'lakekeeper' catalog (location.catalog), not in AWS Glue. Lake Formation governs only Glue Data Catalog tables, so its grants, LF-tags and filters would have no table to name and would not be applied. Govern access in the lakekeeper catalog itself and remove governance.lakeFormation from this binding. Remove location.catalog (or set it to 'glue') to keep the table in the Glue catalog under Lake Formation. ``` - **Has `accessPolicy.grants` reported as unenforced**, with a warning that points at the catalog rather than at Lake Formation grants: ```text - 1. accessPolicy.grants are not enforced on aws binding(s) orders (lakekeeper): - their Iceberg tables live in a catalog other than Glue, and the AWS emitter - writes access only as Lake Formation grants on a Glue table. Grant access in - that catalog. + 1. accessPolicy.grants are not enforced on aws binding(s) orders (lakekeeper): their Iceberg tables live in a catalog other than Glue, and the AWS emitter writes access only as Lake Formation grants on a Glue table. Grant access in that catalog. ``` Grant access in the catalog itself; forge-cli writes no grant there. ### Upgrading an AWS contract that names another catalog -0.19.0 and earlier created a Glue database and table for every AWS Iceberg expose that names `location.database` and `location.table`, whatever its `location.catalog`. After the upgrade, the OpenTofu state of such a contract still holds them while the module no longer declares them, so the next `tofu plan` would destroy them, and destroying a Glue database deletes every table in it, including tables created outside FLUID. `fluid apply` stops before it plans (abridged): +0.19.0 and earlier created a Glue database and table for every AWS Iceberg expose that names `location.database` and `location.table`, whatever its `location.catalog`. After the upgrade, the OpenTofu state of such a contract still holds them while the module no longer declares them, so the next `tofu plan` would destroy them, and destroying a Glue database deletes every table in it, including tables created outside FLUID. `fluid apply` stops before it plans (the `remediation` list, which repeats the commands, is abridged): ```text ❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED] kind: iceberg-catalog-move - error: iceberg catalog move blocked — this contract's OpenTofu state holds 2 -Glue catalog resource(s) for Iceberg table(s) that now live in another catalog: -... + error: iceberg catalog move blocked — this contract's OpenTofu state holds 2 Glue catalog resource(s) for Iceberg table(s) that now live in another catalog: + + exposes[orders]: location.catalog lakekeeper + aws_glue_catalog_database.bronze_orders_stream_streaming aws_glue_catalog_table.bronze_orders_stream_streaming_orders -... + +forge-cli no longer creates a Glue table for an Iceberg table in a non-Glue catalog (it was a second claim on the table's name, without its metadata), so applying now would plan to DESTROY these resources, and destroying a Glue database deletes every table in it, including tables created outside FLUID. + Drop each from this contract's state, then re-run apply: -... + + tofu -chdir=.fluid/iac/aws/bronze_orders_stream state rm aws_glue_catalog_database.bronze_orders_stream_streaming + tofu -chdir=.fluid/iac/aws/bronze_orders_stream state rm aws_glue_catalog_table.bronze_orders_stream_streaming_orders + +`tofu state rm` touches ZERO bytes of infrastructure: the resources stay in AWS, and only this contract's claim on them is released. Delete a Glue table by hand once nothing reads it, and a Glue database only once it holds nothing else. If the table belongs in Glue, remove location.catalog (or set it to 'glue') instead. + remediation: [...] ``` +*(forge-cli fix/iceberg-catalog-followups, unreleased)* The `exposes[orders]` line keeps its expose id, and each printed `tofu state rm` command stays on one line, so a copied line runs with its address. On #707 alone the line printed as `exposes: location.catalog lakekeeper`, and a long command could wrap onto a second line. + 1. Run the `tofu state rm` commands it prints, one per Glue resource, in the form `tofu -chdir= state rm
` (each argument shell-quoted when it needs to be). For the example contract: ```bash @@ -462,7 +462,46 @@ Drop each from this contract's state, then re-run apply: 2. Run `fluid apply` again. 3. Delete the Glue table by hand once nothing reads it, and the Glue database only once it holds nothing else. -If the table belongs in Glue after all, remove `location.catalog` (or set it to `glue`) instead. There is no flag that skips the guard. Before it raises, the apply logs a structured `iceberg_catalog_move_blocked` event with the addresses, the exposes and the commands. The guard runs only on AWS, and only for an Iceberg expose that names a `location.database` and a catalog other than Glue. If the state cannot be listed, or the check itself fails, the apply continues without it (the failure is logged at debug level), and the [OpenTofu data-loss gate](../cli/apply.md#opentofu-data-loss-gate) still stops a plan that would destroy the resources. The same pattern guards [packaging ownership transitions](../cli/generate-iac.md#ownership-transitions). +If the table belongs in Glue after all, remove `location.catalog` (or set it to `glue`) instead. There is no flag that skips the guard. Before it raises, the apply logs a structured `iceberg_catalog_move_blocked` event with the addresses, the exposes and the commands. On AWS the guard concerns an Iceberg expose that names a `location.database` and a catalog other than Glue. The same guard covers [Snowflake](#upgrading-a-snowflake-contract-that-names-another-catalog). + +*(forge-cli fix/iceberg-catalog-followups, unreleased)* A moved expose's Glue resources are flagged only when its own Glue table is in state, which shows that an earlier release created them for this expose. A Glue database that only a since-removed expose created, a parquet expose say, is not a catalog move: the plan's destroy of it is left to the [OpenTofu data-loss gate](../cli/apply.md#opentofu-data-loss-gate). On #707 alone the guard flagged such a database, and the apply could not get past it. + +*(forge-cli fix/iceberg-catalog-followups, unreleased)* When the guard cannot read the state, or its check fails, the apply continues without it and logs an `iceberg_catalog_move_probe_skipped` WARNING that names the reason and the exposes, for example: + +```text +{"time": "...", "level": "WARNING", "name": "fluid.cli", "message": "iceberg_catalog_move_probe_skipped", "provider": "aws", "reason": "the state could not be read: `tofu state pull` failed: Error: Failed to load state: AccessDenied: Access Denied", "exposes": [{"expose": "orders", "catalog": "lakekeeper"}], "remedy": "if the plan below destroys resources an older forge-cli created for these exposes, do NOT pass --allow-data-loss: release them with `tofu state rm` and re-run apply"} +``` + +The [OpenTofu data-loss gate](../cli/apply.md#opentofu-data-loss-gate) still stops a plan that would destroy the resources. Release them with `tofu state rm` as above rather than passing `--allow-data-loss`. On #707 alone a failed check was logged at debug level, and a state that could not be read was not reported. The same pattern guards [packaging ownership transitions](../cli/generate-iac.md#ownership-transitions). + +### Upgrading a Snowflake contract that names another catalog + +*(forge-cli fix/iceberg-catalog-followups, unreleased)* 0.19.0 and earlier created a Snowflake EXTERNAL VOLUME for a `platform: snowflake` Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest`, and dbt wrote the table onto it as a Snowflake-managed table. The module now creates no volume for those catalogs, since Snowflake does not manage them. The guard covers `hive`, `jdbc`, `hadoop` and `dynamodb` too, which also got a volume and which `fluid validate` now refuses. A volume the expose names itself, in `binding.icebergConfig.properties.external_volume`, is not concerned: no release created it. When the contract's state still holds the volume, `fluid apply` stops before it plans instead of dropping it (the `remediation` list is abridged): + +```text +❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED] + kind: iceberg-catalog-move + error: iceberg catalog move blocked — this contract's OpenTofu state holds 1 Snowflake EXTERNAL VOLUME(s) for Iceberg table(s) that now live in another catalog: + + exposes[orders_iceberg]: location.catalog lakekeeper + + snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL + +forge-cli no longer creates an EXTERNAL VOLUME for an Iceberg table in a catalog Snowflake does not manage (dbt now writes it as an externally cataloged table, not a Snowflake-managed one on a volume), so applying now would plan to DROP these volumes, and any Snowflake-managed Iceberg table an earlier dbt run wrote onto one still uses it. + +Drop each from this contract's state, then re-run apply: + + tofu -chdir=.fluid/iac/snowflake/sales_orders_lake state rm snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL + +`tofu state rm` touches ZERO bytes of infrastructure: the resources stay in Snowflake, and only this contract's claim on them is released. Drop a volume by hand (DROP EXTERNAL VOLUME) only once no Iceberg table uses it. If the table belongs in Snowflake's own catalog, remove location.catalog (or set it to 'snowflake') instead. + remediation: [...] +``` + +1. Run the printed command, `tofu -chdir=.fluid/iac/snowflake/ state rm snowflake_external_volume.`. It changes nothing in Snowflake. +2. Run `fluid apply` again. +3. Drop the volume by hand (`DROP EXTERNAL VOLUME`) only once no Iceberg table uses it. A Snowflake-managed table an earlier dbt run wrote onto it still does. + +If the table belongs in Snowflake's own catalog, remove `location.catalog` (or set it to `snowflake`) instead. The volume is named per contract (`FLUID__VOL`), not per expose, so the Snowflake guard has no per-expose resource to check: the volume is flagged when it is in state and a moved expose would have derived it. ### Other commands @@ -483,6 +522,7 @@ Each emitter classified `location.catalog` by hand, and they disagreed: - An unknown value fell back differently in each emitter, so a typo split one table across catalogs. - `fluid policy compile` compiled every Iceberg expose off GCP to AWS S3 and Glue grants, whatever its platform. - A crash inside an Iceberg or Confluent check of `fluid validate` was printed only with `--verbose`, and the contract passed unchecked. +- Neither streaming runner checked its Iceberg sink before it created anything. ## Masking at landing diff --git a/docs/cli/apply.md b/docs/cli/apply.md index c4329f6..e8a6bd1 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -270,12 +270,37 @@ Without the guard `tofu` would find nothing in the new region, create the resour ```text ❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED] kind: iceberg-catalog-move - error: iceberg catalog move blocked — this contract's OpenTofu state holds 2 -Glue catalog resource(s) for Iceberg table(s) that now live in another catalog: + error: iceberg catalog move blocked — this contract's OpenTofu state holds 2 Glue catalog resource(s) for Iceberg table(s) that now live in another catalog: + + exposes[orders]: location.catalog lakekeeper + + aws_glue_catalog_database.bronze_orders_stream_streaming + aws_glue_catalog_table.bronze_orders_stream_streaming_orders ... ``` -The message lists the addresses and a `tofu -chdir=.fluid/iac/aws/ state rm
` command for each. Run them, which changes nothing in AWS, then apply again. No flag skips the guard. The steps, and what to delete by hand afterwards, are in [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). +The message lists the addresses and a `tofu -chdir=.fluid/iac/aws/ state rm
` command for each. Run them, which changes nothing in AWS, then apply again. No flag skips the guard. The steps, and what to delete by hand afterwards, are in [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). *(forge-cli fix/iceberg-catalog-followups, unreleased)* The expose's resources are flagged only when its own Glue table is in state, so a Glue database that a since-removed expose created is left to the [data-loss gate](#opentofu-data-loss-gate) instead of blocking the apply. + +*(forge-cli fix/iceberg-catalog-followups, unreleased)* The guard covers Snowflake too. 0.19.0 and earlier created a Snowflake EXTERNAL VOLUME for a `platform: snowflake` Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest` (and for `hive`, `jdbc`, `hadoop` and `dynamodb`, which `fluid validate` now refuses). The module no longer creates one for those catalogs, so when state still holds the volume, the apply stops instead of dropping it: + +```text +❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED] + kind: iceberg-catalog-move + error: iceberg catalog move blocked — this contract's OpenTofu state holds 1 Snowflake EXTERNAL VOLUME(s) for Iceberg table(s) that now live in another catalog: + + exposes[orders_iceberg]: location.catalog lakekeeper + + snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL +... +Drop each from this contract's state, then re-run apply: + + tofu -chdir=.fluid/iac/snowflake/sales_orders_lake state rm snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL +... +``` + +Run the printed `tofu -chdir=.fluid/iac/snowflake/ state rm snowflake_external_volume.`, which changes nothing in Snowflake, then apply again. Drop the volume by hand (`DROP EXTERNAL VOLUME`) only once no Iceberg table uses it: a Snowflake-managed table an earlier dbt run wrote onto it still does. See [Upgrading a Snowflake contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-a-snowflake-contract-that-names-another-catalog). + +*(forge-cli fix/iceberg-catalog-followups, unreleased)* When the guard cannot read the state, or its check fails, the apply goes on to plan, and logs an `iceberg_catalog_move_probe_skipped` WARNING with the reason and the exposes. The data-loss gate still stops a plan that destroys the resources; release them with `tofu state rm` rather than passing `--allow-data-loss`. On #707 alone a failed check was logged at debug level. ## Remote state @@ -494,7 +519,7 @@ Each failure prints the typed event name and an `[ERR_]` slug, and exits | `opentofu_data_loss_gate` | An OpenTofu plan destroys a resource and `--allow-data-loss` is not set. | | `opentofu_region_moved` | State holds the contract's resources in another region than its bindings name. | | `packaging_transition_blocked` | A container's ownership flips under existing state. See [Packaging modes](#packaging-modes). | -| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707))* State holds Glue resources an earlier release created for an AWS Iceberg table that now lives in another catalog. See [Iceberg catalog-move guard](#iceberg-catalog-move-guard). | +| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707))* State holds Glue resources an earlier release created for an AWS Iceberg table that now lives in another catalog. *(forge-cli fix/iceberg-catalog-followups, unreleased)* Or it holds the Snowflake EXTERNAL VOLUME an earlier release created for a Snowflake Iceberg table in an external catalog. See [Iceberg catalog-move guard](#iceberg-catalog-move-guard). | | `opentofu_dataset_access_unreconciled` | State holds a BigQuery dataset an older forge-cli applied with an authoritative access list, and every entry of it is a grant the contract no longer makes. Revoke those entries on the dataset by hand, or keep one of them in `accessPolicy` for one apply. | | `apply_state_backend_invalid`, `state_shared_with_another_provider`, `state_migration_*` | See [Remote state](#remote-state). | | `generate_iac_provider_mismatch` | `--provider` contradicts the contract's binding. | diff --git a/docs/cli/diff.md b/docs/cli/diff.md index 5dbbf3b..970c1cc 100644 --- a/docs/cli/diff.md +++ b/docs/cli/diff.md @@ -105,7 +105,14 @@ For each expose, `diff` reads the target the way `apply` writes it and compares - AWS Glue tables; - BigQuery tables. -An expose bound to anything else, such as Snowflake, is reported `not_checked`, and so is an expose whose contract declares no schema. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* So is an AWS Iceberg expose whose `location.catalog` names a catalog other than Glue, with the detail `table lives in Iceberg catalog ; Glue is not inspected`: `fluid apply` creates no Glue table for it, and a Glue table of the same name would belong to someone else. On 0.19.0 and earlier, `diff` looked such a table up in Glue. Each expose ends in exactly one status: +An expose bound to anything else, such as Snowflake, is reported `not_checked`, and so is an expose whose contract declares no schema. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* So is an AWS Iceberg expose whose `location.catalog` names a catalog other than Glue, with the detail `table lives in Iceberg catalog ; Glue is not inspected`: `fluid apply` creates no Glue table for it, and a Glue table of the same name would belong to someone else. On 0.19.0 and earlier, `diff` looked such a table up in Glue. *(forge-cli fix/iceberg-catalog-followups, unreleased)* On AWS, a `location.catalog` value FLUID does not know is refused before anything is compared, because the AWS planner `diff` runs cannot tell whether the table is in Glue: + +```text +❌ diff_failed [ERR_DIFF_FAILED] + error: exposes[orders] names Iceberg catalog 'lakekeper' (location.catalog), which FLUID does not know, so it cannot tell whether the table belongs in AWS Glue, and planning it would provision no Glue database or table. Use one of: bigquery, dynamodb, glue, hadoop, hive, jdbc, lakekeeper, nessie, polaris, rest, snowflake-managed, unity, iceberg-rest, snowflake. Or remove location.catalog to keep the table in the Glue catalog. +``` + +Each expose ends in exactly one status: | Status | Meaning | Counts as drift | | --- | --- | --- | diff --git a/docs/cli/generate-iac.md b/docs/cli/generate-iac.md index 5bdb3bf..0dda48a 100644 --- a/docs/cli/generate-iac.md +++ b/docs/cli/generate-iac.md @@ -122,14 +122,12 @@ Principals are refused when they cannot be real identities. A principal in a res ```text ❌ unsupported_binding [ERR_UNSUPPORTED_BINDING] kind: principal-placeholder - error: exposes accessPolicy: principal 'group:data-platform@northwind.example' -resolves to 'group:data-platform@northwind.example', a placeholder: .example is -a reserved top-level domain (RFC 2606), so no real identity has it, and BigQuery -refuses an access entry for an identity that does not exist. - remediation: ["Map the logical principal in this environment's -binding.principals to the real group or service account it stands for, ... + error: exposes[customers] accessPolicy: principal 'group:data-platform@northwind.example' resolves to 'group:data-platform@northwind.example', a placeholder: .example is a reserved top-level domain (RFC 2606), so no real identity has it, and BigQuery refuses an access entry for an identity that does not exist. + remediation: ["Map the logical principal in this environment's binding.principals to the real group or service account it stands for, ... ``` +*(forge-cli fix/iceberg-catalog-followups, unreleased)* The `error:` line names the expose, `exposes[customers]`, and a value is no longer wrapped at the terminal width. 0.19.0 and earlier print `exposes accessPolicy: ...`, without the expose. + Other refusal kinds exist for principals and governance fields: `principal-unmapped`, `column-restriction-*` and `encryption-kms-*`. See the [GCP provider page](../providers/gcp.md) for the provider's own behaviour. ### Access grants on GCP *(0.17.0)* @@ -176,7 +174,7 @@ An Iceberg expose needs infrastructure dbt refuses to create, and the plugins de The emitters are emit-when-derivable: an Iceberg expose missing a required input (an S3 warehouse without `iam_role_arn`, no derivable bucket name, an illegal `external_volume` override) used to emit **nothing, silently** — the failure surfaced only at `dbt run`. Since `0.14.0`, [`fluid validate`](./validate.md#iceberg-prerequisite-checks-since-0-14-0) errors on exactly those cases before any emit. One `--strict` caveat: a Snowflake catalog whose auth is secret-bearing (`polaris` / `unity` / `rest` / `nessie`) draws a *warning* — FLUID does not emit those catalog integrations because the emitted module is credential-free — and `fluid validate --strict` promotes it to an error. -With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unreleased), every plugin reads `location.catalog` through one table of catalog kinds. The Snowflake warning then covers every catalog Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`), and `hive`, `jdbc`, `hadoop` or `dynamodb` on Snowflake is an error. The `aws` plugin keeps an Iceberg expose's S3 bucket but creates no Glue database or table for it unless its catalog is Glue, and `fluid apply` stops with `iceberg_catalog_move_blocked` when state holds Glue resources an earlier release created for such a table. See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). +With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unreleased), every plugin reads `location.catalog` through one table of catalog kinds. The Snowflake warning then covers every catalog Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`), and `hive`, `jdbc`, `hadoop` or `dynamodb` on Snowflake is an error. The `aws` plugin keeps an Iceberg expose's S3 bucket but creates no Glue database or table for it unless its catalog is Glue, and `fluid apply` stops with `iceberg_catalog_move_blocked` when state holds Glue resources an earlier release created for such a table. The `snowflake` plugin creates no EXTERNAL VOLUME for an Iceberg expose in a catalog Snowflake does not manage, and *(forge-cli fix/iceberg-catalog-followups, unreleased)* `fluid apply` stops the same way when state holds the volume an earlier release created for a `lakekeeper`, `bigquery` or `iceberg-rest` expose; see [Iceberg catalog-move guard](./apply.md#iceberg-catalog-move-guard). See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). ## Packaging modes diff --git a/docs/cli/validate.md b/docs/cli/validate.md index 3c0ca58..2c045ff 100644 --- a/docs/cli/validate.md +++ b/docs/cli/validate.md @@ -27,14 +27,16 @@ A contract that breaks a rule prints each finding and exits `1`: ```text ❌ Invalid FLUID contract (2 error(s)) (schema v0.7.6) -Validation completed in 0.020s +Validation completed in 0.001s Validation Errors: ================== - 1. exposes accessPolicy: principal 'group:analysts@northwind.example' resolves to 'group:analysts@northwind.example', a placeholder: ... - 2. exposes is a GCP gcs_bucket binding and declares lifecycle.expire, which the GCP emitter applies only to BigQuery tables. It would not be enforced. Move the policy to a BigQuery table expose, or remove it from this one. + 1. exposes[customers] accessPolicy: principal 'group:analysts@northwind.example' resolves to 'group:analysts@northwind.example', a placeholder: ... + 2. exposes[exports] is a GCP gcs_bucket binding and declares lifecycle.expire, which the GCP emitter applies only to BigQuery tables. It would not be enforced. Move the policy to a BigQuery table expose, or remove it from this one. ``` +*(forge-cli fix/iceberg-catalog-followups, unreleased)* A finding prints its square brackets, so each one names its expose: `exposes[customers]`. forge-cli 0.19.0 and earlier read bracketed text as console markup and dropped it, so the same findings print as `exposes accessPolicy: ...` and `exposes is a GCP gcs_bucket binding ...`, and a [bundle finding](#validating-a-bundle-tgz) loses its `[error]` or `[warning]` severity. `--format json` was not affected. Since the fix, a finding is printed on one line and the terminal wraps it. + More invocations: ```bash @@ -219,7 +221,7 @@ fluid validate runtime/bundle.tgz ```text ✅ Bundle pass: /path/to/proj/runtime/bundle.tgz - digest: sha256:20a68c3100200465659a4394e0deefec0231665168a3162736d2f080d8b21812 + digest: sha256:c86cb86ccfd50fc12f934c329db845b4248d1ed6a9074aab5195b6879ebecae0 issues: 0 total (0 error, 0 warning, 0 info) ``` @@ -238,19 +240,19 @@ fluid validate runtime/bundle.tgz --env prod ```text ❌ Bundle fail: /path/to/proj/runtime/bundle.tgz - digest: sha256:20a68c3100200465659a4394e0deefec0231665168a3162736d2f080d8b21812 + digest: sha256:c86cb86ccfd50fc12f934c329db845b4248d1ed6a9074aab5195b6879ebecae0 issues: 1 total (1 error, 0 warning, 0 info) - bundle-env: MANIFEST.json: the bundle was built for env 'dev' but this stage was asked for env 'prod'. A bundle is never re-overlaid; rebuild it with `fluid bundle --env prod --format tgz`, or pass the env it was built for. + [error] bundle-env: MANIFEST.json: the bundle was built for env 'dev' but this stage was asked for env 'prod'. A bundle is never re-overlaid; rebuild it with `fluid bundle --env prod --format tgz`, or pass the env it was built for. ``` A bundle built from the governance example above fails on its contract rules, the same two errors as validating the file: ```text -❌ Bundle fail: gov.tgz - digest: sha256:a094ac5263cada6f6462b08ae57acec818abd8f6afa0e228a2f9b2ef41090cbc +❌ Bundle fail: /path/to/gov.tgz + digest: sha256:f90911539a301cc3e358ce272e57836d80bac2c93aeb52e85851787bce58f2f3 issues: 2 total (2 error, 0 warning, 0 info) - contract: contract.resolved.yaml: exposes accessPolicy: principal 'group:analysts@northwind.example' resolves to ... - contract: contract.resolved.yaml: exposes is a GCP gcs_bucket binding and declares lifecycle.expire, ... + [error] contract: contract.resolved.yaml: exposes[customers] accessPolicy: principal 'group:analysts@northwind.example' resolves to ... + [error] contract: contract.resolved.yaml: exposes[exports] is a GCP gcs_bucket binding and declares lifecycle.expire, ... ``` `--report PATH` writes the full structured report as JSON, on pass and on fail, so a CI job can upload it as an artifact. The report has `bundleDigest`, `input`, `strict`, `status`, `summary` and `issues[]`; each issue carries `file`, `validator`, `severity`, `message` and a `code`. The `BUNDLE-ENV-MISMATCH` finding is written to the report like any other. `--report` has an effect only for a `.tgz` bundle: `fluid validate contract.fluid.yaml --report out.json` writes no file as of 0.18.1. @@ -290,7 +292,7 @@ With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unre ## Iceberg catalog checks ::: warning Not in a release yet -These checks come with [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707), which no release includes yet. +These checks come with [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707), which no release includes yet, and with the follow-up fixes in forge-cli fix/iceberg-catalog-followups, marked *(forge-cli fix/iceberg-catalog-followups, unreleased)*. ::: `binding.location.catalog` names the Iceberg catalog that owns an expose's table, and every emitter now reads it through one table. [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog) has that table, how spellings fold, and a worked Lakekeeper example. `fluid validate` refuses a contract whose catalog the emitters would disagree about: @@ -301,13 +303,15 @@ These checks come with [forge-cli #707](https://github.com/Agenticstiger/forge-c - a `sink.catalog` that names another catalog than the expose is an error, because dbt and the modules read only the expose. - an `iceberg_catalog_overrides` entry or a hand-written sink config that would leave the connector with both `type` and `catalog-impl` is an error, because Apache Iceberg refuses that catalog and the sink never starts. - `nessie` on a `kafka-connect` build is a warning: the stock Apache Iceberg Kafka Connect runtime has no Nessie client. + - *(forge-cli fix/iceberg-catalog-followups, unreleased)* `bigquery` on a `kafka-connect` build is a warning: the sink config sets `iceberg.catalog.type=bigquery`, which the published Apache Iceberg Kafka Connect sink (1.9.2) cannot load; Iceberg adds the type in 1.10. This warning and the Nessie one follow the catalog type that reaches the worker, so a hand-written `sink_connector_config` that sets `iceberg.catalog.type: rest` draws neither. + - *(forge-cli fix/iceberg-catalog-followups, unreleased)* on `platform: gcp`, an Iceberg expose the build writes must name its `location.catalog`, or it is an error: the sink would write through a REST catalog while dbt-bigquery and the GCP module create a BigLake table. See [How a value is read](../advanced/source-aligned-acquisition.md#how-a-value-is-read). - **Snowflake.** On `platform: snowflake`, a catalog that Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`) draws the warning described above, which `--strict` turns into an error. `hive`, `jdbc`, `hadoop` and `dynamodb` are an error, because Snowflake has no catalog integration for them. A `lakekeeper` expose is no longer asked for an `s3://` or `gs://` warehouse, and two `catalog: snowflake` exposes that derive one EXTERNAL VOLUME on different storage are caught here rather than failing `fluid apply` mid-emit. -- **AWS.** An Iceberg expose in a catalog other than Glue cannot carry `governance.lakeFormation`, `policy.authz.columnRestrictions` or `policy.authz.rowFilters`: each is refused by catalog name. `accessPolicy.grants` on it draws a warning to grant access in that catalog. See [On AWS](../advanced/source-aligned-acquisition.md#on-aws-a-table-in-another-catalog). +- **AWS.** An Iceberg expose in a catalog other than Glue cannot carry `governance.lakeFormation`, `policy.authz.columnRestrictions` or `policy.authz.rowFilters`: each is refused by catalog name. `accessPolicy.grants` on it draws a warning to grant access in that catalog. See [On AWS](../advanced/source-aligned-acquisition.md#on-aws-a-table-in-another-catalog). *(forge-cli fix/iceberg-catalog-followups, unreleased)* An unknown catalog value draws only the unknown-catalog error, not these refusals as well. - **GCP.** A `platform: gcp` expose in a catalog other than `bigquery` may give the catalog's warehouse name; a warehouse in another object store (`s3://`, `abfss://`) is an error. A crash inside the Iceberg sink, Confluent or Iceberg prerequisite check now fails validation, with an error such as `Iceberg sink check could not run (...); the contract was NOT checked for streaming-sink defects`. On 0.19.0 and earlier the crash was printed only with `--verbose`, and the contract passed unchecked. -The Kafka Connect runner and the embedded Debezium Server runner run the same streaming-sink checks before they create anything, so a contract that skipped `fluid validate` still fails before any Connect REST call or `application.properties` write. +The Kafka Connect runner and the embedded Debezium Server runner run the same streaming-sink checks before they create anything, so a contract that skipped `fluid validate` still fails before any Connect REST call or `application.properties` write. *(forge-cli fix/iceberg-catalog-followups, unreleased)* That holds for a build that declares `sink.format: iceberg`, with a derived or a hand-written sink config, and for an embedded Debezium Server build that derives its sink. On #707 alone a hand-written config was not checked by the runners. [What each kind produces](../advanced/source-aligned-acquisition.md#what-each-kind-produces) lists the builds. ## GCP binding checks (since 0.15.0) diff --git a/docs/providers/aws.md b/docs/providers/aws.md index 5a9a72c..6c69278 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -746,10 +746,7 @@ A pipeline that runs `fluid validate --strict` fails on it; stage 2 of the [11-s *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* An Iceberg expose in a catalog other than Glue cannot take Lake Formation grants, so its warning asks you to grant access in that catalog instead: ```text - 1. accessPolicy.grants are not enforced on aws binding(s) orders (lakekeeper): -their Iceberg tables live in a catalog other than Glue, and the AWS emitter -writes access only as Lake Formation grants on a Glue table. Grant access in -that catalog. + 1. accessPolicy.grants are not enforced on aws binding(s) orders (lakekeeper): their Iceberg tables live in a catalog other than Glue, and the AWS emitter writes access only as Lake Formation grants on a Glue table. Grant access in that catalog. ``` `fluid policy-compile` still reads `accessPolicy.grants` and writes IAM bindings, which you can use as a starting point for IAM policies. Nothing applies them: @@ -953,6 +950,7 @@ The release notes for [`0.16.0`](../RELEASE_NOTES_0.16.0.md) and [`0.17.0`](../R - **`0.17.0`:** a column restriction on an aws binding with no Lake Formation grants is refused, and `accessPolicy.grants` on an aws binding with no Lake Formation grants warns; see [Column restrictions](#column-restrictions) and [accessPolicy on AWS](#accesspolicy-on-aws). - **`0.18.0`:** contract SQL runs in a DuckDB sandbox and `$ref` stays inside the contract's directory tree; see [Where a build lands data](#where-a-build-lands-data). - **Unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707):** an Iceberg expose whose `location.catalog` is not `glue` keeps its S3 bucket and loses its Glue database and table. On a contract an earlier release applied, `fluid apply` stops with `iceberg_catalog_move_blocked` and prints the `tofu state rm` commands that release them; see [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). A Kafka Connect sink on Glue starts, where it failed at startup before. +- **Unreleased, forge-cli fix/iceberg-catalog-followups:** the `iceberg_catalog_move_blocked` guard flags an expose's Glue database and table only when that expose's own Glue table is in state, so a Glue database left by a removed expose no longer blocks the apply. When the guard cannot read the state, it logs an `iceberg_catalog_move_probe_skipped` WARNING. `fluid diff` refuses an Iceberg `location.catalog` value FLUID does not know, as `fluid validate` and `fluid apply` do, and a typo no longer also draws a Lake Formation refusal. See [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). ## How far this has been exercised diff --git a/docs/providers/gcp.md b/docs/providers/gcp.md index 4c50a69..4ac3496 100644 --- a/docs/providers/gcp.md +++ b/docs/providers/gcp.md @@ -319,7 +319,19 @@ exposes: Retention, keys and column restrictions are refused on an Iceberg binding: they apply only to a BigQuery table. -**A table in another catalog.** *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* BigLake metastore is the catalog this section describes, and the one an Iceberg expose gets with no `location.catalog` or with `catalog: bigquery`. An expose that names another catalog (`lakekeeper`, `rest`, `nessie`, ...) is not a BigLake table: `catalogs.yml` leaves it out, with a warning, instead of having dbt write a second table under the name that catalog owns. `fluid validate` accepts that catalog's warehouse name (`warehouse: analytics`), and refuses a warehouse in another object store (`s3://`, `abfss://`), since a `platform: gcp` table cannot live there. A streaming sink with no `location.catalog` on GCP writes through a REST catalog, not BigLake, so name the catalog when a sink and dbt share the table. See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). On 0.19.0 and earlier, an expose naming another catalog became a BigLake table, and a warehouse name failed validation. +**A table in another catalog.** *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* BigLake metastore is the catalog this section describes, and the one an Iceberg expose gets with no `location.catalog` or with `catalog: bigquery`. An expose that names another catalog (`lakekeeper`, `rest`, `nessie`, ...) is not a BigLake table: `catalogs.yml` leaves it out, with a warning, instead of having dbt write a second table under the name that catalog owns. `fluid validate` accepts that catalog's warehouse name (`warehouse: analytics`), and refuses a warehouse in another object store (`s3://`, `abfss://`), since a `platform: gcp` table cannot live there. A streaming sink with no `location.catalog` on GCP writes through a REST catalog, not BigLake. See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). On 0.19.0 and earlier, an expose naming another catalog became a BigLake table, and a warehouse name failed validation. + +**A streaming sink must name the catalog.** *(forge-cli fix/iceberg-catalog-followups, unreleased)* An Iceberg expose that a Kafka Connect build or an embedded Debezium Server build writes must set `location.catalog`: `bigquery` for BigLake metastore, or the REST kind your catalog is. Without it, `fluid validate` fails, and so does the run, before it creates anything: + +```text + 1. iceberg sink (build 'stream_events'): the GCP Iceberg expose sets no binding.location.catalog, so it is read two ways: the sink would write through a REST catalog (the 'gcp' platform default) while dbt-bigquery and the GCP IaC, which read only binding.location.catalog, create a BigLake metastore table. Set binding.location.catalog: bigquery, or the REST kind your catalog is (e.g. rest, lakekeeper) +``` + +An expose that no streaming sink writes may still leave `location.catalog` out; it is a BigLake table. With `catalog: bigquery`, a Kafka Connect build whose sink config sends `iceberg.catalog.type=bigquery` (the derived config does) draws a warning, since the published Apache Iceberg Kafka Connect sink (1.9.2) cannot load that type, which Iceberg adds in 1.10: + +```text + 1. iceberg sink (build 'stream_events'): the sink config sets iceberg.catalog.type=bigquery, which the published Apache Iceberg Kafka Connect sink (1.9.2 on Confluent Hub) cannot load: Iceberg's CatalogUtil gains the bigquery type in 1.10. Run a sink built from Iceberg >= 1.10, or the connector fails at start +``` --- @@ -504,12 +516,11 @@ accessPolicy: On GCP, a principal in a reserved top-level domain (`.example`, `.test`, `.invalid`, `.localhost`) or one that is not an IAM member at all (`group:data-platform` with no domain, a bare `analysts`, `role:analyst`) is refused by `fluid validate`, `fluid generate iac` and `fluid apply`: ```text - 1. exposes accessPolicy: principal 'group:data-analysts@company.example' -resolves to 'group:data-analysts@company.example', a placeholder: .example is a -reserved top-level domain (RFC 2606), so no real identity has it, and BigQuery -refuses an access entry for an identity that does not exist. ... + 1. exposes[orders] accessPolicy: principal 'group:data-analysts@company.example' resolves to 'group:data-analysts@company.example', a placeholder: .example is a reserved top-level domain (RFC 2606), so no real identity has it, and BigQuery refuses an access entry for an identity that does not exist. ... ``` +*(forge-cli fix/iceberg-catalog-followups, unreleased)* The message names the expose, `exposes[orders]`. 0.19.0 and earlier print `exposes accessPolicy: ...`, without it. + Map such a logical principal to a real identity in [`binding.principals`](#logical-principals-binding-principals-0-7-6), or to `[]` when it has no identity on GCP. #### First apply after upgrading from 0.16.x or earlier diff --git a/docs/providers/snowflake.md b/docs/providers/snowflake.md index 40f5144..7b31072 100644 --- a/docs/providers/snowflake.md +++ b/docs/providers/snowflake.md @@ -596,7 +596,7 @@ Other keys under `binding.properties` (`table_type`, `data_retention_time_in_day An expose with `binding.format: iceberg` on `platform: snowflake` closes the dbt Iceberg loop in two halves: - **`fluid generate transformation`** emits dbt's `catalogs.yml` (v1 catalogs schema, Snowflake adapter) into the generated project. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* Catalogs external to Snowflake (`location.catalog: glue`, `rest` or `iceberg_rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`) map to `catalog_type: iceberg_rest`. No `location.catalog`, or `snowflake` (`snowflake-managed`), is Snowflake-managed (Horizon) and maps to `built_in`. `hive`, `jdbc`, `hadoop` and `dynamodb`, which Snowflake has no catalog integration for, are left out of the file with a warning, and `fluid validate` refuses them. Any other value is refused by `fluid validate`. The full table, and how spellings fold, is in [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). On 0.19.0 and earlier, only `glue`, `polaris`, `unity`, `rest`, `iceberg_rest` and `nessie` mapped to `iceberg_rest`, and anything else, `lakekeeper` and `hive` included, became `built_in`. -- **`fluid apply`** provisions the prerequisites dbt refuses to create: the **EXTERNAL VOLUME** for Snowflake-managed catalogs (needs an `s3://` or `gs://` `location.warehouse`, plus `location.iam_role_arn` for S3), and the **AWS Glue CATALOG INTEGRATION** for `location.catalog: glue` (needs `location.iam_role_arn` — the role Snowflake assumes — and `location.account`, the AWS account id). It creates nothing for the other `iceberg_rest` catalogs listed above: their integrations authenticate with a secret, and the emitted module is credential-free. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* On 0.19.0 and earlier, a `lakekeeper` expose got an EXTERNAL VOLUME, and `fluid validate` asked it for an `s3://` or `gs://` warehouse. +- **`fluid apply`** provisions the prerequisites dbt refuses to create: the **EXTERNAL VOLUME** for Snowflake-managed catalogs (needs an `s3://` or `gs://` `location.warehouse`, plus `location.iam_role_arn` for S3), and the **AWS Glue CATALOG INTEGRATION** for `location.catalog: glue` (needs `location.iam_role_arn` — the role Snowflake assumes — and `location.account`, the AWS account id). It creates nothing for the other `iceberg_rest` catalogs listed above: their integrations authenticate with a secret, and the emitted module is credential-free. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* On 0.19.0 and earlier, a `lakekeeper` expose got an EXTERNAL VOLUME, and `fluid validate` asked it for an `s3://` or `gs://` warehouse. *(forge-cli fix/iceberg-catalog-followups, unreleased)* While state still holds such a volume, `fluid apply` stops before it plans; see [Upgrading an Iceberg expose in an external catalog](#upgrading-an-iceberg-expose-in-an-external-catalog). ```yaml exposes: @@ -617,6 +617,31 @@ Both halves derive names through one deterministic naming helper, so the EXTERNA Since `0.14.0`, `fluid validate` errors when an Iceberg expose is missing one of the required inputs above instead of letting the emitters silently skip it — see [Iceberg prerequisite checks](/forge_docs/cli/validate.html#iceberg-prerequisite-checks-since-0-14-0). CI note: under `--strict`, Snowflake catalogs that authenticate with secrets (`polaris` / `unity` / `rest` / `nessie`) now fail validation, because the emitted OpenTofu module is credential-free. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* The same warning covers `lakekeeper` and `bigquery`, and `hive`, `jdbc`, `hadoop` or `dynamodb` on Snowflake is an error with or without `--strict`. +### Upgrading an Iceberg expose in an external catalog + +*(forge-cli fix/iceberg-catalog-followups, unreleased)* 0.19.0 and earlier created an EXTERNAL VOLUME for an Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest`, and dbt wrote a Snowflake-managed table onto it. `hive`, `jdbc`, `hadoop` and `dynamodb` got one too; `fluid validate` now refuses them. The module now creates no volume for these catalogs. When the contract's OpenTofu state still holds the volume, the plan would drop it, so `fluid apply` stops before it plans (abridged): + +```text +❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED] + kind: iceberg-catalog-move + error: iceberg catalog move blocked — this contract's OpenTofu state holds 1 Snowflake EXTERNAL VOLUME(s) for Iceberg table(s) that now live in another catalog: + + exposes[orders_iceberg]: location.catalog lakekeeper + + snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL +... +Drop each from this contract's state, then re-run apply: + + tofu -chdir=.fluid/iac/snowflake/sales_orders_lake state rm snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL +... +``` + +1. Run the printed command, in the form `tofu -chdir=.fluid/iac/snowflake/ state rm snowflake_external_volume.`. It changes nothing in Snowflake; it releases this contract's claim on the volume. +2. Run `fluid apply` again. +3. Drop the volume by hand (`DROP EXTERNAL VOLUME`) only once no Iceberg table uses it. A Snowflake-managed table an earlier dbt run wrote onto it still does. + +If the table belongs in Snowflake's own catalog, remove `location.catalog` (or set it to `snowflake`) instead. A volume you name in `binding.icebergConfig.properties.external_volume` is never flagged, because no release created it. The same guard covers Glue resources on AWS; see [Iceberg catalog-move guard](../cli/apply.md#iceberg-catalog-move-guard). + ## See Also - [Snowflake Team Collaboration Walkthrough](/forge_docs/walkthrough/snowflake) - Role-based PR review example for Snowflake teams From 71c2e4821b4b1b85eabdff2d23aa2f8c1412694a Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:14:00 +0200 Subject: [PATCH 2/2] docs(iceberg): link the follow-up behaviour to forge-cli #709 --- docs/advanced/production-troubleshooting.md | 4 ++-- docs/advanced/source-aligned-acquisition.md | 20 ++++++++++---------- docs/cli/apply.md | 8 ++++---- docs/cli/diff.md | 2 +- docs/cli/generate-iac.md | 4 ++-- docs/cli/validate.md | 12 ++++++------ docs/providers/aws.md | 2 +- docs/providers/gcp.md | 4 ++-- docs/providers/snowflake.md | 4 ++-- 9 files changed, 30 insertions(+), 30 deletions(-) diff --git a/docs/advanced/production-troubleshooting.md b/docs/advanced/production-troubleshooting.md index 83d16e8..0d81af9 100644 --- a/docs/advanced/production-troubleshooting.md +++ b/docs/advanced/production-troubleshooting.md @@ -107,8 +107,8 @@ Remote OpenTofu state is keyed per contract and provider. See [Environment varia | `state_migration_unverified` | After the copy, the new key holds different resources than the old one. The old object is untouched | Compare the two states before re-running | | `state_migration_probe_failed` | The old key could not be read | Check access to the bucket and the key named in the message | | `opentofu_region_moved` | State holds this contract's resources in a region other than the one the bindings now name. Applying would create them again and leave the originals unmanaged | If they should stay, set the binding's `location.region` to the region the error names. If they should move, empty and remove them there first (`tofu destroy` in the state directory named, with `AWS_REGION` set to the old region), then apply again | -| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) and forge-cli fix/iceberg-catalog-followups)* An Iceberg expose's `location.catalog` names a catalog an earlier release did not honour, and state still holds what that release created for it: a Glue database and table on AWS, or an EXTERNAL VOLUME on Snowflake. The plan would destroy them, and destroying a Glue database deletes every table in it | Run the `tofu -chdir= state rm
` commands the error prints, which change nothing in the cloud, then apply again. Do not pass `--allow-data-loss`. If the table belongs in Glue or in Snowflake's own catalog, remove `location.catalog` instead. Steps: [Upgrading an AWS contract that names another catalog](./source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog), [Upgrading a Snowflake contract that names another catalog](./source-aligned-acquisition.md#upgrading-a-snowflake-contract-that-names-another-catalog) | -| `iceberg_catalog_move_probe_skipped` (WARNING) | *(unreleased, forge-cli fix/iceberg-catalog-followups)* The catalog-move guard could not read the state, or its check failed, so the apply went on without it. The `reason` field says why | If the plan then destroys Glue resources or an EXTERNAL VOLUME of the exposes the warning names, stop: release them with `tofu state rm` and apply again, rather than passing `--allow-data-loss` to the data-loss gate | +| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) and [forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709))* An Iceberg expose's `location.catalog` names a catalog an earlier release did not honour, and state still holds what that release created for it: a Glue database and table on AWS, or an EXTERNAL VOLUME on Snowflake. The plan would destroy them, and destroying a Glue database deletes every table in it | Run the `tofu -chdir= state rm
` commands the error prints, which change nothing in the cloud, then apply again. Do not pass `--allow-data-loss`. If the table belongs in Glue or in Snowflake's own catalog, remove `location.catalog` instead. Steps: [Upgrading an AWS contract that names another catalog](./source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog), [Upgrading a Snowflake contract that names another catalog](./source-aligned-acquisition.md#upgrading-a-snowflake-contract-that-names-another-catalog) | +| `iceberg_catalog_move_probe_skipped` (WARNING) | *(unreleased, [forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709))* The catalog-move guard could not read the state, or its check failed, so the apply went on without it. The `reason` field says why | If the plan then destroys Glue resources or an EXTERNAL VOLUME of the exposes the warning names, stop: release them with `tofu state rm` and apply again, rather than passing `--allow-data-loss` to the data-loss gate | ## Contract load and overlay errors diff --git a/docs/advanced/source-aligned-acquisition.md b/docs/advanced/source-aligned-acquisition.md index 82c341d..840cfc3 100644 --- a/docs/advanced/source-aligned-acquisition.md +++ b/docs/advanced/source-aligned-acquisition.md @@ -218,7 +218,7 @@ For the DuckDB engine, each connection runs in DuckDB's own sandbox ([DuckDB san ## Iceberg catalogs (`location.catalog`) ::: warning Not in a release yet -This section describes forge-cli [PR #707](https://github.com/Agenticstiger/forge-cli/pull/707) and the follow-up fixes in forge-cli fix/iceberg-catalog-followups, marked *(forge-cli fix/iceberg-catalog-followups, unreleased)*. No release includes either yet. forge-cli 0.19.0 and earlier behave as described in [On 0.19.0 and earlier](#on-0-19-0-and-earlier), at the end of this section. +This section describes forge-cli [PR #707](https://github.com/Agenticstiger/forge-cli/pull/707) and the follow-up fixes in [forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), marked *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)*. No release includes either yet. forge-cli 0.19.0 and earlier behave as described in [On 0.19.0 and earlier](#on-0-19-0-and-earlier), at the end of this section. ::: An Iceberg expose names the catalog that owns its table in `binding.location.catalog`. The streaming sinks (Kafka Connect and Debezium Server), dbt's `catalogs.yml`, the AWS and Confluent modules, `fluid policy compile`, `fluid diff`, `fluid test` and `fluid validate` read the value through one table in forge-cli (`fluid_build/providers/_iceberg_catalog.py`), so they agree on which catalog holds the table. The Snowflake module reads it for the Iceberg prerequisites only: the EXTERNAL VOLUME and the Glue catalog integration follow the table. It still emits a `snowflake_database`, `snowflake_schema` and `snowflake_table` for the expose's `location.database`, `location.schema` and `location.table`, whatever the catalog. @@ -316,7 +316,7 @@ The one resource is `aws_s3_bucket.bronze_orders_stream_acme_lake`. Without `buc ### How a value is read - **Spelling.** Case, surrounding whitespace, and `-` against `_` are folded, so `Lakekeeper` is `lakekeeper` and `SNOWFLAKE_MANAGED` is `snowflake-managed`. Two spellings are aliases: `iceberg-rest` (or `iceberg_rest`) is `rest`, and `snowflake` is `snowflake-managed`. -- **No value.** An Iceberg expose with no `location.catalog` gets its platform's default: `glue` on `platform: aws`, `snowflake-managed` on `platform: snowflake`, and `rest` on any other platform. On `platform: gcp` the default means two things: a streaming sink writes through a REST catalog, while dbt-bigquery and the GCP module create a BigLake table. *(forge-cli fix/iceberg-catalog-followups, unreleased)* So a `platform: gcp` Iceberg expose that a streaming sink writes must name its catalog. Without one, `fluid validate` fails, and so does the Kafka Connect or embedded Debezium Server run before it creates anything. Set `catalog: bigquery`, or the REST kind your catalog is. #707 alone accepted it. An expose that no streaming sink writes may still leave the catalog out. The error, for a Kafka Connect build `stream_events` writing a GCP expose with no catalog: +- **No value.** An Iceberg expose with no `location.catalog` gets its platform's default: `glue` on `platform: aws`, `snowflake-managed` on `platform: snowflake`, and `rest` on any other platform. On `platform: gcp` the default means two things: a streaming sink writes through a REST catalog, while dbt-bigquery and the GCP module create a BigLake table. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* So a `platform: gcp` Iceberg expose that a streaming sink writes must name its catalog. Without one, `fluid validate` fails, and so does the Kafka Connect or embedded Debezium Server run before it creates anything. Set `catalog: bigquery`, or the REST kind your catalog is. #707 alone accepted it. An expose that no streaming sink writes may still leave the catalog out. The error, for a Kafka Connect build `stream_events` writing a GCP expose with no catalog: ```text 1. iceberg sink (build 'stream_events'): the GCP Iceberg expose sets no binding.location.catalog, so it is read two ways: the sink would write through a REST catalog (the 'gcp' platform default) while dbt-bigquery and the GCP IaC, which read only binding.location.catalog, create a BigLake metastore table. Set binding.location.catalog: bigquery, or the REST kind your catalog is (e.g. rest, lakekeeper) @@ -324,7 +324,7 @@ The one resource is `aws_s3_bucket.bronze_orders_stream_acme_lake`. Without `buc Without `uri` and `warehouse`, the same build also draws the `rest` errors for them. When a `sink.catalog` or a hand-written sink config sets the catalog, the parentheses name it instead of the platform default. - **`sink.catalog`.** A streaming build may set `properties.sink.catalog`, but it must name the expose's catalog, because dbt and the modules read only the expose. Spellings are compared after folding. Leave it unset and the sink uses the expose's catalog. -- **Any other value** is refused by `fluid validate` on every Iceberg expose, with the accepted spellings. A `platform: confluent` expose has its own rule, [below](#other-commands). `fluid apply` on AWS refuses the value too, as `unknown-iceberg-catalog`, because whether a Glue table exists depends on it. *(forge-cli fix/iceberg-catalog-followups, unreleased)* So does the native AWS planner that [`fluid diff`](../cli/diff.md) runs; on #707 alone it planned the S3 buckets and no Glue table. A typo on an expose that also declares `governance.lakeFormation` draws only the two errors below, not a Lake Formation refusal that names the typo as a catalog. +- **Any other value** is refused by `fluid validate` on every Iceberg expose, with the accepted spellings. A `platform: confluent` expose has its own rule, [below](#other-commands). `fluid apply` on AWS refuses the value too, as `unknown-iceberg-catalog`, because whether a Glue table exists depends on it. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* So does the native AWS planner that [`fluid diff`](../cli/diff.md) runs; on #707 alone it planned the S3 buckets and no Glue table. A typo on an expose that also declares `governance.lakeFormation` draws only the two errors below, not a Lake Formation refusal that names the typo as a catalog. ```text ❌ Invalid FLUID contract (2 error(s)) (schema v0.7.5) @@ -361,7 +361,7 @@ A `sink.catalog` that names another catalog than the expose: How to read the columns: -- **Sink catalog selector.** Kafka Connect takes the key with the `iceberg.catalog.` prefix (`iceberg.catalog.type`, `iceberg.catalog.catalog-impl`), and Debezium Server with `debezium.sink.iceberg.`. The `type` values are catalog types Apache Iceberg defines. Iceberg has no `lakekeeper`, `polaris` or `unity` type, so those catalogs are reached over Iceberg REST, and no `dynamodb` type, so DynamoDB is selected by class. `type=bigquery` needs an Iceberg runtime of 1.10 or later. For `nessie` the sink uses Iceberg's Nessie client, which the stock Apache Iceberg Kafka Connect runtime does not bundle, so `fluid validate` warns on a `kafka-connect` build. *(forge-cli fix/iceberg-catalog-followups, unreleased)* A `kafka-connect` build that sends `type=bigquery` draws a warning too, because the published Apache Iceberg Kafka Connect sink predates it: +- **Sink catalog selector.** Kafka Connect takes the key with the `iceberg.catalog.` prefix (`iceberg.catalog.type`, `iceberg.catalog.catalog-impl`), and Debezium Server with `debezium.sink.iceberg.`. The `type` values are catalog types Apache Iceberg defines. Iceberg has no `lakekeeper`, `polaris` or `unity` type, so those catalogs are reached over Iceberg REST, and no `dynamodb` type, so DynamoDB is selected by class. `type=bigquery` needs an Iceberg runtime of 1.10 or later. For `nessie` the sink uses Iceberg's Nessie client, which the stock Apache Iceberg Kafka Connect runtime does not bundle, so `fluid validate` warns on a `kafka-connect` build. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* A `kafka-connect` build that sends `type=bigquery` draws a warning too, because the published Apache Iceberg Kafka Connect sink predates it: ```text 1. iceberg sink (build 'stream_events'): the sink config sets iceberg.catalog.type=bigquery, which the published Apache Iceberg Kafka Connect sink (1.9.2 on Confluent Hub) cannot load: Iceberg's CatalogUtil gains the bigquery type in 1.10. Run a sink built from Iceberg >= 1.10, or the connector fails at start @@ -371,10 +371,10 @@ How to read the columns: - **dbt `catalogs.yml` on Snowflake** and **Snowflake module** apply to `platform: snowflake`. The **Snowflake module** column lists the Iceberg prerequisite the module creates; for every kind it also emits the database, schema and `snowflake_table` the binding's `location` names. Apart from `glue`, Snowflake reaches the `iceberg_rest` kinds through a catalog integration that authenticates with a secret. The module is credential-free, so it creates none, and `fluid validate` warns (an error under `--strict`). Snowflake has no catalog integration for `hive`, `jdbc`, `hadoop` or `dynamodb`, so an expose naming one fails `fluid validate`. The prerequisites each emitted object needs are in [Iceberg tables via dbt](../providers/snowflake.md#iceberg-tables-via-dbt-since-0-13-1). - **AWS module** applies to `platform: aws`: the bucket is the one the binding names. See [On AWS](#on-aws-a-table-in-another-catalog). - **A streaming sink needs** the listed `binding.location` keys when a Kafka Connect build, or a Debezium Server build in `embedded` mode, writes the expose. Debezium in `bring-your-own` or `managed` mode creates only the source connector, so these checks do not apply to it. The Kafka Connect runner and the embedded Debezium Server runner run the same checks before they create anything, so a contract `fluid validate` refuses also fails its run, for these builds: - - *(forge-cli fix/iceberg-catalog-followups, unreleased)* a build that declares `sink.format: iceberg`, whether the runner derives the sink config or pushes a hand-written one: `sink_connector_config` on Kafka Connect, `server.sink.config` on an embedded Debezium Server build whose `server.sink.type` is `iceberg` (the default). On #707 alone the runners checked only a config they derived, so a hand-written config that `fluid validate` refuses was still deployed; + - *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* a build that declares `sink.format: iceberg`, whether the runner derives the sink config or pushes a hand-written one: `sink_connector_config` on Kafka Connect, `server.sink.config` on an embedded Debezium Server build whose `server.sink.type` is `iceberg` (the default). On #707 alone the runners checked only a config they derived, so a hand-written config that `fluid validate` refuses was still deployed; - an embedded Debezium Server build that derives its sink: `server.sink.type: iceberg` (the default) with no hand-written `server.sink.config`, or with `server.sink.iceberg_sink_enabled: true`. - *(forge-cli fix/iceberg-catalog-followups, unreleased)* In a contract with several builds, each of these two runners reads its properties from the build it runs, the build the checks read. They used to read the first build's. + *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* In a contract with several builds, each of these two runners reads its properties from the build it runs, the build the checks read. They used to read the first build's. ### Kafka Connect: `catalog-impl` or `type`, never both @@ -449,7 +449,7 @@ Drop each from this contract's state, then re-run apply: remediation: [...] ``` -*(forge-cli fix/iceberg-catalog-followups, unreleased)* The `exposes[orders]` line keeps its expose id, and each printed `tofu state rm` command stays on one line, so a copied line runs with its address. On #707 alone the line printed as `exposes: location.catalog lakekeeper`, and a long command could wrap onto a second line. +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The `exposes[orders]` line keeps its expose id, and each printed `tofu state rm` command stays on one line, so a copied line runs with its address. On #707 alone the line printed as `exposes: location.catalog lakekeeper`, and a long command could wrap onto a second line. 1. Run the `tofu state rm` commands it prints, one per Glue resource, in the form `tofu -chdir= state rm
` (each argument shell-quoted when it needs to be). For the example contract: @@ -464,9 +464,9 @@ Drop each from this contract's state, then re-run apply: If the table belongs in Glue after all, remove `location.catalog` (or set it to `glue`) instead. There is no flag that skips the guard. Before it raises, the apply logs a structured `iceberg_catalog_move_blocked` event with the addresses, the exposes and the commands. On AWS the guard concerns an Iceberg expose that names a `location.database` and a catalog other than Glue. The same guard covers [Snowflake](#upgrading-a-snowflake-contract-that-names-another-catalog). -*(forge-cli fix/iceberg-catalog-followups, unreleased)* A moved expose's Glue resources are flagged only when its own Glue table is in state, which shows that an earlier release created them for this expose. A Glue database that only a since-removed expose created, a parquet expose say, is not a catalog move: the plan's destroy of it is left to the [OpenTofu data-loss gate](../cli/apply.md#opentofu-data-loss-gate). On #707 alone the guard flagged such a database, and the apply could not get past it. +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* A moved expose's Glue resources are flagged only when its own Glue table is in state, which shows that an earlier release created them for this expose. A Glue database that only a since-removed expose created, a parquet expose say, is not a catalog move: the plan's destroy of it is left to the [OpenTofu data-loss gate](../cli/apply.md#opentofu-data-loss-gate). On #707 alone the guard flagged such a database, and the apply could not get past it. -*(forge-cli fix/iceberg-catalog-followups, unreleased)* When the guard cannot read the state, or its check fails, the apply continues without it and logs an `iceberg_catalog_move_probe_skipped` WARNING that names the reason and the exposes, for example: +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* When the guard cannot read the state, or its check fails, the apply continues without it and logs an `iceberg_catalog_move_probe_skipped` WARNING that names the reason and the exposes, for example: ```text {"time": "...", "level": "WARNING", "name": "fluid.cli", "message": "iceberg_catalog_move_probe_skipped", "provider": "aws", "reason": "the state could not be read: `tofu state pull` failed: Error: Failed to load state: AccessDenied: Access Denied", "exposes": [{"expose": "orders", "catalog": "lakekeeper"}], "remedy": "if the plan below destroys resources an older forge-cli created for these exposes, do NOT pass --allow-data-loss: release them with `tofu state rm` and re-run apply"} @@ -476,7 +476,7 @@ The [OpenTofu data-loss gate](../cli/apply.md#opentofu-data-loss-gate) still sto ### Upgrading a Snowflake contract that names another catalog -*(forge-cli fix/iceberg-catalog-followups, unreleased)* 0.19.0 and earlier created a Snowflake EXTERNAL VOLUME for a `platform: snowflake` Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest`, and dbt wrote the table onto it as a Snowflake-managed table. The module now creates no volume for those catalogs, since Snowflake does not manage them. The guard covers `hive`, `jdbc`, `hadoop` and `dynamodb` too, which also got a volume and which `fluid validate` now refuses. A volume the expose names itself, in `binding.icebergConfig.properties.external_volume`, is not concerned: no release created it. When the contract's state still holds the volume, `fluid apply` stops before it plans instead of dropping it (the `remediation` list is abridged): +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* 0.19.0 and earlier created a Snowflake EXTERNAL VOLUME for a `platform: snowflake` Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest`, and dbt wrote the table onto it as a Snowflake-managed table. The module now creates no volume for those catalogs, since Snowflake does not manage them. The guard covers `hive`, `jdbc`, `hadoop` and `dynamodb` too, which also got a volume and which `fluid validate` now refuses. A volume the expose names itself, in `binding.icebergConfig.properties.external_volume`, is not concerned: no release created it. When the contract's state still holds the volume, `fluid apply` stops before it plans instead of dropping it (the `remediation` list is abridged): ```text ❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED] diff --git a/docs/cli/apply.md b/docs/cli/apply.md index e8a6bd1..876b438 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -279,9 +279,9 @@ Without the guard `tofu` would find nothing in the new region, create the resour ... ``` -The message lists the addresses and a `tofu -chdir=.fluid/iac/aws/ state rm
` command for each. Run them, which changes nothing in AWS, then apply again. No flag skips the guard. The steps, and what to delete by hand afterwards, are in [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). *(forge-cli fix/iceberg-catalog-followups, unreleased)* The expose's resources are flagged only when its own Glue table is in state, so a Glue database that a since-removed expose created is left to the [data-loss gate](#opentofu-data-loss-gate) instead of blocking the apply. +The message lists the addresses and a `tofu -chdir=.fluid/iac/aws/ state rm
` command for each. Run them, which changes nothing in AWS, then apply again. No flag skips the guard. The steps, and what to delete by hand afterwards, are in [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The expose's resources are flagged only when its own Glue table is in state, so a Glue database that a since-removed expose created is left to the [data-loss gate](#opentofu-data-loss-gate) instead of blocking the apply. -*(forge-cli fix/iceberg-catalog-followups, unreleased)* The guard covers Snowflake too. 0.19.0 and earlier created a Snowflake EXTERNAL VOLUME for a `platform: snowflake` Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest` (and for `hive`, `jdbc`, `hadoop` and `dynamodb`, which `fluid validate` now refuses). The module no longer creates one for those catalogs, so when state still holds the volume, the apply stops instead of dropping it: +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The guard covers Snowflake too. 0.19.0 and earlier created a Snowflake EXTERNAL VOLUME for a `platform: snowflake` Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest` (and for `hive`, `jdbc`, `hadoop` and `dynamodb`, which `fluid validate` now refuses). The module no longer creates one for those catalogs, so when state still holds the volume, the apply stops instead of dropping it: ```text ❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED] @@ -300,7 +300,7 @@ Drop each from this contract's state, then re-run apply: Run the printed `tofu -chdir=.fluid/iac/snowflake/ state rm snowflake_external_volume.`, which changes nothing in Snowflake, then apply again. Drop the volume by hand (`DROP EXTERNAL VOLUME`) only once no Iceberg table uses it: a Snowflake-managed table an earlier dbt run wrote onto it still does. See [Upgrading a Snowflake contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-a-snowflake-contract-that-names-another-catalog). -*(forge-cli fix/iceberg-catalog-followups, unreleased)* When the guard cannot read the state, or its check fails, the apply goes on to plan, and logs an `iceberg_catalog_move_probe_skipped` WARNING with the reason and the exposes. The data-loss gate still stops a plan that destroys the resources; release them with `tofu state rm` rather than passing `--allow-data-loss`. On #707 alone a failed check was logged at debug level. +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* When the guard cannot read the state, or its check fails, the apply goes on to plan, and logs an `iceberg_catalog_move_probe_skipped` WARNING with the reason and the exposes. The data-loss gate still stops a plan that destroys the resources; release them with `tofu state rm` rather than passing `--allow-data-loss`. On #707 alone a failed check was logged at debug level. ## Remote state @@ -519,7 +519,7 @@ Each failure prints the typed event name and an `[ERR_]` slug, and exits | `opentofu_data_loss_gate` | An OpenTofu plan destroys a resource and `--allow-data-loss` is not set. | | `opentofu_region_moved` | State holds the contract's resources in another region than its bindings name. | | `packaging_transition_blocked` | A container's ownership flips under existing state. See [Packaging modes](#packaging-modes). | -| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707))* State holds Glue resources an earlier release created for an AWS Iceberg table that now lives in another catalog. *(forge-cli fix/iceberg-catalog-followups, unreleased)* Or it holds the Snowflake EXTERNAL VOLUME an earlier release created for a Snowflake Iceberg table in an external catalog. See [Iceberg catalog-move guard](#iceberg-catalog-move-guard). | +| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707))* State holds Glue resources an earlier release created for an AWS Iceberg table that now lives in another catalog. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* Or it holds the Snowflake EXTERNAL VOLUME an earlier release created for a Snowflake Iceberg table in an external catalog. See [Iceberg catalog-move guard](#iceberg-catalog-move-guard). | | `opentofu_dataset_access_unreconciled` | State holds a BigQuery dataset an older forge-cli applied with an authoritative access list, and every entry of it is a grant the contract no longer makes. Revoke those entries on the dataset by hand, or keep one of them in `accessPolicy` for one apply. | | `apply_state_backend_invalid`, `state_shared_with_another_provider`, `state_migration_*` | See [Remote state](#remote-state). | | `generate_iac_provider_mismatch` | `--provider` contradicts the contract's binding. | diff --git a/docs/cli/diff.md b/docs/cli/diff.md index 970c1cc..b2d127d 100644 --- a/docs/cli/diff.md +++ b/docs/cli/diff.md @@ -105,7 +105,7 @@ For each expose, `diff` reads the target the way `apply` writes it and compares - AWS Glue tables; - BigQuery tables. -An expose bound to anything else, such as Snowflake, is reported `not_checked`, and so is an expose whose contract declares no schema. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* So is an AWS Iceberg expose whose `location.catalog` names a catalog other than Glue, with the detail `table lives in Iceberg catalog ; Glue is not inspected`: `fluid apply` creates no Glue table for it, and a Glue table of the same name would belong to someone else. On 0.19.0 and earlier, `diff` looked such a table up in Glue. *(forge-cli fix/iceberg-catalog-followups, unreleased)* On AWS, a `location.catalog` value FLUID does not know is refused before anything is compared, because the AWS planner `diff` runs cannot tell whether the table is in Glue: +An expose bound to anything else, such as Snowflake, is reported `not_checked`, and so is an expose whose contract declares no schema. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* So is an AWS Iceberg expose whose `location.catalog` names a catalog other than Glue, with the detail `table lives in Iceberg catalog ; Glue is not inspected`: `fluid apply` creates no Glue table for it, and a Glue table of the same name would belong to someone else. On 0.19.0 and earlier, `diff` looked such a table up in Glue. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* On AWS, a `location.catalog` value FLUID does not know is refused before anything is compared, because the AWS planner `diff` runs cannot tell whether the table is in Glue: ```text ❌ diff_failed [ERR_DIFF_FAILED] diff --git a/docs/cli/generate-iac.md b/docs/cli/generate-iac.md index 0dda48a..e70af01 100644 --- a/docs/cli/generate-iac.md +++ b/docs/cli/generate-iac.md @@ -126,7 +126,7 @@ Principals are refused when they cannot be real identities. A principal in a res remediation: ["Map the logical principal in this environment's binding.principals to the real group or service account it stands for, ... ``` -*(forge-cli fix/iceberg-catalog-followups, unreleased)* The `error:` line names the expose, `exposes[customers]`, and a value is no longer wrapped at the terminal width. 0.19.0 and earlier print `exposes accessPolicy: ...`, without the expose. +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The `error:` line names the expose, `exposes[customers]`, and a value is no longer wrapped at the terminal width. 0.19.0 and earlier print `exposes accessPolicy: ...`, without the expose. Other refusal kinds exist for principals and governance fields: `principal-unmapped`, `column-restriction-*` and `encryption-kms-*`. See the [GCP provider page](../providers/gcp.md) for the provider's own behaviour. @@ -174,7 +174,7 @@ An Iceberg expose needs infrastructure dbt refuses to create, and the plugins de The emitters are emit-when-derivable: an Iceberg expose missing a required input (an S3 warehouse without `iam_role_arn`, no derivable bucket name, an illegal `external_volume` override) used to emit **nothing, silently** — the failure surfaced only at `dbt run`. Since `0.14.0`, [`fluid validate`](./validate.md#iceberg-prerequisite-checks-since-0-14-0) errors on exactly those cases before any emit. One `--strict` caveat: a Snowflake catalog whose auth is secret-bearing (`polaris` / `unity` / `rest` / `nessie`) draws a *warning* — FLUID does not emit those catalog integrations because the emitted module is credential-free — and `fluid validate --strict` promotes it to an error. -With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unreleased), every plugin reads `location.catalog` through one table of catalog kinds. The Snowflake warning then covers every catalog Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`), and `hive`, `jdbc`, `hadoop` or `dynamodb` on Snowflake is an error. The `aws` plugin keeps an Iceberg expose's S3 bucket but creates no Glue database or table for it unless its catalog is Glue, and `fluid apply` stops with `iceberg_catalog_move_blocked` when state holds Glue resources an earlier release created for such a table. The `snowflake` plugin creates no EXTERNAL VOLUME for an Iceberg expose in a catalog Snowflake does not manage, and *(forge-cli fix/iceberg-catalog-followups, unreleased)* `fluid apply` stops the same way when state holds the volume an earlier release created for a `lakekeeper`, `bigquery` or `iceberg-rest` expose; see [Iceberg catalog-move guard](./apply.md#iceberg-catalog-move-guard). See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). +With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unreleased), every plugin reads `location.catalog` through one table of catalog kinds. The Snowflake warning then covers every catalog Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`), and `hive`, `jdbc`, `hadoop` or `dynamodb` on Snowflake is an error. The `aws` plugin keeps an Iceberg expose's S3 bucket but creates no Glue database or table for it unless its catalog is Glue, and `fluid apply` stops with `iceberg_catalog_move_blocked` when state holds Glue resources an earlier release created for such a table. The `snowflake` plugin creates no EXTERNAL VOLUME for an Iceberg expose in a catalog Snowflake does not manage, and *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* `fluid apply` stops the same way when state holds the volume an earlier release created for a `lakekeeper`, `bigquery` or `iceberg-rest` expose; see [Iceberg catalog-move guard](./apply.md#iceberg-catalog-move-guard). See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). ## Packaging modes diff --git a/docs/cli/validate.md b/docs/cli/validate.md index 2c045ff..7dc5d1c 100644 --- a/docs/cli/validate.md +++ b/docs/cli/validate.md @@ -35,7 +35,7 @@ Validation Errors: 2. exposes[exports] is a GCP gcs_bucket binding and declares lifecycle.expire, which the GCP emitter applies only to BigQuery tables. It would not be enforced. Move the policy to a BigQuery table expose, or remove it from this one. ``` -*(forge-cli fix/iceberg-catalog-followups, unreleased)* A finding prints its square brackets, so each one names its expose: `exposes[customers]`. forge-cli 0.19.0 and earlier read bracketed text as console markup and dropped it, so the same findings print as `exposes accessPolicy: ...` and `exposes is a GCP gcs_bucket binding ...`, and a [bundle finding](#validating-a-bundle-tgz) loses its `[error]` or `[warning]` severity. `--format json` was not affected. Since the fix, a finding is printed on one line and the terminal wraps it. +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* A finding prints its square brackets, so each one names its expose: `exposes[customers]`. forge-cli 0.19.0 and earlier read bracketed text as console markup and dropped it, so the same findings print as `exposes accessPolicy: ...` and `exposes is a GCP gcs_bucket binding ...`, and a [bundle finding](#validating-a-bundle-tgz) loses its `[error]` or `[warning]` severity. `--format json` was not affected. Since the fix, a finding is printed on one line and the terminal wraps it. More invocations: @@ -292,7 +292,7 @@ With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unre ## Iceberg catalog checks ::: warning Not in a release yet -These checks come with [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707), which no release includes yet, and with the follow-up fixes in forge-cli fix/iceberg-catalog-followups, marked *(forge-cli fix/iceberg-catalog-followups, unreleased)*. +These checks come with [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707), which no release includes yet, and with the follow-up fixes in [forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), marked *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)*. ::: `binding.location.catalog` names the Iceberg catalog that owns an expose's table, and every emitter now reads it through one table. [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog) has that table, how spellings fold, and a worked Lakekeeper example. `fluid validate` refuses a contract whose catalog the emitters would disagree about: @@ -303,15 +303,15 @@ These checks come with [forge-cli #707](https://github.com/Agenticstiger/forge-c - a `sink.catalog` that names another catalog than the expose is an error, because dbt and the modules read only the expose. - an `iceberg_catalog_overrides` entry or a hand-written sink config that would leave the connector with both `type` and `catalog-impl` is an error, because Apache Iceberg refuses that catalog and the sink never starts. - `nessie` on a `kafka-connect` build is a warning: the stock Apache Iceberg Kafka Connect runtime has no Nessie client. - - *(forge-cli fix/iceberg-catalog-followups, unreleased)* `bigquery` on a `kafka-connect` build is a warning: the sink config sets `iceberg.catalog.type=bigquery`, which the published Apache Iceberg Kafka Connect sink (1.9.2) cannot load; Iceberg adds the type in 1.10. This warning and the Nessie one follow the catalog type that reaches the worker, so a hand-written `sink_connector_config` that sets `iceberg.catalog.type: rest` draws neither. - - *(forge-cli fix/iceberg-catalog-followups, unreleased)* on `platform: gcp`, an Iceberg expose the build writes must name its `location.catalog`, or it is an error: the sink would write through a REST catalog while dbt-bigquery and the GCP module create a BigLake table. See [How a value is read](../advanced/source-aligned-acquisition.md#how-a-value-is-read). + - *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* `bigquery` on a `kafka-connect` build is a warning: the sink config sets `iceberg.catalog.type=bigquery`, which the published Apache Iceberg Kafka Connect sink (1.9.2) cannot load; Iceberg adds the type in 1.10. This warning and the Nessie one follow the catalog type that reaches the worker, so a hand-written `sink_connector_config` that sets `iceberg.catalog.type: rest` draws neither. + - *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* on `platform: gcp`, an Iceberg expose the build writes must name its `location.catalog`, or it is an error: the sink would write through a REST catalog while dbt-bigquery and the GCP module create a BigLake table. See [How a value is read](../advanced/source-aligned-acquisition.md#how-a-value-is-read). - **Snowflake.** On `platform: snowflake`, a catalog that Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`) draws the warning described above, which `--strict` turns into an error. `hive`, `jdbc`, `hadoop` and `dynamodb` are an error, because Snowflake has no catalog integration for them. A `lakekeeper` expose is no longer asked for an `s3://` or `gs://` warehouse, and two `catalog: snowflake` exposes that derive one EXTERNAL VOLUME on different storage are caught here rather than failing `fluid apply` mid-emit. -- **AWS.** An Iceberg expose in a catalog other than Glue cannot carry `governance.lakeFormation`, `policy.authz.columnRestrictions` or `policy.authz.rowFilters`: each is refused by catalog name. `accessPolicy.grants` on it draws a warning to grant access in that catalog. See [On AWS](../advanced/source-aligned-acquisition.md#on-aws-a-table-in-another-catalog). *(forge-cli fix/iceberg-catalog-followups, unreleased)* An unknown catalog value draws only the unknown-catalog error, not these refusals as well. +- **AWS.** An Iceberg expose in a catalog other than Glue cannot carry `governance.lakeFormation`, `policy.authz.columnRestrictions` or `policy.authz.rowFilters`: each is refused by catalog name. `accessPolicy.grants` on it draws a warning to grant access in that catalog. See [On AWS](../advanced/source-aligned-acquisition.md#on-aws-a-table-in-another-catalog). *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* An unknown catalog value draws only the unknown-catalog error, not these refusals as well. - **GCP.** A `platform: gcp` expose in a catalog other than `bigquery` may give the catalog's warehouse name; a warehouse in another object store (`s3://`, `abfss://`) is an error. A crash inside the Iceberg sink, Confluent or Iceberg prerequisite check now fails validation, with an error such as `Iceberg sink check could not run (...); the contract was NOT checked for streaming-sink defects`. On 0.19.0 and earlier the crash was printed only with `--verbose`, and the contract passed unchecked. -The Kafka Connect runner and the embedded Debezium Server runner run the same streaming-sink checks before they create anything, so a contract that skipped `fluid validate` still fails before any Connect REST call or `application.properties` write. *(forge-cli fix/iceberg-catalog-followups, unreleased)* That holds for a build that declares `sink.format: iceberg`, with a derived or a hand-written sink config, and for an embedded Debezium Server build that derives its sink. On #707 alone a hand-written config was not checked by the runners. [What each kind produces](../advanced/source-aligned-acquisition.md#what-each-kind-produces) lists the builds. +The Kafka Connect runner and the embedded Debezium Server runner run the same streaming-sink checks before they create anything, so a contract that skipped `fluid validate` still fails before any Connect REST call or `application.properties` write. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* That holds for a build that declares `sink.format: iceberg`, with a derived or a hand-written sink config, and for an embedded Debezium Server build that derives its sink. On #707 alone a hand-written config was not checked by the runners. [What each kind produces](../advanced/source-aligned-acquisition.md#what-each-kind-produces) lists the builds. ## GCP binding checks (since 0.15.0) diff --git a/docs/providers/aws.md b/docs/providers/aws.md index 6c69278..cf65798 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -950,7 +950,7 @@ The release notes for [`0.16.0`](../RELEASE_NOTES_0.16.0.md) and [`0.17.0`](../R - **`0.17.0`:** a column restriction on an aws binding with no Lake Formation grants is refused, and `accessPolicy.grants` on an aws binding with no Lake Formation grants warns; see [Column restrictions](#column-restrictions) and [accessPolicy on AWS](#accesspolicy-on-aws). - **`0.18.0`:** contract SQL runs in a DuckDB sandbox and `$ref` stays inside the contract's directory tree; see [Where a build lands data](#where-a-build-lands-data). - **Unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707):** an Iceberg expose whose `location.catalog` is not `glue` keeps its S3 bucket and loses its Glue database and table. On a contract an earlier release applied, `fluid apply` stops with `iceberg_catalog_move_blocked` and prints the `tofu state rm` commands that release them; see [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). A Kafka Connect sink on Glue starts, where it failed at startup before. -- **Unreleased, forge-cli fix/iceberg-catalog-followups:** the `iceberg_catalog_move_blocked` guard flags an expose's Glue database and table only when that expose's own Glue table is in state, so a Glue database left by a removed expose no longer blocks the apply. When the guard cannot read the state, it logs an `iceberg_catalog_move_probe_skipped` WARNING. `fluid diff` refuses an Iceberg `location.catalog` value FLUID does not know, as `fluid validate` and `fluid apply` do, and a typo no longer also draws a Lake Formation refusal. See [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). +- **Unreleased, [forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709):** the `iceberg_catalog_move_blocked` guard flags an expose's Glue database and table only when that expose's own Glue table is in state, so a Glue database left by a removed expose no longer blocks the apply. When the guard cannot read the state, it logs an `iceberg_catalog_move_probe_skipped` WARNING. `fluid diff` refuses an Iceberg `location.catalog` value FLUID does not know, as `fluid validate` and `fluid apply` do, and a typo no longer also draws a Lake Formation refusal. See [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). ## How far this has been exercised diff --git a/docs/providers/gcp.md b/docs/providers/gcp.md index 4ac3496..bc56e98 100644 --- a/docs/providers/gcp.md +++ b/docs/providers/gcp.md @@ -321,7 +321,7 @@ Retention, keys and column restrictions are refused on an Iceberg binding: they **A table in another catalog.** *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* BigLake metastore is the catalog this section describes, and the one an Iceberg expose gets with no `location.catalog` or with `catalog: bigquery`. An expose that names another catalog (`lakekeeper`, `rest`, `nessie`, ...) is not a BigLake table: `catalogs.yml` leaves it out, with a warning, instead of having dbt write a second table under the name that catalog owns. `fluid validate` accepts that catalog's warehouse name (`warehouse: analytics`), and refuses a warehouse in another object store (`s3://`, `abfss://`), since a `platform: gcp` table cannot live there. A streaming sink with no `location.catalog` on GCP writes through a REST catalog, not BigLake. See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). On 0.19.0 and earlier, an expose naming another catalog became a BigLake table, and a warehouse name failed validation. -**A streaming sink must name the catalog.** *(forge-cli fix/iceberg-catalog-followups, unreleased)* An Iceberg expose that a Kafka Connect build or an embedded Debezium Server build writes must set `location.catalog`: `bigquery` for BigLake metastore, or the REST kind your catalog is. Without it, `fluid validate` fails, and so does the run, before it creates anything: +**A streaming sink must name the catalog.** *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* An Iceberg expose that a Kafka Connect build or an embedded Debezium Server build writes must set `location.catalog`: `bigquery` for BigLake metastore, or the REST kind your catalog is. Without it, `fluid validate` fails, and so does the run, before it creates anything: ```text 1. iceberg sink (build 'stream_events'): the GCP Iceberg expose sets no binding.location.catalog, so it is read two ways: the sink would write through a REST catalog (the 'gcp' platform default) while dbt-bigquery and the GCP IaC, which read only binding.location.catalog, create a BigLake metastore table. Set binding.location.catalog: bigquery, or the REST kind your catalog is (e.g. rest, lakekeeper) @@ -519,7 +519,7 @@ On GCP, a principal in a reserved top-level domain (`.example`, `.test`, `.inval 1. exposes[orders] accessPolicy: principal 'group:data-analysts@company.example' resolves to 'group:data-analysts@company.example', a placeholder: .example is a reserved top-level domain (RFC 2606), so no real identity has it, and BigQuery refuses an access entry for an identity that does not exist. ... ``` -*(forge-cli fix/iceberg-catalog-followups, unreleased)* The message names the expose, `exposes[orders]`. 0.19.0 and earlier print `exposes accessPolicy: ...`, without it. +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The message names the expose, `exposes[orders]`. 0.19.0 and earlier print `exposes accessPolicy: ...`, without it. Map such a logical principal to a real identity in [`binding.principals`](#logical-principals-binding-principals-0-7-6), or to `[]` when it has no identity on GCP. diff --git a/docs/providers/snowflake.md b/docs/providers/snowflake.md index 7b31072..858c93d 100644 --- a/docs/providers/snowflake.md +++ b/docs/providers/snowflake.md @@ -596,7 +596,7 @@ Other keys under `binding.properties` (`table_type`, `data_retention_time_in_day An expose with `binding.format: iceberg` on `platform: snowflake` closes the dbt Iceberg loop in two halves: - **`fluid generate transformation`** emits dbt's `catalogs.yml` (v1 catalogs schema, Snowflake adapter) into the generated project. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* Catalogs external to Snowflake (`location.catalog: glue`, `rest` or `iceberg_rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`) map to `catalog_type: iceberg_rest`. No `location.catalog`, or `snowflake` (`snowflake-managed`), is Snowflake-managed (Horizon) and maps to `built_in`. `hive`, `jdbc`, `hadoop` and `dynamodb`, which Snowflake has no catalog integration for, are left out of the file with a warning, and `fluid validate` refuses them. Any other value is refused by `fluid validate`. The full table, and how spellings fold, is in [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog). On 0.19.0 and earlier, only `glue`, `polaris`, `unity`, `rest`, `iceberg_rest` and `nessie` mapped to `iceberg_rest`, and anything else, `lakekeeper` and `hive` included, became `built_in`. -- **`fluid apply`** provisions the prerequisites dbt refuses to create: the **EXTERNAL VOLUME** for Snowflake-managed catalogs (needs an `s3://` or `gs://` `location.warehouse`, plus `location.iam_role_arn` for S3), and the **AWS Glue CATALOG INTEGRATION** for `location.catalog: glue` (needs `location.iam_role_arn` — the role Snowflake assumes — and `location.account`, the AWS account id). It creates nothing for the other `iceberg_rest` catalogs listed above: their integrations authenticate with a secret, and the emitted module is credential-free. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* On 0.19.0 and earlier, a `lakekeeper` expose got an EXTERNAL VOLUME, and `fluid validate` asked it for an `s3://` or `gs://` warehouse. *(forge-cli fix/iceberg-catalog-followups, unreleased)* While state still holds such a volume, `fluid apply` stops before it plans; see [Upgrading an Iceberg expose in an external catalog](#upgrading-an-iceberg-expose-in-an-external-catalog). +- **`fluid apply`** provisions the prerequisites dbt refuses to create: the **EXTERNAL VOLUME** for Snowflake-managed catalogs (needs an `s3://` or `gs://` `location.warehouse`, plus `location.iam_role_arn` for S3), and the **AWS Glue CATALOG INTEGRATION** for `location.catalog: glue` (needs `location.iam_role_arn` — the role Snowflake assumes — and `location.account`, the AWS account id). It creates nothing for the other `iceberg_rest` catalogs listed above: their integrations authenticate with a secret, and the emitted module is credential-free. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* On 0.19.0 and earlier, a `lakekeeper` expose got an EXTERNAL VOLUME, and `fluid validate` asked it for an `s3://` or `gs://` warehouse. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* While state still holds such a volume, `fluid apply` stops before it plans; see [Upgrading an Iceberg expose in an external catalog](#upgrading-an-iceberg-expose-in-an-external-catalog). ```yaml exposes: @@ -619,7 +619,7 @@ Since `0.14.0`, `fluid validate` errors when an Iceberg expose is missing one of ### Upgrading an Iceberg expose in an external catalog -*(forge-cli fix/iceberg-catalog-followups, unreleased)* 0.19.0 and earlier created an EXTERNAL VOLUME for an Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest`, and dbt wrote a Snowflake-managed table onto it. `hive`, `jdbc`, `hadoop` and `dynamodb` got one too; `fluid validate` now refuses them. The module now creates no volume for these catalogs. When the contract's OpenTofu state still holds the volume, the plan would drop it, so `fluid apply` stops before it plans (abridged): +*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* 0.19.0 and earlier created an EXTERNAL VOLUME for an Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest`, and dbt wrote a Snowflake-managed table onto it. `hive`, `jdbc`, `hadoop` and `dynamodb` got one too; `fluid validate` now refuses them. The module now creates no volume for these catalogs. When the contract's OpenTofu state still holds the volume, the plan would drop it, so `fluid apply` stops before it plans (abridged): ```text ❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED]