Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/advanced/production-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ Remote OpenTofu state is keyed per contract and provider. See [Environment varia
| `state_migration_unverified` | After the copy, the new key holds different resources than the old one. The old object is untouched | Compare the two states before re-running |
| `state_migration_probe_failed` | The old key could not be read | Check access to the bucket and the key named in the message |
| `opentofu_region_moved` | State holds this contract's resources in a region other than the one the bindings now name. Applying would create them again and leave the originals unmanaged | If they should stay, set the binding's `location.region` to the region the error names. If they should move, empty and remove them there first (`tofu destroy` in the state directory named, with `AWS_REGION` set to the old region), then apply again |
| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) and [forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709))* An Iceberg expose's `location.catalog` names a catalog an earlier release did not honour, and state still holds what that release created for it: a Glue database and table on AWS, or an EXTERNAL VOLUME on Snowflake. The plan would destroy them, and destroying a Glue database deletes every table in it | Run the `tofu -chdir=<module directory> state rm <address>` commands the error prints, which change nothing in the cloud, then apply again. Do not pass `--allow-data-loss`. If the table belongs in Glue or in Snowflake's own catalog, remove `location.catalog` instead. Steps: [Upgrading an AWS contract that names another catalog](./source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog), [Upgrading a Snowflake contract that names another catalog](./source-aligned-acquisition.md#upgrading-a-snowflake-contract-that-names-another-catalog) |
| `iceberg_catalog_move_probe_skipped` (WARNING) | *(unreleased, [forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709))* The catalog-move guard could not read the state, or its check failed, so the apply went on without it. The `reason` field says why | If the plan then destroys Glue resources or an EXTERNAL VOLUME of the exposes the warning names, stop: release them with `tofu state rm` and apply again, rather than passing `--allow-data-loss` to the data-loss gate |

## Contract load and overlay errors

Expand Down
136 changes: 88 additions & 48 deletions docs/advanced/source-aligned-acquisition.md

Large diffs are not rendered by default.

33 changes: 29 additions & 4 deletions docs/cli/apply.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,12 +270,37 @@ Without the guard `tofu` would find nothing in the new region, create the resour
```text
❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED]
kind: iceberg-catalog-move
error: iceberg catalog move blocked — this contract's OpenTofu state holds 2
Glue catalog resource(s) for Iceberg table(s) that now live in another catalog:
error: iceberg catalog move blocked — this contract's OpenTofu state holds 2 Glue catalog resource(s) for Iceberg table(s) that now live in another catalog:

exposes[orders]: location.catalog lakekeeper

aws_glue_catalog_database.bronze_orders_stream_streaming
aws_glue_catalog_table.bronze_orders_stream_streaming_orders
...
```

The message lists the addresses and a `tofu -chdir=.fluid/iac/aws/<safe-id> state rm <address>` command for each. Run them, which changes nothing in AWS, then apply again. No flag skips the guard. The steps, and what to delete by hand afterwards, are in [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog).
The message lists the addresses and a `tofu -chdir=.fluid/iac/aws/<safe-id> state rm <address>` command for each. Run them, which changes nothing in AWS, then apply again. No flag skips the guard. The steps, and what to delete by hand afterwards, are in [Upgrading an AWS contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-an-aws-contract-that-names-another-catalog). *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The expose's resources are flagged only when its own Glue table is in state, so a Glue database that a since-removed expose created is left to the [data-loss gate](#opentofu-data-loss-gate) instead of blocking the apply.

*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The guard covers Snowflake too. 0.19.0 and earlier created a Snowflake EXTERNAL VOLUME for a `platform: snowflake` Iceberg expose whose `location.catalog` is `lakekeeper` or `bigquery`, or is spelled `iceberg-rest` (and for `hive`, `jdbc`, `hadoop` and `dynamodb`, which `fluid validate` now refuses). The module no longer creates one for those catalogs, so when state still holds the volume, the apply stops instead of dropping it:

```text
❌ iceberg_catalog_move_blocked [ERR_ICEBERG_CATALOG_MOVE_BLOCKED]
kind: iceberg-catalog-move
error: iceberg catalog move blocked — this contract's OpenTofu state holds 1 Snowflake EXTERNAL VOLUME(s) for Iceberg table(s) that now live in another catalog:

exposes[orders_iceberg]: location.catalog lakekeeper

snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL
...
Drop each from this contract's state, then re-run apply:

tofu -chdir=.fluid/iac/snowflake/sales_orders_lake state rm snowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL
...
```

Run the printed `tofu -chdir=.fluid/iac/snowflake/<safe-id> state rm snowflake_external_volume.<name>`, which changes nothing in Snowflake, then apply again. Drop the volume by hand (`DROP EXTERNAL VOLUME`) only once no Iceberg table uses it: a Snowflake-managed table an earlier dbt run wrote onto it still does. See [Upgrading a Snowflake contract that names another catalog](../advanced/source-aligned-acquisition.md#upgrading-a-snowflake-contract-that-names-another-catalog).

*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* When the guard cannot read the state, or its check fails, the apply goes on to plan, and logs an `iceberg_catalog_move_probe_skipped` WARNING with the reason and the exposes. The data-loss gate still stops a plan that destroys the resources; release them with `tofu state rm` rather than passing `--allow-data-loss`. On #707 alone a failed check was logged at debug level.

## Remote state

Expand Down Expand Up @@ -494,7 +519,7 @@ Each failure prints the typed event name and an `[ERR_<EVENT>]` slug, and exits
| `opentofu_data_loss_gate` | An OpenTofu plan destroys a resource and `--allow-data-loss` is not set. |
| `opentofu_region_moved` | State holds the contract's resources in another region than its bindings name. |
| `packaging_transition_blocked` | A container's ownership flips under existing state. See [Packaging modes](#packaging-modes). |
| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707))* State holds Glue resources an earlier release created for an AWS Iceberg table that now lives in another catalog. See [Iceberg catalog-move guard](#iceberg-catalog-move-guard). |
| `iceberg_catalog_move_blocked` | *(unreleased, [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707))* State holds Glue resources an earlier release created for an AWS Iceberg table that now lives in another catalog. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* Or it holds the Snowflake EXTERNAL VOLUME an earlier release created for a Snowflake Iceberg table in an external catalog. See [Iceberg catalog-move guard](#iceberg-catalog-move-guard). |
| `opentofu_dataset_access_unreconciled` | State holds a BigQuery dataset an older forge-cli applied with an authoritative access list, and every entry of it is a grant the contract no longer makes. Revoke those entries on the dataset by hand, or keep one of them in `accessPolicy` for one apply. |
| `apply_state_backend_invalid`, `state_shared_with_another_provider`, `state_migration_*` | See [Remote state](#remote-state). |
| `generate_iac_provider_mismatch` | `--provider` contradicts the contract's binding. |
Expand Down
9 changes: 8 additions & 1 deletion docs/cli/diff.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,14 @@ For each expose, `diff` reads the target the way `apply` writes it and compares
- AWS Glue tables;
- BigQuery tables.

An expose bound to anything else, such as Snowflake, is reported `not_checked`, and so is an expose whose contract declares no schema. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* So is an AWS Iceberg expose whose `location.catalog` names a catalog other than Glue, with the detail `table lives in Iceberg catalog <kind>; Glue is not inspected`: `fluid apply` creates no Glue table for it, and a Glue table of the same name would belong to someone else. On 0.19.0 and earlier, `diff` looked such a table up in Glue. Each expose ends in exactly one status:
An expose bound to anything else, such as Snowflake, is reported `not_checked`, and so is an expose whose contract declares no schema. *(forge-cli [#707](https://github.com/Agenticstiger/forge-cli/pull/707), unreleased)* So is an AWS Iceberg expose whose `location.catalog` names a catalog other than Glue, with the detail `table lives in Iceberg catalog <kind>; Glue is not inspected`: `fluid apply` creates no Glue table for it, and a Glue table of the same name would belong to someone else. On 0.19.0 and earlier, `diff` looked such a table up in Glue. *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* On AWS, a `location.catalog` value FLUID does not know is refused before anything is compared, because the AWS planner `diff` runs cannot tell whether the table is in Glue:

```text
❌ diff_failed [ERR_DIFF_FAILED]
error: exposes[orders] names Iceberg catalog 'lakekeper' (location.catalog), which FLUID does not know, so it cannot tell whether the table belongs in AWS Glue, and planning it would provision no Glue database or table. Use one of: bigquery, dynamodb, glue, hadoop, hive, jdbc, lakekeeper, nessie, polaris, rest, snowflake-managed, unity, iceberg-rest, snowflake. Or remove location.catalog to keep the table in the Glue catalog.
```

Each expose ends in exactly one status:

| Status | Meaning | Counts as drift |
| --- | --- | --- |
Expand Down
12 changes: 5 additions & 7 deletions docs/cli/generate-iac.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,14 +122,12 @@ Principals are refused when they cannot be real identities. A principal in a res
```text
❌ unsupported_binding [ERR_UNSUPPORTED_BINDING]
kind: principal-placeholder
error: exposes accessPolicy: principal 'group:data-platform@northwind.example'
resolves to 'group:data-platform@northwind.example', a placeholder: .example is
a reserved top-level domain (RFC 2606), so no real identity has it, and BigQuery
refuses an access entry for an identity that does not exist.
remediation: ["Map the logical principal in this environment's
binding.principals to the real group or service account it stands for, ...
error: exposes[customers] accessPolicy: principal 'group:data-platform@northwind.example' resolves to 'group:data-platform@northwind.example', a placeholder: .example is a reserved top-level domain (RFC 2606), so no real identity has it, and BigQuery refuses an access entry for an identity that does not exist.
remediation: ["Map the logical principal in this environment's binding.principals to the real group or service account it stands for, ...
```

*([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* The `error:` line names the expose, `exposes[customers]`, and a value is no longer wrapped at the terminal width. 0.19.0 and earlier print `exposes accessPolicy: ...`, without the expose.

Other refusal kinds exist for principals and governance fields: `principal-unmapped`, `column-restriction-*` and `encryption-kms-*`. See the [GCP provider page](../providers/gcp.md) for the provider's own behaviour.

### Access grants on GCP *(0.17.0)*
Expand Down Expand Up @@ -176,7 +174,7 @@ An Iceberg expose needs infrastructure dbt refuses to create, and the plugins de

The emitters are emit-when-derivable: an Iceberg expose missing a required input (an S3 warehouse without `iam_role_arn`, no derivable bucket name, an illegal `external_volume` override) used to emit **nothing, silently** — the failure surfaced only at `dbt run`. Since `0.14.0`, [`fluid validate`](./validate.md#iceberg-prerequisite-checks-since-0-14-0) errors on exactly those cases before any emit. One `--strict` caveat: a Snowflake catalog whose auth is secret-bearing (`polaris` / `unity` / `rest` / `nessie`) draws a *warning* — FLUID does not emit those catalog integrations because the emitted module is credential-free — and `fluid validate --strict` promotes it to an error.

With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unreleased), every plugin reads `location.catalog` through one table of catalog kinds. The Snowflake warning then covers every catalog Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`), and `hive`, `jdbc`, `hadoop` or `dynamodb` on Snowflake is an error. The `aws` plugin keeps an Iceberg expose's S3 bucket but creates no Glue database or table for it unless its catalog is Glue, and `fluid apply` stops with `iceberg_catalog_move_blocked` when state holds Glue resources an earlier release created for such a table. See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog).
With [forge-cli #707](https://github.com/Agenticstiger/forge-cli/pull/707) (unreleased), every plugin reads `location.catalog` through one table of catalog kinds. The Snowflake warning then covers every catalog Snowflake reaches over Iceberg REST (`rest`, `lakekeeper`, `polaris`, `unity`, `nessie`, `bigquery`), and `hive`, `jdbc`, `hadoop` or `dynamodb` on Snowflake is an error. The `aws` plugin keeps an Iceberg expose's S3 bucket but creates no Glue database or table for it unless its catalog is Glue, and `fluid apply` stops with `iceberg_catalog_move_blocked` when state holds Glue resources an earlier release created for such a table. The `snowflake` plugin creates no EXTERNAL VOLUME for an Iceberg expose in a catalog Snowflake does not manage, and *([forge-cli #709](https://github.com/Agenticstiger/forge-cli/pull/709), unreleased)* `fluid apply` stops the same way when state holds the volume an earlier release created for a `lakekeeper`, `bigquery` or `iceberg-rest` expose; see [Iceberg catalog-move guard](./apply.md#iceberg-catalog-move-guard). See [Iceberg catalogs](../advanced/source-aligned-acquisition.md#iceberg-catalogs-location-catalog).

## Packaging modes

Expand Down
Loading
Loading