Skip to content

docs(iceberg): document the forge-cli catalog follow-up fixes (fix/iceberg-catalog-followups) - #145

Merged
fas89 merged 2 commits into
mainfrom
docs/iceberg-catalog-followups
Oct 7, 2026
Merged

fas89 merged 2 commits into
mainfrom
docs/iceberg-catalog-followups

Conversation

@fas89

@fas89 fas89 commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Companion docs for the forge-cli fix wave on branch fix/iceberg-catalog-followups (head b87fd3a7, "fix(iceberg): close the gaps the post-merge inspection of #707 found"). It follows #144, which documented forge-cli #707.

The fix wave is in no release, and its forge-cli PR has no number yet. Each new statement is marked (forge-cli fix/iceberg-catalog-followups, unreleased). When the PR number is known, replace the substring forge-cli fix/iceberg-catalog-followups with [forge-cli #NNN](https://github.com/Agenticstiger/forge-cli/pull/NNN) across docs/.

Related CLI PR

  • CLI branch: forge-cli fix/iceberg-catalog-followups (PR number to be added)

Pages changed

Page Change
advanced/source-aligned-acquisition.md Regenerated every quoted output. The Lake Formation refusal now reads exposes[orders] declares .... Narrowed the "one table" sentence: the Snowflake module reads location.catalog only for the EXTERNAL VOLUME and the Glue catalog integration, and still emits snowflake_database/snowflake_schema/snowflake_table whatever the catalog. The GCP no-catalog case is now an error, quoted. Added the bigquery Kafka Connect warning (table row and quote). States exactly which builds the runner preflight covers, and that each runner reads the build it runs. fluid diff's planner refuses a typo, and a typo draws no Lake Formation refusal. Rewrote the AWS guard output and added the Glue-table evidence rule and the iceberg_catalog_move_probe_skipped WARNING. New heading Upgrading a Snowflake contract that names another catalog.
cli/apply.md Iceberg catalog-move guard: regenerated AWS output, the evidence rule, the Snowflake volume path (message and tofu state rm shape), and the probe WARNING. The iceberg_catalog_move_blocked error row covers Snowflake.
providers/snowflake.md New heading Upgrading an Iceberg expose in an external catalog, with the real message and the tofu state rm command. A pointer to it from the fluid apply bullet.
providers/gcp.md A GCP Iceberg expose that a streaming sink writes must name its catalog (error quoted), plus the catalog: bigquery Kafka Connect warning. Regenerated the placeholder-principal output (exposes[orders]).
providers/aws.md Regenerated the accessPolicy.grants warning. A new "Upgrading" bullet: the evidence rule, the probe WARNING, and fluid diff refusing an unknown catalog.
cli/validate.md Regenerated the first example and the bundle examples (exposes[customers], [error] severity), with a note on what 0.19.0 and earlier print. Iceberg catalog checks: the bigquery warning, the GCP rule, the unknown-catalog/Lake Formation change, and the preflight scope.
cli/generate-iac.md Regenerated the principal-placeholder output (exposes[customers]). The Snowflake volume guard, in the #707 paragraph.
cli/diff.md fluid diff on AWS refuses an unknown location.catalog (diff_failed), with the real message.
advanced/production-troubleshooting.md Rows for iceberg_catalog_move_blocked and iceberg_catalog_move_probe_skipped in "State and region errors". This is the CLI's fallback docs page (_DOC_FALLBACK in fluid_build/_errors.py).

No page moved, and no heading was renamed or removed. forge_docs URLs and routed anchors are a CLI contract (_DOC_ROUTES). This PR adds two headings: one in source-aligned-acquisition.md and one in snowflake.md.

How the claims were checked

Every behavioural claim traces to the CHANGELOG [Unreleased] entries or the code at b87fd3a7. The "0.19.0 and earlier" statements were checked against the v0.19.0 tag, which is an ancestor of b87fd3a7. Every quoted output below was produced by forge-cli at b87fd3a7 (editable install in the forge-cli worktree's .venv):

  • fluid validate: the Lakekeeper example and its variants (no uri, typo, sink.catalog: glue, governance.lakeFormation, typo plus Lake Formation, accessPolicy, override with both selectors). Also a GCP Kafka Connect build with no catalog, the same with catalog: bigquery, and the governance example with its two bundles (fluid bundle --format tgz).
  • fluid generate iac: the placeholder-principal refusal.
  • fluid diff on AWS: the unknown-catalog refusal.
  • AWS guard: fluid apply against a moto server with real tofu 1.12.0. The contract was applied with the Glue catalog first, then switched to catalog: lakekeeper. The guard fired, and the printed tofu state rm lines were run as copied. The next apply then planned past the guard, and the Glue table stayed in moto.
  • Snowflake guard: fluid apply with real tofu and the real Snowflake provider (init only). There is no Snowflake account, so the state file was written by hand. It holds the four addresses fluid generate iac emits for the same contract without location.catalog. The guard fired, the printed command removed the volume, and the next apply got past the guard (its plan then failed on missing credentials, as expected). The same setup with catalog: hive also fires the guard.
  • iceberg_catalog_move_probe_skipped WARNING: from fluid apply run in-process. The only stub replaced the tofu shell-outs, with tofu state pull failing (AccessDenied).
  • Native table: the snowflake_table claim was read from fluid generate iac for catalog: lakekeeper.

The commit's own tests pass at b87fd3a7: 246 tests across the sink validation, deriver, Debezium, catalog-move, wiring, bracket-output, planner and unknown-catalog files.

Areas Updated

  • Getting Started
  • CLI Reference
  • Provider Docs
  • Walkthroughs / Examples
  • Contributing / Community Docs
  • Other (advanced/source-aligned-acquisition, advanced/production-troubleshooting)

Checklist

  • I previewed the change locally with npm run docs:dev or built it with npm run docs:build
  • I checked links, page titles, and navigation where relevant
  • Screenshots or command output are updated if needed (command output regenerated from forge-cli at b87fd3a7)

Checks run locally:

Check Result
npm ci && npm run docs:build pass (Node 25 locally; CI uses Node 20)
node scripts/check-dist-links.mjs clean: 137,793 references, 242 pages
base-prefix check (copied from link-check.yml) clean, 48 routes watched
lychee --offline source check (mirror and canaries from link-check.yml; lychee 0.24.2) 0 errors, canaries good=0 and bad=2
python scripts/check_cli_docs.py (incl. --version-only), with data-product-forge==0.18.1 pass
python scripts/check_providers.py pass
python scripts/gen_contract_reference.py --check pass
Every new or linked in-page anchor resolves to an id in the built HTML

Notes

  • Merge with, or after, the forge-cli fix PR. If that PR changes before it merges, these pages need the same change.
  • The forge-cli CHANGELOG's Snowflake upgrade note lists rest, iceberg_rest, polaris, unity and nessie among the catalogs that "got a Snowflake EXTERNAL VOLUME from earlier releases". The code (_SNOWFLAKE_NO_VOLUME_BEFORE in iac/catalog_moves.py) says those never got one, and the guard does not fire for them. It fires for lakekeeper, bigquery, the iceberg-rest spelling, and hive/jdbc/hadoop/dynamodb. These pages follow the code.
  • iceberg_catalog_move_blocked has no entry in forge-cli's error catalog, so the CLI prints no docs link for it. The troubleshooting row is there for a reader who searches the event name.
  • Out of scope, and still showing bracket-stripped output (exposes accessPolicy: ...): recipes/one-contract-two-clouds.md, concepts/governance-parity.md, concepts/governance-policy.md and reference/preview-fields.md.

fas89 added 2 commits October 7, 2026 13:10
Companion to forge-cli fix/iceberg-catalog-followups (b87fd3a7), which
closes the gaps the post-merge inspection of #707 found. Marked
"(forge-cli fix/iceberg-catalog-followups, unreleased)".

- Quoted CLI output regenerated from b87fd3a7: findings keep their
  square brackets (exposes[orders], [error]) and print on one line.
- The streaming runners check a hand-written sink config too, for a
  build that declares sink.format: iceberg, and read the build they run.
- GCP: an Iceberg expose a streaming sink writes must name its catalog.
- catalog: bigquery on Kafka Connect warns (published sink 1.9.2).
- Catalog-move guard: Snowflake EXTERNAL VOLUMEs, the Glue-table
  evidence rule, the iceberg_catalog_move_probe_skipped WARNING, and
  both events on the troubleshooting page.
- The Snowflake module still emits a native snowflake_table whatever
  the catalog; fluid diff refuses an unknown catalog value.
@fas89
fas89 merged commit fbf7d11 into main Oct 7, 2026
9 checks passed
fas89 added a commit that referenced this pull request Oct 7, 2026
…orge-cli #709) (#146)

Follow-up to #145 for forge-cli #709 at 6e6eb000 ("four gaps the defect
scan found in this PR's own checks"). Quoted output regenerated from
6e6eb000.

- Catalog-move guard: a Glue database is flagged when the moved expose's
  own Glue table is in state, or when the expose names no table (its
  database is all an earlier release created for it). The Snowflake
  volume is found by its contract-derived name, so an upgrade that also
  changed location.warehouse to a catalog name or dropped iam_role_arn
  is still stopped. (apply, source-aligned-acquisition, aws, snowflake,
  production-troubleshooting)
- GCP: an expose with no location.catalog is refused when any catalog
  other than BigLake reaches the worker, by type or by catalog-impl,
  Glue included (sink.catalog: glue). (gcp, validate,
  source-aligned-acquisition)
- New rule: a sink_connector_config, server.sink.config or
  iceberg_catalog_overrides whose type or catalog-impl selects another
  catalog than the expose's is an error on every platform; a REST
  endpoint fronting Glue is catalog: rest. The Nessie and BigQuery
  warnings follow a catalog-impl class too. (validate,
  source-aligned-acquisition, gcp)

This branch was successfully deployed

1 active deployment
github-pages — 71c2e482 Deployed Oct 7, 2026 by fas89 via deploy #179
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant