Repository navigation
docs(iceberg): correct the catalog-move, GCP and sink-config rules (forge-cli #709) - #146
Merged
Merged
Conversation
…orge-cli #709) Follow-up to #145 for forge-cli #709 at 6e6eb000 ("four gaps the defect scan found in this PR's own checks"). Quoted output regenerated from 6e6eb000. - Catalog-move guard: a Glue database is flagged when the moved expose's own Glue table is in state, or when the expose names no table (its database is all an earlier release created for it). The Snowflake volume is found by its contract-derived name, so an upgrade that also changed location.warehouse to a catalog name or dropped iam_role_arn is still stopped. (apply, source-aligned-acquisition, aws, snowflake, production-troubleshooting) - GCP: an expose with no location.catalog is refused when any catalog other than BigLake reaches the worker, by type or by catalog-impl, Glue included (sink.catalog: glue). (gcp, validate, source-aligned-acquisition) - New rule: a sink_connector_config, server.sink.config or iceberg_catalog_overrides whose type or catalog-impl selects another catalog than the expose's is an error on every platform; a REST endpoint fronting Glue is catalog: rest. The Nessie and BigQuery warnings follow a catalog-impl class too. (validate, source-aligned-acquisition, gcp)
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #145. Corrects the pages #145 wrote for forge-cli #709, whose branch
fix/iceberg-catalog-followupsnow ends at 6e6eb000 ("fix(iceberg): four gaps the defect scan found in this PR's own checks"). Same marker style: (forge-cli #709, unreleased). No page or heading moved or renamed.cli/apply.html#iceberg-catalog-move-guard, which the CLI routes to, is unchanged.What changed
1. Catalog-move guard (
cli/apply.md,advanced/source-aligned-acquisition.md,providers/aws.md,providers/snowflake.md,advanced/production-troubleshooting.md)location.databaseand nolocation.table. An earlier release created only the database for that expose, so the database is flagged when state holds it and the module no longer declares it. docs(iceberg): document the forge-cli catalog follow-up fixes (fix/iceberg-catalog-followups) #145 said "only when its own Glue table is in state". Code:catalog_moves._detect, which skips an expose only when it has evidence resources (own) and none of them is in state._snowflake_volume_before), so it also stops an upgrade whose edit changedlocation.warehouseto the catalog's warehouse name or removedlocation.iam_role_arn. docs(iceberg): document the forge-cli catalog follow-up fixes (fix/iceberg-catalog-followups) #145 said the volume is flagged "when a moved expose would have derived it".2. GCP "must name the catalog" (
providers/gcp.md,cli/validate.md,advanced/source-aligned-acquisition.md)typeor acatalog-implclass selects it (_reaching_catalog,_IMPL_CATALOGS). It refuses every catalog but BigLake, so Glue is refused too, fromsink.catalog: glueor from a hand-writtenGlueCatalogcatalog-impl. Quoted the real error forsink.catalog: glue.3. New rule: a hand-written selector must match the expose (
cli/validate.md,advanced/source-aligned-acquisition.md, plus one sentence inproviders/gcp.md)sink_connector_config,iceberg_catalog_overridesor embedded Debeziumserver.sink.configwhosetype/catalog-implselects another catalog than the expose's is an error on every platform. A REST endpoint that fronts Glue (Glue's Iceberg REST endpoint) iscatalog: rest. Quoted the real errors: AWS Glue default withiceberg.catalog.type: rest, and GCPcatalog: bigquerywithiceberg.catalog.type: rest.4. Guidance that contradicted these
iceberg.catalog.type: restfor abigqueryornessieexpose draws neither [warning]": still true for the warnings, but that config is now an error. Both pages now say so.catalog-implclass too (NessieCatalog,BigQueryMetastoreCatalog), not onlytype.production-troubleshooting.md: for a table-less expose, the guard flags a Glue database only, not a database and a table.Evidence
Built from the forge-cli worktree's
.venv/bin/fluid(FLUID Forge CLI v0.19.1.dev6, editable install at 6e6eb000). Every new quoted block is the exactfluid validateoutput for that contract. The guard claims were run throughdetect_catalog_moves/guard_catalog_moveswith the realAwsIacPlugin/SnowflakeIacPlugin:catalog: lakekeeper,database: streaming, notable, state = bucket + database:('aws_glue_catalog_database.bronze_orders_stream_streaming',)flagged.catalog: lakekeeper,warehouse: analytics(a name), noiam_role_arn, state = the volume: today's emit derives no volume, and the guard still flagssnowflake_external_volume.sales_orders_lake_vol_FLUID_SALES_ORDERS_LAKE_VOL.catalog: reston AWS with a hand-writtentype: restvalidates clean. Its module holds onlyaws_s3_bucket, andgovernance.lakeFormationon it is refused, which is what the "a table in another catalog" sentence says.iceberg_sink_preflightreturns the same errors for the AWS, GCP and Debezium contracts.Checks
npm ci&&npm run docs:build: exit 0 (236 pages; the only warning is vite's existing chunk-size notice)node scripts/check-dist-links.mjs: exit 0, canaries ok, 137,794 absolute references across 242 built pages all clean.github/workflows/link-check.yml: exit 0 ("Watching 48 top-level routes. All internal absolute body links carry the /forge_docs/ base.")#how-a-value-is-read,#on-aws-a-table-in-another-catalog,validate.html#iceberg-catalog-checks) exist in the built HTMLpackage*.jsonbumps)