Repository navigation
fix(iceberg): sinks write the expose their build names; dynamodb/jdbc/bigquery sinks start; Confluent reads as Glue; policy compile speaks up - #710
Merged
Conversation
…sh, and word the Snowflake volume guard without blaming an expose
RT-707-3: `fluid policy compile` wrote its warnings only into bindings.json
and logged a count at INFO (routed to DEBUG), so a grant that compiled to no
table grant printed nothing and exited 0. Each warning now goes to stderr
through the CLI warning helper, exit stays 0, and `fluid policy apply` echoes
the warnings stored in bindings.json. "No grants found in accessPolicy" (a
legitimate no-op) stays in the file and is not shown as a WARNING on compile,
apply or stage 3.
A crash inside compile_policy was swallowed into {"bindings": {"bindings": []}}
and exit 0. It now fails with the new policy_compiler_crashed error, writes
nothing, and names the schema type errors at the values the compiler reads
(the grants, their permissions, each binding and its location), or the
compiler's own exception when there are none. Schema errors elsewhere, such as
an extra key on a grant, are not named. The error routes to the policy compile
page's Errors section.
JRN-707-6: the Snowflake EXTERNAL VOLUME guard detects what it did before. Its
message no longer says forge-cli stopped creating the volume or lists the moved
exposes as the tables it was held for: the volume is keyed by the contract, so
the message names both possible causes (an upgrade from a release that gave a
non-Snowflake-managed catalog a volume, or a Snowflake-managed expose removed
or re-cataloged in this change) and keeps the tofu state rm remediation. The
AWS message, which has per-expose evidence, is unchanged.
… outputs pick Every Kafka Connect and embedded Debezium Iceberg sink resolved its table from exposes[0], so a second CDC build whose outputs named another Iceberg expose streamed into the first expose's table while validate, the preflight and the run all passed. A sink config forge-cli derives now resolves its exposes from build.outputs through resolve_iceberg_sink_exposes: - Kafka Connect derives one iceberg.tables entry: exactly one Iceberg sink expose in outputs, or no outputs and exactly one in the contract. Anything else is a hard validate error. - Embedded Debezium Server derives one table-namespace and one catalog: the exposes outputs pick (all of them with no outputs) must share one database and resolve to one catalog, else a hard error. Validate, the runner preflight, both runners and the Kafka Connect late-arrival target read iceberg_sink_target, which applies this join to derived configs. A hand-written config, or an override that sets iceberg.tables or table-namespace, is checked against the first Iceberg sink expose as before.
…d to start; confluent defaults to glue CON-3: Iceberg's DynamoDbCatalog and JdbcCatalog refuse to initialise without a warehouse, but the kind table asked nothing (dynamodb) or only uri (jdbc) of them, and the resolver ignored location.bucket for them. Both now require a warehouse. The resolver takes location.warehouse, else derives one from an explicit location.bucket (and path, as the Glue helper builds it) with the platform scheme (aws s3://, gcp gs://), never from the account-derived fallback bucket. When none can be derived, validate errors and the runner preflight refuses. Both the Kafka Connect and the embedded Debezium configs emit the warehouse. CON-4: BigQueryMetastoreCatalog refuses to initialise without gcp.bigquery.project-id. location.project now maps to it (a missing project is a validate error and a preflight refusal), location.region maps to gcp.bigquery.location instead of the AWS-only client.region, and the gs:// storage URI dbt-bigquery and the GCP IaC use becomes the warehouse, with a validate warning when none can be derived. The Kafka Connect deriver now forwards the resolver's catalog-specific properties under the iceberg.catalog. prefix, as the Debezium deriver already did. CON-5: default_catalog_kind returns glue for platform confluent, the catalog the Tableflow IaC publishes an absent-catalog table to. Policy compile now grants the Glue table instead of dropping every grant with a false "cataloged in 'rest'" warning, and dbt-snowflake keeps the Glue linked-database type.
…og needs to start The DynamoDB / JDBC warehouse and the BigQuery gcp.bigquery.project-id requirements read only binding.location, so validate and the preflight refused a config whose pushed form carried the key: a Kafka Connect iceberg_catalog_overrides or sink_connector_config entry, or a Debezium server.sink.config entry. Main accepted these. Check 4 now counts the key when an override map sets the catalog property under the runtime's prefix (iceberg.catalog.<prop> on Kafka Connect, bare on Debezium Server), and still errors when neither the binding nor an override supplies it. The refusal names the override remedy. The BigQuery no-warehouse warning understated the Kafka Connect case: its auto-create calls createNamespace before every createTable, and BigQueryMetastoreCatalog.createNamespace refuses without a warehouse, so table auto-creation fails even in a dataset that exists (apache-iceberg-1.10.0 IcebergWriterFactory.java:89, :122-137; BigQueryMetastoreCatalog.java:197-208, :298-303). The Debezium Server sink creates a namespace only when it is missing, so its wording keeps the dataset storage URI exception (debezium-server-iceberg 1.2.0.Final IcebergUtil.java:98-126).
…d only by the runner preflight
A dynamodb or jdbc expose whose location.bucket is an unresolved {{ env.* }}
template was treated as having no bucket: fluid validate failed in a shell
without the variable and asked for the bucket the contract already sets.
Validate now warns and names the unset variables; the runner preflight, in
the environment the sink config is derived in, refuses the build.
A {{ env.* }} bucket whose variable is set but empty resolves to an empty
bucket, which derives no warehouse. Validate warned only for unset
variables, so the empty case fell through to the error that asks for the
bucket the contract already sets. The check now reads the raw template
and names every variable that is unset or empty.
…tten
The acquisition dispatcher resolves {{ env.X }} before the runner sees the
contract, turning an unset or empty X into an empty string. The preflight
then passed acme-{{ env.LAKE_ENV }}-lake as acme--lake and the sink was
pushed a warehouse in a bucket the contract does not name. The dispatcher
now runs the streaming runners inside contract_as_written, so the preflight
reads the bucket template, names the variable and refuses the build.
The bigquery deriver renders the bucket's env templates and derives no
warehouse from one that does not resolve; an empty variable counts as
unset for every bucket-derived warehouse. The Debezium bigquery
no-warehouse warning says the server refuses to boot, since
debezium.sink.iceberg.warehouse has no default.
…en sink ignores it
run_builds_from_args resolves a plan-embedded contract with
resolve_contract_env_templates before dispatching, and that resolver
renders an empty variable to "". The dispatcher then handed that
partially resolved contract to contract_as_written, so with LAKE_ENV
empty acme-{{ env.LAKE_ENV }}-lake reached the preflight as acme--lake
and the sink was pushed s3://acme--lake/... . The dispatcher now takes
the contract as written from its caller, and the binding match strips
strings the way the plan resolver does.
The preflight also refused a hand-written (non-deriving) sink config
when a bucket variable it never reads was unset. Only a deriving build
now waits on the bucket's variables.
…out auto-create only warns A whitespace-only location.warehouse reached the sink verbatim. It now counts as unset, so a DynamoDB or JDBC sink derives the warehouse from the bucket or is refused. A Kafka Connect BigQuery sink reads the warehouse only to auto-create tables (iceberg.tables.auto-create-enabled defaults to false, apache-iceberg-1.10.0 IcebergSinkConfig.java:154-159). When its bucket variable is unset or empty, the run now refuses only with auto-create on; otherwise it warns and pushes no warehouse rather than one in a bucket the contract does not name.
…unts as missing 5047720 dropped the derived warehouse when the bucket variable was unset, but iceberg_catalog_overrides is already merged into the derived config, so an operator's override warehouse was dropped too and a DynamoDB, JDBC or BigQuery sink was pushed without one. The runner now drops only a warehouse check 4 would have derived: none for a build that does not derive, or whose overrides set the warehouse. A whitespace-only location.project passed as gcp.bigquery.project-id. The project and region are stripped, and check 4 tests the stripped value, so a blank project is the missing-project error.
…n-wave # Conflicts: # fluid_build/build_runners/kafka_connect/iceberg_sink_validation.py
…es of one database is one catalog A DynamoDB or JDBC warehouse derived from location.bucket is a per-table prefix, as Glue's is, so two such exposes of one database no longer read as two catalogs to the embedded Debezium join: only their bucket is compared. A BigQuery project or location that differs is named as the binding's project or region, not as catalog properties. Adds tests for the expose resolver and the catalog derivation together.
…zium sink over differing derived warehouses policy compile read only location.table for a Confluent Tableflow expose, so it granted a Glue table nothing publishes, or with no table a database-wide glue.table with table null. It now grants <database>.<topic>, the table the Tableflow IaC names (_topic_name: topic > table > exposeId), in location.database or, with none, the Kafka cluster id database Tableflow defaults to; with neither it warns and grants no Glue table. An embedded Debezium sink over several DynamoDB / JDBC exposes used the first expose's bucket-derived table prefix as the catalog warehouse, so the catalog created every other table inside that prefix. Derived warehouses are compared whole again, and differing ones are refused by validate, the preflight and the runner, naming each expose's warehouse.
…ived DynamoDB/JDBC warehouses are refused
fas89
added
the
ci:integration-emulated
fas89
had a problem deploying
to
integration-emulated
October 8, 2026 23:50 — with 
GitHub Actions
Failure
GitHub Actions
Failure
github-actions
Bot
added
provider
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
…h, never the value The schema type error policy_compiler_crashed printed included the value as written, so a connection URL with a password in a binding's location reached stderr and CI logs. It now names the path and the expected type only.
fas89
had a problem deploying
to
integration-emulated
October 9, 2026 00:04 — with
GitHub Actions
Error
GitHub Actions
Error
fas89
had a problem deploying
to
integration-emulated
October 9, 2026 00:04 — with
GitHub Actions
Error
GitHub Actions
Error
github-actions
Bot
removed
the
ci:integration-emulated
fas89
added
the
ci:integration-emulated
fas89
had a problem deploying
to
integration-emulated
October 9, 2026 00:06 — with
GitHub Actions
Failure
GitHub Actions
Failure
github-actions
Bot
removed
the
needs-docs
The sink named its table's namespace from location.database only. A GCP binding names its dataset in location.dataset, so a BigQuery or other GCP Iceberg sink pushed iceberg.tables=None.<table> and validate passed. The namespace is now location.database, else location.dataset (the BigQuery metastore catalog maps a namespace to a dataset, apache-iceberg-1.10.0 BigQueryMetastoreCatalog.java:149, :200, :307). A derived sink whose expose names no namespace, or a derived Kafka Connect sink with no table, is refused by validate and the run preflight.
fas89
had a problem deploying
to
integration-emulated
October 9, 2026 09:27 — with
GitHub Actions
Error
GitHub Actions
Error
fas89
had a problem deploying
to
integration-emulated
October 9, 2026 09:27 — with
GitHub Actions
Error
GitHub Actions
Error
github-actions
Bot
removed
the
ci:integration-emulated
fas89
added
the
ci:integration-emulated
fas89
had a problem deploying
to
integration-emulated
October 11, 2026 07:39 — with
GitHub Actions
Error
GitHub Actions
Error
fas89
had a problem deploying
to
integration-emulated
October 11, 2026 07:39 — with
GitHub Actions
Error
GitHub Actions
Error
github-actions
Bot
removed
the
ci:integration-emulated
fas89
added
the
ci:integration-emulated
fas89
had a problem deploying
to
integration-emulated
October 11, 2026 07:40 — with
GitHub Actions
Error
GitHub Actions
Error
…t with pydantic 2.14 mcp 1.20 and 1.21 import eval_type_backport from pydantic._internal._typing_extra, which pydantic 2.14 removed, so mcp.server.fastmcp fails to import with current pydantic (verified: mcp 1.20 imports with pydantic 2.13.5 and fails with 2.14.0; 1.22 is the first 1.x that imports with 2.14.0). The floor CI leg force-reinstalled mcp==1.20 with the newest pydantic and was red on main. The pin is now mcp>=1.22,<3.0 and the leg tests 1.22.
github-actions
Bot
added
ci
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes six defects found by the defect scan, plus one found while writing the docs, of the Iceberg catalog-kind change set (#707 and #709). The owner chose the scope on 2026-10-07: the High plus five Mediums ship now, and the Lows stay on Trello. Defect fixes only: no new CLI flags and no schema changes (EIC freeze).
exposes[0]. A second CDC build streamed its records into the first expose's table, andvalidate, the preflight and the run all passed.outputspick. An ambiguous build is a hard error invalidateand the run preflight. A derived embedded Debezium sink may write several outputs that share one database and one catalog. One resolver is used by validate, the preflight, both runners and the late-arrival target. Hand-written configs and overrides that seticeberg.tables/table-namespacebehave as before.dynamodbandjdbcsinks got no warehouse. Iceberg'sDynamoDbCatalogandJdbcCatalogrefuse to initialise without one, so the connector never started.location.warehouse, else from an explicitlocation.bucket(s3://on AWS,gs://on GCP), else an override. With none,validateerrors and the run refuses. A{{ env.* }}bucket whose variable is unset or empty: validate warns, and the run refuses instead of deriving a bucket the contract does not name.bigquerysink had nogcp.bigquery.project-id(required byBigQueryMetastoreCatalog) and no warehouse, and it got the AWS-onlyclient.region.location.projectmaps togcp.bigquery.project-id(required),location.regionmaps togcp.bigquery.location, and the warehouse is thegs://storage dbt-bigquery and the GCP IaC use, or a warning when there is none.client.regionis no longer emitted.<location.database>.<table>. A GCP binding names its BigQuery dataset inlocation.dataset, so a GCP Iceberg sink pushediceberg.tables=None.<table>, and validate passed.location.database, elselocation.dataset(the BigQuery metastore catalog maps a namespace to a dataset:BigQueryMetastoreCatalog.java:149, :200, :307). A derived sink with no namespace is refused, as is a derived Kafka Connect sink with no table.platform: confluentexpose with no catalog read asrestto policy compile and dbt, but the Tableflow IaC published it to Glue, so policy compile dropped every grant with a false warning.glueeverywhere. Policy compile grants the Glue table Tableflow publishes (the topic's table, inlocation.databaseor the Kafka cluster id database).fluid policy compilenever showed its warnings, and a compiler crash became an emptybindings.jsonwith exit 0.fluid policy applyechoes the stored warnings. A crash exits non-zero with an error.The Upgrade notes in
CHANGELOG.mdlist the contracts that validated before and now fail (all of them already failed at connector start) and the config output changes.Borrow before build
Every third-party contract the fix relies on was read from upstream source at the released tag, and is cited in the code:
apache-iceberg-1.10.0:DynamoDbCatalog.javaandJdbcCatalog.java: the warehouse is required, and the table layout is under it.BigQueryMetastoreCatalog.java:gcp.bigquery.project-idis required,gcp.bigquery.location, and when the warehouse is used.IcebergSinkConfig.java:154-159:iceberg.tables.auto-create-enableddefaults to false.1.2.0.FinalIcebergConfig.java:debezium.sink.iceberg.warehousehas no default.How it was built and verified
tests/build_runners/test_fw2_integration_seams.py. A seam review and a security review then ran on the merged diff. Their High (a multi-expose Debezium sink used one expose's table prefix as the catalog warehouse) and Mediums (the Tableflow grant table) are fixed here.Local CI
scripts/mypy_strict.py: no issues.pytest tests -n auto) passes, apart from these local-environment failures, which also fail on main:tests/iac/test_iac_moto_e2e.py: up to 3 tests (OpenTofu 1.12 with a fresh moto).tests/test_coding_agent_live.py: 1 test (the localclaudeCLI).tests/test_docs_reminder.pyandtests/test_ci_release_workflow.py: 8 tests each. The local Homebrew node cannot loadlibsimdjson, and the same tests fail on a clean origin/main worktree.Fixed during review
policy_compiler_crashedechoed a wrong-typed value, so a connection URL with a password could reach stderr. It now names the path and the expected type only (2ed842b2).fluid validatehas the same echo on main; that is on Trello.CI fix included
MCP SDK Drift (mcp 1.20)was red on main as well as here. mcp 1.20 and 1.21 importeval_type_backportfrom pydantic's private_typing_extramodule, which pydantic 2.14 removed. Tested in throwaway uv environments: mcp 1.20 imports with pydantic 2.13.5 and fails with 2.14.0, and 1.22 is the first 1.x release that imports with 2.14.0. Themcpfloor is now>=1.22,<3.0, and the drift leg tests 1.22. Reproduced locally:Tested live
This change touches the streaming-sink paths. The
ci:integration-emulatedlabel runs the Lakekeeper + Kafka Connect live lane (tests/integration/test_lakekeeper_kafka_connect_live.py) as the gate, per the owner's "CI as the gate" decision. No local Docker was used.Follow-ups
The low and pre-existing findings from the verification rounds go to Trello. Docs: forge_docs #147, to merge after this PR.