`), an AWS KMS key or alias ARN, or a Google Cloud KMS key name (`projects//locations//keyRings//cryptoKeys/`). The schema refuses an `alias/…` or `arn:…` value on a `gcp` binding, and a `projects/…` value on an `aws` binding. In the 0.7.6 schema's description, `product` means a customer-managed key created for this product, and `none` means the platform's default encryption.
+
+**`principals`** maps the principals the contract names — in `accessPolicy.grants[].principal` and `exposes[].policy.authz.columnRestrictions[].principal` — to identities on this binding's platform, so the base contract stays cloud-neutral and each environment binds it. A value is one identity or a list of them; `[]` says the principal has no identity on this platform. On a `gcp` binding each identity must be an IAM member (`user:`, `group:`, `serviceAccount:` or `domain:`), and on an `aws` binding an IAM principal ARN; the schema enforces both. The schema's description adds that, when the block is present, every principal the contract names for the expose must be mapped; the reference implementation refuses an unmapped one.
+
+**`governance.lakeFormation.bucketPolicy`** (`cross-account` (default) | `none` | `all-grantees`) decides which Lake Formation grantees also get a statement in the S3 bucket policy emitted beside the grants. A bucket-policy statement lets its principal read the objects directly, which bypasses Lake Formation's column, row and cell filters; `none` emits no bucket policy at all.
+
+::: danger An emitted bucket policy is authoritative
+The 0.7.6 schema's description of `bucketPolicy` ends: "An emitted aws_s3_bucket_policy is authoritative: it replaces every other statement on the bucket. When no grantee is left none is emitted." Applying a contract that emits a bucket policy therefore replaces every other statement on that bucket.
+
+Whether one is emitted depends on the value. `cross-account`, **the default**, gives a statement only to principals "whose ARN names an AWS account other than the one running the apply, decided at plan time against the caller identity"; same-account grantees get none, "since on a registered location Lake Formation vends them credentials". So a contract that never sets `bucketPolicy` still has an emitted policy replace every other statement on the bucket as soon as one of its grantees is in another AWS account. `all-grantees` gives every grantee a statement, same-account included, "which is the output before this field existed and reopens the direct S3 read path". `none` emits no bucket policy, so nothing is replaced.
+
+A statement (`s3:ListBucket` and `s3:GetBucketLocation` on the bucket, `s3:GetObject` under `location.path`) "lets its principal read the objects straight from S3, which skips Lake Formation's column, row and cell filters and its revocations": Lake Formation's filters do not apply to that read, and revoking the grant in Lake Formation does not close it.
+
+The schema describes `none` as "the tightest setting when the location is registered (registerLocation) and every reader, cross-account ones included, queries through a Lake Formation integrated engine, or when the bucket policy is managed elsewhere". Like the other 0.7.6 descriptions of what `fluid apply` emits, this is a non-normative description of the reference implementation (see the tip above).
+:::
+
+## `exposes[].lifecycle.expire`
+
+
+```yaml
+lifecycle:
+ retention: P400D # ISO-8601 duration
+ expire: true # default false: delete data older than retention
+```
+
+An expose's `lifecycle` has the same fields as the contract-root `lifecycle`, plus `expire`. With `expire: false` — the default — `retention` is a declaration only, so a contract that already declares a retention period does not start deleting data. With `expire: true`, data older than `retention` is deleted. How a platform expires data is implementation-defined. The reference implementation honours `expire` on an AWS S3 binding and on a BigQuery table, and refuses it on other GCP bindings rather than ignoring it.
+
+## Typed masking parameters
+
+
+```yaml
+policy:
+ privacy:
+ masking:
+ - { column: phone, strategy: mask, params: { keepLast: 4 } } # ********4567
+ - { column: email, strategy: hash, params: { saltEnv: FLUID_PII_HASH_SECRET } }
+```
+
+| Param | Strategy | Meaning |
+|---|---|---|
+| `keepFirst`, `keepLast` | `mask` | Leading / trailing characters left unmasked (0–64; defaults 0 and 4). |
+| `saltEnv` | `hash` | Name of the environment variable that holds the salt (default `FLUID_PII_HASH_SECRET`). |
+| `keyEnv` | `tokenize`, `encrypt` | Name of the environment variable that holds the key. |
+
+`params` stays open (`additionalProperties: true`). The typed keys name secrets by environment variable rather than carrying them; the schema's description adds that the reference implementation's runner refuses any other key, including a literal salt or key. The 0.7.6 description of `strategy` also gives each strategy's output form — for example, `hash` is the lowercase hex SHA-256 of salt followed by value — and notes that `k_anonymity` is accepted by the schema but is a property of a whole table, not a per-value transform.
+
+## Column restrictions: semantics written down
+
+`exposes[].policy.authz.columnRestrictions` exists in 0.7.5 with the one-line description "Column-level access control". The 0.7.6 description defines it:
+
+- A column named in any restriction is restricted.
+- `access: deny` — the principal may not read the columns. `access: allow` — the columns are readable only by the principals an `allow` names.
+- A deny beats an allow, and a restriction never grants access: the readers are still the expose's readers.
+- Principals are logical, resolved per platform through `binding.principals`.
+
+---
+
+## Two complete contracts
+
+Both validate against the 0.7.6 JSON Schema (Draft 2020-12) and with the reference implementation, `data-product-forge` 0.18.1 (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.6)`).
+
+**BigQuery:** column restriction, mapped principals, a product-owned key, expiring partitions, a percentile measure, packaging and a consumer.
+
+```yaml
+fluidVersion: "0.7.6"
+kind: DataProduct
+id: sales.gold.orders
+name: Orders
+metadata:
+ owner: { team: sales-analytics, email: sales-analytics@example.com }
+accessPolicy:
+ grants:
+ - principal: "group:analysts"
+ permissions: [read]
+ - principal: "group:stewards"
+ permissions: [read]
+exposes:
+ - exposeId: orders
+ kind: table
+ contract:
+ schema:
+ - { name: order_id, type: STRING, required: true }
+ - { name: order_date, type: DATE, required: true }
+ - { name: customer_email, type: STRING }
+ - { name: amount, type: NUMERIC }
+ policy:
+ authz:
+ columnRestrictions:
+ - principal: "group:analysts"
+ columns: [customer_email]
+ access: deny
+ semantics:
+ measures:
+ - name: p95_amount
+ agg: percentile
+ expr: amount
+ aggParams: { percentile: 0.95 }
+ lifecycle:
+ retention: P400D
+ expire: true
+ binding:
+ platform: gcp
+ format: bigquery_table
+ location: { project: acme-analytics, dataset: sales, table: orders, region: europe-west1 }
+ encryption: { kms: product }
+ principals:
+ "group:analysts": "group:analysts@example.com"
+ "group:stewards": "group:data-stewards@example.com"
+packaging:
+ mode: isolated
+consumers:
+ - name: weekly_revenue_dashboard
+ label: Weekly Revenue Dashboard
+ type: dashboard
+ maturity: high
+ owner: { team: finance-analytics }
+ exposeIds: [orders]
+```
+
+**S3 + Lake Formation:** a cross-mesh upstream pin, masking parameters, an existing key, a shared bucket and no bucket policy.
+
+```yaml
+fluidVersion: "0.7.6"
+kind: DataProduct
+id: sales.silver.customers
+name: Customers
+metadata:
+ owner: { team: sales-data }
+consumes:
+ - productId: crm.bronze.accounts
+ exposeId: accounts
+ upstreamWorkspace: crm-mesh
+ upstreamDigest: "sha256:4f1c9a0d6e2b7c8a1f3e5d9b0c2a4e6f8d1b3c5e7a9f0b2d4c6e8a1b3d5f7091"
+exposes:
+ - exposeId: customers
+ kind: table
+ contract:
+ schema:
+ - { name: customer_id, type: STRING, required: true }
+ - { name: phone, type: STRING }
+ - { name: email, type: STRING }
+ policy:
+ privacy:
+ masking:
+ - { column: phone, strategy: mask, params: { keepLast: 4 } }
+ - { column: email, strategy: hash, params: { saltEnv: FLUID_PII_HASH_SECRET } }
+ binding:
+ platform: aws
+ format: parquet
+ location: { bucket: acme-lake, path: sales/customers/, region: eu-west-1, database: sales, table: customers }
+ encryption: { kms: "alias/acme-lake" }
+ packaging: { mode: shared, pool: lake-pool-eu, containers: { bucket: shared } }
+ governance:
+ lakeFormation:
+ registerLocation: true
+ bucketPolicy: none
+ grants:
+ - principal: "arn:aws:iam::123456789012:role/analyst"
+ permissions: [SELECT]
+```
+
+The example sets `bucketPolicy: none` on purpose. Its bucket is `shared`, so by the `packaging` section above it already exists and belongs to the platform, and the product writes into it without owning it; the location is registered (`registerLocation: true`), so Lake Formation governs the `analyst` grant. An emitted bucket policy would replace every other statement on that bucket and would give a grantee a path to the objects that skips Lake Formation's filters (see [the box above](#binding-fields-encryption-principals-lake-formation-bucket-policy)). With the default, `cross-account`, a grantee in another AWS account would get exactly such a statement. `none` is the value for a registered location whose readers query through a Lake Formation integrated engine, or whose bucket policy is managed elsewhere.
+
+Remove `upstreamDigest` from the second contract and it becomes invalid: `'upstreamDigest' is a dependency of 'upstreamWorkspace'` (keyword `dependentRequired`, at `/consumes/0`).
+
+---
+
+## References
+
+- [Schema Versions](/fluid/schema/versions) — status of every version, and how to choose `fluidVersion`.
+- [Changelog: 0.7.5 → 0.7.6](/fluid/schema/changelog#_0-7-5-to-0-7-6-preview-declarative-packaging-modes).
+- Reference implementation: the [governance parity notes](https://github.com/Agenticstiger/forge-cli/blob/main/docs/governance-parity.md) describe what `data-product-forge` emits and verifies for retention, encryption, column restrictions and principals on AWS and GCP.
diff --git a/docs/releases/README.md b/docs/releases/README.md
index 05cb8bb..0908f15 100644
--- a/docs/releases/README.md
+++ b/docs/releases/README.md
@@ -1,21 +1,24 @@
# What's New
-Release notes for the FLUID specification, newest first. Each entry links to the full page and the auto-generated schema diff.
+Release notes for the FLUID specification, newest first. Each entry links to its full page and to the auto-generated schema diff.
-| Version | Codename | Compatibility | Summary |
-|---|---|---|---|
-| **[0.7.5](/fluid/releases/0.7.5)** *(latest)* | Streaming Kafka → Iceberg Sink & Confluent Tableflow | ✅ Additive over 0.7.4 | Adds an opt-in Iceberg streaming sink on the `kafka-connect` acquisition pattern (`iceberg_sink_enabled`, `streamingSink`, `sink_topics`, `iceberg_catalog_overrides`) and the `confluent` Cloud Tableflow binding platform (`environment_id` / `kafka_cluster_id` / `confluent_role_arn`). |
-| **[0.7.4](/fluid/releases/0.7.4)** | Runtime agentPolicy Enforcement at the MCP Gateway | ✅ Additive over 0.7.3 | Makes an expose agent-consumable over MCP and turns `policy.agentPolicy` into a gate the MCP output-port gateway enforces on every read. Adds the optional `exposes[].mcp` block, the `postgres` binding platform, and the `postgres_table` / `athena_table` / `glue_table` formats. |
-| **[0.7.3](/fluid/releases/0.7.3)** | Source-Aligned Acquisition | ✅ Additive over 0.7.2 | Ingestion becomes a first-class FLUID concept: the `acquisition` build pattern, six ingestion engines, delivery guarantees + DLQ, schema-evolution policy, supply-chain image signing, and a top-level `retention` block. |
-| **[0.7.2](/fluid/releases/0.7.2)** | Semantic Truth Engine | ✅ Additive over 0.7.1 | A per-expose `semantics` block (entities, measures, dimensions, metrics) gives agents and BI tools a verbatim metric definition, plus `binding.icebergConfig` for Iceberg table specifics. |
-| **[0.7.1](/fluid/releases/0.7.1)** | Agentic Governance + Provider-First Orchestration | ✅ Additive over 0.5.7 | `agentPolicy`, `sovereignty`, root-level `accessPolicy`, and provider-first orchestration tasks. |
+**0.7.5 is the latest stable version. 0.7.6 is a preview**: a contract uses it only by declaring `fluidVersion: "0.7.6"`, and it can still change before it is promoted. See [Stable and preview versions](/fluid/schema/versions#stable-and-preview-versions).
+
+| Version | Status | Codename | Compatibility | Summary |
+|---|---|---|---|---|
+| **[0.7.6](/fluid/releases/0.7.6)** | **Preview** | Declarative Packaging Modes | ✅ Additive over 0.7.5 | Adds top-level `packaging` and `consumers`, cross-mesh pins on `consumes[]` (`upstreamWorkspace` + `upstreamDigest`), `aggParams` on semantic measures, and on bindings `encryption.kms`, `principals` and Lake Formation `bucketPolicy`; an expose's `lifecycle` gains `expire`, and masking rules gain typed `params`. |
+| **[0.7.5](/fluid/releases/0.7.5)** | **Stable (latest)** | Streaming Kafka → Iceberg Sink & Confluent Tableflow | ✅ Additive over 0.7.4 | Adds an opt-in Iceberg streaming sink on the `kafka-connect` acquisition engine, the `confluent` (Tableflow) and `pgvector` binding platforms, the `pgvector_table` format and `binding.vectorConfig`, and location fields for Tableflow, Iceberg catalogs, Redshift Serverless and Kinesis. |
+| **[0.7.4](/fluid/releases/0.7.4)** | Stable | Runtime agentPolicy Enforcement at the MCP Gateway | ✅ Additive over 0.7.3 | Adds the optional `exposes[].mcp` block, the `postgres` binding platform, and the `postgres_table` / `athena_table` / `glue_table` formats. |
+| **[0.7.3](/fluid/releases/0.7.3)** | Stable | Source-Aligned Acquisition | ✅ Additive over 0.7.2 | The `acquisition` build pattern and six ingestion engines, top-level `retention`, `governance` (AWS Lake Formation) and `extensions`, `binding.governance`, `metadata.productType`, and `exposes[].contract.schemaPolicy`. |
+| **[0.7.2](/fluid/releases/0.7.2)** | Stable | Semantic Truth Engine | ⚠️ One narrowing: `notification` became a closed object | A per-expose `semantics` block, `binding.icebergConfig`, the top-level `orchestration` key, and adapter-qualified dbt engines (`build.engine: dbt-`). |
+| **[0.7.1](/fluid/releases/0.7.1)** | Stable | Agentic Governance + Provider-First Orchestration | ✅ Additive over 0.5.7 | `exposes[].policy.agentPolicy`, top-level `sovereignty` and `accessPolicy`, and provider-action tasks under `builds[].execution.orchestration`. |
---
## Reading the compatibility column
-The 0.7.1 → 0.7.5 line is strictly **additive**: bump `fluidVersion` and every prior contract still validates.
+"Additive" means what [`scripts/check-compat.py`](https://github.com/open-data-protocol/fluid/blob/main/scripts/check-compat.py) checks on every pull request: no document that validates under one version stops validating under the next. Every transition in the table is additive except **0.7.1 → 0.7.2**, where `$defs/notification` gained `additionalProperties: false`. That break is recorded in [`scripts/compat-waivers.txt`](https://github.com/open-data-protocol/fluid/blob/main/scripts/compat-waivers.txt) and explained in the [0.7.2 note](/fluid/releases/0.7.2#backward-compatibility-with-0-7-1-one-narrowing-change).
-**0.7.5 continues that streak.** It only adds optional fields, enum values, and a wider `engine` shape, so every valid 0.7.4 contract validates as 0.7.5 unchanged. See [What's New in 0.7.5](/fluid/releases/0.7.5#backward-compatibility-fully-additive) for the details.
+Additive does **not** mean that an older `fluidVersion` may use newer fields. A document is validated against the schema of the version it declares, so to use a field you declare the version that added it. See [Choosing `fluidVersion`](/fluid/schema/versions#choosing-fluidversion).
-For the field-by-field machine diff of every version, see the [**Schema Changelog**](/fluid/schema/changelog).
+For the field-by-field history, see the [**Schema Changelog**](/fluid/schema/changelog).
diff --git a/docs/schema/README.md b/docs/schema/README.md
index e7f469d..3e885f7 100644
--- a/docs/schema/README.md
+++ b/docs/schema/README.md
@@ -1,19 +1,20 @@
# Schema Reference
-Everything you need to read, write, and validate a FLUID contract. The latest schema version is **0.7.4**.
+Everything you need to read, write, and validate a FLUID contract. **The latest stable schema is 0.7.5; 0.7.6 is a preview** (see [Versions](/fluid/schema/versions#stable-and-preview-versions)).
| Page | Use it for |
|---|---|
| [**Anatomy**](/fluid/schema/anatomy) | A guided tour of every top-level block, in authoring order, with a "what / when / why" per block. Start here if you've never written a contract. |
| [**Cheatsheet**](/fluid/schema/cheatsheet) | One row per field — meaning, required flag, and version added. The fast lookup table while you author. |
| [**Minimal Valid Contract**](/fluid/schema/minimal-contract) | The smallest file that validates, plus a glance at every top-level block. |
-| [**Full Specification**](/fluid/schema/specification) | The narrative protocol specification. |
-| [**Versions**](/fluid/schema/versions) | Every published schema version, linked to its JSON Schema and generated HTML reference. |
-| [**Changelog**](/fluid/schema/changelog) | Version-to-version diffs, including the breaking changes in 0.7.4. |
+| [**Specification**](/fluid/schema/specification) | Validation semantics, document structure, and how multi-file composition relates to validity. |
+| [**Versions**](/fluid/schema/versions) | Every published schema version and its status, how to choose `fluidVersion`, and known caveats. |
+| [**Changelog**](/fluid/schema/changelog) | What changed between consecutive versions, including the pre-0.7.1 breaking changes and the one narrowing in 0.7.2. |
## Machine-readable artifacts
-- **Raw JSON Schema (0.7.4):** [`/fluid/schema/fluid-schema-0.7.4.json`](/fluid/schema/fluid-schema-0.7.4.json) — point your validator here.
-- **Generated HTML reference (0.7.4):** [`/fluid/specs/0.7.4/fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html) — the authoritative field-by-field rendering of every type, enum, and validation rule.
+- **JSON Schema (0.7.5, stable):** [`/fluid/schema/fluid-schema-0.7.5.json`](/fluid/schema/fluid-schema-0.7.5.json) — point your validator here.
+- **Generated HTML reference (0.7.5):** [`/fluid/specs/0.7.5/fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html) — every type, enum and validation rule, rendered from the schema.
+- **Preview (0.7.6):** [`fluid-schema-0.7.6.json`](/fluid/schema/fluid-schema-0.7.6.json) · [`specs/0.7.6/fluid-spec.html`](/fluid/specs/0.7.6/fluid-spec.html).
-For older versions, see [**Versions**](/fluid/schema/versions).
+Validate a document against the schema of the `fluidVersion` it declares. For older versions, see [**Versions**](/fluid/schema/versions).
diff --git a/docs/schema/anatomy.md b/docs/schema/anatomy.md
index 4b52a41..f5b1787 100644
--- a/docs/schema/anatomy.md
+++ b/docs/schema/anatomy.md
@@ -1,10 +1,13 @@
# Schema Anatomy
-A FLUID contract is one YAML file. This page walks **every top-level block** in v0.7.4, in the order you'd typically author them, with a one-paragraph "what / when / why" for each and a link into the generated reference doc for the field-by-field details.
+A FLUID contract is one YAML file. This page walks **every top-level block** of the latest stable schema, **0.7.5**, in the order you'd typically author them, with a one-paragraph "what / when / why" for each and a link into the generated reference doc for the field-by-field details.
> 📋 If you just want a one-line-per-field table, see the [**Cheatsheet**](/fluid/schema/cheatsheet).
> 🟢 If you want a copy-pasteable minimal example, see the [**Minimal Valid Contract**](/fluid/schema/minimal-contract).
-> 📚 Full field-by-field reference: [`specs/0.7.4/fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html).
+> 📚 Full field-by-field reference: [`specs/0.7.5/fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html).
+> 🧪 Blocks marked **0.7.6 preview** need `fluidVersion: "0.7.6"` and can still change — see [0.7.6 (preview)](/fluid/releases/0.7.6).
+
+The YAML blocks on this page fit together: merged in order into the [minimal contract](/fluid/schema/minimal-contract), they form one contract that validates against the 0.7.5 schema.
---
@@ -13,13 +16,14 @@ A FLUID contract is one YAML file. This page walks **every top-level block** in
```
┌────────────────────────── IDENTITY ──────────────────────────┐
│ fluidVersion kind id name description domain │
-│ metadata { owner, layer, businessContext } │
+│ metadata { owner, layer, productType, businessContext } │
│ tags labels │
└──────────────────────────────────────────────────────────────┘
┌────── INTERFACE (what you publish) ───────┐ ┌── INTAKE ────┐
│ exposes[] │ │ consumes[] │
-│ ├─ contract (schema, dq, policy) │ │ │
+│ ├─ contract (schema, dq, schemaPolicy)│ │ │
+│ ├─ policy (authz, privacy, agent…) │ │ │
│ ├─ semantics (entities, measures, …) │ │ │
│ ├─ mcp (MCP gateway opt-in) ⭐0.7.4│ │ │
│ └─ binding (platform, location, …) │ │ │
@@ -28,14 +32,14 @@ A FLUID contract is one YAML file. This page walks **every top-level block** in
┌────── IMPLEMENTATION (how it's built) ───────────────────────┐
│ build { pattern, engine, properties, capabilities, … } │
│ └─ acquisition (source-aligned ingestion) ⭐ 0.7.3 │
-│ orchestration { engine, tasks } ⭐ 0.7.0+ │
+│ orchestration { engine, airflow, … } top-level ⭐ 0.7.2 │
└──────────────────────────────────────────────────────────────┘
┌────── GOVERNANCE (rules around the product) ─────────────────┐
│ sovereignty (jurisdiction) top-level ⭐ 0.7.1 │
│ accessPolicy (IAM grants) top-level ⭐ 0.7.1 │
│ retention (data/log TTLs) top-level ⭐ 0.7.3 │
-│ governance (AWS Lake Formation) top-level ⭐ 0.4.0 │
+│ governance (AWS Lake Formation) top-level ⭐ 0.7.3 │
│ exposes[].policy.agentPolicy (AI/LLM) per-expose ⭐ 0.7.1 │
│ lineage schemaEvolution observability │
└──────────────────────────────────────────────────────────────┘
@@ -44,14 +48,18 @@ A FLUID contract is one YAML file. This page walks **every top-level block** in
│ lifecycle { state } environments { dev, staging, prod } │
│ machineLearning { ... } docs { ... } │
└──────────────────────────────────────────────────────────────┘
+
+ 0.7.6 preview adds: packaging, consumers[], consumes[].upstream*,
+ binding.{encryption, principals, packaging}, lifecycle.expire
```
---
-## 1. Identity — *who is this product?*
+## 1. Identity: *who is this product?*
+
```yaml
-fluidVersion: "0.7.4" # required
+fluidVersion: "0.7.5" # required
kind: DataProduct # required — DataProduct | MLPipeline
id: finance.gold.customer_360 # required — globally unique, dot-separated
name: "Customer 360" # required — display name
@@ -67,30 +75,33 @@ labels: { team: customer-analytics, criticality: high, cost-center: engineering
---
-## 2. `metadata` — *who owns it, where does it sit in the medallion?*
+## 2. `metadata`: *who owns it, where does it sit in the medallion?*
+
```yaml
metadata: # required (only metadata.owner is required inside)
- owner: # required
- team: customer-analytics # required
+ owner: # required — no member of owner is required by the schema
+ team: customer-analytics # name a team: tools route alerts and ownership by it
email: customer-analytics@company.com
slack: "#customer-analytics-eng"
layer: Gold # free-form string; convention: Bronze | Silver | Gold
+ productType: CDP # SDP | ADP | CDP (source-aligned, aggregated, consumption-aligned)
businessContext:
domain: "Customer Experience"
subdomain: "Customer Intelligence"
```
**What:** ownership and business context that doesn't fit in `id`.
-**When:** always — `metadata` is required, and its only required sub-field is `owner.team`.
+**When:** always — `metadata` is required, and inside it only `owner` is. The schema requires no member of `owner`, so `owner: {}` validates; name at least a `team`.
**Why:** every alert, catalog page, and ownership audit traces back here. Without an `owner.email`, a broken product has no human to wake up.
> 💡 `metadata.layer` is a **free-form string** (not an enum). `Bronze`/`Silver`/`Gold` is the conventional medallion vocabulary, but the schema doesn't enforce it.
---
-## 3. `consumes` — *what upstream products do I need?*
+## 3. `consumes`: *what upstream products do I need?*
+
```yaml
consumes:
- productId: finance.bronze.raw_payments
@@ -102,14 +113,17 @@ consumes:
**What:** explicit, version-constrained dependencies on other FLUID products.
**When:** any product that isn't strictly source-aligned (i.e. not `build.pattern: acquisition`).
-**Why:** the orchestrator builds the DAG from `consumes`. No `consumes` → no automatic upstream waiting, no automatic lineage.
+**Why:** the orchestrator builds the DAG from `consumes`. No `consumes` → no automatic upstream waiting, no automatic lineage. Each entry can also state `qosExpectations` (`freshnessMax`, `maxStaleness`, `minCompleteness`) and `requiredPolicies`.
+
+> 🧪 **0.7.6 preview:** `upstreamWorkspace` and `upstreamDigest` pin an upstream that lives in another mesh; setting the first requires the second. See [0.7.6](/fluid/releases/0.7.6#cross-mesh-pins-on-consumes).
---
-## 4. `exposes` — *what does this product publish?*
+## 4. `exposes`: *what does this product publish?*
This is the heart of the contract. A product can expose multiple ports (e.g. a Snowflake table **and** a Kafka stream **and** an API).
+
```yaml
exposes:
- exposeId: customer_profiles # required
@@ -130,12 +144,21 @@ exposes:
type: valid_values # type ∈ freshness | completeness | uniqueness | valid_values | accuracy | schema | anomaly_detection | drift_detection
selector: "ltv >= 0"
severity: error # severity ∈ info | warn | error | critical
- policy:
- privacy:
- masking:
- - { column: email, strategy: tokenize }
- authn: iam
- authz: { readers: [analytics-team, ml-agents] }
+ schemaPolicy: evolve_safe # strict | discover_and_freeze | evolve_safe | evolve_all
+
+ # ── POLICY — a sibling of contract, not inside it ─────────
+ policy:
+ authn: iam
+ authz: { readers: [analytics-team, ml-agents] }
+ privacy:
+ masking:
+ - { column: email, strategy: tokenize }
+
+ # ── QoS ───────────────────────────────────────────────────
+ qos:
+ availability: "99.9%" # a percentage string
+ freshnessSLO: PT1H # ISO-8601 durations
+ latencyP95: PT2S
# ── 4b. SEMANTICS (⭐ 0.7.2) ─────────────────────────────
semantics:
@@ -155,42 +178,42 @@ exposes:
# ── 4d. BINDING ───────────────────────────────────────────
binding:
- platform: gcp # gcp | aws | azure | snowflake | databricks | kafka | local | kubernetes | postgres ⭐0.7.4 | other
- format: bigquery_table # bigquery_table | snowflake_table | iceberg | delta_table | parquet | http_api | kafka_topic | postgres_table ⭐0.7.4 | athena_table ⭐0.7.4 | glue_table ⭐0.7.4 | …
- location: { project: company-data, dataset: gold_customer, table: profiles_v1 }
- icebergConfig: # ⭐ 0.7.2 — only when format = iceberg
- writeVersion: 2
- fileFormat: parquet
- partitionSpec: [ { sourceColumn: created_at, transform: month } ]
+ platform: gcp # gcp | aws | azure | snowflake | databricks | kafka | confluent ⭐0.7.5 | local | kubernetes | postgres ⭐0.7.4 | pgvector ⭐0.7.5 | other
+ format: bigquery_table # bigquery_table | snowflake_table | iceberg | delta_table | parquet | http_api | kafka_topic | postgres_table ⭐0.7.4 | athena_table ⭐0.7.4 | glue_table ⭐0.7.4 | pgvector_table ⭐0.7.5 | …
+ location: { project: company-data, dataset: gold_customer, table: profiles_v1, region: europe-west1 }
```
-### `contract` — *the data shape and the rules around it*
+### `contract`: *the data shape and the rules around it*
-- **`schema`**: columns, types, nullability, sensitivity (`cleartext` | `tokenized` | `hashed` | `encrypted` | …), tags.
+- **`schema`**: columns, types, `required`, sensitivity (`cleartext` | `tokenized` | `pseudonymized` | `encrypted` | `pii` | …), tags.
- **`dq.rules`**: declarative data-quality assertions (`valid_values`, `freshness`, `completeness`, `uniqueness`, `accuracy`, `anomaly_detection`, `drift_detection`, `schema`).
-- **`policy`**: privacy treatments, RBAC readers/writers, column-level redaction, and the per-expose `agentPolicy` (see §7).
+- **`schemaPolicy`** (0.7.3+): how the output schema may change.
+
+`contract` is closed: `policy` is **not** a member of it. Privacy treatments, readers/writers, column restrictions and the per-expose `agentPolicy` go in **`exposes[].policy`**, next to `contract` (see §7). `qos` states service levels for the port: `availability`, `freshnessSLO`, `dataLossSLO`, `latencyP95`, `completenessTarget`, `errorBudget`.
-### `semantics` ⭐ 0.7.2 — *the business meaning layer*
+### `semantics` (0.7.2): *the business meaning layer*
`entities`, `measures`, `dimensions`, `metrics` — the same primitives as dbt MetricFlow and Snowflake Semantic Views. An LLM or BI tool reading this block can answer *"what is our MRR?"* without inventing SQL.
-### `mcp` ⭐ 0.7.4 — *opt this expose into the MCP output-port gateway*
+### `mcp` (0.7.4): *opt this expose into the MCP output-port gateway*
**What:** the `exposes[].mcp` block declares this expose as agent-consumable over MCP. Its two sub-blocks are `sampling.maxRows` (integer ≥1, default 100 — a hard cap on the `sample` tool so an over-curious agent can't pull petabyte-scale rows) and `classification.dataClass` (`public` | `internal` | `confidential` | `restricted` — an advisory hint surfaced on the `describe` tool so consumer agents can declare downstream handling).
**When:** any expose you want served through `fluid mcp output-port serve` and reachable by AI agents over MCP.
-**Why:** the *presence* of a (non-empty) `mcp` block is what flips on runtime enforcement. Once present, the expose's `policy.agentPolicy` (`allowedModels` / `deniedModels`, `allowedUseCases` / `deniedUseCases`, token caps) is enforced by the Fluid MCP gateway **on every read** — closing the gap where those fields were previously declarative metadata only. The `agentPolicy` shape itself is unchanged from 0.7.3; 0.7.4 makes it a runtime gate.
+**Why:** the *presence* of a (non-empty) `mcp` block is what flips on runtime enforcement. Once present, the expose's `policy.agentPolicy` (`allowedModels` / `deniedModels`, `allowedUseCases` / `deniedUseCases`, token caps) is enforced **on every read** by an MCP gateway that implements 0.7.4 (the reference implementation's does) — where before those fields were declarative metadata only. The `agentPolicy` shape itself is unchanged from 0.7.3; 0.7.4 makes it a runtime gate.
> 🔗 See the [**MCP how-to**](/fluid/how-to/mcp) for an end-to-end walkthrough of serving an expose over MCP with governed agent access.
-### `binding` — *where the data physically lives*
+### `binding`: *where the data physically lives*
+
+Platform + format + location. Plus `icebergConfig` (0.7.2+) when format is `iceberg` (write version, file format, `partitionSpec`, `sortOrder`), so the contract owns table-format details too. **0.7.4** adds the `postgres` platform and the `postgres_table` / `athena_table` / `glue_table` formats. **0.7.5** adds the `confluent` (Tableflow) and `pgvector` platforms, `pgvector_table`, `vectorConfig`, and location fields for Iceberg catalogs, Redshift Serverless and Kinesis — see [0.7.5](/fluid/releases/0.7.5). Optional `binding.governance` carries per-resource AWS Lake Formation settings.
-Platform + format + location. Plus `icebergConfig` (0.7.2+) when format is `iceberg`, so the contract owns table-format details too. **0.7.4** adds the `postgres` platform and the `postgres_table` / `athena_table` / `glue_table` formats (additive — `snowflake_view` and the `redshift_*` formats remain valid). Optional `binding.governance` carries per-resource AWS Lake Formation grants.
+> 🧪 **0.7.6 preview** adds `binding.encryption.kms`, `binding.principals` (logical principal → platform identity), `binding.packaging`, and Lake Formation `bucketPolicy`; a bucket policy it emits is **authoritative** and replaces every other statement on the bucket. See [0.7.6](/fluid/releases/0.7.6#binding-fields-encryption-principals-lake-formation-bucket-policy).
---
-## 5. `build` — *how the product is produced*
+## 5. `build`: *how the product is produced*
Four patterns, validated conditionally:
@@ -201,10 +224,11 @@ Four patterns, validated conditionally:
| `multi-stage` | Multi-step pipeline (bronze → silver → gold) in one product. |
| `acquisition` ⭐ 0.7.3 | Source-aligned ingestion from external systems (Postgres, Salesforce, Kafka, files, …). |
+
```yaml
build:
pattern: acquisition # ⭐ 0.7.3 — picks acquisitionPattern shape for build.properties below
- engine: airbyte # dbt | sql | python | spark | glue | custom | duckdb | airbyte | meltano | dlt | kafka-connect | debezium
+ engine: airbyte # dbt | dbt- | sql | python | spark | glue | custom | duckdb | airbyte | meltano | dlt | kafka-connect | debezium
capabilities: [incremental_dedup, schema_evolution, dlp_scan]
properties: # ← schema validates this as acquisitionPattern when pattern=acquisition
source:
@@ -212,11 +236,11 @@ build:
mode: incremental_dedup # full_refresh | incremental_append | incremental_dedup | incremental_merge | cdc | streaming
cursor_field: updated_at
connection:
- secretRef: "vault://pg-prod-readonly" # URI form required: vault:// aws:// gcp:// azure:// env://
+ secretRef: "vault://pg-prod-readonly" # must be a URI (://…), e.g. vault://, env://
streams: [public.customers, public.orders]
sink:
- format: iceberg
- partitionBy: ["day(ingested_at)"] # array of strings (function-form). NOT the object-form used by binding.icebergConfig.partitionSpec
+ format: bigquery_table # iceberg | delta | parquet | csv | json | snowflake_table | bigquery_table | redshift_table | duckdb_table
+ partitionBy: ["day(ingested_at)"] # array of strings (function-form) — not the object form of binding.icebergConfig.partitionSpec
delivery:
guarantee: at_least_once # at_most_once | at_least_once | exactly_once
idempotencyKey: "${stream}|${batch_id}"
@@ -241,39 +265,52 @@ build:
> 🧭 **Where `acquisitionPattern` actually lives:** in the schema, `build.properties` is validated against a different shape depending on `build.pattern` (`hybrid-reference` → `hybridReferencePattern`, `embedded-logic` → `embeddedLogicPattern`, `multi-stage` → `multiStagePattern`, `acquisition` → `acquisitionPattern`). The pattern name is the discriminator; `properties` is always the payload key.
-**When:** any product that is built rather than virtual.
+**When:** any product that is built rather than virtual. Use `build` for one build, or `builds[]` (each with an `id`) for several; both have the same shape.
---
-## 6. `orchestration` — *who actually runs the build?*
+## 6. `orchestration`: *who actually runs the build?*
+
```yaml
orchestration:
- engine: airflow # airflow | dagster | prefect | kubeflow | custom | none
+ engine: airflow # required — airflow | dagster | prefect | kubeflow | custom | none
+ mode: generated # generated | manual | hybrid
generateOnChange: true
- tasks:
- - taskId: ensure_warehouse
- type: provider_action # provider_action | fluid_validate | fluid_plan | fluid_apply | fluid_execute | fluid_verify | fluid_dq_check | bash | python | sensor | …
- provider: snowflake.warehouse
- action: ensure
- parameters: { name: ANALYTICS_WH, size: medium }
- - taskId: load_customers
- type: provider_action
- provider: snowflake.table
- action: ensure
- buildStepRef: customers_ingest
- dependsOn: [ensure_warehouse]
+ airflow:
+ dagId: customer_360 # required when airflow is present
+ dagConfig: { schedule: "0 2 * * *", catchup: false }
+ tasks:
+ - taskId: ensure_warehouse
+ type: provider_action # provider_action | fluid_validate | fluid_plan | fluid_apply | fluid_execute | fluid_verify | fluid_dq_check | bash | python | sensor | …
+ operator: ProviderActionOperator
+ provider: snowflake # aws | gcp | azure | snowflake | databricks | kafka | kubernetes | local | custom
+ action: warehouse.ensure # .
+ params: { name: ANALYTICS_WH, size: medium }
+ - taskId: load_customers
+ type: provider_action
+ operator: ProviderActionOperator
+ provider: snowflake
+ action: table.ensure
+ params: { name: CUSTOMERS }
+ buildStepRef: customers_ingest
+ dependencies: [ensure_warehouse]
```
-**What:** how the build is scheduled and runs. Provider actions (0.7.0+) let you target cloud primitives directly without wrapper operators.
-**Why:** the orchestrator config used to be the part most likely to drift from the contract. Embedding it here keeps "what runs" and "what gets produced" in one file.
+**What:** how the build is scheduled and run. The top-level `orchestration` key exists from 0.7.2; the same block can sit under `build.execution.orchestration` (from 0.7.1).
+**Why:** the orchestrator config used to be the part most likely to drift from the contract. Embedding it keeps "what runs" and "what gets produced" in one file.
+
+> ⚠️ **Only `orchestration.airflow.tasks[]` is checked.** `orchestration` is an open object, so a top-level `orchestration.tasks` list — which some FLUID tooling reads — is accepted with **no validation at all**: a typo in it is not reported. The key names in such a list are implementation-defined. In the reference implementation, the AWS and GCP code generators read `params` and `dependsOn` from `orchestration.tasks[]`, while its Snowflake Airflow generator reads `parameters`.
+>
+> Name each task's **`operator`**, as above. In the 0.7.1 to 0.7.6 schemas the conditional rules for `fluid_execute`, `bash` and `python` tasks also match a task that has no `operator`, so such a task must satisfy all three at once and is in practice rejected.
---
-## 7. Governance — `sovereignty`, `accessPolicy`, `governance`, and `exposes[].policy.agentPolicy`
+## 7. Governance: `sovereignty`, `accessPolicy`, `governance`, and `exposes[].policy.agentPolicy`
-The agentic-era governance layer is mostly 0.7.1 (`sovereignty`, `accessPolicy`, `agentPolicy`), plus the older top-level `governance` block (AWS Lake Formation, since 0.4.0). **Three blocks are top-level (`sovereignty`, `accessPolicy`, `governance`); `agentPolicy` is per-expose** — it lives inside `exposes[].policy.agentPolicy`, not at the root, even though some prior release notes show it at the top level.
+The agentic-era governance layer is mostly 0.7.1 (`sovereignty`, `accessPolicy`, `agentPolicy`), plus the top-level `governance` block (AWS Lake Formation, since 0.7.3; an unrelated `governance` block existed from 0.0.1 to 0.4.0 and was removed in 0.5.7). **Three blocks are top-level (`sovereignty`, `accessPolicy`, `governance`); `agentPolicy` is per-expose** — it lives inside `exposes[].policy.agentPolicy`, not at the root (earlier versions of the 0.7.1 release note showed it at the top level).
+
```yaml
sovereignty: # ⭐ 0.7.1 — where data may live (top-level)
jurisdiction: EU
@@ -294,9 +331,11 @@ accessPolicy: # ⭐ 0.7.1 — IAM grants genera
permissions: [write, insert, update]
conditions: { ipRanges: ["10.0.0.0/8"] }
-governance: # ⭐ 0.4.0 — account/project-wide AWS Lake Formation (top-level)
+governance: # ⭐ 0.7.3 — account-wide AWS Lake Formation (top-level)
lakeFormation: # admins + LF-tag definitions (per-resource grants live under binding.governance)
- admins: ["arn:aws:iam::123456789012:role/LakeAdmins"]
+ admins: # AUTHORITATIVE: replaces the account's admin list — see the warning below
+ - "arn:aws:iam::123456789012:role/LakeAdmins"
+ - "arn:aws:iam::123456789012:role/fluid-deployer" # the role that runs the apply
tagDefinitions: # tag key -> allowed values (emitted as aws_lakeformation_lf_tag)
classification: [public, internal, confidential, restricted]
@@ -326,14 +365,21 @@ exposes:
purposeLimitation: "Customer support chatbot only — not for marketing."
```
-**Why:** the contract becomes the source of truth for compliance reviewers. Drift between "what the contract says" and "what IAM bindings exist in prod" disappears.
+**Why:** the contract becomes the source of truth for compliance reviewers, and drift between "what the contract says" and "what IAM bindings exist" becomes checkable.
+
+::: danger `governance.lakeFormation.admins` replaces the admin list
+`admins` is **authoritative, not additive**. Applying it sets the account's Lake Formation admins to exactly this list, so any admin you do not list is **removed** — including the role or user that runs the apply, after which Lake Formation grants and registrations that need an admin can fail. List every principal that must stay an admin, the deploying identity included. (The 0.7.3–0.7.5 schemas were first published with a description saying the opposite; see the [Changelog](/fluid/schema/changelog#corrections-to-published-schemas).) In the reference implementation, applying this block also clears the account's other data-lake settings that it does not set: the create-database and create-table default permissions, trusted resource owners and parameters. Destroying the emitted `aws_lakeformation_data_lake_settings` resource empties the admin list and resets `CROSS_ACCOUNT_VERSION` to 1. See the field's description in the [schema](/fluid/specs/0.7.5/fluid-spec.html).
+:::
+
+> 🧪 **0.7.6 preview** adds `binding.principals`, which maps the logical principals named in `accessPolicy` and `columnRestrictions` to each platform's identities, writes down what `columnRestrictions` means (deny beats allow; a restriction never grants access), and adds typed masking `params`. See [0.7.6](/fluid/releases/0.7.6).
-> ⭐ **0.7.4 — runtime enforcement.** When an expose also carries an [`mcp` block](#mcp-0-7-4-opt-this-expose-into-the-mcp-output-port-gateway), the `agentPolicy` above stops being passive metadata: the Fluid MCP gateway enforces `allowedModels` / `deniedModels` and `allowedUseCases` / `deniedUseCases` on every read. The `agentPolicy` shape is unchanged — 0.7.4 only changes *when* it bites.
+> ⭐ **0.7.4 — runtime enforcement.** When an expose also carries an [`mcp` block](#mcp-0-7-4-opt-this-expose-into-the-mcp-output-port-gateway), the `agentPolicy` above stops being passive metadata: an MCP gateway that implements 0.7.4, such as the reference implementation's, enforces `allowedModels` / `deniedModels` and `allowedUseCases` / `deniedUseCases` on every read. The `agentPolicy` shape is unchanged — 0.7.4 only changes *when* it bites.
---
-## 8. `retention` ⭐ 0.7.3 — *TTLs for everything around the product*
+## 8. `retention` (0.7.3): *TTLs for everything around the product*
+
```yaml
retention: # ISO-8601 durations; defaults shown
runState: P30D
@@ -342,8 +388,8 @@ retention: # ISO-8601 durations; defaults sh
dlq: P180D
```
-**What:** how long the platform retains operational artifacts.
-**Why:** before 0.7.3 these knobs were scattered across Airflow, S3 lifecycle rules, Datadog, etc. The retention sweeper now reads this block.
+**What:** how long the platform retains operational records — run state, run logs, lineage events, dead-letter records. It does not govern the product's data; that is `lifecycle.retention`.
+**Why:** before 0.7.3 these knobs were scattered across tools. One block in the contract states them once.
---
@@ -351,15 +397,15 @@ retention: # ISO-8601 durations; defaults sh
These are the deeper-cut blocks — every one is opt-in:
-- **`lineage`** — upstream/downstream graph, optional field-level mappings.
-- **`schemaEvolution`** — compatibility strategy (`backward`, `forward`, `full`), version pinning.
-- **`observability`** — SLIs and dashboard refs (default SLIs auto-generated).
-- **`machineLearning`** — model spec for ML-product kinds (features, training, inference).
-- **`environments`** — per-env (`dev` / `staging` / `prod`) override blocks.
-- **`lifecycle`** — `state: preview | active | deprecated | retired` + deprecation date.
-- **`docs`** — external doc URLs (runbook, design doc, dashboard).
+- **`lineage`** — `granularity` (`table_level` | `field_level`), `upstream[]` (with optional `fieldMappings`) and `downstream[]`.
+- **`schemaEvolution`** — `strategy` (`semantic_versioning` | `date_based` | `sequential`), `compatibility` (`backward_compatible` | `forward_compatible` | `full_compatible` | `breaking`) and `changePolicy`.
+- **`observability`** (per expose) — `metrics`, `onBreach` actions, `defaultSLIs`, and (0.7.3+) `alert` channels.
+- **`machineLearning`** — `enabled`, `framework` (`mlflow` | `kubeflow` | `sagemaker` | `vertex_ai` | `custom`), `models[]`.
+- **`environments`** — per-environment overrides of `metadata` and `exposes`.
+- **`lifecycle`** — `state: preview | active | deprecated | retired`, `retention` (an ISO-8601 duration) and `deprecationPolicy` (`noticePeriod`, `contact`, `replacement`). Note that `preview` here is a *product's* state; it has nothing to do with preview *schema versions*. 🧪 In the 0.7.6 preview, an expose's `lifecycle` also takes `expire: true`, which deletes data older than `retention`.
+- **`docs`** — `homepage`, `runbook`, `dictionary`, `changeLog`.
-For the field-by-field detail on each, see the generated reference: [`specs/0.7.4/fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html).
+For the field-by-field detail on each, see the generated reference: [`specs/0.7.5/fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html).
---
@@ -371,4 +417,4 @@ If you've never written a FLUID contract:
2. Copy the [Minimal Valid Contract](/fluid/schema/minimal-contract) and run it.
3. Walk through the [**Examples**](/fluid/examples/) — each example adds one block from this anatomy until you reach a production-grade source-aligned acquisition product.
4. Use the [**Cheatsheet**](/fluid/schema/cheatsheet) as a lookup table when you're authoring.
-5. Drop into the generated [`fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html) for exact types and enums.
+5. Drop into the generated [`fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html) for exact types and enums.
diff --git a/docs/schema/changelog.md b/docs/schema/changelog.md
index a862623..75ee722 100644
--- a/docs/schema/changelog.md
+++ b/docs/schema/changelog.md
@@ -1,70 +1,149 @@
# Changelog
-Human-readable diffs between consecutive versions of the FLUID schema. Each entry links to the full auto-generated change list on GitHub. The latest version is **0.7.4**.
+Human-readable summaries of what changed between consecutive versions of the FLUID schema. Each entry links to the full auto-generated change list on GitHub. **0.7.5 is the latest stable version; 0.7.6 is a preview** (see [Versions](/fluid/schema/versions#stable-and-preview-versions)).
-## 0.7.3 → 0.7.4 — Runtime agentPolicy Enforcement at the MCP Gateway
+"No narrowing change" below means that `python3 scripts/check-compat.py --from --to ` reports none: every document valid under the earlier version stays valid under the later one (once it declares the later `fluidVersion`). Each summary lists the property paths present in the later schema and absent in the earlier one, plus new enum values.
-[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.3-to-0.7.4.md)
+## 0.7.5 to 0.7.6 (preview): Declarative Packaging Modes
-This release makes an expose *agent-consumable over MCP* and turns `policy.agentPolicy` into a gate the Fluid MCP output-port gateway enforces on every read. The `policy.agentPolicy` **shape is unchanged** from 0.7.3 — the enforcement is behavioral, not a new field. **Fully backward-compatible: this release is purely additive, so every valid 0.7.3 contract validates as 0.7.4 unchanged.**
+[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.5-to-0.7.6.md) · [Release note →](/fluid/releases/0.7.6)
+
+No narrowing change. **Preview:** the 0.7.6 schema can still change before it is promoted.
+
+**Added**
+
+- Top-level `packaging` (`mode`, `pool`, `poolManifest`, `containers`) and its per-binding override `exposes[].binding.packaging`.
+- Top-level `consumers[]` (`name` and `type` required; `label`, `owner`, `url`, `maturity`, `description`, `exposeIds`).
+- `consumes[].upstreamWorkspace` and `consumes[].upstreamDigest` (`sha256:` + 64 lowercase hex), with `dependentRequired: {upstreamWorkspace: [upstreamDigest]}` — the first JSON Schema keyword in a FLUID schema that Draft 7 does not have.
+- `exposes[].semantics.measures[].aggParams` (`percentile`, `useDiscretePercentile`).
+- `exposes[].binding.encryption.kms`, with platform rules: no AWS key reference on a `gcp` binding, no Cloud KMS name on an `aws` binding.
+- `exposes[].binding.principals`, with identity patterns for `gcp` (IAM members) and `aws` (IAM ARNs).
+- `exposes[].binding.governance.lakeFormation.bucketPolicy` (`cross-account` | `none` | `all-grantees`; `cross-account` is the default). Its description says that an emitted `aws_s3_bucket_policy` is authoritative and replaces every other statement on the bucket; see the [0.7.6 notes](/fluid/releases/0.7.6#binding-fields-encryption-principals-lake-formation-bucket-policy).
+- `exposes[].lifecycle` now refers to a new `exposeLifecycle` definition: the root `lifecycle` fields plus `expire` (default `false`).
+- Typed `exposes[].policy.privacy.masking[].params`: `keepFirst`, `keepLast`, `saltEnv`, `keyEnv`.
+- `fluidVersion` accepts `"0.7.6"`.
+
+**Described:** `exposes[].policy.authz.columnRestrictions` gains a description of its semantics (deny beats allow; a restriction never grants access; principals are logical).
+
+## 0.7.4 to 0.7.5: Streaming Kafka to Iceberg Sink & Confluent Tableflow
+
+[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.4-to-0.7.5.md) · [Release note →](/fluid/releases/0.7.5)
+
+No narrowing change.
**Added**
-- **`exposes[].mcp`** (new `exposeMcp` block) opts an expose into the MCP gateway. `mcp.sampling.maxRows` (integer ≥ 1, default 100) hard-caps the `sample` tool; `mcp.classification.dataClass` (`public` / `internal` / `confidential` / `restricted`) is an advisory hint surfaced on `describe`.
-- **`binding.platform`** gains `postgres`.
-- **`binding.format`** gains `postgres_table`, `athena_table`, `glue_table` (added alongside the existing formats, including `snowflake_view` and the `redshift_*` values).
-- **`fluidVersion`** moves from `const: "0.7.3"` to `enum: ["0.7.3", "0.7.4"]`, so existing `0.7.3` values still validate.
+- On the `kafka-connect` acquisition engine (`build(s).properties.kafka-connect`): `iceberg_sink_enabled`, `sink_topics`, `streamingSink`, `iceberg_catalog_overrides`.
+- `binding.platform` values `confluent` and `pgvector`; `binding.format` value `pgvector_table`; `binding.vectorConfig`.
+- `binding.location` fields: `environment_id`, `kafka_cluster_id`, `confluent_role_arn` (Confluent Tableflow); `catalog`, `warehouse`, `uri`, `partitionBy` (Iceberg catalogs); `namespace`, `workgroup`, `iam_role_arn`, `external_schema`, `glue_database` (Redshift Serverless and external schemas); `stream` (Kinesis).
+- `fluidVersion` accepts `"0.7.5"`.
-**Unchanged & still valid.** Nothing was removed — the top-level `governance` block and `binding.governance` (AWS Lake Formation), the `snowflake_view` / `redshift_table` / `redshift_serverless` / `redshift_external_schema` formats, the `athena` / `glue` / `redshift` runtimes, `datamesh_manager` catalog registration, and the `opentofu` deployment target all remain valid in 0.7.4.
+**Re-published.** The 0.7.5 schema first published here was a snapshot taken while 0.7.5 was still a preview in the reference implementation. It lacked `pgvector`, `pgvector_table`, `vectorConfig` and the Iceberg-catalog, Redshift Serverless and Kinesis location fields, which the stable 0.7.5 has. The published file has been re-vendored from forge-cli v0.18.1, which carries the stable 0.7.5; every document the snapshot accepted is still accepted.
-**Upgrading.** Set `fluidVersion: "0.7.4"` (or keep `"0.7.3"` — it still validates). There are no removed fields to migrate off.
+## 0.7.3 to 0.7.4: Runtime agentPolicy Enforcement at the MCP Gateway
-## 0.7.2 → 0.7.3 — Source-Aligned Acquisition
+[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.3-to-0.7.4.md) · [Release note →](/fluid/releases/0.7.4)
-[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.2-to-0.7.3.md)
+No narrowing change. The gate flags `fluidVersion`, which became an enum; the waiver in `scripts/compat-waivers.txt` records why that is intended.
-Adds the `acquisition` build pattern, six ingestion engines, `build.capabilities`, delivery guarantees + DLQ, source-side `schemaEvolution`, supply-chain image signatures, and the top-level `retention` block. Additive over 0.7.2.
+**Added**
+
+- `exposes[].mcp` (`sampling.maxRows`, `classification.dataClass`).
+- `binding.platform` value `postgres`; `binding.format` values `postgres_table`, `athena_table`, `glue_table`.
+- Acquisition `catalog.register` values `glue`, `snowflake_horizon`, `unity`.
+- `fluidVersion` becomes `enum: ["0.7.3", "0.7.4"]`.
+
+`policy.agentPolicy` is unchanged in shape; 0.7.4's headline change is how the reference implementation's MCP gateway enforces it.
+
+## 0.7.2 to 0.7.3: Source-Aligned Acquisition
-## 0.7.1 → 0.7.2 — Semantic Truth Engine
+[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.2-to-0.7.3.md) · [Release note →](/fluid/releases/0.7.3)
-[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.1-to-0.7.2.md)
+No narrowing change. The gate flags the identifier pattern (upper-case letters became allowed); the waiver records the exhaustive check that shows the new pattern only widens.
-Adds the per-expose `semantics` block (entities, measures, dimensions, metrics) and `binding.icebergConfig`. Additive over 0.7.1.
+**Added**
+
+- `build.pattern` value `acquisition`, the acquisition properties (`source`, `sink`, `delivery`, `schemaEvolution`, `preLand`, `quality`, `cost`, `catalog`, `concurrency`, `lineage`, and one key per engine: `duckdb`, `airbyte`, `meltano`, `dlt`, `kafka-connect`, `debezium`), the matching `build.engine` values, and `build.capabilities`.
+- Top-level `retention`, `governance` (AWS Lake Formation `admins` and `tagDefinitions`) and `extensions`.
+- `exposes[].binding.governance` (per-resource Lake Formation settings).
+- `metadata.productType`, `metadata.classification`, `metadata.experimental`.
+- `exposes[].contract.schemaPolicy`, `exposes[].observability.alert`.
+- `binding.format` values `snowflake_view`, `redshift_table`, `redshift_serverless`, `redshift_external_schema`; `runtime.platform` values `athena`, `glue`, `redshift`.
-## 0.5.7 → 0.7.1 — Agentic Governance & Provider-First Orchestration
+## 0.7.1 to 0.7.2: Semantic Truth Engine
-[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.5.7-to-0.7.1.md)
+[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.1-to-0.7.2.md) · [Release note →](/fluid/releases/0.7.2)
-Adds `exposes[].policy.agentPolicy`, top-level `sovereignty`, top-level `accessPolicy`, and provider-first `orchestration` tasks.
+**One narrowing change:** `$defs/notification` gained `additionalProperties: false`, so an extra member on a `notifications[]` entry that 0.7.1 accepted is rejected by 0.7.2. It shipped before the gate existed and is recorded in `scripts/compat-waivers.txt`.
+
+**Added**
-## 0.4.0 → 0.5.7
+- `exposes[].semantics` (entities, measures, dimensions, metrics).
+- `exposes[].binding.icebergConfig`, `exposes[].binding.properties`, and `binding.format` value `iceberg`.
+- Top-level `orchestration` (an open object; see the note under [Cheatsheet → orchestration](/fluid/schema/cheatsheet#orchestration-fields)).
+- `exposes[].description`, `exposes[].crawler`, `exposes[].iceberg`, `exposes[].contract.quality`.
+- `metadata.provenance`, `metadata.tags`.
+- `hybrid-reference` build properties `models`, `select`, `target`.
+- `build.engine` accepts adapter-qualified dbt engines matching `^dbt-[a-z0-9]+([_-][a-z0-9]+)*$` (for example `dbt-databricks`).
+- `accessPolicy.grants[].permissions` value `create`.
+
+## 0.5.7 to 0.7.1: Agentic Governance & Provider-First Orchestration
+
+[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.5.7-to-0.7.1.md) · [Release note →](/fluid/releases/0.7.1)
+
+No narrowing change. 0.7.1 is the first version the compatibility promise covers; the gate runs from here on.
+
+**Added**
+
+- `exposes[].policy.agentPolicy`, top-level `sovereignty`, top-level `accessPolicy`.
+- `build(s).execution.orchestration`, including Airflow DAG settings and `provider_action` tasks under `airflow.tasks[]`.
+- Runtime fields (`platform`, `image`, `executor`, `serviceAccount`) and trigger fields (`cron`, `timezone`, `datasets`, `datasetsOperator`, `timetable`); trigger types `dataset`, `schedule_and_dataset`, `timetable`.
+
+---
+
+## Before 0.7.1: history before the compatibility promise
+
+These versions predate the promise, and several transitions removed or narrowed fields. The gate is not run over them in CI; the result of running it by hand is given for each.
+
+### 0.4.0 to 0.5.7 (breaking)
[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.4.0-to-0.5.7.md)
-Adds `builds`, `machineLearning`, `environments`, `lifecycle`, `docs`, and `extensions`.
+`check-compat` reports breaking changes. The contract was restructured around today's shape: `exposes[]` gains `exposeId`, `kind`, `contract` and `binding`; `consumes[]` gains `productId`, `exposeId` and `versionConstraint`; `build` gains `id`, `pattern`, `engine` and `properties`. Added at the top level: `builds`, `tags`, `labels`, `lineage`, `schemaEvolution`, `machineLearning`, `environments`, `lifecycle`, `docs`. **Removed** at the top level: `accessPolicy`, `governance`, `operations`, `security`, `slo` (`accessPolicy` returns in 0.7.1 and a different `governance` in 0.7.3).
-## 0.3.0 → 0.4.0
+### 0.3.0 to 0.4.0
[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.3.0-to-0.4.0.md)
-Adds `lineage`, `governance`, and `schemaEvolution`.
+Apart from the `$id`, the only change is in `build.transformation`: its `properties` are now chosen by `pattern` through `if`/`then` rules instead of a `oneOf`. `check-compat` reports no narrowing change.
-## 0.2.0 → 0.3.0
+### 0.2.0 to 0.3.0 (breaking)
[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.2.0-to-0.3.0.md)
-Adds the `build` block.
+`check-compat` reports breaking changes. `build` is restructured into `build.transformation` (with build patterns) and `build.execution`, replacing `build.engine`, `runtime`, `trigger`, `retries` and `notifications` at the `build` level. `governance` gains `lineage`, `regulatory` and `stewardship` in place of `rules`.
-## 0.1.1 → 0.2.0
+### 0.1.1 to 0.2.0 (breaking)
[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.1.1-to-0.2.0.md)
-Adds the `consumes` block.
+`check-compat` reports breaking changes: patterns are added to ids, `domain`, column names and access principals, bounds to SLA and lifecycle numbers, and a pattern to cron triggers. `exposes[].mappings` is added.
-## 0.1.0 → 0.1.1
+### 0.1.0 to 0.1.1
[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.1.0-to-0.1.1.md)
-## 0.0.1 → 0.1.0
+No narrowing change. Note that 0.1.0 and 0.1.1 declare the same `$id` (see [Versions](/fluid/schema/versions#_0-4-0-and-earlier-ids-that-do-not-resolve)).
+
+### 0.0.1 to 0.1.0 (breaking)
[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.0.1-to-0.1.0.md)
+
+`check-compat` reports breaking changes. Top-level `conformance`, `dynamicPolicies` and `extensions` are removed (`extensions` returns in 0.7.3), and `slo` is added. `consumes` and `build` exist from 0.0.1 on.
+
+---
+
+## Corrections to published schemas
+
+- **Lake Formation `admins` (0.7.3, 0.7.4, 0.7.5).** These schemas were first published with a description of `governance.lakeFormation.admins` saying that principals not listed are not removed. That is the opposite of what applying it does: the list is **authoritative**, and an admin not listed — including the identity running the apply — loses Lake Formation admin. The description was corrected in the reference implementation and the published files carry the corrected text. Only the description changed; the same documents validate.
+- **0.7.5** was re-published with the stable content (above).
diff --git a/docs/schema/cheatsheet.md b/docs/schema/cheatsheet.md
index c4f0be0..b3047a5 100644
--- a/docs/schema/cheatsheet.md
+++ b/docs/schema/cheatsheet.md
@@ -1,276 +1,299 @@
# Schema Cheatsheet
-One row per top-level field in the latest schema (**0.7.4**). Use this as a lookup table when you're reading or writing a `.fluid.yml` — find the field on the left, get a one-line meaning, find out if it's required, and jump to the deep-dive.
+One row per field of the latest stable schema, **0.7.5**: a one-line meaning, whether it is required, and the version it first appeared in. Rows marked 🧪 are in the **0.7.6 preview** only; a contract needs `fluidVersion: "0.7.6"` to use them, and they can still change.
> 🧭 For a guided tour with context per block, read the [**Schema Anatomy**](/fluid/schema/anatomy).
-> 📚 For the exhaustive field-by-field reference, see [`specs/0.7.4/fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html).
+> 📚 For the exhaustive field-by-field reference, see [`specs/0.7.5/fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html) (stable) or [`specs/0.7.6/fluid-spec.html`](/fluid/specs/0.7.6/fluid-spec.html) (preview).
+
+"Since" is the version from which the field has been in the schema without interruption; where a member of the same name existed earlier and was removed, the row says so. Contracts before 0.5.7 had a different overall shape (see the [Changelog](/fluid/schema/changelog#before-0-7-1-history-before-the-compatibility-promise)), so for those blocks the current *shape* is newer than the name.
---
## Top-level fields
-| Field | Purpose | Required | Added in | Type |
+The root object is closed: a top-level key not listed here makes the document invalid.
+
+| Field | Purpose | Required | Since | Type |
|---|---|:-:|:-:|---|
-| `fluidVersion` | Which contract version this file targets. Accepts `"0.7.4"` (or `"0.7.3"`). | ✅ | 0.1.0 | `enum` |
-| `kind` | Product kind: `DataProduct` \| `MLPipeline`. | ✅ | 0.1.0 | `enum` |
-| `id` | Globally unique, dot-separated product id. The public address of this product. | ✅ | 0.1.0 | `string` |
-| `name` | Human-readable display name. | ✅ | 0.1.0 | `string` |
-| `description` | Business-facing summary. | | 0.1.0 | `string` |
-| `domain` | Owning business domain (e.g. `Finance`, `Customer Experience`). | | 0.1.0 | `string` |
-| `tags` | Free-text categorization labels. | | 0.1.0 | `string[]` |
-| `labels` | Key/value categorization for catalog and IAM. | | 0.1.0 | `map` |
-| `metadata` | Owner, layer, business context. | ✅ | 0.1.0 | `object` |
-| `consumes` | Upstream FLUID products this product depends on (with version constraints). | | 0.2.0 | `object[]` |
-| `exposes` | Ports this product publishes (table / api / stream / feature_store / …). | ✅ | 0.1.0 | `object[]` |
-| `build` | How the product is produced (pattern + engine + properties). | | 0.3.0 | `object` |
-| `builds` | Multi-build orchestration (when one product has several builds). | | 0.5.7 | `object[]` |
-| `orchestration` | Schedule + tasks for the build. Provider-first since 0.7.0. | | 0.7.0 | `object` |
-| `sovereignty` ⭐ | Data residency & jurisdictional compliance. | | **0.7.1** | `object` |
-| `accessPolicy` ⭐ | Root-level IAM grants — generates cloud IAM bindings. | | **0.7.1** | `object` |
-| `retention` ⭐ | TTLs for runState / runLogs / lineage / dlq (ISO-8601). | | **0.7.3** | `object` |
-| `lineage` | Upstream/downstream graph + optional field-level mappings. | | 0.4.0 | `object` |
-| `governance` | Contract-level (account/project-wide) governance — AWS Lake Formation admins + LF-tag definitions. Per-resource grants live under `binding.governance`. | | 0.4.0 | `object` |
-| `schemaEvolution` | Compatibility strategy (`backward` / `forward` / `full`). | | 0.4.0 | `object` |
-| `machineLearning` | Model spec for ML-product kinds (features, training, inference). | | 0.5.7 | `object` |
-| `environments` | Per-environment overrides (`dev` / `staging` / `prod`). | | 0.5.7 | `object` |
-| `lifecycle` | `state: preview \| active \| deprecated \| retired` + deprecation date. | | 0.5.7 | `object` |
-| `docs` | External documentation URLs (runbook, design doc, dashboard). | | 0.5.7 | `object` |
-| `extensions` | Escape hatch for vendor-specific additions. | | 0.5.7 | `object` |
+| `fluidVersion` | The schema version this document declares; it selects the schema the document is validated against. 0.7.5 accepts `"0.7.3"`, `"0.7.4"`, `"0.7.5"` — but see [Choosing `fluidVersion`](/fluid/schema/versions#choosing-fluidversion). | ✅ | 0.0.1 | `enum` |
+| `kind` | `DataProduct` \| `MLPipeline`. | ✅ | 0.0.1 | `enum` |
+| `id` | Globally unique product id — the product's public address. | ✅ | 0.0.1 | identifier |
+| `name` | Human-readable display name. | ✅ | 0.0.1 | `string` |
+| `metadata` | Owner, layer, product type, business context. | ✅ | 0.0.1 | `object` |
+| `exposes` | Ports this product publishes. | ✅ | 0.0.1 | `object[]` |
+| `description` | Business-facing summary. | | 0.0.1 | `string` |
+| `domain` | Owning business domain. | | 0.0.1 | `string` |
+| `consumes` | Upstream FLUID products this product reads. | | 0.0.1 | `object[]` |
+| `build` | How the product is produced (pattern + engine + properties). | | 0.0.1 | `object` |
+| `builds` | Several builds for one product, same shape as `build`. | | 0.5.7 | `object[]` |
+| `tags` | Lower-case categorization tags (`^[a-z0-9][a-z0-9-]*[a-z0-9]$`). | | 0.5.7 | `string[]` |
+| `labels` | Key/value labels. | | 0.5.7 | `map` |
+| `lineage` | `granularity`, `upstream[]` (with `fieldMappings`), `downstream[]`. | | 0.5.7 | `object` |
+| `schemaEvolution` | `strategy` and `compatibility` of the product's schema changes. | | 0.5.7 | `object` |
+| `machineLearning` | `enabled`, `framework`, `models[]`. | | 0.5.7 | `object` |
+| `environments` | Per-environment overrides of `metadata` and `exposes`. | | 0.5.7 | `map` |
+| `lifecycle` | `state`, `retention`, `deprecationPolicy`. | | 0.5.7 | `object` |
+| `docs` | `homepage`, `runbook`, `dictionary`, `changeLog`. | | 0.5.7 | `object` |
+| `sovereignty` | Jurisdiction and data residency. | | **0.7.1** | `object` |
+| `accessPolicy` | Access grants for the product. (A different `accessPolicy` existed from 0.0.1 to 0.4.0.) | | **0.7.1** | `object` |
+| `orchestration` | Scheduling and workflow-engine settings. Open object — see [below](#orchestration-fields). | | **0.7.2** | `object` |
+| `retention` | ISO-8601 TTLs for run state, run logs, lineage events and DLQ records. | | **0.7.3** | `object` |
+| `governance` | Account-wide AWS Lake Formation settings: `admins`, `tagDefinitions`. (A different `governance` existed from 0.0.1 to 0.4.0.) | | **0.7.3** | `object` |
+| `extensions` | Vendor- or plugin-namespaced configuration. Open object. (Also in 0.0.1; absent from 0.1.0 to 0.7.2.) | | **0.7.3** | `object` |
+| 🧪 `packaging` | Container ownership: `mode` (`isolated` \| `shared`), `pool`, `poolManifest`, `containers`. | | 0.7.6 | `object` |
+| 🧪 `consumers` | Declared downstream artifacts: `name` and `type` required; `label`, `owner`, `url`, `maturity`, `exposeIds`. | | 0.7.6 | `object[]` |
---
-## `metadata` (required) — fields
+## `metadata` (required) fields
-`metadata` itself is required at the top level, but only `metadata.owner` is required inside it.
+`metadata` is required, and inside it only `owner` is. The schema requires **no** member of `owner`, so `owner: {}` validates; name at least a `team`.
-| Field | Purpose | Required |
-|---|---|:-:|
-| `metadata.owner` | Owning team object. | ✅ |
-| `metadata.owner.team` | Owning team handle. | ✅ |
-| `metadata.owner.email` | Owner email for alerts and audit. | |
-| `metadata.owner.slack` | Owner Slack channel. | |
-| `metadata.layer` | Medallion layer label. **Free-form string** — `Bronze`/`Silver`/`Gold` is convention, not enum. | |
-| `metadata.businessContext.domain` | Business domain. | |
-| `metadata.businessContext.subdomain` | Business subdomain. | |
-| `metadata.provenance` | Auto-injected generation envelope (tool + version + timestamp). Machine-written. | |
+| Field | Purpose | Required | Since |
+|---|---|:-:|:-:|
+| `metadata.owner` | Owner object: `team`, `email`, `slack`, `oncall`. | ✅ | 0.0.1 |
+| `metadata.layer` | Layer label. **Free-form string** — `Bronze`/`Silver`/`Gold` is convention, not enum. | | 0.0.1 |
+| `metadata.productType` | `SDP` (source-aligned) \| `ADP` (aggregated) \| `CDP` (consumption-aligned). | | 0.7.3 |
+| `metadata.classification` | `public` \| `internal` \| `confidential` \| `restricted`. (A `classification` also existed in 0.0.1.) | | 0.7.3 |
+| `metadata.experimental` | Experimental features the contract opts into. | | 0.7.3 |
+| `metadata.businessContext` | `domain`, `subdomain`, `businessCapability`, `valueStream`. | | 0.5.7 |
+| `metadata.tags` | Tags on the metadata. (Also existed from 0.0.1 to 0.4.0.) | | 0.7.2 |
+| `metadata.createdAt` | Creation time (`format: date-time`, an annotation). | | 0.5.7 |
+| `metadata.provenance` | Generation envelope written by tooling. | | 0.7.2 |
---
-## `exposes[]` — fields
+## `exposes[]` fields
-Each entry in `exposes[]` requires `exposeId`, `kind`, `contract`, and `binding`.
+Each entry requires `exposeId`, `kind`, `contract` and `binding`, and is closed.
| Field | Purpose | Required |
|---|---|:-:|
| `exposes[].exposeId` | Stable id of this port within the product. | ✅ |
| `exposes[].kind` | `table` \| `view` \| `api` \| `file` \| `stream` \| `topic` \| `feature_store` \| `model` \| `vector` \| `graph` \| `time_series` \| `other`. | ✅ |
-| `exposes[].contract` | Schema + dq + policy. Must define at least `schema` or `openapiRef`. | ✅ |
-| `exposes[].binding` | Where it lives — `platform`, `format`, `location`. | ✅ |
-| `exposes[].description` | What this port is. *(NEW in 0.7.2)* | |
+| `exposes[].contract` | The data's shape and promises. Must contain `schema` or `openapiRef`. | ✅ |
+| `exposes[].binding` | Where it lives — `platform`, `format`, `location` (all three required). | ✅ |
+| `exposes[].policy` | **Sibling of `contract`**, not inside it: `authn`, `authz`, `privacy`, `classification`, `agentPolicy`. | |
+| `exposes[].title`, `exposes[].description` | Display name and description of the port (`description` since 0.7.2; it also existed from 0.1.0 to 0.4.0). | |
| `exposes[].version` | Semver of this port. | |
-| `exposes[].contract.schema[]` | Column list with type, required, sensitivity, tags. | (one of) |
-| `exposes[].contract.openapiRef` | Reference to an external OpenAPI spec (for `kind: api`). | (one of) |
-| `exposes[].contract.dq.rules[]` | Declarative quality assertions. | |
-| `exposes[].contract.policy` | Privacy + RBAC + column-level access. | |
-| `exposes[].semantics` ⭐ 0.7.2 | Entities, measures, dimensions, metrics. | |
-| `exposes[].mcp` ⭐ 0.7.4 | Opts this expose into the Fluid MCP output-port gateway. Presence enables runtime `agentPolicy` enforcement on every read. | |
-| `exposes[].mcp.sampling.maxRows` ⭐ 0.7.4 | Hard cap for the `sample` MCP tool. Integer ≥ 1, default `100`. | |
-| `exposes[].mcp.classification.dataClass` ⭐ 0.7.4 | Advisory data class surfaced on `describe`: `public` \| `internal` \| `confidential` \| `restricted`. | |
-| `exposes[].binding.platform` | `gcp` \| `aws` \| `azure` \| `snowflake` \| `databricks` \| `kafka` \| `local` \| `kubernetes` \| `postgres` ⭐ 0.7.4 \| `other`. | |
-| `exposes[].binding.format` | `bigquery_table` \| `snowflake_table` \| `snowflake_view` \| `iceberg` \| `delta_table` \| `parquet` \| `csv` \| `json` \| `http_api` \| `grpc_api` \| `kafka_topic` \| `pubsub_topic` \| `gcs_file` \| `s3_file` \| `redshift_table` \| `redshift_serverless` \| `redshift_external_schema` \| `postgres_table` ⭐ 0.7.4 \| `athena_table` ⭐ 0.7.4 \| `glue_table` ⭐ 0.7.4 \| `other`. | |
-| `exposes[].binding.icebergConfig` ⭐ 0.7.2 | Iceberg specifics when `format: iceberg`. | |
-| `exposes[].binding.governance` | Per-resource AWS Lake Formation governance — `registerLocation`, principal grants, tag associations, row/column filters. Account-wide settings live in the top-level `governance` block. | |
-| `exposes[].qos` | Availability, freshness, latency targets. | |
+| `exposes[].contract.schema[]` | Columns: `name` and `type` required; `required`, `description`, `sensitivity`, `semanticType`, `businessName`, `businessDefinition`, `validationRules`, `tags`, `labels`. | (one of) |
+| `exposes[].contract.openapiRef` | Reference to an OpenAPI document (for `kind: api`). | (one of) |
+| `exposes[].contract.dq.rules[]` | Declarative quality assertions (below). | |
+| `exposes[].contract.schemaPolicy` | `strict` \| `discover_and_freeze` \| `evolve_safe` \| `evolve_all` (since 0.7.3). | |
+| `exposes[].contract.schemaSignature` | `sha256:` + 64 hex — a digest of the schema. | |
+| `exposes[].qos` | `availability` (a percentage string such as `"99.9%"`), `freshnessSLO`, `latencyP95` (ISO-8601 durations), `dataLossSLO`, `completenessTarget`, `errorBudget`. | |
+| `exposes[].semantics` | Entities, measures, dimensions, metrics (since 0.7.2; a `semantics` member with a different shape existed from 0.1.0 to 0.4.0). | |
+| `exposes[].mcp` | Serve the port to AI agents over MCP (since 0.7.4): `sampling.maxRows` (integer ≥ 1), `classification.dataClass` (`public` \| `internal` \| `confidential` \| `restricted`). | |
+| `exposes[].lifecycle`, `exposes[].observability`, `exposes[].docs` | Per-port lifecycle, observability (`metrics`, `onBreach`, `defaultSLIs`, `alert`) and docs. | |
+| `exposes[].binding.platform` | `gcp` \| `aws` \| `azure` \| `snowflake` \| `databricks` \| `kafka` \| `confluent` (0.7.5) \| `local` \| `kubernetes` \| `postgres` (0.7.4) \| `pgvector` (0.7.5) \| `other`. | ✅ |
+| `exposes[].binding.format` | `bigquery_table` \| `snowflake_table` \| `snowflake_view` \| `iceberg` \| `delta_table` \| `parquet` \| `csv` \| `json` \| `http_api` \| `grpc_api` \| `kafka_topic` \| `pubsub_topic` \| `gcs_file` \| `s3_file` \| `redshift_table` \| `redshift_serverless` \| `redshift_external_schema` \| `postgres_table` (0.7.4) \| `athena_table` (0.7.4) \| `glue_table` (0.7.4) \| `pgvector_table` (0.7.5) \| `other`. | ✅ |
+| `exposes[].binding.location` | Platform address; no member is required by the schema. 0.7.5 adds `environment_id`, `kafka_cluster_id`, `confluent_role_arn` (Tableflow), `catalog`, `warehouse`, `uri`, `partitionBy` (Iceberg catalogs), `namespace`, `workgroup`, `iam_role_arn`, `external_schema`, `glue_database` (Redshift Serverless), `stream` (Kinesis). | ✅ |
+| `exposes[].binding.icebergConfig` | Iceberg table settings when `format: iceberg` (since 0.7.2). | |
+| `exposes[].binding.vectorConfig` | Vector output port for `pgvector` (since 0.7.5): `dimensions` (required, 1–16000), `embeddingModel`, `vectorType`, `indexType`, `distanceMetric`, `hnsw`, `ivfflat`, `table`, `sourceKeyColumn`. | |
+| `exposes[].binding.governance` | Per-resource AWS Lake Formation settings: `registerLocation`, `grants[]`, `tags`, `rowFilter` (since 0.7.3). Account-wide settings live in the top-level `governance` block. | |
+| 🧪 `exposes[].binding.encryption.kms` | `product` \| `none` \| an AWS KMS alias or ARN \| a Cloud KMS key name. | |
+| 🧪 `exposes[].binding.principals` | Logical principal → identity (or list) on this platform. | |
+| 🧪 `exposes[].binding.packaging` | Per-binding override of `packaging`. | |
+| 🧪 `exposes[].binding.governance.lakeFormation.bucketPolicy` | `cross-account` (default) \| `none` \| `all-grantees`. **Authoritative** — an emitted bucket policy replaces every other statement on the bucket; see the [0.7.6 notes](/fluid/releases/0.7.6#binding-fields-encryption-principals-lake-formation-bucket-policy). | |
+| 🧪 `exposes[].lifecycle.expire` | `true` deletes data older than `retention` (default `false`). | |
| `exposes[].tags`, `exposes[].labels` | Port-level categorization. | |
-### `contract.dq.rules[]` — fields
+### `contract.dq.rules[]` fields
| Field | Purpose | Required |
|---|---|:-:|
-| `dq.rules[].id` | Rule id (unique within the contract). | ✅ |
+| `dq.rules[].id` | Rule id. | ✅ |
| `dq.rules[].type` | `freshness` \| `completeness` \| `uniqueness` \| `valid_values` \| `accuracy` \| `schema` \| `anomaly_detection` \| `drift_detection`. | ✅ |
| `dq.rules[].severity` | `info` \| `warn` \| `error` \| `critical`. | ✅ |
-| `dq.rules[].selector` | SQL-like predicate (e.g. `"amount > 0"`). | |
+| `dq.rules[].selector` | Predicate or column the rule applies to (e.g. `"amount > 0"`). | |
| `dq.rules[].threshold` | Numeric threshold for the operator. | |
| `dq.rules[].operator` | `>=` \| `>` \| `<=` \| `<` \| `==` \| `!=`. | |
| `dq.rules[].window` | ISO-8601 duration window (e.g. `PT15M`). | |
-| `dq.rules[].description` | Human-readable purpose. | |
---
-## `build` — fields
+## `build` / `builds[]` fields
+
+A build is closed and has no required member.
| Field | Purpose | Required |
|---|---|:-:|
-| `build.pattern` | `hybrid-reference` \| `embedded-logic` \| `multi-stage` \| `acquisition` ⭐ 0.7.3. | |
-| `build.engine` | `dbt` \| `sql` \| `python` \| `spark` \| `glue` \| `custom` \| `duckdb` ⭐ \| `airbyte` ⭐ \| `meltano` ⭐ \| `dlt` ⭐ \| `kafka-connect` ⭐ \| `debezium` ⭐ (⭐ added 0.7.3). | |
-| `build.capabilities` ⭐ 0.7.3 | What the build asks for: `full_refresh`, `incremental_dedup`, `cdc`, `schema_discovery`, `dlp_scan`, `exactly_once`, … | |
-| `build.properties` | Pattern-specific properties (validated conditionally). | |
-| `build.execution.trigger` | `schedule` / `event` / `manual`. | |
-| `build.execution.runtime` | Where it runs (platform + resources). | |
-| `build.execution.retries` | Retry policy. | |
-| `build.outputs` | List of `exposeId`s this build produces. | |
-| `build.dependencies` | Other builds this one waits on. | |
-| `build.properties` | Pattern-specific payload — schema validates this against `acquisitionPattern` when `pattern: acquisition` (see below). | (when `pattern: acquisition`) |
+| `build.id` | Build id; give one to each entry of `builds[]`. | |
+| `build.pattern` | `hybrid-reference` \| `embedded-logic` \| `multi-stage` \| `acquisition` (0.7.3). Selects the shape of `properties`. | |
+| `build.engine` | `dbt` \| `sql` \| `python` \| `spark` \| `glue` \| `custom` \| `duckdb` \| `airbyte` \| `meltano` \| `dlt` \| `kafka-connect` \| `debezium` (the last six since 0.7.3), or `dbt-` such as `dbt-databricks` (since 0.7.2). | |
+| `build.capabilities` | What the build asks of its runner: `full_refresh`, `incremental_append`, `incremental_dedup`, `incremental_merge`, `cdc`, `streaming`, `schema_discovery`, `schema_evolution`, `dlp_scan`, `at_most_once`, `at_least_once`, `exactly_once` (since 0.7.3). | |
+| `build.properties` | Pattern-specific settings. `hybrid-reference` requires `model`; `embedded-logic` requires `sql`; `acquisition` requires `source` (below). | |
+| `build.execution.trigger` | `type`: `schedule` \| `event` \| `manual` \| `dependency` \| `dataset` \| `schedule_and_dataset` \| `timetable`, plus `schedule`, `cron`, `timezone`, … | |
+| `build.execution.runtime` | Where it runs: `platform`, `resources`, `image`, `executor`, `serviceAccount`, `timeout`. | |
+| `build.execution.retries` | `maxAttempts`, `backoffStrategy` (`fixed` \| `exponential` \| `linear`), `initialDelay`, `maxDelay`. | |
+| `build.execution.notifications[]` | `type` (`email` \| `slack` \| `webhook` \| `pagerduty`), `target`, `condition` (`success` \| `failure` \| `always`). Closed since 0.7.2. | |
+| `build.execution.orchestration` | The same shape as top-level `orchestration`. | |
+| `build.outputs` | The `exposeId`s this build produces. | |
+| `build.repository` | Where referenced code lives. | |
---
-## `build.properties` when `pattern: acquisition` ⭐ 0.7.3 — fields
+## `build.properties` when `pattern: acquisition` fields
-When `build.pattern: acquisition`, the schema validates `build.properties` against the `acquisitionPattern` shape below. (For other patterns, `properties` conforms to `hybridReferencePattern` / `embeddedLogicPattern` / `multiStagePattern` respectively.)
+When `pattern: acquisition` (0.7.3+), the schema validates `properties` against the acquisition shape below. (For the other patterns it uses the hybrid-reference, embedded-logic and multi-stage shapes.)
| Field (under `build.properties`) | Purpose | Required |
|---|---|:-:|
-| `source.kind` | `filesystem` \| `postgres` \| `mysql` \| `sqlite` \| `http` \| `salesforce` \| `stripe` \| … (free string with example values, no enum). | ✅ |
+| `source.kind` | Source system — a free string, e.g. `postgres`, `kafka`, `salesforce`. | ✅ |
| `source.mode` | `full_refresh` \| `incremental_append` \| `incremental_dedup` \| `incremental_merge` \| `cdc` \| `streaming`. | ✅ |
-| `source.cursor_field` | Column used as incremental cursor. | |
-| `source.connection` | Connection details. `secretRef` must be a URI: `vault://...`, `aws://...`, `gcp://...`, `azure://...`, or `env://...`. Inline secrets are rejected by the validator. | |
-| `source.streams` | List of streams/tables/objects to ingest. | |
+| `source.cursor_field` | Column used as the incremental cursor. | |
+| `source.connection` | Connection details (open object). `secretRef` must be a URI matching `://…`; the schema's description names `vault://`, `aws://`, `gcp://`, `azure://` and `env://`. | |
+| `source.streams` | Streams / tables / objects to ingest. | |
| `sink.format` | `iceberg` \| `delta` \| `parquet` \| `csv` \| `json` \| `snowflake_table` \| `bigquery_table` \| `redshift_table` \| `duckdb_table`. | |
-| `sink.partitionBy` | Array of **strings** (function-form, e.g. `["day(ingested_at)"]`). Note: `binding.icebergConfig.partitionSpec` uses an object form — different shape. | |
-| `sink.catalog` | Catalog identifier (e.g. `glue`, `snowflake_horizon`). | |
-| `delivery.guarantee` | `at_most_once` \| `at_least_once` \| `exactly_once`. | |
-| `delivery.idempotencyKey` | Template, e.g. `"${stream}\|${batch_id}"`. | |
-| `delivery.dlq.enabled` | Boolean — DLQ on/off (default `true`). | |
-| `delivery.dlq.sink.format` | `parquet` \| `json` \| `ndjson`. | |
-| `delivery.dlq.sink.location` | DLQ destination URI. | |
-| `delivery.dlq.maxRecordsBeforeAbort` | Integer (default 10000). | |
-| `delivery.dlq.alertOn` | List: `pii_classification_failed` \| `schema_violation` \| `destination_write_failed` \| `quality_gate_failed`. | |
-| `schemaEvolution.policy` | `strict` \| `discover_and_freeze` \| `evolve_safe` \| `evolve_all`. | |
-| `schemaEvolution.onAddedColumn` | `include` \| `warn` \| `fail`. | |
-| `schemaEvolution.onRemovedColumn` | `drop` \| `warn` \| `fail`. | |
-| `schemaEvolution.onTypeChange` | `cast` \| `warn` \| `fail`. | |
-| `schemaEvolution.sourceFingerprint` | `required` \| `optional` \| `disabled` (default: `required`). | |
+| `sink.catalog` | `rest` \| `glue` \| `nessie` \| `unity` \| `snowflake-managed` \| `hive`. | |
+| `sink.partitionBy` | Array of **strings** (function form, e.g. `["day(ingested_at)"]`) — not the object form of `binding.icebergConfig.partitionSpec`. | |
+| `delivery.guarantee` | `at_most_once` \| `at_least_once` (default) \| `exactly_once`. | |
+| `delivery.idempotencyKey` | Key template (default `{run_id}:{stream}:{record_pk}`). | |
+| `delivery.dlq` | `enabled` (default `true`), `sink.format` (`parquet` \| `json` \| `ndjson`), `sink.location`, `maxRecordsBeforeAbort` (default 10000), `alertOn` (`pii_classification_failed` \| `schema_violation` \| `destination_write_failed` \| `quality_gate_failed`). | |
+| `schemaEvolution.policy` | `strict` (default) \| `discover_and_freeze` \| `evolve_safe` \| `evolve_all`. | |
+| `schemaEvolution.onAddedColumn` / `onRemovedColumn` / `onTypeChange` | `include` \| `warn` \| `fail` / `drop` \| `warn` \| `fail` / `cast` \| `warn` \| `fail`. | |
+| `schemaEvolution.sourceFingerprint` | `required` (default) \| `optional` \| `disabled`. | |
| `preLand` | Hook chain: `dlp_scan`, `tokenize_pii`, `quality_gate`, `emit_lineage_input`. | |
-| `quality` | Pre-land quality gates (gate-or-quarantine semantics). | |
-| `cost.budget.monthly` | `{ rows, bytes (e.g. "50GB"), computeMinutes }`. | |
-| `cost.budget.onExceed` | `warn` (default) \| `abort`. | |
-| `cost.chargeback` | `{ team, project, costCenter }`. | |
-| `catalog.register` | Catalog auto-registration targets: `datahub`, `openmetadata`, `datamesh_manager`, `unity`, `glue`, `snowflake_horizon`. | |
-| `concurrency.lock` | Single-flight lock scope (`product` or `build`). | |
-| `lineage.emit` | Whether to emit OpenLineage events (default: `true`). | |
-| `` (`airbyte` \| `meltano` \| `dlt` \| `duckdb` \| `kafka-connect` \| `debezium`) | Engine-specific config block. | |
-| `.deployment.mode` | `embedded` \| `bring-your-own` \| `managed`. | |
-| `.deployment.managed.target` | When `managed`: `docker` \| `kubernetes` \| `terraform` \| `opentofu`. | |
-| `.image_signature.verifier` | `cosign` (only value today, default `cosign`). | (prod) |
-| `.image_signature.publicKey` | Public key reference for signature verification. | (prod) |
-| `.image_signature.slsaProvenance` | `required` \| `optional` (default) \| `disabled`. | (prod) |
+| `quality` | Pre-land quality `gates[]` (`rule`, `severity` required), `onError`, `anomalies[]`. | |
+| `cost.budget` | `monthly` (`rows`, `bytes`, `computeMinutes`) and `onExceed` (`warn` (default) \| `abort`). | |
+| `cost.chargeback` | `team`, `project`, `costCenter`. | |
+| `catalog.register` | `datahub`, `openmetadata`, `datamesh_manager`, `unity`, `glue`, `snowflake_horizon` (the last three since 0.7.4). | |
+| `concurrency.lock` | `scope` (`product` (default) \| `build`), `timeout`, `onContended` (`abort` \| `queue` \| `replace`). | |
+| `lineage.emit` | Emit lineage events (default `true`). | |
+| `` (`duckdb` \| `airbyte` \| `meltano` \| `dlt` \| `kafka-connect` \| `debezium`) | Engine-specific settings. | |
+| `.deployment.mode` | `embedded` (default) \| `bring-your-own` \| `managed`; with `managed`, `managed.target` is `docker` \| `kubernetes` \| `terraform` \| `opentofu`. | |
+| `.image_signature` | `verifier` (`cosign`), `publicKey`, `slsaProvenance` (`required` \| `optional` (default) \| `disabled`). | |
+| `kafka-connect.iceberg_sink_enabled`, `sink_topics`, `streamingSink`, `iceberg_catalog_overrides` | Opt-in Iceberg streaming sink (since 0.7.5). | |
---
-## `exposes[].mcp` ⭐ 0.7.4 — fields
-
-The presence of a (non-empty) `mcp` block opts the expose into the Fluid MCP output-port gateway, where `policy.agentPolicy` is enforced at runtime on every read. The `agentPolicy` shape is unchanged from 0.7.3.
+## `exposes[].policy` fields
| Field | Purpose | Required |
|---|---|:-:|
-| `exposes[].mcp.sampling.maxRows` | Hard cap for the `sample` MCP tool (defends against an over-curious agent). Integer ≥ 1, default `100`. | |
-| `exposes[].mcp.classification.dataClass` | Advisory data class surfaced on the `describe` tool: `public` \| `internal` \| `confidential` \| `restricted`. The real gate is `policy.agentPolicy`. | |
-
----
+| `exposes[].policy.authn` | `oidc` \| `oauth2` \| `api_key` \| `none` \| `custom` \| `iam` \| `jwt`. | |
+| `exposes[].policy.authz.readers` / `writers` | Principals with read / write access. | |
+| `exposes[].policy.authz.columnRestrictions[]` | `principal`, `columns`, `access` (`allow` \| `deny`). 🧪 The 0.7.6 schema defines its meaning: a column named in any restriction is restricted; `deny` removes access; `allow` limits the columns to the principals an `allow` names; deny beats allow; a restriction never grants access. | |
+| `exposes[].policy.privacy.masking[]` | `{column, strategy}` required; `strategy` is `mask` \| `hash` \| `tokenize` \| `encrypt` \| `k_anonymity`; `params` is open. 🧪 0.7.6 types `params`: `keepFirst`, `keepLast` (mask), `saltEnv` (hash), `keyEnv` (tokenize, encrypt). | |
+| `exposes[].policy.privacy.rowLevelPolicy.expression` | Provider-specific row filter predicate (required inside `rowLevelPolicy`). | |
+| `exposes[].policy.classification` | `Public` \| `Internal` \| `Confidential` \| `Restricted`. | |
+| `exposes[].policy.agentPolicy` | AI-consumer policy (below). | |
-## `exposes[].policy.agentPolicy` ⭐ 0.7.1 — fields
+### `exposes[].policy.agentPolicy` (0.7.1) fields
-> ⚠️ **Location.** `agentPolicy` is **per-expose**, nested under `exposes[].policy.agentPolicy`. It is not a top-level property in the schema (despite the 0.7.1 release notes showing it that way).
+> ⚠️ **Location.** `agentPolicy` is **per-expose**, under `exposes[].policy.agentPolicy`. It is not a top-level property.
>
-> ⭐ **0.7.4 — runtime enforcement.** When the same expose carries an `mcp` block, these `allowedModels` / `deniedModels` and `allowedUseCases` / `deniedUseCases` fields are enforced by the Fluid MCP gateway on every read (no shape change).
+> **0.7.4 — runtime enforcement.** When the same expose carries an `mcp` block, an MCP gateway that implements 0.7.4 enforces the model and use-case lists on every read; the reference implementation's gateway does. The shape is unchanged.
-| Field | Purpose | Required |
+| Field | Purpose | Default |
|---|---|:-:|
-| `…policy.agentPolicy.allowedModels` | Whitelist of AI model ids (e.g. `gpt-4`, `claude-3-opus`). Empty array = no AI access. | |
-| `…policy.agentPolicy.deniedModels` | Blacklist (takes precedence over `allowedModels`). | |
-| `…policy.agentPolicy.maxTokensPerRequest` | Per-request token cap (positive integer). | |
-| `…policy.agentPolicy.maxTokensPerDay` | Per-day token cap. | |
-| `…policy.agentPolicy.allowedUseCases` | Permitted use cases from the schema's controlled vocabulary: `inference` \| `reasoning` \| `analysis` \| `summarization` \| `classification` \| `embedding` \| `search` \| `qa` \| `code_generation` \| `fine_tuning` \| `training` \| `rag`. **Custom strings are rejected.** | |
-| `…policy.agentPolicy.deniedUseCases` | Same controlled vocabulary as `allowedUseCases`. Useful pattern: allow `inference, qa, rag`; deny `training, fine_tuning`. | |
-| `…policy.agentPolicy.canReason` | Boolean — allow chain-of-thought / multi-step reasoning? (default `false`) | |
-| `…policy.agentPolicy.canStore` | Boolean — allow caching / persistence? (default `false`) | |
-| `…policy.agentPolicy.retentionPolicy.maxRetentionDays` | Max days AI may retain data (`0` = no retention). | |
-| `…policy.agentPolicy.retentionPolicy.requireDeletion` | Boolean — must AI delete after use? (default `true`) | |
-| `…policy.agentPolicy.auditRequired` | Boolean — log all AI consumption? (default `true`) | |
-| `…policy.agentPolicy.purposeLimitation` | Free-text purpose statement, e.g. `"Customer support chatbot only"`. | |
-
-## `exposes[].policy` — sibling fields
+| `…agentPolicy.allowedModels` | Model ids allowed to read the expose (lower-case ids such as `gpt-4`, `claude-3-opus`). | |
+| `…agentPolicy.deniedModels` | Model ids denied. | |
+| `…agentPolicy.maxTokensPerRequest` / `maxTokensPerDay` | Token caps (integers ≥ 1). | |
+| `…agentPolicy.allowedUseCases` / `deniedUseCases` | From a fixed vocabulary: `inference` \| `reasoning` \| `analysis` \| `summarization` \| `classification` \| `embedding` \| `search` \| `qa` \| `code_generation` \| `fine_tuning` \| `training` \| `rag`. **Other strings are invalid.** | |
+| `…agentPolicy.canReason` | Allow multi-step reasoning over the data. | `false` |
+| `…agentPolicy.canStore` | Allow caching or persisting the data. | `false` |
+| `…agentPolicy.retentionPolicy` | `maxRetentionDays` (≥ 0), `requireDeletion`. | `requireDeletion: true` |
+| `…agentPolicy.auditRequired` | Log AI consumption. | `true` |
+| `…agentPolicy.purposeLimitation` | Free-text purpose statement. | |
+
+---
+
+## `consumes[]` fields
| Field | Purpose | Required |
|---|---|:-:|
-| `exposes[].policy.authn` | `oidc` \| `oauth2` \| `api_key` \| `none` \| `custom` \| `iam` \| `jwt`. | |
-| `exposes[].policy.authz.readers` | List of principals with read access. | |
-| `exposes[].policy.authz.writers` | List of principals with write access. | |
-| `exposes[].policy.authz.columnRestrictions[]` | Per-principal column allow/deny rules. | |
-| `exposes[].policy.privacy.masking[]` | Column-level masking rules: `{column, strategy: mask\|hash\|tokenize\|encrypt\|k_anonymity}`. | |
-| `exposes[].policy.privacy.rowLevelPolicy.expression` | Provider-specific row-level filter predicate. | |
-| `exposes[].policy.classification` | Data classification label. | |
+| `consumes[].productId` | Upstream product id. | ✅ |
+| `consumes[].exposeId` | Upstream port. | ✅ |
+| `consumes[].versionConstraint` | Semver range, e.g. `^2.0.0`. | |
+| `consumes[].qosExpectations` | `freshnessMax`, `maxStaleness` (durations), `minCompleteness` (0–1). | |
+| `consumes[].requiredPolicies` | Policies the upstream must carry. | |
+| `consumes[].purpose` | Why this product reads the upstream. | |
+| 🧪 `consumes[].upstreamWorkspace` | The other mesh that owns the upstream. Requires `upstreamDigest`. | |
+| 🧪 `consumes[].upstreamDigest` | `sha256:` + 64 lowercase hex — the upstream contract this product was composed against. | |
---
-## `sovereignty` ⭐ 0.7.1 — fields
+## `sovereignty` (0.7.1) fields
-| Field | Purpose | Required |
-|---|---|:-:|
-| `sovereignty.jurisdiction` | Required legal jurisdiction (e.g. `EU`, `US`, `Multi-Region`). | |
-| `sovereignty.allowedRegions` | Explicit allowed cloud regions. | |
-| `sovereignty.deniedRegions` | Explicit denied cloud regions. | |
-| `sovereignty.dataResidency` | Boolean — must data stay within jurisdiction? | |
-| `sovereignty.crossBorderTransfer` | Boolean — is cross-border transfer permitted? | |
-| `sovereignty.transferMechanisms` | If permitted, legal mechanisms (`SCCs`, `BCRs`, …). | |
-| `sovereignty.regulatoryFramework` | Frameworks: `GDPR`, `HIPAA`, `CCPA`, `LGPD`. | |
-| `sovereignty.enforcementMode` | `strict` (block apply) \| `advisory` (warn only) \| `audit`. | |
-| `sovereignty.validationRequired` | Whether `binding` regions are validated against this block at apply-time. | |
+| Field | Purpose |
+|---|---|
+| `sovereignty.jurisdiction` | `EU` \| `US` \| `UK` \| `CA` \| `AU` \| `JP` \| `CN` \| `IN` \| `BR` \| `Global` \| `Multi-Region`. |
+| `sovereignty.allowedRegions` / `deniedRegions` | Cloud regions allowed / denied. |
+| `sovereignty.dataResidency` | Boolean — must data stay within the jurisdiction? |
+| `sovereignty.crossBorderTransfer` | Boolean — is cross-border transfer permitted? |
+| `sovereignty.transferMechanisms` | `SCCs` \| `BCRs` \| `Adequacy` \| `DPF` \| `Consent` \| `Derogation`. |
+| `sovereignty.regulatoryFramework` | `GDPR` \| `CCPA` \| `CPRA` \| `HIPAA` \| `PIPEDA` \| `LGPD` \| `PDPA` \| `POPIA` \| `DPA` \| `APPI`. |
+| `sovereignty.enforcementMode` | `strict` \| `advisory` \| `audit`. |
+| `sovereignty.validationRequired` | Whether bindings are checked against this block. |
---
-## `accessPolicy` ⭐ 0.7.1 — fields
+## `accessPolicy` (0.7.1) fields
-Each grant requires only `principal`; everything else is opt-in.
+Each grant requires only `principal`, and is closed.
| Field | Purpose | Required |
|---|---|:-:|
-| `accessPolicy.grants[].principal` | `user:`, `group:`, `serviceAccount:` identifier. | ✅ |
-| `accessPolicy.grants[].permissions` | List of permissions: `read` \| `select` \| `query` \| `write` \| `insert` \| `update` \| `delete` \| `create` \| `admin` \| `manage`. | |
-| `accessPolicy.grants[].resources` | JSONPath expressions selecting which exposes are in scope. | |
-| `accessPolicy.grants[].conditions` | Conditional access (IP ranges, time windows). | |
+| `accessPolicy.grants[].principal` | The principal, e.g. `group:analysts@example.com`. | ✅ |
+| `accessPolicy.grants[].permissions` | `read` \| `select` \| `query` \| `write` \| `insert` \| `update` \| `delete` \| `create` (0.7.2) \| `admin` \| `manage`. | |
+| `accessPolicy.grants[].resources` | Strings selecting the exposes in scope, e.g. JSONPath. | |
+| `accessPolicy.grants[].conditions` | Open object for conditional access (IP ranges, time windows). | |
---
-## `retention` ⭐ 0.7.3 — fields
+## `governance` (0.7.3) fields
-| Field | Purpose | Default |
+| Field | Purpose |
+|---|---|
+| `governance.lakeFormation.admins` | IAM ARNs. **Authoritative:** applying it replaces the account's Lake Formation admin list, so list every admin, including the identity that applies it. The schema's description adds that applying it also clears the account's create-database and create-table default permissions, trusted resource owners and parameters, and that destroying the emitted `aws_lakeformation_data_lake_settings` resource empties the admin list and resets `CROSS_ACCOUNT_VERSION` to 1. |
+| `governance.lakeFormation.tagDefinitions` | LF-tag key → allowed values. |
+
+---
+
+## `retention` (0.7.3) fields
+
+| Field | Purpose | Default (schema description) |
|---|---|:-:|
-| `retention.runState` | How long the orchestrator keeps run state. | `P30D` |
+| `retention.runState` | How long run state is kept. | `P30D` |
| `retention.runLogs` | How long run logs are kept. | `P90D` |
| `retention.lineage` | How long emitted lineage events are kept. | `P365D` |
-| `retention.dlq` | How long DLQ records are kept before purge. | `P180D` |
+| `retention.dlq` | How long DLQ records are kept. | `P180D` |
-All values are [ISO-8601 durations](https://en.wikipedia.org/wiki/ISO_8601#Durations).
+All values are [ISO-8601 durations](https://en.wikipedia.org/wiki/ISO_8601#Durations). `retention` covers operational records, not the product's data; for data, see `lifecycle.retention` (and, 🧪 in 0.7.6, `exposes[].lifecycle.expire`).
---
-## `orchestration` — fields
+## `orchestration` fields
| Field | Purpose | Required |
|---|---|:-:|
| `orchestration.engine` | `airflow` \| `dagster` \| `prefect` \| `kubeflow` \| `custom` \| `none`. | ✅ |
-| `orchestration.mode` | Pull or push generation. | |
-| `orchestration.generateOnChange` | Auto-regenerate DAGs when contract changes. | |
-| `orchestration.tasks[].taskId` | Task id within the DAG. | |
-| `orchestration.tasks[].type` | `provider_action` \| `fluid_validate` \| `fluid_plan` \| `fluid_apply` \| `fluid_execute` \| `fluid_verify` \| `fluid_dq_check` \| `bash` \| `python` \| `branch_python` \| `sensor` \| `email` \| `http` \| `snowflake_query` \| `bigquery_query` \| `bigquery_job` \| `glue_job` \| `databricks_job` \| `custom`. | |
-| `orchestration.tasks[].provider` | For `provider_action`: e.g. `aws.s3`, `snowflake.table`. | |
-| `orchestration.tasks[].action` | For `provider_action`: e.g. `ensure_bucket`, `ensure`. | |
-| `orchestration.tasks[].parameters` | Action parameters. | |
-| `orchestration.tasks[].dependsOn` | Upstream task ids. | |
-| `orchestration.tasks[].buildStepRef` | Reference into `build` step. | |
+| `orchestration.mode` | `generated` \| `manual` \| `hybrid`. | |
+| `orchestration.generateOnChange` | Regenerate the workflow when the contract changes. | |
+| `orchestration.airflow.dagId` | DAG id; required when `airflow` is present. | |
+| `orchestration.airflow.dagConfig` | `schedule`, `startDate`, `catchup`, `maxActiveRuns`, … | |
+| `orchestration.airflow.tasks[].taskId` | Task id (lower-case). | ✅ |
+| `orchestration.airflow.tasks[].type` | `provider_action` \| `fluid_validate` \| `fluid_plan` \| `fluid_apply` \| `fluid_execute` \| `fluid_verify` \| `fluid_dq_check` \| `bash` \| `python` \| `branch_python` \| `sensor` \| `email` \| `http` \| `snowflake_query` \| `bigquery_query` \| `bigquery_job` \| `glue_job` \| `databricks_job` \| `custom`. | |
+| `orchestration.airflow.tasks[].operator` | Operator name. **Set it** — see the note below. | |
+| `orchestration.airflow.tasks[].provider` | For `provider_action`: `aws` \| `gcp` \| `azure` \| `snowflake` \| `databricks` \| `kafka` \| `kubernetes` \| `local` \| `custom`. | (provider_action) |
+| `orchestration.airflow.tasks[].action` | For `provider_action`: `.`, e.g. `s3.ensure_bucket`. | (provider_action) |
+| `orchestration.airflow.tasks[].params` | Action parameters (open object). | (provider_action) |
+| `orchestration.airflow.tasks[].dependencies` | Upstream task ids. | |
+| `orchestration.airflow.tasks[].buildStepRef` | Reference into a build step. | |
+
+> ⚠️ **`orchestration.tasks` is not validated.** `orchestration` is an open object up to 0.7.6, so a top-level `orchestration.tasks` list is accepted without any check. Its keys are implementation-defined: the reference implementation's AWS and GCP code generators read `params` and `dependsOn` there, and its Snowflake Airflow generator reads `parameters`.
+>
+> **Name each Airflow task's `operator`.** In the 0.7.1 to 0.7.6 schemas the conditional rules for `fluid_execute`, `bash` and `python` tasks also match a task with no `operator`, so a task that has no `operator` and carries `params` must satisfy all three at once and is in practice rejected. A `provider_action` task always carries `params`, so it is always affected. A task with neither `operator` nor `params` validates.
---
-## `lifecycle` — values
-
-`lifecycle.state` ∈ `preview` \| `active` \| `deprecated` \| `retired`.
+## `lifecycle` values
-A `deprecated` product still serves reads but new `consumes:` references should fail validation in CI.
+`lifecycle.state` ∈ `preview` \| `active` \| `deprecated` \| `retired` — the state of the *product*, unrelated to preview *schema versions*. Other members: `retention` (ISO-8601 duration) and `deprecationPolicy` (`noticePeriod`, `contact`, `replacement`).
---
## Where each field is exhaustively documented
-The auto-generated reference at [`specs/0.7.4/fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html) is the authoritative source for every field's exact type, enum values, validation rules, and examples.
+The auto-generated reference at [`specs/0.7.5/fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html) renders every field's exact type, enum values and validation rules from the schema.
diff --git a/docs/schema/minimal-contract.md b/docs/schema/minimal-contract.md
index 619a7d1..505fbe9 100644
--- a/docs/schema/minimal-contract.md
+++ b/docs/schema/minimal-contract.md
@@ -1,65 +1,13 @@
# Minimal Contract & FLUID at a Glance
-Two views of the contract for v0.7.4: the smallest file that validates, and a one-screen map of every top-level block.
-
-## FLUID at a Glance
-
-A FLUID contract is one YAML file. The shape below shows **every top-level block** in v0.7.4, with one-line meaning and the version each was introduced. Required blocks are marked `[req]`; everything else is opt-in.
-
-```yaml
-fluidVersion: "0.7.4" # [req] which contract version this file targets
-kind: DataProduct # [req] DataProduct | MLPipeline
-id: domain.layer.name # [req] globally unique product id
-name: "Human-readable name" # [req] display name
-description: "..." # business-facing summary
-domain: "Finance" # owning business domain
-tags: [pii, gold-layer] # free-text categorization
-labels: { team: analytics } # key/value categorization
-
-metadata: # [req] only metadata.owner is required
- owner: { team, email } # [req] team is the one truly required field
- layer: Gold # free-form; convention: Bronze | Silver | Gold
-
-consumes: [ ... ] # upstream FLUID products you depend on
-exposes: [ ... ] # [req] ports you publish (each requires exposeId+kind+contract+binding)
- └── contract # [req] schema columns (or openapiRef), dq rules
- └── semantics # ⭐ 0.7.2 — entities, measures, dimensions, metrics
- └── policy # authn, authz, privacy, classification, agentPolicy (⭐ 0.7.1)
- └── mcp # ⭐ 0.7.4 — opt this expose into the MCP output-port gateway
- └── binding # [req] where it lives (platform + format + location, + icebergConfig*)
-
-build: # how the product is produced
- pattern: hybrid-reference # | embedded-logic | multi-stage | acquisition (⭐ 0.7.3)
- engine: dbt | sql | python | spark | glue | custom # transformation engines
- | duckdb | airbyte | meltano | dlt | kafka-connect | debezium # ⭐ 0.7.3 acquisition
- properties: { ... } # pattern-specific (⭐ 0.7.3: acquisitionPattern)
-
-orchestration: { engine: airflow | dagster | prefect | kubeflow | custom | none, tasks: [...] } # ⭐ 0.7.0+
-sovereignty: { jurisdiction, allowedRegions, deniedRegions, enforcementMode, … } # ⭐ 0.7.1
-accessPolicy: { grants: [{ principal, permissions, resources, conditions }] } # ⭐ 0.7.1
-retention: { runState, runLogs, lineage, dlq } # ⭐ 0.7.3
-lineage: { upstream, downstream, fieldLevel }
-schemaEvolution: { strategy, compatibility }
-machineLearning: { ... }
-environments: { dev: {...}, staging: {...}, prod: {...} }
-lifecycle: { state: preview | active | deprecated | retired }
-docs: { ... }
-```
-
-::: tip `agentPolicy` location
-AI/LLM consumption policy lives **per-expose** under `exposes[].policy.agentPolicy` — not at the root. (Earlier release notes show it at the top level; the schema has never accepted it there.) As of 0.7.4, when an expose also carries an `mcp` block, that `agentPolicy` is enforced at runtime by the Fluid MCP gateway on every read. See [Anatomy §7](/fluid/schema/anatomy#_7-governance-sovereignty-accesspolicy-and-exposes-policy-agentpolicy) for the correct shape.
-:::
-
-For a one-line-per-field reference with required/optional flags, see the [**Cheatsheet**](/fluid/schema/cheatsheet). For a tour of each block with deep-dive links, see the [**Anatomy**](/fluid/schema/anatomy).
-
----
+Two views of a contract on the latest stable schema, **0.7.5**: the smallest file that validates, and a one-screen map of every top-level block.
## Minimal Valid Contract
-The smallest file that passes JSON Schema validation against v0.7.4 — every other top-level block is opt-in:
+The smallest file that passes JSON Schema validation against 0.7.5 — every other block is opt-in:
```yaml
-fluidVersion: "0.7.4"
+fluidVersion: "0.7.5"
kind: DataProduct
id: demo.bronze.hello_world
name: "Hello World"
@@ -77,6 +25,68 @@ exposes:
location: { path: "./hello.parquet" }
```
-**What you can drop:** `description`, `domain`, `tags`, `labels`, `consumes`, `build`, `orchestration`, all governance blocks (`agentPolicy`, `sovereignty`, `accessPolicy`, `retention`), `exposes[].mcp`, `lineage`, `lifecycle`, `environments`, `docs`. Everything else is layered on as you need it.
+It validates against `fluid-schema-0.7.5.json` with any JSON Schema Draft 2020-12 validator, and with the reference implementation (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.5)`).
+
+**What is required:** the six top-level keys above; inside `metadata`, only `owner` (the schema requires no member of `owner`, but name a `team`); and inside each expose, `exposeId`, `kind`, `contract` (with `schema` or `openapiRef`) and `binding` (with `platform`, `format` and `location`).
See the [**Examples**](/fluid/examples/) for the step-by-step progression from this minimal file to a production source-aligned acquisition product.
+
+---
+
+## FLUID at a Glance
+
+Every top-level block in 0.7.5, with the version each first appeared in (in its current form). `[req]` marks required blocks. This is a **map, not a contract** — the placeholders are not valid values.
+
+```text
+fluidVersion: "0.7.5" [req] the schema version this file declares
+kind: DataProduct [req] DataProduct | MLPipeline
+id: domain.layer.name [req] globally unique product id
+name: "Human-readable name" [req] display name
+description, domain business-facing summary, owning domain
+tags: [pii, gold-layer] lower-case categorization tags
+labels: { team: analytics } key/value labels
+
+metadata: [req] only metadata.owner is required
+ owner: { team, email, slack, oncall } [req] owner itself; none of its members
+ layer, productType, classification, businessContext, …
+
+exposes: [ ... ] [req] ports you publish; each requires exposeId, kind, contract, binding
+ ├── contract [req] schema columns (or openapiRef), dq rules, schemaPolicy
+ ├── policy authn, authz, privacy, classification, agentPolicy (0.7.1)
+ ├── semantics entities, measures, dimensions, metrics (0.7.2)
+ ├── qos availability, freshnessSLO, latencyP95, …
+ ├── mcp serve the port to AI agents over MCP (0.7.4)
+ └── binding [req] platform + format + location (+ icebergConfig, vectorConfig, governance)
+
+consumes: [ ... ] upstream FLUID products you read
+build / builds[] how the product is produced
+ pattern: hybrid-reference | embedded-logic | multi-stage | acquisition (0.7.3)
+ engine: dbt | dbt- | sql | python | spark | glue | custom
+ | duckdb | airbyte | meltano | dlt | kafka-connect | debezium (0.7.3)
+ properties: { ... } shape chosen by pattern
+
+orchestration { engine, mode, airflow, … } (0.7.2, top level)
+sovereignty { jurisdiction, allowedRegions, deniedRegions, … } (0.7.1)
+accessPolicy { grants: [{ principal, permissions, resources, conditions }] } (0.7.1)
+governance { lakeFormation: { admins, tagDefinitions } } (0.7.3)
+retention { runState, runLogs, lineage, dlq } (0.7.3)
+extensions { ... } (0.7.3)
+lineage { granularity, upstream, downstream }
+schemaEvolution { strategy, compatibility, changePolicy }
+machineLearning { enabled, framework, models }
+environments { dev: {...}, staging: {...}, prod: {...} }
+lifecycle { state, retention, deprecationPolicy }
+docs { homepage, runbook, dictionary, changeLog }
+
+0.7.6 preview only (fluidVersion "0.7.6"):
+packaging { mode, pool, containers }
+consumers [ { name, type, ... } ]
+```
+
+Field-level lineage is `lineage.granularity: field_level` with `fieldMappings` on each `upstream[]` entry; there is no `fieldLevel` member.
+
+::: tip `agentPolicy` location
+AI/LLM consumption policy lives **per-expose** under `exposes[].policy.agentPolicy` — not at the root; the schema has never accepted it there. See [Anatomy §7](/fluid/schema/anatomy#_7-governance-sovereignty-accesspolicy-governance-and-exposes-policy-agentpolicy) for the shape.
+:::
+
+For a one-line-per-field reference with required flags, see the [**Cheatsheet**](/fluid/schema/cheatsheet). For a tour of each block, see the [**Anatomy**](/fluid/schema/anatomy).
diff --git a/docs/schema/specification.md b/docs/schema/specification.md
index 85d62c4..54d7835 100644
--- a/docs/schema/specification.md
+++ b/docs/schema/specification.md
@@ -1,88 +1,29 @@
-# Fluid Protocol Specification
+# FLUID Specification
-::: tip Latest schema version
-The latest published JSON Schema is **0.7.5**. This page is the narrative protocol specification; for the version-specific, field-by-field reference see the [**Cheatsheet**](/fluid/schema/cheatsheet), the [**Anatomy**](/fluid/schema/anatomy), and the [**Versions**](/fluid/schema/versions) index (raw JSON Schema + generated HTML per version).
+::: tip Versions
+**Latest stable: 0.7.5. Preview: 0.7.6.** This page specifies the structure of a FLUID document as defined by the **0.7.5** JSON Schema. For every version's schema and generated field-by-field reference, see [**Versions**](/fluid/schema/versions); for what is new in the preview, see [0.7.6 (preview)](/fluid/releases/0.7.6).
:::
-This document provides the complete, official specification for the FLUID (Federated Layered Unified Interchange Definition) protocol. It is intended for data architects, platform engineers, and developers who are building the next generation of data infrastructure, as well as for vendors seeking to make their tools compliant with this open standard.
+FLUID (Federated Layered Unified Interchange Definition) is an open, declarative specification for **data products**, written in YAML or JSON and kept in version control. It is not a platform or a single tool: it is a shared language that tools read to build, deploy, govern and serve a data product.
-### The Strategic Imperative: A Protocol for the Agentic Era
-The contemporary enterprise is shifting from process automation to an Agentic Ecosystem, where autonomous AI agents drive operations with unprecedented speed and intelligence. This paradigm shift, enabled by communication standards like the Model Context Protocol (MCP), exposes a foundational vulnerability in modern data architecture: the lack of a common language for defining, governing, and interacting with data assets.
+A FLUID document describes one data product: what it **consumes** (its inputs), what it **exposes** (its output ports, each with a contract and a binding to where the data lives), how it is **built**, and the policies around it — access, sovereignty, AI-agent use, retention. This page is for anyone implementing a FLUID-aware tool or checking that one conforms.
-Today's data landscape is a fragmented collection of imperative pipelines, siloed tool configurations, and implicit knowledge. This static, brittle foundation cannot support the dynamic, real-time demands of an agentic workforce. Agents require a data fabric that is not only accessible but also discoverable, trustworthy, and context-aware.
-
-FLUID is the standard designed to create this fabric. It addresses this challenge by providing a declarative, universal protocol for defining Data Products. It is the missing piece of the puzzle, serving as the foundational layer that makes an organization truly MCP-ready. While MCP standardizes how agents communicate, FLUID standardizes the trustworthy Data Products they communicate with.
-
-### What is FLUID?
-FLUID is an open, declarative specification, written in YAML and managed in version control. It is not a platform or a single tool, but a shared language that enables a decentralized ecosystem of compliant tools to work in concert.
-
-It re-frames the data lifecycle around the concept of a Data Product: a versioned, autonomous asset with a clearly defined interface, contract, and implementation. By unifying the definition of what a data product consumes (its dependencies), what it exposes (its public interface), and how it is built (its implementation logic), FLUID provides a holistic, auditable, and machine-readable blueprint for every data asset in the enterprise.
-
-This document details the full specification for this protocol, providing the technical foundation required to build the governable, scalable, and agent-ready data ecosystems of the future.
-
-> 📖 *FLUID Data Products* is [available on Amazon](https://amzn.eu/d/ikMlWNV).
-
-🌊 FLUID: Federated Layered Unified Interchange Definition
-
----
-
-## 🧭 Core Principles
-
-- **Data as a Product**
- Data is a first-class asset with a clear owner, a versioned interface, and a machine-readable contract.
-
-- **Declarative, Not Imperative**
- Contracts define the desired end state of a data product. The FLUID-aware framework is responsible for the implementation.
-
-- **Contracts as Code**
- Governance (schema, quality, build, privacy) is embedded directly into version-controlled files, enabling automated, proactive enforcement.
-
-- **Federated Ownership**
- Data products are owned and managed by the domain teams who know the data best, enabling a true, scalable Data Mesh.
+**What defines conformance.** The published JSON Schemas define which documents are valid FLUID documents, and the [conformance corpus](https://github.com/open-data-protocol/fluid/blob/main/tests/README.md) pins that behaviour case by case, so that "FLUID-conformant" can be checked without any particular implementation ([GOVERNANCE.md](https://github.com/open-data-protocol/fluid/blob/main/GOVERNANCE.md)). [`data-product-forge`](/fluid/concepts/forge-cli) is the reference implementation: implementation #1 under test, not the referee. Where this page and a schema disagree, the schema is authoritative and this page is wrong.
---
-## 🏗️ Contract Structure: Monolithic vs. Modular
-
-The FLUID specification is designed for flexibility.
-A data product contract can be defined in a single, **monolithic file** or composed from multiple, specialized files.
-
-### Monolithic Structure (For Simplicity)
-
-For simple data products owned by a single team, all definitions can be contained within a single root `fluid.yml` file.
+## Core principles
-```
-/dp-simple-product/
-└── 📄 fluid.yml # All definitions are inline.
-```
+- **Data as a product** — data is a first-class asset with an owner, a versioned interface and a machine-readable contract.
+- **Declarative, not imperative** — a document states the desired end state; FLUID-aware tools decide how to reach it.
+- **Contracts as code** — schema, quality, build and policy live in version-controlled files, so they can be checked automatically.
+- **Federated ownership** — data products are owned by the domain teams that know the data.
---
-### Modular Structure (For Complexity & Federation)
+## Documents and files
-For complex, enterprise-grade products with multiple stakeholders, the contract can be broken into logical, linked files.
-This is the recommended best practice.
-
-The root `fluid.yml` acts as a "table of contents," referencing detailed configuration files stored in a dedicated `.fluid/` directory using the `$ref` keyword.
-
-```
-/dp-complex-product/
-│
-├── 📄 fluid.yml # The main entrypoint, contains high-level identity.
-│
-└── 📁 .fluid/ # A dedicated folder for all contract details.
- ├── 📄 consumes.yml
- ├── 📄 build.yml
- ├── 📄 exposes.yml
- ├── 📄 schema.yml
- └── 📄 quality.yml
-```
-
-## Preamble
-
-This document provides the complete, official specification for the FLUID (Federated Layered Unified Interchange Definition) protocol. It is intended for data architects, platform engineers, and developers who are building the next generation of data infrastructure, as well as for vendors seeking to make their tools compliant with this open standard.
-
----
+A FLUID document is a single YAML or JSON object. **The file name is not normative.** Pages on this site name files `*.fluid.yml`; the reference implementation scaffolds and looks for `contract.fluid.yaml`. Either is fine.
## Validation semantics
@@ -98,7 +39,7 @@ This is the Draft 2020-12 default, and FLUID keeps it for two reasons of its own
The first is that conformance must not depend on which validator you run. `format` vocabularies are optional in Draft 2020-12 and implementations differ widely in which formats they recognise and what they pull in to check them. Were FLUID to make `format` assertive, the same document could be conformant in one language and non-conformant in another, which would defeat the purpose of publishing a conformance corpus at all.
-The second is that the choice is not symmetric. Declaring `format` assertive would invalidate documents that are valid today — a narrowing, which [GOVERNANCE.md](https://github.com/open-data-protocol/fluid/blob/main/GOVERNANCE.md) forbids between versions and `scripts/check-compat.py` enforces on every pull request. Annotation-only is therefore the only reading available to a pre-1.0 specification that has already published twelve schema versions. A future version may add assertive checking behind a new, opt-in keyword; it may not retroactively sharpen this one.
+The second is that the choice is not symmetric. Declaring `format` assertive would invalidate documents that are valid today — a narrowing, which [GOVERNANCE.md](https://github.com/open-data-protocol/fluid/blob/main/GOVERNANCE.md) forbids between versions and `scripts/check-compat.py` enforces on every pull request. Annotation-only is therefore the only reading available to a pre-1.0 specification that has already published many schema versions. A future version may add assertive checking behind a new, opt-in keyword; it may not retroactively sharpen this one.
Implementations that do want to assert formats are served by the conformance corpus rather than left to guess: the cases that depend on assertion live in [`tests/optional/`](https://github.com/open-data-protocol/fluid/tree/main/tests/optional), separated from the core tier for exactly this reason, and `conformance/run.py` runs them under a format-asserting validator.
@@ -108,225 +49,188 @@ FLUID defines a **field** named `format` in several places — `exposes[].bindin
The collision of names is unfortunate and is called out here because it is easy to read "`format` is an annotation" as applying to them. It does not. The sentence above is about the JSON Schema *keyword*; this paragraph is about a FLUID *field* that happens to share its spelling.
----
-
-## 1. Specification Root
-
-The FLUID definition is a YAML or JSON file (`.fluid.yml` OR `.fluid.json`) with the following root-level objects.
-
-| Key | Type | Required | Description |
-|----------------|----------------|----------|-----------------------------------------------------------------------------|
-| fluidVersion | String | Yes | The version of the FLUID specification this file adheres to (e.g., 0.7.4). |
-| kind | String | Yes | The type of data product definition. See section 1.1. |
-| id | String | Yes | A globally unique, versioned id for the data product (customer360_v1). |
-| name | String | Yes | A human-readable name for the data product (e.g., "Customer 360"). |
-| description | String | Yes | A brief description of the product's purpose. |
-| domain | String | Yes | The business domain that owns this product (e.g., "Marketing"). |
-| metadata | Object | Yes | Identification, ownership, and classification information. See section 1.2. |
-| consumes | Object / List | Yes | Defines the input data sources needed to build the product. See section 1.4.|
-| build | Object | Yes | Contains the implementation logic for how the product is built. See 1.5. |
-| exposes | Object / List | Yes | Defines the public output interface(s) of the product. See section 1.3. |
-| accessPolicy | Object | No | Defines the static access control policies for the data product. See 1.6. |
-| dynamicPolicies| Object | No | Defines context-aware access policies that adapt at runtime. See 1.7. |
-| operations | Object | No | Defines SLAs, lifecycle, and observability characteristics. See 1.8. |
-| extensions | Object | No | Registers required external plugins. See section 1.9. |
-
----
-
-### 1.1 kind Enumeration
-
-The `kind` key specifies the nature of the data product.
-
-- **DataProduct**: A standard, materialized data asset.
-- **VirtualDataProduct**: A product that exists only as a logical view or query, without its own physical storage.
-- **EgressFlow**: A product specifically designed to export data securely to an external system.
-
----
-
-### 1.2 metadata Block
-
-| Key | Type | Required | Description |
-|-------------------|----------------|----------|-----------------------------------------------------------------------------|
-| owner | Object | Yes | Ownership details (e.g., { team: 'finance', email: 'finance@company.com' }).|
-| layer | String | No | The architectural layer (e.g., "Gold", "Silver", "Bronze") |
-| status | String | No | The lifecycle status (e.g., "Published", "Development", "Deprecated"). |
-| sensitivity_level | String | No | The overall data classification (e.g., "Confidential", "Public"). |
-| cost_center | String | No | An identifier for automated FinOps cost attribution. |
-| tags | Map[String] | No | Key-value pairs for categorization (e.g., layer: gold, domain: finance). |
-| classification | String | Yes | Default privacy level: public, internal, confidential, restricted. |
-| purpose | Object | No | Detailed, machine-readable description of the business purpose. |
-| version | String | No | (Recommended) Semantic version of this data product definition (e.g., 1.0.0)|
-
-#### 1.2.1 purpose Block
-
-| Key | Type | Required | Description |
-|--------------------|----------------|----------|-----------------------------------------------------------------------------|
-| business_purpose | String | Yes | A clear statement of why this data product exists. |
-| use_cases | List | No | A list of specific business use cases it supports. |
-| target_group | List | No | The intended audience for this product. |
-| limitations | String | No | Any known limitations or constraints of the data. |
+### The declared `fluidVersion` selects the schema
----
-
-### 1.3 exposes Block (The Output Port)
-
-Defines the public interface of the data product. This is what consumers interact with.
-
-| Key | Type | Required | Description |
-|----------|--------|----------|--------------------------------------------------------------------|
-| name | String | If list | A unique name for this output port within the product. |
-| location | Object | Yes | The physical or virtual location where the data product is materialized. See 1.3.1. |
-| contract | Object | Yes | The schema, quality, and privacy promises for this output. See 1.3.2. |
-
-#### 1.3.1 location Object
-
-| Key | Type | Required | Description |
-|------------|--------|--------------|--------------------------------------------------------------------|
-| type | String | Yes | bigquery, s3, snowflake, gcs, iceberg, api, kafka, or virtual. |
-| connection | String | Yes | Reference to a secret in a vault (e.g., secret:gcp-prod-dwh-key). |
-| format | Object | If not virtual| Describes the data format (e.g., { type: 'parquet' }). |
-| properties | Object | Yes | Technology-specific properties (e.g., project, dataset, table). |
-
-#### 1.3.2 contract Object
-
-| Key | Type | Required | Description |
-|-------------|----------|--------------|--------------------------------------------------------------------|
-| inheritFrom | String | No | dbt, fluid-product, or openApi. Populates the contract from a source.|
-| model / spec| String | If inheriting| The dbt model name or path to the OpenAPI spec file/URL. |
-| schema | Object | Yes | Defines the columns and data types. See 1.3.2.1. |
-| quality | List | No | List of data quality rules to enforce. See 1.3.2.2. |
-| privacy | List | No | List of privacy classifications and treatments. See 1.3.2.3. |
-| semantics | Object | No | Adds machine-readable meaning to the data. See 1.3.2.4. |
-
-##### 1.3.2.1 schema.columns Array
-
-| Key | Type | Required | Description |
-|---------|---------|----------|----------------------------------------------|
-| name | String | Yes | Column name. |
-| type | String | Yes | Data type (STRING, INT64, NUMERIC, TIMESTAMP, JSON, BOOLEAN, DATE). |
-| nullable| Boolean | No | true by default. |
-
-##### 1.3.2.2 quality Array Item
-
-| Key | Type | Required | Description |
-|----------|--------|--------------|--------------------------------------------------------------------|
-| rule | String | Yes | not_null, unique, regex_match, in_set, or a custom SQL expression. |
-| columns | List | If applicable| Column(s) to apply the rule to. |
-| pattern/set| String/List|If applicable| Parameters for regex_match or in_set. |
-| onFailure| Object | Yes | action (reject_row, quarantine_row, fail_pipeline, alert) and optional notifications. |
-
-##### 1.3.2.3 privacy Array Item
-
-| Key | Type | Required | Description |
-|---------------|--------|--------------|--------------------------------------------------------------------|
-| classification| String | No | PII, SPI, Confidential. Overrides metadata.classification. |
-| columns | List | Yes | Column(s) to apply the treatment to. Can be ['*']. |
-| treatment | Object | Yes | type (hashing, masking, encryption, tokenization) and properties. |
-
-##### 1.3.2.4 semantics Object
-
-| Key | Type | Description |
-|---------------|--------|--------------------------------------------------------------------|
-| ontology | String | Reference to an external ontology (e.g., URL to an OWL or RDF file).|
-| classifications| List | Maps columns to terms in a business glossary or formal ontology. |
+Because the schema is chosen by the document's own `fluidVersion`, a later schema accepting an earlier `fluidVersion` value is not a verdict on documents that declare that earlier version. The 0.7.5 schema's `fluidVersion` enum lists `"0.7.3"`, `"0.7.4"` and `"0.7.5"`; a document declaring `"0.7.4"` is still validated against the 0.7.4 schema, so it cannot use a field that only 0.7.5 defines. See [Choosing `fluidVersion`](/fluid/schema/versions#choosing-fluidversion).
----
-
-### 1.4 consumes Block (The Input Port)
-
-| Key | Type | Required | Description |
-|-----------------|--------|--------------|--------------------------------------------------------------------|
-| type | String | Yes | gcs, kafka, s3, api, postgres-cdc, sftp, or fluid-product. |
-| name | String | If type: fluid-product | The dataProduct name of the upstream FLUID definition. |
-| alias | String | If list | A local alias to refer to this source in the build block. |
-| onUpstreamChange| String | No | Action on upstream contract change: fail, alert, triggerRebuild. |
-| connection | String | If physical type | Reference to a secret in a vault. |
-| format | Object | If physical type | Describes the data format (e.g., { type: 'json' }). |
-| properties | Object | If physical type | Technology-specific properties (e.g., Kafka topic, API endpoint). |
-
----
-
-### 1.5 build Block (The Implementation)
+### Known interoperability issue: one `pattern` is not an ECMA-262 regular expression
-| Key | Type | Required | Description |
-|-----------------|--------|--------------|---------------------------------------------------------------------------------|
-| engine | String | Yes | The engine to use: sql, python, dbt, dbt-cloud, spark-sql. |
-| script | String | Yes | The specific asset to execute (e.g., a script path or dbt model selector). |
-| trigger | Object | Yes | Defines how the build is initiated (schedule, event, manual). |
-| runtime | Object | Yes | The underlying compute platform where the build will run (airflow, gcp-cloud-run). |
-| dependencies | Object | No | Defines the execution dependencies on other data products. |
-| retries | Object | No | Configuration for handling transient failures: count, delay. |
-| notifications | Object | No | Defines how to send alerts (channel, target) on onSuccess or onFailure. |
+Draft 2020-12 says a `pattern` SHOULD be a valid ECMA-262 regular expression. In the 0.7.2 to 0.7.6 schemas, the second alternative of the column `type` (`$defs/column/properties/type`) begins with the inline flag `(?i)`, which ECMA-262 does not have: JavaScript's `RegExp` rejects it as an invalid group. Python's `re`, which the conformance tooling in this repository uses, accepts it and matches case-insensitively. A validator that compiles patterns as ECMA-262 may therefore refuse that pattern, or the schema, where a Python-based validator does not. This is recorded here rather than resolved: changing the pattern would be a schema change, and schemas are changed upstream (see [CONTRIBUTING.md](https://github.com/open-data-protocol/fluid/blob/main/CONTRIBUTING.md)).
---
-### 1.6 accessPolicy (Static Access)
-
-| Key | Type | Required | Description |
-|------------|--------|----------|--------------------------------------------------------------------|
-| visibility | String | No | Default discoverability: private, internal. |
-| grants | List | Yes | A list of static, explicit access grants. |
-
-#### 1.6.1 grants Array Item
-| Key | Type | Required | Description |
-|------------|--------|----------|--------------------------------------------------------------------|
-| principal | String | Yes | The actor receiving the grant. Format: `user:`, `group:`, `agent:`. |
-| permissions| List | Yes | Rights granted: readData, readMetadata, manage. |
-| scope | Object | No | Fine-grained access. See 1.6.2. |
+## Composing a document from several files
-#### 1.6.2 scope Object
+FLUID defines **one document**. The standard does not define how a document may be assembled from several files; any such mechanism is **implementation-defined**, and nothing in this section is normative.
-| Key | Type | Required | Description |
-|-------------|--------|----------|--------------------------------------------------------------------|
-| columns | List | No | Allow-list of columns the principal can view. |
-| rowFilter | String | No | SQL WHERE clause for secure view. |
-| privacyView | String | No | treated (default, views post-privacy data), cleartext (views pre-privacy data). |
+What follows from the rest of this specification:
----
+1. **A composed root is not a FLUID document.** The published schemas have no `$ref` property in a document, and the root object and most nested objects are closed (`additionalProperties: false`), so a file that stands in `{ $ref: ... }` for a block fails validation. Splitting a valid contract into fragments with the reference implementation's `fluid split` and validating the root file directly against the 0.7.5 schema gives errors such as `Additional properties are not allowed ('$ref' was unexpected)`.
+2. **Validate the resolved document.** Conformance is a property of the single document a composition mechanism produces. Resolve first, then validate that result against the schema of its `fluidVersion`.
-### 1.7 dynamicPolicies (Adaptive Access)
+### How the reference implementation composes contracts (non-normative)
-| Key | Type | Required | Description |
-|-------|------|----------|--------------------------------------------------------------------|
-| rules | List | Yes | A list of contextual access rules, evaluated in order. |
+`data-product-forge` resolves `$ref` nodes before it validates, plans or applies a contract, so the rest of its pipeline sees one document. As of data-product-forge 0.18:
-#### 1.7.1 rules Array Item
+- A `$ref` node is an object whose only key is `$ref`; its value is a path to a YAML or JSON file, resolved relative to the file that contains it, optionally followed by `#` and an [RFC 6901](https://www.rfc-editor.org/rfc/rfc6901) JSON Pointer. A referenced YAML file must hold an object, not a list. Same-document refs (`#/...`) are left as written; cycles and nesting deeper than 20 levels are errors.
+- **Refs are confined to the root contract's directory tree.** A ref that resolves outside it — through `..` or a symlink — is refused, as are absolute paths and URLs (`https://`, `file://`, `s3://`, …). Setting `FLUID_REF_ROOT` (or passing `ref_root=` to the loader) widens the root to a directory that contains the contract, for monorepos that share fragments; a `FLUID_REF_ROOT` that does not contain the contract is ignored with a `ref_root_env_ignored` warning.
+- `fluid split` turns a single-file contract into a root plus a `fragments/` directory (`fragments/exposes/.yaml`, `fragments/builds/.yaml`, `fragments/sovereignty.yaml`, `fragments/access-policy.yaml`); `fluid bundle` resolves a root back into one document.
+- Its `.fluid/` directories hold the tool's own runtime state (receipts, run records, staging data), which its generated `.gitignore` partly excludes from version control. They are not a place for contract fragments.
-| Key | Type | Required | Description |
-|-----------|--------|----------|--------------------------------------------------------------------|
-| name | String | Yes | A descriptive name for the policy rule. |
-| condition | String | Yes | An expression evaluated against the agent's context. Supports interpolation. |
-| grant | Object | Yes | The permissions and scope to grant if the condition is met. |
+Details: [Composing a contract from fragments](https://agenticstiger.github.io/forge_docs/concepts/contract-refs.html), [`fluid split`](https://agenticstiger.github.io/forge_docs/cli/split.html) and [`fluid bundle`](https://agenticstiger.github.io/forge_docs/cli/bundle.html) in the reference implementation's documentation.
---
-### 1.8 operations Block
-
-| Key | Type | Required | Description |
-|-------------|--------|----------|--------------------------------------------------------------------|
-| sla | Object | No | Defines the Service Level Agreements for this product. See 1.8.1. |
-| lifecycle | Object | No | Manages data retention and archival policies. See 1.8.2. |
-| observability| Object| No | Configures logging, alerting, and monitoring. See 1.8.3. |
-
-#### 1.8.1 sla Object
-
-Defines cost, latency, freshness, accuracy, sustainability, and feedbackSignals.
-
-#### 1.8.2 lifecycle Object
-
-Defines retention (period, condition) and archival (trigger, destination).
-
-#### 1.8.3 observability Object
-
-Defines logging (level, destination) and alerting (onFailure notifications).
+## 1. Document structure (0.7.5)
+
+The tables summarise the 0.7.5 schema. They name every top-level member and the main nested blocks; the [generated reference](/fluid/specs/0.7.5/fluid-spec.html) has every field. **Closed** means `additionalProperties: false`: a member the table does not list makes the document invalid.
+
+### 1.1 Root
+
+The root object is **closed**.
+
+| Member | Type | Required | Meaning |
+|---|---|:-:|---|
+| `fluidVersion` | string, one of `"0.7.3"`, `"0.7.4"`, `"0.7.5"` | ✅ | The schema version the document declares. It selects the schema the document is validated against. |
+| `kind` | `DataProduct` \| `MLPipeline` | ✅ | The kind of product. |
+| `id` | identifier | ✅ | Globally unique product id, e.g. `finance.gold.customer_360`. |
+| `name` | string | ✅ | Display name. |
+| `metadata` | object (§1.2) | ✅ | Ownership and classification. |
+| `exposes` | array of expose (§1.3) | ✅ | The product's output ports. |
+| `description`, `domain` | string | | Business description and owning domain. |
+| `tags` | array of unique tags | | Each tag lower-case: `^[a-z0-9][a-z0-9-]*[a-z0-9]$` or a single character. |
+| `labels` | map of string → string | | Key/value labels. |
+| `consumes` | array of consume (§1.4) | | Upstream data products this product reads. |
+| `build` | build (§1.5) | | How the product is built. |
+| `builds` | array of build (§1.5) | | Several builds for one product. |
+| `orchestration` | object (§1.6) | | Scheduling and workflow-engine settings. |
+| `accessPolicy` | object (§1.7) | | Access grants for the product. |
+| `sovereignty` | object (§1.7) | | Jurisdiction and data-residency constraints. |
+| `governance` | object (§1.7) | | Account-wide AWS Lake Formation settings. |
+| `retention` | object | | ISO-8601 durations for operational records: `runState`, `runLogs`, `lineage`, `dlq`. |
+| `lifecycle` | object | | `state` (`preview` \| `active` \| `deprecated` \| `retired`), `retention`, `deprecationPolicy`. |
+| `lineage` | object | | `granularity` (`table_level` \| `field_level`), `upstream[]`, `downstream[]`. |
+| `schemaEvolution` | object | | `strategy` (`semantic_versioning` \| `date_based` \| `sequential`), `compatibility` (`backward_compatible` \| `forward_compatible` \| `full_compatible` \| `breaking`), `changePolicy`. |
+| `machineLearning` | object | | `enabled`, `framework`, `models[]`. |
+| `environments` | map of name → environment | | Per-environment overrides of `metadata` and `exposes`. |
+| `docs` | object | | `homepage`, `runbook`, `dictionary`, `changeLog`. |
+| `extensions` | object (open) | | Vendor- or plugin-namespaced configuration. |
+
+An **identifier** matches `^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$` (or is a single letter, digit or underscore). A **duration** is an ISO-8601 duration such as `P30D` or `PT15M`.
+
+### 1.2 `metadata`
+
+**Closed.** `owner` is required; no member of `owner` is.
+
+| Member | Type | Meaning |
+|---|---|---|
+| `owner` (required) | object, closed: `team`, `email`, `slack`, `oncall` | Who owns the product. `email` carries `format: email`, which is an annotation (see [Validation semantics](#validation-semantics)). |
+| `layer` | string | Free-form layer label; `Bronze` / `Silver` / `Gold` is a convention, not an enum. |
+| `productType` | `SDP` \| `ADP` \| `CDP` | Source-aligned, aggregated or consumption-aligned data product. |
+| `classification` | `public` \| `internal` \| `confidential` \| `restricted` | Default classification. Optional. |
+| `businessContext` | object: `domain`, `subdomain`, `businessCapability`, `valueStream` | Business context. |
+| `experimental` | array of unique strings | Experimental features the document opts into. |
+| `createdAt` | string, `format: date-time` | Creation time. |
+| `provenance` | object | Generation envelope written by tooling (tool, version, command, time). |
+| `tags` | array of tags | |
+
+### 1.3 `exposes[]` — output ports
+
+Each expose is **closed** and requires `exposeId`, `kind`, `contract` and `binding`.
+
+| Member | Type | Meaning |
+|---|---|---|
+| `exposeId` (required) | identifier | Id of the port, unique within the product. |
+| `kind` (required) | `table` \| `view` \| `api` \| `file` \| `stream` \| `topic` \| `feature_store` \| `model` \| `vector` \| `graph` \| `time_series` \| `other` | What the port is. |
+| `contract` (required) | object, closed | The data's shape and promises. Must contain `schema` or `openapiRef` (or both). |
+| `binding` (required) | object, closed | Where the data lives (§1.3.2). |
+| `policy` | object, closed | `authn`, `authz` (`readers`, `writers`, `columnRestrictions`), `privacy` (`masking[]`, `rowLevelPolicy`), `classification`, `agentPolicy`. |
+| `semantics` | object, closed | Business meaning: `entities`, `measures`, `dimensions`, `metrics`. |
+| `qos` | object, closed | `availability`, `freshnessSLO`, `dataLossSLO`, `latencyP95`, `completenessTarget`, `errorBudget`. |
+| `mcp` | object, closed | `sampling.maxRows`, `classification.dataClass` for serving the port to AI agents over MCP. |
+| `lifecycle`, `observability`, `docs` | object | Per-port lifecycle, observability and documentation. |
+| `title`, `description`, `version` | string (`version` is semver) | |
+| `crawler`, `iceberg` | object | AWS Glue crawler and Iceberg table-maintenance settings. |
+| `tags`, `labels` | | |
+
+#### 1.3.1 `contract`
+
+| Member | Type | Meaning |
+|---|---|---|
+| `schema` | array of column | The columns. |
+| `openapiRef` | string | Reference to an OpenAPI document, for `kind: api`. |
+| `dq` | object: `rules[]`, `monitoring` | Data-quality rules. Each rule is closed and requires `id`, `type` (`freshness` \| `completeness` \| `uniqueness` \| `valid_values` \| `accuracy` \| `schema` \| `anomaly_detection` \| `drift_detection`) and `severity` (`info` \| `warn` \| `error` \| `critical`). |
+| `schemaPolicy` | `strict` \| `discover_and_freeze` \| `evolve_safe` \| `evolve_all` | How the output schema may change. |
+| `schemaSignature` | `sha256:` + 64 hex | A digest of the schema. |
+| `guarantees`, `quality` | object, array | Compatibility promises and additional quality checks. |
+
+A **column** is closed and requires `name` and `type`. `type` is a type name such as `string`, `int64`, `numeric`, `timestamp` or `json` from a fixed list, matched case-insensitively and optionally with parameters (`VARCHAR(255)`, `DECIMAL(10,2)`). Other members: `required` (boolean), `description`, `sensitivity` (`none` \| `internal` \| `confidential` \| `restricted` \| `pii` \| `phi` \| `cleartext` \| `treated` \| `anonymized` \| `pseudonymized` \| `tokenized` \| `encrypted`), `semanticType`, `businessName`, `businessDefinition`, `validationRules`, `tags`, `labels`.
+
+#### 1.3.2 `binding`
+
+Closed; requires `platform`, `format` and `location`.
+
+| Member | Type | Meaning |
+|---|---|---|
+| `platform` (required) | `gcp` \| `aws` \| `azure` \| `snowflake` \| `databricks` \| `kafka` \| `confluent` \| `local` \| `kubernetes` \| `postgres` \| `pgvector` \| `other` | The platform. |
+| `format` (required) | `bigquery_table`, `snowflake_table`, `snowflake_view`, `gcs_file`, `s3_file`, `http_api`, `grpc_api`, `pubsub_topic`, `kafka_topic`, `delta_table`, `iceberg`, `parquet`, `csv`, `json`, `redshift_table`, `redshift_serverless`, `redshift_external_schema`, `postgres_table`, `athena_table`, `glue_table`, `pgvector_table`, `other` | The physical format. |
+| `location` (required) | object, closed | Platform-specific address. Members: `account`, `project`, `dataset`, `database`, `schema`, `table`, `bucket`, `path`, `gateway`, `baseUrl`, `topic`, `subscription`, `region`, `zone`, `environment_id`, `kafka_cluster_id`, `confluent_role_arn`, `stream`, `namespace`, `workgroup`, `iam_role_arn`, `external_schema`, `glue_database`, `catalog`, `warehouse`, `uri`, `partitionBy`. None is required by the schema. |
+| `icebergConfig` | object | Iceberg table settings (write version, file format, partition spec, sort order). |
+| `vectorConfig` | object, closed; `dimensions` required | Vector / embeddings output-port settings. |
+| `governance` | object: `lakeFormation` | Per-resource AWS Lake Formation settings: `registerLocation`, `grants[]`, `tags`, `rowFilter`. |
+| `properties` | object (open) | Platform-specific extra properties. |
+| `tags`, `labels` | | |
+
+### 1.4 `consumes[]` — input ports
+
+Each entry is **closed** and requires `productId` and `exposeId`: the upstream product and the port it reads. Optional: `versionConstraint` (a semver range such as `^2.0.0`), `qosExpectations` (`freshnessMax`, `maxStaleness`, `minCompleteness`), `requiredPolicies[]`, `purpose`, `tags`, `labels`.
+
+### 1.5 `build` and `builds[]`
+
+A build is **closed**; no member is required.
+
+| Member | Type | Meaning |
+|---|---|---|
+| `id` | identifier | Build id (useful when there are several). |
+| `pattern` | `hybrid-reference` \| `embedded-logic` \| `multi-stage` \| `acquisition` | Selects the shape of `properties`. |
+| `engine` | `dbt`, `sql`, `python`, `spark`, `glue`, `custom`, `duckdb`, `airbyte`, `meltano`, `dlt`, `kafka-connect`, `debezium`, or `dbt-` | The engine. |
+| `properties` | object | The pattern's settings (below). |
+| `execution` | object, closed | `trigger`, `runtime`, `retries`, `notifications[]`, `orchestration`. |
+| `capabilities` | array | What the build asks of its runner, e.g. `incremental_dedup`, `cdc`, `streaming`, `exactly_once`. |
+| `repository`, `description` | string | |
+| `outputs` | array of identifier | The `exposeId`s the build produces. |
+| `dependencies`, `transformations` | array | |
+
+`properties` is validated according to `pattern`:
+
+| `pattern` | `properties` shape | Required |
+|---|---|---|
+| `hybrid-reference` | `model`, `target`, `select`, `models`, `vars`, `materializations` — a reference to a model kept elsewhere, such as a dbt project | `model` |
+| `embedded-logic` | `sql`, `language` (`sql` \| `flink_sql` \| `pyspark` \| `scala` \| `python` \| `r`), `parameters` | `sql` |
+| `multi-stage` | `stages[]`, `orchestration` | — |
+| `acquisition` | `source` (`kind`, `mode` required), `sink`, `delivery`, `schemaEvolution`, `preLand`, `quality`, `cost`, `catalog`, `concurrency`, `lineage`, and one settings key per engine: `duckdb`, `airbyte`, `meltano`, `dlt`, `kafka-connect`, `debezium` | `source` |
+
+### 1.6 `orchestration`
+
+Requires `engine` (`airflow` \| `dagster` \| `prefect` \| `kubeflow` \| `custom` \| `none`). Defined members: `mode` (`generated` \| `manual` \| `hybrid`), `generateOnChange`, and per-engine settings `airflow` (which requires `dagId`, and holds `tasks[]`), `dagster`, `prefect`.
+
+The object is **open**: it does not set `additionalProperties`, so members it does not define — such as an `orchestration.tasks` list — are accepted **without being checked**. The only task shape the schema checks is `orchestration.airflow.tasks[]` (and the same under `build.execution.orchestration`). See the [Cheatsheet](/fluid/schema/cheatsheet#orchestration-fields).
+
+### 1.7 Access and governance
+
+| Block | Shape |
+|---|---|
+| `accessPolicy` | Closed; `grants[]`, each closed with `principal` (required), `permissions` (`read`, `select`, `query`, `write`, `insert`, `update`, `delete`, `create`, `admin`, `manage`), `resources` (array of strings, e.g. JSONPath selecting exposes) and `conditions` (open object). |
+| `sovereignty` | Closed: `jurisdiction`, `allowedRegions`, `deniedRegions`, `dataResidency` (boolean), `crossBorderTransfer` (boolean), `transferMechanisms`, `regulatoryFramework`, `enforcementMode` (`strict` \| `advisory` \| `audit`), `validationRequired`. |
+| `exposes[].policy.agentPolicy` | Closed: `allowedModels`, `deniedModels`, `maxTokensPerRequest`, `maxTokensPerDay`, `allowedUseCases` / `deniedUseCases` (from `inference`, `reasoning`, `analysis`, `summarization`, `classification`, `embedding`, `search`, `qa`, `code_generation`, `fine_tuning`, `training`, `rag`), `canReason`, `canStore`, `retentionPolicy`, `auditRequired`, `purposeLimitation`. It exists only per expose, not at the root. |
+| `governance.lakeFormation` | Closed: `admins` (IAM ARNs) and `tagDefinitions` (tag key → allowed values). `admins` is **authoritative**: applying it replaces the account's Lake Formation admin list. The schema's description adds that applying it also clears the account's create-database and create-table default permissions, trusted resource owners and parameters, and that destroying the emitted `aws_lakeformation_data_lake_settings` resource empties the admin list and resets `CROSS_ACCOUNT_VERSION` to 1. |
---
-### 1.9 extensions Block
+## Further reading
-| Key | Type | Description |
-|----------------------|------|--------------------------------------------------------------------|
-| customTransformations| List | A list of custom transformation engines the build requires. |
-| policyEngines | List | A list of external policy engines (e.g., OPA) needed to evaluate policies. |
-| observabilityHooks | List | A list of custom hooks to send metrics and traces to external systems. |
+- [Anatomy](/fluid/schema/anatomy) — a guided tour of the blocks, with examples.
+- [Cheatsheet](/fluid/schema/cheatsheet) — one row per field.
+- [Changelog](/fluid/schema/changelog) — what changed in each version.
+- *FLUID Data Products* (book) is [available on Amazon](https://amzn.eu/d/ikMlWNV).
diff --git a/docs/schema/versions.md b/docs/schema/versions.md
index feeb138..6658993 100644
--- a/docs/schema/versions.md
+++ b/docs/schema/versions.md
@@ -1,20 +1,69 @@
# Schema Versions
-Every published version of the FLUID JSON Schema. Point your validator at a version's JSON Schema; read its generated HTML reference for the field-by-field detail. The latest version is **0.7.5**.
-
-| Version | JSON Schema | HTML Reference |
-|---|---|---|
-| **0.7.5** **(latest)** | [`fluid-schema-0.7.5.json`](/fluid/schema/fluid-schema-0.7.5.json) | [`0.7.5/fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html) |
-| 0.7.4 | [`fluid-schema-0.7.4.json`](/fluid/schema/fluid-schema-0.7.4.json) | [`0.7.4/fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html) |
-| 0.7.3 | [`fluid-schema-0.7.3.json`](/fluid/schema/fluid-schema-0.7.3.json) | [`0.7.3/fluid-spec.html`](/fluid/specs/0.7.3/fluid-spec.html) |
-| 0.7.2 | [`fluid-schema-0.7.2.json`](/fluid/schema/fluid-schema-0.7.2.json) | [`0.7.2/fluid-spec.html`](/fluid/specs/0.7.2/fluid-spec.html) |
-| 0.7.1 | [`fluid-schema-0.7.1.json`](/fluid/schema/fluid-schema-0.7.1.json) | [`0.7.1/fluid-spec.html`](/fluid/specs/0.7.1/fluid-spec.html) |
-| 0.5.7 | [`fluid-schema-0.5.7.json`](/fluid/schema/fluid-schema-0.5.7.json) | [`0.5.7/fluid-spec.html`](/fluid/specs/0.5.7/fluid-spec.html) |
-| 0.4.0 | [`fluid-schema-0.4.0.json`](/fluid/schema/fluid-schema-0.4.0.json) | [`0.4.0/fluid-spec.html`](/fluid/specs/0.4.0/fluid-spec.html) |
-| 0.3.0 | [`fluid-schema-0.3.0.json`](/fluid/schema/fluid-schema-0.3.0.json) | [`0.3.0/fluid-spec.html`](/fluid/specs/0.3.0/fluid-spec.html) |
-| 0.2.0 | [`fluid-schema-0.2.0.json`](/fluid/schema/fluid-schema-0.2.0.json) | [`0.2.0/fluid-spec.html`](/fluid/specs/0.2.0/fluid-spec.html) |
-| 0.1.1 | [`fluid-schema-0.1.1.json`](/fluid/schema/fluid-schema-0.1.1.json) | — |
-| 0.1.0 | [`fluid-schema-0.1.0.json`](/fluid/schema/fluid-schema-0.1.0.json) | — |
-| 0.0.1 | [`fluid-schema-0.0.1.json`](/fluid/schema/fluid-schema-0.0.1.json) | — |
-
-For what changed between any two versions, see the [**Changelog**](/fluid/schema/changelog).
+Every published version of the FLUID JSON Schema. Point your validator at a version's JSON Schema; read its generated HTML reference for the field-by-field detail.
+
+- **Latest stable: 0.7.5.** Use it for new contracts.
+- **Preview: 0.7.6.** Published so that it can be read and validated against. It can still change before it is promoted. See [Stable and preview versions](#stable-and-preview-versions).
+
+| Version | Status | JSON Schema | HTML Reference |
+|---|---|---|---|
+| **0.7.6** | **Preview** | [`fluid-schema-0.7.6.json`](/fluid/schema/fluid-schema-0.7.6.json) | [`0.7.6/fluid-spec.html`](/fluid/specs/0.7.6/fluid-spec.html) |
+| **0.7.5** | **Stable (latest)** | [`fluid-schema-0.7.5.json`](/fluid/schema/fluid-schema-0.7.5.json) | [`0.7.5/fluid-spec.html`](/fluid/specs/0.7.5/fluid-spec.html) |
+| 0.7.4 | Stable | [`fluid-schema-0.7.4.json`](/fluid/schema/fluid-schema-0.7.4.json) | [`0.7.4/fluid-spec.html`](/fluid/specs/0.7.4/fluid-spec.html) |
+| 0.7.3 | Stable | [`fluid-schema-0.7.3.json`](/fluid/schema/fluid-schema-0.7.3.json) | [`0.7.3/fluid-spec.html`](/fluid/specs/0.7.3/fluid-spec.html) |
+| 0.7.2 | Stable | [`fluid-schema-0.7.2.json`](/fluid/schema/fluid-schema-0.7.2.json) | [`0.7.2/fluid-spec.html`](/fluid/specs/0.7.2/fluid-spec.html) |
+| 0.7.1 | Stable — [see note](#_0-7-1-two-documents-share-one-id) | [`fluid-schema-0.7.1.json`](/fluid/schema/fluid-schema-0.7.1.json) | [`0.7.1/fluid-spec.html`](/fluid/specs/0.7.1/fluid-spec.html) |
+| 0.5.7 | Historical | [`fluid-schema-0.5.7.json`](/fluid/schema/fluid-schema-0.5.7.json) | [`0.5.7/fluid-spec.html`](/fluid/specs/0.5.7/fluid-spec.html) |
+| 0.4.0 | Historical | [`fluid-schema-0.4.0.json`](/fluid/schema/fluid-schema-0.4.0.json) | [`0.4.0/fluid-spec.html`](/fluid/specs/0.4.0/fluid-spec.html) |
+| 0.3.0 | Historical | [`fluid-schema-0.3.0.json`](/fluid/schema/fluid-schema-0.3.0.json) | [`0.3.0/fluid-spec.html`](/fluid/specs/0.3.0/fluid-spec.html) |
+| 0.2.0 | Historical | [`fluid-schema-0.2.0.json`](/fluid/schema/fluid-schema-0.2.0.json) | [`0.2.0/fluid-spec.html`](/fluid/specs/0.2.0/fluid-spec.html) |
+| 0.1.1 | Historical | [`fluid-schema-0.1.1.json`](/fluid/schema/fluid-schema-0.1.1.json) | — |
+| 0.1.0 | Historical | [`fluid-schema-0.1.0.json`](/fluid/schema/fluid-schema-0.1.0.json) | — |
+| 0.0.1 | Historical | [`fluid-schema-0.0.1.json`](/fluid/schema/fluid-schema-0.0.1.json) | — |
+
+For what changed between any two versions, see the [**Changelog**](/fluid/schema/changelog). For the release notes, see [**What's New**](/fluid/releases/).
+
+---
+
+## Stable and preview versions
+
+| Status | Meaning |
+|---|---|
+| **Stable** | Published and settled. Each stable version is checked on every pull request by [`scripts/check-compat.py`](https://github.com/open-data-protocol/fluid/blob/main/scripts/check-compat.py): a document valid under one version must stay valid under the next (see [GOVERNANCE.md](https://github.com/open-data-protocol/fluid/blob/main/GOVERNANCE.md#versioning-and-compatibility)). |
+| **Preview** | Published so that its `$id` resolves and a document that declares it can be validated by any JSON Schema validator. It is **not settled**: until it is promoted to stable, its fields and their meaning can still change, and the file at its URL is replaced when they do. |
+| **Historical** | 0.5.7 and earlier. These predate the compatibility promise, which the gate enforces from 0.7.1 on. Several of those transitions removed or narrowed fields; the [Changelog](/fluid/schema/changelog) marks which. |
+
+### What "preview" means for you
+
+- **A contract opts in explicitly**, by declaring `fluidVersion: "0.7.6"`. Nothing else selects a preview.
+- **Expect to re-validate.** The compatibility gate compares two *versions*. It cannot compare one revision of the 0.7.6 file with an earlier revision of the same file, so a contract written against an earlier revision of 0.7.6 may need changes after a later one. The schemas are authored in the reference implementation, [forge-cli](https://github.com/Agenticstiger/forge-cli), where `fluid_build/schemas/fluid-schema-0.7.6.json` has changed several times since 0.7.6 opened; its history is the record of those changes.
+- **Promotion** makes a preview the latest stable version and is announced in its release notes. 0.7.5 went through the same path: it was an opt-in preview in forge-cli from 0.9.0 and was promoted to stable in forge-cli 0.12.0, when 0.7.6 opened as the next preview.
+
+### How the reference implementation treats a preview (non-normative)
+
+[`data-product-forge`](/fluid/concepts/forge-cli) bundles 0.7.6 and validates a contract that declares it (`✅ Valid FLUID contract (schema v0.7.6)`), but never chooses it on its own: its newest *stable* bundled version is its default, and `fluid init --quickstart` writes `fluidVersion: 0.7.5`. This describes one implementation. The standard's rule is only the one in [Validation semantics](/fluid/schema/specification#validation-semantics): a document is validated against the schema of the version it declares.
+
+---
+
+## Choosing `fluidVersion`
+
+A document is validated against the schema whose version matches its `fluidVersion` ([Validation semantics](/fluid/schema/specification#validation-semantics)). Three consequences are easy to miss:
+
+1. **To use a field, declare the version that defines it.** The 0.7.5 schema's own `fluidVersion` enum also accepts `"0.7.3"` and `"0.7.4"`. That does not make a document declaring `"0.7.4"` valid when it uses a 0.7.5 field: its matching schema is 0.7.4, which rejects the field. For example, `binding.platform: confluent` (added in 0.7.5) in a contract declaring `"0.7.4"` is invalid, although the 0.7.5 schema alone would accept it. The reference implementation agrees: it reports `'confluent' is not one of [...]` against schema v0.7.4.
+2. **Point your editor at the matching schema.** In a `# yaml-language-server: $schema=…` line, use the URL of the version the file declares. Pointing an older file at a newer schema hides exactly the mistake in (1).
+3. **Upgrading is changing the number.** Every stable transition from 0.7.2 on keeps every valid document valid, so moving a valid contract to a later stable version is a one-line change. The one recorded exception is 0.7.1 → 0.7.2, where `notification` became a closed object (see the [0.7.2 release note](/fluid/releases/0.7.2#backward-compatibility-with-0-7-1-one-narrowing-change)).
+
+---
+
+## 0.7.1: two documents share one `$id`
+
+The 0.7.1 file published here and the 0.7.1 file the reference implementation bundles carry the same `$id` (`…/schema/fluid-schema-0.7.1.json`) but are different documents, and they disagree in both directions:
+
+- The reference implementation's copy accepts fields the published file rejects — for example top-level `orchestration`, `binding.icebergConfig` and `exposes[].contract.quality`.
+- It also rejects documents the published file accepts — for example an extra member on a `notifications[]` entry, or a column `type` outside its list of type names.
+
+This repository does not re-synchronise 0.7.1 (see [CONTRIBUTING.md](https://github.com/open-data-protocol/fluid/blob/main/CONTRIBUTING.md#the-schemas-are-not-authored-here)), so the file above is the 0.7.1 that this site documents. If a 0.7.1 result has to be reproducible across tools, validate against this file, or move the contract to 0.7.2 or later, where the published files and the reference implementation's copies agree on which documents are valid.
+
+## 0.4.0 and earlier: `$id`s that do not resolve
+
+The schemas from 0.0.1 to 0.4.0 declare `$id`s on `open-data-protocol.org`, a host that does not resolve, and their `$id`s do not follow the FLUID version (0.1.0 and 0.1.1 both declare `…/fluid.schema.v2.0.json`, 0.2.0 declares `…/fluid.schema.v2.3.json`). Two different documents sharing an `$id` cannot both be registered in a validator that keys schemas by `$id`. Load these files by their URL on this site instead. They are kept unchanged as history; from 0.5.7 on, every `$id` is the file's own URL here.