Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

---

FLUID is one YAML file that describes a data product end to end — **schema, build, orchestration, agentic governance, sovereignty, and semantics**. Write it once; validate it, compile it, and deploy it anywhere. It compiles to Bitol ODPS + ODCS for catalog interop via the reference compiler, [`forge-cli`](https://github.com/Agenticstiger/forge-cli).
FLUID is one YAML file that describes a data product end to end — **schema, build, orchestration, agentic governance, sovereignty, and semantics**. Write it once; validate it, compile it, and deploy it anywhere. It compiles to Bitol ODPS + ODCS for catalog interop via the reference implementation, [`forge-cli`](https://github.com/Agenticstiger/forge-cli).

## How the pieces fit

Expand Down Expand Up @@ -61,14 +61,25 @@ exposes:

## JSON Schema

- **Latest:** `https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.5.json`
- Use it in your editor — add this line to any `.fluid.yml`:
- **Latest stable: 0.7.5** — `https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.5.json`
- **Preview: 0.7.6** — `https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.6.json`. A contract uses it only by declaring `fluidVersion: "0.7.6"`, and it can still change before it is promoted to stable. See **[Stable and preview versions](https://open-data-protocol.github.io/fluid/schema/versions#stable-and-preview-versions)** and **[What's new in 0.7.6](https://open-data-protocol.github.io/fluid/releases/0.7.6)**.
- A document is validated against the schema of the `fluidVersion` it declares. To use a field, declare the version that added it — see **[Choosing `fluidVersion`](https://open-data-protocol.github.io/fluid/schema/versions#choosing-fluidversion)**.
- Editor support — add this line to a file that declares `fluidVersion: "0.7.5"` (use the URL matching the version the file declares):
```yaml
# yaml-language-server: $schema=https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.5.json
```
- All versions, diffs, and the generated HTML reference: **[Schema → Versions](https://open-data-protocol.github.io/fluid/schema/versions)**.

> **Note:** 0.7.5 ("Streaming Kafka → Iceberg Sink & Confluent Tableflow") is **additive and fully backward-compatible** with 0.7.4 — every valid 0.7.4 contract still validates. See **[What's New in 0.7.5](https://open-data-protocol.github.io/fluid/releases/0.7.5)**.
0.7.5 is additive over 0.7.4: every valid 0.7.4 contract stays valid when it declares `"0.7.5"`. See **[What's New in 0.7.5](https://open-data-protocol.github.io/fluid/releases/0.7.5)**.

## Validate a contract

With a JSON Schema Draft 2020-12 validator, against the schema of the declared version (validators that compile `pattern` as ECMA-262, as JavaScript ones do, should read the [known interoperability issue](https://open-data-protocol.github.io/fluid/schema/specification#known-interoperability-issue-one-pattern-is-not-an-ecma-262-regular-expression) first). Or with the reference implementation, [`data-product-forge`](https://open-data-protocol.github.io/fluid/concepts/forge-cli):

```bash
pip install data-product-forge
fluid validate contract.fluid.yaml
```

## Build the docs locally

Expand Down
20 changes: 17 additions & 3 deletions docs/.vuepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,23 +75,36 @@ export default defineUserConfig({
{ text: 'Core Principles', link: '/concepts/principles' },
{ text: 'Agentic-Native Layer', link: '/concepts/agentic-native' },
{ text: 'FLUID vs ODCS / ODPS', link: '/concepts/comparisons' },
{ text: 'Reference Implementation', link: '/concepts/forge-cli' },
],
},
{
text: 'Schema',
children: [
{ text: 'Anatomy', link: '/schema/anatomy' },
{ text: 'Cheatsheet', link: '/schema/cheatsheet' },
{ text: 'Full Specification', link: '/schema/specification' },
{ text: 'Minimal Contract', link: '/schema/minimal-contract' },
{ text: 'Specification', link: '/schema/specification' },
{ text: 'Versions', link: '/schema/versions' },
{ text: 'JSON Schema 0.7.5 ↗', link: 'https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.5.json', target: '_blank' },
{ text: 'Reference (HTML) ↗', link: 'https://open-data-protocol.github.io/fluid/specs/0.7.5/fluid-spec.html', target: '_blank' },
{ text: 'Changelog', link: '/schema/changelog' },
{ text: 'JSON Schema 0.7.5 (stable) ↗', link: 'https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.5.json', target: '_blank' },
{ text: 'Reference 0.7.5 (HTML) ↗', link: 'https://open-data-protocol.github.io/fluid/specs/0.7.5/fluid-spec.html', target: '_blank' },
{ text: 'Preview: 0.7.6', link: '/releases/0.7.6' },
],
},
{ text: 'Examples', link: '/examples/' },
{ text: 'How-to', link: '/how-to/' },
{ text: "What's New", link: '/releases/' },
{ text: 'Deck', link: '/deck/' },
{
text: 'Project',
children: [
{ text: 'Contributing', link: '/contributing/' },
{ text: 'Governance ↗', link: 'https://github.com/open-data-protocol/fluid/blob/main/GOVERNANCE.md', target: '_blank' },
{ text: 'Conformance Corpus ↗', link: 'https://github.com/open-data-protocol/fluid/blob/main/tests/README.md', target: '_blank' },
{ text: 'Vision', link: '/vision/' },
],
},
{ text: 'GitHub', link: 'https://github.com/open-data-protocol/fluid' },
],

Expand Down Expand Up @@ -162,6 +175,7 @@ export default defineUserConfig({
text: "What's New",
children: [
'/releases/README.md',
'/releases/0.7.6.md',
'/releases/0.7.5.md',
'/releases/0.7.4.md',
'/releases/0.7.3.md',
Expand Down
14 changes: 8 additions & 6 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,16 @@ features:
- title: Contract-First
details: One .fluid.yml is the source of truth — schema, quality, build, lineage, governance. Version-controlled and schema-validated.
- title: Agentic-Native
details: agentPolicy, sovereignty, and semantics give LLM agents deterministic answers to PII, allowed-use, metric, and residency questions — enforced at the MCP gateway.
details: agentPolicy, sovereignty, and semantics give LLM agents machine-readable answers to PII, allowed-use, metric, and residency questions — and an MCP gateway can enforce them on every read.
- title: Operational Superset
details: The only open spec covering build + orchestration + source-aligned acquisition alongside contract and governance.
details: Covers build, orchestration and source-aligned acquisition alongside the data contract and its governance.
- title: Interoperable
details: Compiles to Bitol ODPS + ODCS via the forge-cli reference compiler for DataHub / OpenMetadata / Datamesh Manager catalog interop.
details: Compiles to Bitol ODPS + ODCS via the forge-cli reference implementation for DataHub / OpenMetadata / Datamesh Manager catalog interop.
- title: Federated by Design
details: Built for Data Mesh — decentralized ownership, globally unique product ids, one unified fabric.
- title: Open & Apache 2.0-Licensed
details: A community-led protocol. Good ideas backed by real use cases get in.
footer: Apache 2.0 Licensed | © open-data-protocol — Federated Layered Unified Interchange Definition
details: An open specification under the Apache License 2.0, with a public conformance corpus and no CLA.
footer: Licensed under the Apache License 2.0 | Copyright 2025 The FLUID Authors
---

> **Your agents are only as trustworthy as the data products they consume.**
Expand Down Expand Up @@ -65,5 +65,7 @@ exposes:
- **[Guide](/fluid/guide/)** — what FLUID is, the quickstart, and the FAQ.
- **[Concepts](/fluid/concepts/)** — the agentic-native layer and how FLUID compares to ODCS / ODPS.
- **[Schema Reference](/fluid/schema/anatomy)** — every top-level block, a cheatsheet, and the full specification.
- **[What's New in 0.7.5](/fluid/releases/0.7.5)** — streaming Kafka → Iceberg sink + Confluent Cloud Tableflow.
- **[What's New in 0.7.5](/fluid/releases/0.7.5)** — the latest stable version: streaming Kafka → Iceberg sink, Confluent Cloud Tableflow, and a pgvector output port.
- **[0.7.6 (preview)](/fluid/releases/0.7.6)** — packaging modes, declared consumers, cross-mesh pins; opt-in, and still subject to change.
- **[See the deck](/fluid/deck/)** — the FLUID story in slides.
- **[Contributing & governance](/fluid/contributing/)** — how the standard is maintained, and how to take part.
2 changes: 1 addition & 1 deletion docs/concepts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ A FLUID contract describes a data product end to end: its schema, build logic, d

This structure separates **interface** (what you get) from **implementation** (how it's built), enabling reliable data ecosystems ready for both humans and AI agents.

> The latest schema version is **0.7.4**, which adds runtime `agentPolicy` enforcement at the MCP gateway. Like the 0.7.1 → 0.7.3 line, **0.7.4 is additive and fully backward-compatible** — every valid 0.7.3 contract still validates. See the release notes for details.
> The latest stable schema version is **0.7.5**; **0.7.6** is a preview that a contract uses only by declaring it. Every stable release after 0.7.2 has been additive over its predecessor; the one recorded narrowing is the step into 0.7.2 (0.7.1 → 0.7.2). See [What's New](/fluid/releases/) and [Versions](/fluid/schema/versions).

---

Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/agentic-native.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Each spec was evaluated against the **canonical example file its maintainers pub
| Bitol ODCS v3.1 | [`full-example.odcs.yaml`](https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/examples/all/full-example.odcs.yaml) (seller/payments contract) |
| Bitol ODPS v1.0 | [`customer-data-product.odps.yaml`](https://github.com/bitol-io/open-data-product-standard/blob/main/docs/examples/customer-data-product.odps.yaml) |
| ODPS v4 | [`urbanpulse_final.yml`](https://github.com/Open-Data-Product-Initiative/v4.0/blob/main/source/examples/Refs/urbanpulse_final.yml) (UrbanPulse Events) |
| FLUID v0.7.3 | [Example 10 (Customers CDC)](/fluid/examples/#10-source-aligned-acquisition) |
| FLUID v0.7.3 | [Example 10 (Customers CDC)](/fluid/examples/#_10-source-aligned-acquisition) |

## The four agent failure modes

Expand Down
12 changes: 6 additions & 6 deletions docs/concepts/comparisons.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Four active "open data product" specs share overlapping names and adjacent scope
| **ODCS** (Open Data Contract Standard) | LF AI & Data · Bitol | v3.1.0 — Dec 2025 | Column-level technical contract; producer↔consumer agreement for one dataset |
| **Bitol ODPS** (Open Data Product Standard) | LF AI & Data · Bitol | v1.0.0 — Sep 2025 | Thin product manifest; bundles ODCS contracts via `contractId` on input/output ports |
| **ODPS v4** (Open Data Product Specification) | LF · [Open-Data-Product-Initiative](https://github.com/Open-Data-Product-Initiative/v4.0) | v4.0 — Jul 2025 · v4.1 — Oct 2025 | Business + commercial wrapper: pricing · license · multi-language · marketplace |
| **FLUID** | open-data-protocol | v0.7.3 — this repo | End-to-end operational contract: schema + build + orchestration + agentic governance + sovereignty + semantics |
| **FLUID** | open-data-protocol | v0.7.5 (stable) — this repo | End-to-end operational contract: schema + build + orchestration + agentic governance + sovereignty + semantics |

## How they actually fit together

Expand All @@ -34,9 +34,9 @@ flowchart LR
classDef core fill:#5B8DEF,color:#fff,stroke:#1E3A8A,stroke-width:4px,font-weight:bold
classDef opt fill:#94A3B8,color:#fff,stroke:#475569,stroke-width:1px,stroke-dasharray:6 3

F["FLUID v0.7.3<br/>your .fluid.yml<br/>standalone — complete on its own"]:::core
F["FLUID v0.7.5<br/>your .fluid.yml<br/>standalone — complete on its own"]:::core

FC["forge-cli<br/>reference compiler<br/>(emits IaC + Airflow DAGs)"]:::opt
FC["forge-cli<br/>reference implementation<br/>(emits IaC + Airflow DAGs)"]:::opt
BIT["Bitol ODPS + ODCS<br/>(catalog interop)"]:::opt
V4["ODPS v4 wrapper<br/>(commercial publishing)"]:::opt

Expand All @@ -62,7 +62,7 @@ flowchart LR

## 📊 Capability matrix

Legend: ✅ deterministic in spec · ⚠️ partial · ❌ silent. Headers abbreviated for width: **F** = FLUID v0.7.3 · **ODCS** = Bitol ODCS v3.1 · **ODPS** = Bitol ODPS v1.0 · **v4** = ODPS v4.0. Field-level detail lives in the [**Schema Cheatsheet**](/fluid/schema/cheatsheet) — this matrix is the at-a-glance scoreboard.
Legend: ✅ deterministic in spec · ⚠️ partial · ❌ silent. Headers abbreviated for width: **F** = FLUID v0.7.5 · **ODCS** = Bitol ODCS v3.1 · **ODPS** = Bitol ODPS v1.0 · **v4** = ODPS v4.0. Field-level detail lives in the [**Schema Cheatsheet**](/fluid/schema/cheatsheet) — this matrix is the at-a-glance scoreboard.

### 📐 Data shape & quality

Expand Down Expand Up @@ -111,7 +111,7 @@ Legend: ✅ deterministic in spec · ⚠️ partial · ❌ silent. Headers abbre

> Each capability links to its field-level reference in the [**Schema Cheatsheet**](/fluid/schema/cheatsheet). MetricFlow round-trip is on the FLUID roadmap.

> ⚙️ The matrix above shows what each spec *covers*. **[`forge-cli`](/fluid/concepts/forge-cli)** is the reference compiler that turns a FLUID contract into deployed reality and Bitol-compatible outputs.
> ⚙️ The matrix above shows what each spec *covers*. **[`forge-cli`](/fluid/concepts/forge-cli)** is the reference implementation that turns a FLUID contract into deployed reality and Bitol-compatible outputs.

---

Expand All @@ -138,7 +138,7 @@ The cleanest production stack uses all four where each is strongest:
- [Bitol ODCS — open-data-contract-standard](https://github.com/bitol-io/open-data-contract-standard) (v3.1.0)
- [Bitol ODPS — open-data-product-standard](https://github.com/bitol-io/open-data-product-standard) (v1.0.0)
- [opendataproducts.org ODPS v4](https://opendataproducts.org/v4.0/) · [v4.0 repo](https://github.com/Open-Data-Product-Initiative/v4.0) · [v4.1 release](https://github.com/Open-Data-Product-Initiative/v4.1)
- [forge-cli — the FLUID reference compiler](https://github.com/Agenticstiger/forge-cli) · [forge-docs](https://agenticstiger.github.io/forge_docs/) (emits Bitol ODPS + ODCS)
- [forge-cli — the FLUID reference implementation](https://github.com/Agenticstiger/forge-cli) · [forge-docs](https://agenticstiger.github.io/forge_docs/) (emits Bitol ODPS + ODCS)
- [Linux Foundation AI & Data — Bitol project](https://lfaidata.foundation/projects/bitol/)

---
Expand Down
Loading
Loading