From ad1b88c56f80ab33aab268c1588a9f40e658c5e6 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Fri, 2 Oct 2026 19:29:52 +0200 Subject: [PATCH 1/5] =?UTF-8?q?docs:=200.18.0=20contract=20confinement=20?= =?UTF-8?q?=E2=80=94=20$ref=20root,=20DuckDB=20sandbox,=20contract-load=20?= =?UTF-8?q?API?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to forge-cli #687, #688 and #689, which release together as 0.18.0. Adds four pages and one separate sidebar group; no existing page, heading or the pinned CLI version is changed. - concepts/contract-refs.md: $ref composition, the ref root, FLUID_REF_ROOT and ref_root=, the monorepo migration, OAS-REF-EXTERNAL - advanced/duckdb-sandbox.md: what contract SQL can reach, declared locations, FLUID_DUCKDB_ALLOWED_DIRS, the duckdb>=1.5.0 floor, limits - advanced/contract-loading-api.md: fluid_build.api.load_contract, load_contract_from_text/_dict, LoadedContract, ContractLoadError (API 1.1) - RELEASE_NOTES_0.18.0.md: breaking changes and migration steps Example output was produced by running the 0.18.0 code. --- docs/.vuepress/config.ts | 12 + docs/RELEASE_NOTES_0.18.0.md | 182 ++++++++++++++ docs/advanced/contract-loading-api.md | 305 +++++++++++++++++++++++ docs/advanced/duckdb-sandbox.md | 338 ++++++++++++++++++++++++++ docs/concepts/contract-refs.md | 338 ++++++++++++++++++++++++++ 5 files changed, 1175 insertions(+) create mode 100644 docs/RELEASE_NOTES_0.18.0.md create mode 100644 docs/advanced/contract-loading-api.md create mode 100644 docs/advanced/duckdb-sandbox.md create mode 100644 docs/concepts/contract-refs.md diff --git a/docs/.vuepress/config.ts b/docs/.vuepress/config.ts index cb5ffec..6d5774d 100644 --- a/docs/.vuepress/config.ts +++ b/docs/.vuepress/config.ts @@ -614,6 +614,18 @@ export default defineUserConfig({ '/advanced/v1.5-release-notes.md' ] }, + // CLI 0.18.0: contract confinement (forge-cli #687, #688, #689). + // A group of its own so it merges cleanly with edits to the + // Concepts / Advanced / Project lists. + { + text: 'Contract loading & sandboxing', + children: [ + '/concepts/contract-refs.md', + '/advanced/duckdb-sandbox.md', + '/advanced/contract-loading-api.md', + '/RELEASE_NOTES_0.18.0.md' + ] + }, { text: 'Project', children: [ diff --git a/docs/RELEASE_NOTES_0.18.0.md b/docs/RELEASE_NOTES_0.18.0.md new file mode 100644 index 0000000..c9713c3 --- /dev/null +++ b/docs/RELEASE_NOTES_0.18.0.md @@ -0,0 +1,182 @@ +--- +title: Upgrading to CLI 0.18.0 +description: Breaking changes and migration steps for data-product-forge 0.18.0 β€” $ref confinement, the DuckDB sandbox, and the public contract-loading API. +--- + +# Upgrading to CLI `0.18.0` + +**Status:** Upgrade notes for `data-product-forge` `0.18.0`. The site's +[pinned docs baseline](./RELEASE_NOTES_0.8.3.md) is unchanged by this page. + +## Headline + +A contract can no longer make the engine read the machine it runs on. + +- **`$ref` composes only files inside the contract's own directory tree.** + ([forge-cli #687](https://github.com/Agenticstiger/forge-cli/pull/687)) +- **Contract SQL runs inside DuckDB's own sandbox.** + ([forge-cli #689](https://github.com/Agenticstiger/forge-cli/pull/689)) +- **A new public API loads a contract exactly as `fluid plan` sees it.** + ([forge-cli #688](https://github.com/Agenticstiger/forge-cli/pull/688)) + +These changes come from a security review of the FLUID Command Center, which +runs the engine on user-supplied contracts. The same holes apply to any +service, CI job or shared host that runs contracts its operator did not write. + +## Do I need to change anything? + +| If your contracts… | Then | +|---|---| +| use `$ref` only to files under the contract's own directory | nothing | +| use `$ref: ../…` to reach another product's or a shared directory | [widen the ref root](#ref-to-files-outside-the-contract-directory) | +| use a `$ref` with a URL (`file://`, `https://`) or an absolute path | [copy the fragment in](#ref-to-files-outside-the-contract-directory) | +| run SQL only on files under the contract's directory or workspace | nothing | +| declare inputs, outputs or acquisition sources at absolute paths elsewhere (`/data/landing/*.csv`) | [allow the directory](#sql-or-declarations-that-read-outside-the-contract-directory) | +| call `read_xlsx`, `sqlite_scan`, `ST_Read`, `delta_scan` or `iceberg_scan` in contract SQL | [convert or land the data](#functions-duckdb-used-to-autoload) | +| install DuckDB yourself at a version below 1.5.0 | [upgrade DuckDB](#upgrade-duckdb) | + +## Breaking changes and migration + +### `$ref` to files outside the contract directory + +Every `$ref` must name a file inside the **ref root**, by default the +directory that holds the root contract. URL refs (`file://` included) and +absolute paths are refused; `..` and symlink escapes are refused after +resolution. The check applies to `fluid validate`, `plan`, `apply` and +`bundle`. + +```console +$ fluid validate contract.fluid.yaml +❌ Validation error: contract_load_failed + error: $ref '../shared/policy.yaml#/gold' at JSON pointer '/exposes/0/policy' in + /work/orders/contract.fluid.yaml escapes the ref root /work/orders (after resolving + '..' and symlinks); refs may only name files inside it. ... +``` + +**Migrate a monorepo** that shares fragments across products: set +`FLUID_REF_ROOT` to the narrowest directory that holds the products and their +shared fragments, per command: + +```bash +FLUID_REF_ROOT="$(git rev-parse --show-toplevel)" \ + fluid validate products/orders/contract.fluid.yaml +``` + +From Python, pass `ref_root=` to `load_contract`, `load_with_overlay` or +`compile_contract` in `fluid_build.loader`. + +- A `FLUID_REF_ROOT` that does not contain the contract is ignored for that + contract, with a `ref_root_env_ignored` warning. +- Do not `export` it: every contract under it is widened without a warning. +- URL and absolute-path refs cannot be allowed by any setting. Copy the + fragment under the ref root and refer to it by a relative path. + +`RefConfinementError` subclasses `RefResolutionError`, so existing +`except RefResolutionError` handlers still catch it. + +Full reference: [Composing a contract with `$ref`](./concepts/contract-refs.md). + +### SQL or declarations that read outside the contract directory + +SQL in a contract can read and write only: + +- the contract's directory and its FLUID workspace; +- `./runtime` and the run's scratch directory; +- the locations the contract declares, and only those inside the same roots; +- any directory the operator lists in `FLUID_DUCKDB_ALLOWED_DIRS`. + +A declared input, output or acquisition source outside those roots is refused +before any SQL runs: + +```console +$ fluid apply contract.fluid.yaml --mode amend-and-build --yes +πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + ❌ Failed: 1 action(s) failed + The contract declares '/work/reference/rates.csv' (/work/reference/rates.csv), + outside the directories it may read and write (/work/orders, + /work/orders/runtime, $TMPDIR/fluid_jl4bqbtc). The operator can allow a directory + with FLUID_DUCKDB_ALLOWED_DIRS. +``` + +**Migrate** either by moving the data under the contract's directory or +workspace, or by allowing its directory in the environment that runs `fluid` +(absolute paths, `:`-separated), and declaring the file in the contract: + +```bash +FLUID_DUCKDB_ALLOWED_DIRS=/work/reference \ + fluid apply contract.fluid.yaml --mode amend-and-build --yes +``` + +SQL that reads a path the contract does not declare (`read_csv('/etc/passwd')`, +`../`, an undeclared URL) is refused by DuckDB itself. Declare the location as +an input. + +A declared glob grants the directory above its first wildcard, because DuckDB +expands the glob again when the SQL runs; that directory must itself be +inside the allowed roots. + +Full reference: [DuckDB sandbox for contract SQL](./advanced/duckdb-sandbox.md). + +### Functions DuckDB used to autoload + +`sqlite_scan`, `read_xlsx`, `ST_Read`, `delta_scan` and `iceberg_scan` no +longer work in contract SQL. Extension autoloading is off, and contract SQL +cannot `INSTALL` or `LOAD` an extension. **Migrate** by reading CSV, Parquet +or JSON instead, or by landing the data with an acquisition build first. + +### Upgrade DuckDB + +The `local` extra now requires `duckdb>=1.5.0`, and the engine refuses to open +DuckDB on anything older (`DuckDB 1.4.4 is too old to sandbox contract SQL`). +Older versions let `/./../` and symlinks escape the allowlist. + +```bash +pip install -U 'data-product-forge[local]' +``` + +### A DuckDB database file can be open once per process + +A file DuckDB database already open in the same process can no longer be +opened a second time. It raises a clear `DuckDBSandboxError` instead of +sharing the instance. Two MCP DuckDB drivers bound to the same `.duckdb` file +in one process, or two concurrent `persist=True` local runs, hit this. + +Persistent DuckDB secrets (`~/.duckdb/stored_secrets`) are no longer loaded; +object-store builds use the credential-chain secret the engine creates. + +## Security + +- `$ref` resolution refuses remote, absolute and escaping targets, before it + tests whether the target exists (#687). +- `fluid validate` on a bundle no longer hands an OpenAPI fragment with an + external `$ref` to openapi-spec-validator, which followed `file://` refs. It + reports `OAS-REF-EXTERNAL` instead (#687). +- Every DuckDB connection in the engine goes through one helper, + `secure_duckdb_connect` (#689). It applies DuckDB's built-in sandbox: + `allowed_directories` / `allowed_paths`, `enable_external_access = false`, + autoload and autoinstall off, no persistent secrets, no community + extensions, and `lock_configuration = true`. Contract SQL can no longer + read host files, URLs or other databases, or change those settings. A guard + test fails if a new `duckdb.connect` bypasses the helper. + +## Added + +- `fluid_build.api.load_contract`, `load_contract_from_text` and + `load_contract_from_dict` (#688). They return a `LoadedContract` whose + `contract` is the dict `fluid plan` plans, after parsing, `$ref` + composition, the env overlay and the engine's alias and legacy-`build:` + rewrites, plus its plan-digest canonicalisation (`.digest`). Failures raise + a typed `ContractLoadError`. The `fluid_build.api` version is now `1.1`. + See [Contract loading API](./advanced/contract-loading-api.md). + +## Fixed + +- One failed SQL action no longer makes every later action of the same local + apply fail with "configuration has been locked" (#689). + +## See also + +- [Composing a contract with `$ref`](./concepts/contract-refs.md) +- [DuckDB sandbox for contract SQL](./advanced/duckdb-sandbox.md) +- [Contract loading API](./advanced/contract-loading-api.md) +- [API Stability](./advanced/api-stability.md) diff --git a/docs/advanced/contract-loading-api.md b/docs/advanced/contract-loading-api.md new file mode 100644 index 0000000..e4a6406 --- /dev/null +++ b/docs/advanced/contract-loading-api.md @@ -0,0 +1,305 @@ +--- +title: Contract loading API +description: Load a contract from Python exactly as fluid plan sees it, with fluid_build.api.load_contract, load_contract_from_text and load_contract_from_dict. +--- + +# Contract loading API β€” `fluid_build.api.load_contract` + +`fluid_build.api.load_contract` returns the contract that `fluid plan` plans: +the same dict `plan.json` embeds as `contract`, after every rewrite the engine +makes on the way in. Use it when your code has to agree with the engine about +what a contract *is*: comparing a contract to the plan made from it, diffing +two revisions, rendering a contract in a UI, or checking it in CI. + +Added in CLI 0.18.0, `fluid_build.api` version **1.1** +([forge-cli #688](https://github.com/Agenticstiger/forge-cli/pull/688)). It is +part of the [stable public API](./api-stability.md). + +```python +import fluid_build.api +print(fluid_build.api.__api_version__) +# 1.1 +``` + +## Load a contract file + +The contract below is the one from +[Composing a contract with `$ref`](../concepts/contract-refs.md): an owner, a +build and a policy, each composed from a fragment. + +```python +from fluid_build.api import load_contract + +loaded = load_contract("orders/contract.fluid.yaml") + +loaded.origin # 'file' +loaded.digest # 'sha256:ffb903222a5ab79af24df49dcf2729924b7756dc11e0ab221775df93df6dd391' +loaded.files # contract.fluid.yaml, owner.yaml, parts/build.yaml, parts/policy.yaml +loaded.unresolved_refs # () +loaded.contract["exposes"][0]["policy"] +# {'classification': 'Internal', 'authz': {'readers': ['group:sales-analysts']}} +``` + +### With an environment overlay + +```python +loaded = load_contract("contract.fluid.yaml", env="prod") + +loaded.overlay # PosixPath('/work/orders/overlays/prod.yaml') +loaded.contract["exposes"][0]["binding"]["location"] +# {'path': 'out/prod/summary.csv'} +``` + +`env` is an environment *name*, never a path. The engine turns `env` into +overlay paths (`overlays/.yaml`, `.json`, …), so an env holding `..` +or an absolute path would merge whichever file it named into the contract. An +env that is not a single path component is refused before a file is read: + +```python +load_contract("contract.fluid.yaml", env="../../etc/x") +# ContractLoadError: contract_env_invalid +``` + +Refused: `""` (not read as `None`), `.`, `..`, anything holding `/`, `\` or a +NUL byte, and a drive-qualified name (`C:prod`, on every platform). Every other +string loads exactly as `fluid plan --env` loads it. Pass `None` for no env. + +A `fluid bundle` archive loads the same way, and is refused for an env it was +not built for, as on the CLI: + +```python +load_contract("runtime/bundle.tgz", env="prod") +``` + +## Is this the contract that plan was made from? + +```console +$ fluid plan contract.fluid.yaml --env prod --out runtime/plan.json +``` + +```python +import json +from pathlib import Path + +from fluid_build.api import load_contract +from fluid_build.forge.core.plan_digest import compute_contract_digest + +plan = json.loads(Path("runtime/plan.json").read_text()) +loaded = load_contract("contract.fluid.yaml", env="prod") + +loaded.digest == compute_contract_digest(plan["contract"]) +# True +``` + +Formatting, comments, key order, quoting, Unicode normal form, an alias beside +its canonical value (`format: bigquery-table` / `bigquery_table`) and a legacy +`build:` beside `builds:` do not change the digest. Every other value does. + +**Compare digests, not dicts.** `loaded.contract` keeps a YAML magic-word or +numeric key (`on:`, `no:`, `1:` in an open block such as `extensions`) as the +Python `bool` / `int` the engine plans with, while `plan.json` writes every key +as a string. The digest coerces keys the same way `plan.json` does. + +::: warning `digest` is not `fluid contract digest` +`loaded.digest` is the digest of the **planned** contract: normalised, +composed and overlaid. It is not the value `fluid contract digest` prints, and +not what a federation `upstreamDigest` pins: those hash the file as parsed, +before any alias rewrite, `$ref` or overlay. Do not pin `upstreamDigest` from +`loaded.digest`. +::: + +## Contract text or a parsed dict, without touching the disk + +```python +from fluid_build.api import load_contract_from_dict, load_contract_from_text + +loaded = load_contract_from_text(text) # YAML by default +loaded = load_contract_from_text(text, suffix=".json") # or JSON +loaded = load_contract_from_dict(document) # already parsed +``` + +Neither reads a file. A `$ref` is left in place and listed in +`unresolved_refs`; you decide whether that is an error. For the contract +above, sent as text from a form or a database row: + +```python +submitted = load_contract_from_text(text) +submitted.origin # 'memory' +submitted.unresolved_refs # ('./owner.yaml', 'parts/build.yaml', 'parts/policy.yaml#/gold') +``` + +### Text with its fragments and an overlay + +Pass `base_dir` to resolve file refs against a directory, under the same +[ref root rules](../concepts/contract-refs.md#the-ref-root) as a file load: + +```python +loaded = load_contract_from_text(text, base_dir="orders") +loaded.unresolved_refs # () +loaded.digest == load_contract("orders/contract.fluid.yaml").digest +# True +``` + +```python +loaded = load_contract_from_text( + text, + base_dir="orders", + overlay={"exposes": [{"binding": {"location": {"path": "out/prod/summary.csv"}}}]}, +) +``` + +With `base_dir` set to a contract's directory and `overlay` set to the parsed +overlay file `--env` would select, the result equals +`load_contract(that_file, env=...)`. Without `base_dir`, passing `overlay` for +a document that holds file `$ref` values raises +`contract_overlay_needs_base_dir`, because whether the engine would apply the +overlay depends on what the fragments hold. + +## What "as plan sees it" means + +In the engine's order: + +1. **Parse**: JSON, or YAML through the engine's billion-laughs guard. +2. **`$ref` composition**: each `{"$ref": "./file.yaml#/pointer"}` replaced by + its target, resolved against the contract's directory (or `base_dir`) under + the [ref root](../concepts/contract-refs.md#the-ref-root). Same-document + `#/...` pointers are kept, and listed in `unresolved_refs`. +3. **Overlay**: for `env`, the first of `overlays/.yaml|yml|json`, + `.yaml|yml|json`, `..yaml|yml|json` next to the + contract, deep-merged over the base (objects key by key, lists of objects + by position, anything else replaced). A bundle is never re-overlaid. +4. **Alias values**: human-friendly values rewritten to the schema's enum + value, for example `source.kind: pg` β†’ `postgres`, + `source.mode: incremental` β†’ `incremental_append`, + `binding.format: kafka` β†’ `kafka_topic`, `iceberg-table` β†’ `iceberg`. +5. **Legacy `build:`**: a singular `build:` becomes `builds: [build]`; when + both are present `builds:` wins. + +Step 4 runs before step 5, as in the engine, so an alias under a legacy +singular `build:` is **not** rewritten (and `fluid plan` then rejects it at the +schema gate). Write `builds:` to get alias rewriting for builds. + +**Validation is not part of loading.** A schema-invalid contract loads; +`fluid validate` and `fluid plan` reject it. + +### Edge case: an overlay next to a `$ref` + +When the overlay file, or the merged contract, still holds a `$ref` (an +overlay that references a fragment, or a same-document `#/...` pointer in the +contract), the engine reloads the contract from the base file and the overlay +is dropped: `fluid plan --env prod` plans the base contract. `load_contract` +returns the base too, and says so: `overlay` is `None`, the overlay is not in +`files`, and a `contract_overlay_not_applied` WARNING names the file. Keep +`$ref` out of overlays, and use file refs rather than `#/...` pointers in a +contract that has overlays. + +### `FLUID_REF_ROOT` + +These functions have no `ref_root` argument. They go through the engine's +loader, so `FLUID_REF_ROOT` in the process environment applies to them as it +does to the CLI. To widen the root for one call, use +`fluid_build.loader.load_contract(path, ref_root=...)`; see +[Widening the root for a monorepo](../concepts/contract-refs.md#widening-the-root-for-a-monorepo). + +## Reference + +### `load_contract(path, *, env=None, logger=None) -> LoadedContract` + +Loads a contract file or a bundle through `fluid plan`'s own loader. `path` is +resolved to an absolute path first, as `fluid plan` does. The CLI's gate on +operator-typed paths (no `..`, no symlink) is not applied to `path`: a library +caller chooses its own paths. `env` must be a single path component or `None`. + +### `load_contract_from_text(text, *, suffix=".yaml", base_dir=None, overlay=None, logger=None) -> LoadedContract` + +Parses `text` with the engine's parser (`suffix` picks it as a file extension +would), then loads the result as `load_contract_from_dict` does. + +### `load_contract_from_dict(document, *, base_dir=None, overlay=None, logger=None) -> LoadedContract` + +Loads a parsed document. `document` and `overlay` are never modified. `logger` +receives the `contract_overlay_not_applied` WARNING (default: the +`fluid.api.contract` logger). + +### `LoadedContract` + +A frozen dataclass. + +| Field | Type | Meaning | +|---|---|---| +| `contract` | `dict` | The contract as planned, keys as the engine holds them. A fresh dict each call; yours to mutate. | +| `origin` | `"file"` \| `"bundle"` \| `"memory"` | Which entry point and input shape produced it. | +| `source` | `Path \| None` | The resolved contract or bundle path; `None` for the in-memory forms. | +| `env` | `str \| None` | The env requested. | +| `overlay` | `Path \| None` | The overlay file merged for `env`. Never set for a bundle, or when the engine dropped the overlay. | +| `files` | `tuple[Path, ...]` | Every file composed: the source, each `$ref` target in first-read order, the overlay. Empty for the in-memory forms without `base_dir`. | +| `unresolved_refs` | `tuple[str, ...]` | `$ref` values left in `contract`, in document order. | +| `digest` | `str` (property) | `sha256:` of the planned contract via `compute_contract_digest`. Raises `contract_not_serialisable` when JSON cannot represent the contract. | + +### `ContractLoadError` + +Every failure raises `ContractLoadError` with a stable `event` (safe to route +on), `message`, `path` when there is one, and the engine's exception as +`__cause__`: + +```python +from fluid_build.api import ContractLoadError, load_contract + +try: + load_contract("orders/escape.fluid.yaml") +except ContractLoadError as err: + print(err.event) + # contract_ref_unresolved + print(err) + # $ref '../shared/policy.yaml#/gold' at JSON pointer '/exposes/0/policy' in + # /work/orders/escape.fluid.yaml escapes the ref root /work/orders (after resolving + # '..' and symlinks); refs may only name files inside it. ... +``` + +| `event` | When | +|---|---| +| `contract_not_found` | The contract file does not exist, or `path` / `base_dir` cannot name a file (it holds a NUL byte). | +| `contract_parse_failed` | The text is not valid JSON/YAML, is not UTF-8, or trips the YAML size/anchor guard. | +| `contract_not_a_mapping` | The document (or overlay) root is not an object. | +| `contract_ref_unresolved` | A `$ref` target is missing, cyclic, outside the ref root, or its pointer does not resolve. | +| `contract_env_invalid` | `env` is not a single path component. | +| `contract_overlay_needs_base_dir` | In-memory form: `overlay` given for a document with file `$ref` values but no `base_dir`. | +| `contract_not_serialisable` | Raised by `.digest`: the contract holds a value JSON cannot represent (an unquoted YAML date, a set, binary, a self-referencing alias). `fluid plan` cannot write it either; quote the value. | +| `contract_load_failed` | Any other loader failure. | +| *engine event* | Passed through unchanged, for example `overlay_declared_but_missing`, `bundle_not_found`, `bundle_env_mismatch`, `bundle_manifest_invalid`. | + +## Stability + +`fluid_build.api` follows the SemVer policy in +[API Stability](./api-stability.md); these names were added in 1.1 and are +locked by the API surface snapshot test. The behavioural promise: +`load_contract(path, env=env).contract` is the contract +`fluid plan path --env env` plans, equal to `plan.json["contract"]` once keys +are written as strings, and always equal by `.digest`. + +- The file form calls the engine's loader itself, so a new loader step reaches + it with no change. A test runs the real `fluid plan` on fixtures that + exercise every rewrite and fails if the two differ. +- The in-memory forms have no file to hand the loader, so they replay a fixed + sequence of the loader's steps. Tests pin them to the file form, and a guard + test fails when the engine loader gains, loses or reorders a step that + touches the contract. + +::: tip Not the private helpers +Do not import helpers from `fluid_build._contract_loader` (for example +`_normalize_contract_aliases`) to reproduce this. They are private, their +order matters, and they are not the whole pipeline. +::: + +## Related + +- [Composing a contract with `$ref`](../concepts/contract-refs.md) +- [API Stability](./api-stability.md) +- [DuckDB sandbox](./duckdb-sandbox.md) +- [Upgrading to 0.18.0](../RELEASE_NOTES_0.18.0.md) + +::: tip About the examples +Output on this page was produced by running the snippets against the 0.18.0 +code (forge-cli #687, #688 and #689 together). Long temporary paths are +shortened to `/work`. +::: diff --git a/docs/advanced/duckdb-sandbox.md b/docs/advanced/duckdb-sandbox.md new file mode 100644 index 0000000..29c4997 --- /dev/null +++ b/docs/advanced/duckdb-sandbox.md @@ -0,0 +1,338 @@ +--- +title: DuckDB sandbox for contract SQL +description: What SQL inside a contract can read and write on the local DuckDB engine, how to grant more, and the limits of the sandbox. +--- + +# DuckDB sandbox for contract SQL + +From CLI 0.18.0, every DuckDB connection the engine opens runs inside +DuckDB's own sandbox. SQL in a contract can read and write the contract's +directory and the locations the contract declares. It cannot read the rest of +the host, fetch a URL the contract does not declare, attach another database, +install an extension, or change those settings. + +::: warning Breaking in 0.18.0 +- A declared input, output or acquisition source **outside the allowed + directories is refused before any SQL runs**, including absolute landing + paths such as `/data/landing/*.csv`. The operator allows a directory with + [`FLUID_DUCKDB_ALLOWED_DIRS`](#reading-a-file-from-another-directory). +- **Functions DuckDB used to autoload no longer work in contract SQL**: + `sqlite_scan`, `read_xlsx`, `ST_Read`, `delta_scan`, `iceberg_scan`. +- The `local` extra requires **`duckdb>=1.5.0`**, and the engine refuses to + open DuckDB on anything older. + +Migration steps are in the [0.18.0 upgrade notes](../RELEASE_NOTES_0.18.0.md). +::: + +## Example + +The build below reads the contract's own CSV. It works as before: + +```yaml +# orders/parts/build.yaml +id: summarise +pattern: embedded-logic +engine: sql +properties: + sql: SELECT id, amount * 2 AS doubled FROM read_csv('data/orders.csv') +``` + +```console +$ fluid apply contract.fluid.yaml --mode amend-and-build --yes +πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + βœ… Completed in 0.02s β€” 1 action(s) executed + πŸ“ /work/orders/out/summary.csv +``` + +The same build pointed at a host file is refused by DuckDB: + +```yaml + sql: SELECT * FROM read_csv('/etc/passwd') +``` + +```console +$ fluid apply contract.fluid.yaml --mode amend-and-build --yes +πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + ❌ Failed: 1 action(s) failed + Permission Error: Cannot access file "/etc/passwd" - file system operations are + disabled by configuration DuckDB refused it: contract SQL may only read and write + the locations the contract declares and its own directory (/work/orders, + /work/orders/runtime, $TMPDIR/fluid_gxyqoqe4, /work/orders/out/summary.csv). + Declare the file as an input under the contract's directory or workspace, or move + it there (the operator can allow another directory with FLUID_DUCKDB_ALLOWED_DIRS). +``` + +The list in parentheses is exactly what that build's SQL may touch: the +contract's directory, `./runtime`, the run's scratch directory, and the +declared output. + +## What SQL is refused + +The same message (`DuckDB refused it: …`) follows each of these: + +| SQL in the contract | DuckDB's error | +|---|---| +| `read_csv('/etc/passwd')`, `read_text(...)`, `read_blob(...)`, `read_parquet(...)`, `read_json(...)`, `glob('/etc/*')` on a path outside the list | `Permission Error: Cannot access file …` | +| `read_csv('../shared/x.csv')`, `/./../`, a symlink that points outside | `Permission Error: Cannot access file …` | +| `read_csv('https://example.com/x.csv')`, or any URL the contract does not declare | `File https://… requires the extension httpfs to be loaded` | +| `INSTALL httpfs`, `LOAD …` | `Permission Error: Cannot access directory "~/.duckdb/extensions/…"` | +| `SET enable_external_access = true`, or any other `SET` | `Cannot change configuration option … - the configuration has been locked` | +| `read_xlsx(...)`, `sqlite_scan(...)`, `ST_Read(...)`, `delta_scan(...)`, `iceberg_scan(...)` | `Catalog Error: Table Function with name "read_xlsx" is not in the catalog` | + +`ATTACH` of another database file and `COPY … TO` / `COPY … FROM` a location +outside the list are refused the same way. `~/…` is refused unless the +matching file under `$HOME` is itself in the list. + +### Functions that used to autoload + +DuckDB used to install and load an extension the first time a query called +one of its functions. With autoloading off, functions from extensions the +engine does not load for that build are not in the catalog, even for a file +inside the contract's directory: + +```console + ❌ Failed: 1 action(s) failed + Catalog Error: Table Function with name "read_xlsx" is not in the catalog, but it + exists in the excel extension. ... DuckDB refused it: contract SQL runs with + extension autoloading off and cannot INSTALL or LOAD one, so functions from + extensions the engine does not load (sqlite_scan, read_xlsx, ST_Read, delta_scan, + iceberg_scan) are not available. Read the data as CSV, Parquet or JSON, or land it + with an acquisition build first. +``` + +DuckDB's own hint in that message (`INSTALL excel; LOAD excel;` or +`SET autoload_known_extensions=1`) does not apply: contract SQL cannot run +either. Convert the file to CSV, Parquet or JSON, or land the data with an +acquisition build first. + +## What each kind of SQL can reach + +| Where the SQL runs | It can read and write | +|---|---| +| Embedded-SQL build on the local DuckDB engine (`builds[].properties.sql`) | the contract's directory, the FLUID workspace it sits in (`fluid.workspace.yaml`), `./runtime`, the run's scratch directory, each declared `parameters.inputs[].path`, each resolved `consumes[]` upstream, the expose's landing path, and the `s3://` prefixes those name. A declared local path counts only [inside the allowed directories](#declared-locations-stay-inside-the-allowed-directories). | +| DuckDB acquisition build (`pattern: acquisition`, `engine: duckdb`) | the contract's directory, the declared `source.connection.uri` (or stream paths), and each stream's landing file, each inside the allowed directories. A `mysql` source is attached before the sandbox closes; so is a `sqlite` source, and its file must also be inside the allowed directories. | +| `fluid validate` quality rules, `fluid verify`, `fluid diff` | the one file being checked | +| `fluid contract-tests` local actions | each declared input file and each output file | +| Discovery (`fluid forge data-model from-source`, `discover`) | the one file or URL being introspected; a JDBC source is attached first | +| MCP output port (DuckDB driver) | the bound file | + +## Declared locations stay inside the allowed directories + +Each declared input and output is granted to the build's SQL, and whoever +writes the contract writes the declarations. So a declaration grants a local +path only inside these directories: + +- the contract's directory and the FLUID workspace it sits in; +- `./runtime` and the run's scratch directory; +- the upstream roots in `FLUID_UPSTREAM_CONTRACTS`; +- the directories the operator lists in `FLUID_DUCKDB_ALLOWED_DIRS`. + +Anything else is refused before any SQL runs. Declaring an innocuous glob in +`$HOME` does not make `~/.aws/credentials` readable: + +```yaml +properties: + sql: SELECT content FROM read_text('~/.aws/credentials') + parameters: + inputs: + - name: d + path: /home/me/*.csv +``` + +```console +$ fluid apply contract.fluid.yaml --mode amend-and-build --yes +πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + ❌ Failed: 1 action(s) failed + The contract declares '/home/me/*.csv' (/home/me), outside the directories it may + read and write (/work/orders, /work/orders/runtime, $TMPDIR/fluid_tb4ky08e). The + operator can allow a directory with FLUID_DUCKDB_ALLOWED_DIRS. +``` + +Both sides are compared after resolving symlinks, as DuckDB resolves them: a +symlink inside the contract's directory that points at `/` grants nothing. A +relative declared path is resolved where DuckDB opens it, the working +directory, and is confined the same way. + +What a declaration grants, once allowed: + +| Declared | Granted | +|---|---| +| a file (`/shared/reference/rates.csv`) | that file | +| a glob (`data/*.csv`, `landing/**/*.parquet`) | the directory above its first wildcard (`data/`, `landing/`), which must itself be inside the allowed directories | +| a directory (`data/`) | everything under it | +| an `s3://` URL | its prefix (see [the limits](#limits)) | + +A glob grants its directory, not the files it matches when the run starts, +because DuckDB expands the glob again when the SQL runs. DuckDB checks each +expanded file after resolving symlinks, so a matched symlink that points +outside the directory is refused. + +A `[`, `?` or `*` in the name of a directory that exists (for example a +project checked out under `Proj [old]/`) is part of that name, not a wildcard. + +### `./runtime` that is a symlink + +`./runtime` is granted by convention, not by a declaration. It is granted only +when it is a real directory, or a symlink into the contract's directory or +workspace. A repository that ships `runtime` as a symlink out of itself does +not get that directory granted: the symlink is ignored with a +`local_runtime_not_granted` warning, and SQL that reads or writes under it is +refused. + +### SQLite sources + +A SQLite acquisition source (`source.kind: sqlite`) is confined the same way, +and refused with the same `FLUID_DUCKDB_ALLOWED_DIRS` message outside the +allowed directories. This check is the only one on it: the sqlite scanner +opens the file through its own library, which DuckDB's allowlist does not +bound. + +## Reading a file from another directory + +The operator allows the directory; the contract then declares the file. In +the build below the declared input is available to the SQL as the view +`rates`: + +```yaml +id: summarise +pattern: embedded-logic +engine: sql +properties: + sql: >- + SELECT o.id, o.amount * r.rate AS doubled + FROM read_csv('data/orders.csv') o JOIN rates r USING (id) + parameters: + inputs: + - name: rates + path: /work/reference/rates.csv +``` + +Without the variable, the declaration is refused before any SQL runs: + +```console +$ fluid apply contract.fluid.yaml --mode amend-and-build --yes +πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + ❌ Failed: 1 action(s) failed + The contract declares '/work/reference/rates.csv' (/work/reference/rates.csv), + outside the directories it may read and write (/work/orders, + /work/orders/runtime, $TMPDIR/fluid_jl4bqbtc). The operator can allow a directory + with FLUID_DUCKDB_ALLOWED_DIRS. +``` + +With it, the build runs: + +```console +$ FLUID_DUCKDB_ALLOWED_DIRS=/work/reference \ + fluid apply contract.fluid.yaml --mode amend-and-build --yes +πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + βœ… Completed in 0.02s β€” 1 action(s) executed + πŸ“ /work/orders/out/summary.csv +``` + +The declared file becomes readable. Its neighbours in `/work/reference` do +not, unless the contract declares them too (or declares a glob or the +directory). Reading an undeclared neighbour is refused, and the list in the +message now includes the declared file: + +```console + Permission Error: Cannot access file "/work/reference/neighbour.csv" - file system + operations are disabled by configuration DuckDB refused it: contract SQL may only + read and write the locations the contract declares and its own directory + (/work/orders, /work/orders/runtime, $TMPDIR/fluid_ft7ya_8s, + /work/reference/rates.csv, /work/orders/out/summary.csv). ... +``` + +### `FLUID_DUCKDB_ALLOWED_DIRS` + +- Absolute directories, separated by `:` (`;` on Windows). `~` is expanded. +- Read from the environment of the process that runs the engine. No contract + field can set it. +- A relative entry, or `/`, fails the run: + + ```console + $ FLUID_DUCKDB_ALLOWED_DIRS=relative/dir fluid apply contract.fluid.yaml --mode amend-and-build --yes + ❌ Failed: 1 action(s) failed + FLUID_DUCKDB_ALLOWED_DIRS entry 'relative/dir' is not an absolute path + + $ FLUID_DUCKDB_ALLOWED_DIRS=/ fluid apply contract.fluid.yaml --mode amend-and-build --yes + ❌ Failed: 1 action(s) failed + FLUID_DUCKDB_ALLOWED_DIRS entry '/' is the filesystem root + ``` + +- It lets a contract *declare* a location in that directory. It does not make + the directory readable to SQL that has not declared it. + +A product inside a FLUID workspace can also read its sibling products' files +by path, because the workspace is an allowed directory, and a `consumes[]` +entry resolves to the upstream's landed file. + +## How it works + +`fluid_build/providers/_duckdb_sandbox.py` (`secure_duckdb_connect`) is the one +place the engine calls `duckdb.connect`; a test fails if any other module +opens DuckDB without it. For each connection it: + +1. connects (a file-backed database is opened here and needs no grant); +2. turns off persistent secrets and community extensions, loads the + extensions the call site needs, and runs its set-up (an `ATTACH` of a + declared source, an object-store secret); +3. turns off extension autoinstall and autoload; +4. pins `home_directory` to `$HOME`, then sets `allowed_directories` and + `allowed_paths`; +5. sets `enable_external_access = false`; +6. sets `lock_configuration = true`. + +These are DuckDB's own settings, in the order the +[Securing DuckDB](https://duckdb.org/docs/current/operations_manual/securing_duckdb/overview.html) +guide gives. + +## Limits + +- **DuckDB 1.5.0 or newer.** Up to 1.4.3, `/./../` escaped + `allowed_directories`, and through 1.4.x a symlink inside an allowed + directory reached its target. On an older DuckDB every build fails: + + ```console + πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + ❌ Failed: 1 action(s) failed + DuckDB 1.4.4 is too old to sandbox contract SQL + ``` + + Upgrade with `pip install 'duckdb>=1.5.0'`, or reinstall + `data-product-forge[local]`. +- **A remote prefix bounds the bucket, not the path.** DuckDB 1.5 does not + resolve `..` inside a URL, so a declared `s3://bucket/landing/` also lets the + SQL reach other keys in that bucket with the same credentials. +- **A loaded database scanner is not bounded by the allowlist.** The `sqlite`, + `postgres` and `mysql` extensions open files and sockets through their own + client libraries. The engine loads them only for an acquisition build's + declared source, discovery, and the copilot's sample-rows tool, whose SQL the + engine builds from validated identifiers. Contract SQL never runs on such a + connection. +- **One open connection per database file.** DuckDB shares one instance per + database file within a process, and the sandbox locks that instance. A second + connection to a file that is already open raises `DuckDBSandboxError` + (`DuckDB database … is already open in this process`). Two MCP DuckDB + drivers bound to the same `.duckdb` file in one process, or two concurrent + `persist=True` local runs, hit this. +- **Persistent DuckDB secrets are off.** Secrets saved in + `~/.duckdb/stored_secrets` are no longer loaded; object-store builds use the + credential-chain secret the engine creates. +- **Defense in depth, not isolation.** DuckDB describes these settings as not a + substitute for proper sandboxing. A service that runs other people's + contracts (the Command Center, a shared CI runner) should still run each one + in its own container. + +## Related + +- [Composing a contract with `$ref`](../concepts/contract-refs.md): the matching confinement for contract fragments +- [Local provider](../providers/local.md) +- [Upgrading to 0.18.0](../RELEASE_NOTES_0.18.0.md) + +::: tip About the examples +Output on this page was produced with the 0.18.0 code (forge-cli +[#689](https://github.com/Agenticstiger/forge-cli/pull/689)) and DuckDB 1.5.6. +Only the build block of each run is shown. Long temporary paths are shortened +to `/work` and `$TMPDIR`, `$HOME` is shown as `/home/me`, and long lines are +re-wrapped. +::: diff --git a/docs/concepts/contract-refs.md b/docs/concepts/contract-refs.md new file mode 100644 index 0000000..cea5c33 --- /dev/null +++ b/docs/concepts/contract-refs.md @@ -0,0 +1,338 @@ +--- +title: Composing a contract with $ref +description: Split a contract into fragments with $ref, and the ref root that confines where a fragment may come from. +--- + +# Composing a contract with `$ref` + +A contract can pull any object from another YAML or JSON file with `$ref`. +`fluid validate`, `fluid plan`, `fluid apply` and `fluid bundle` resolve every +ref before they do anything else, so the rest of the pipeline sees one +document. + +::: warning Changed in CLI 0.18.0 +A `$ref` may only name a file inside the contract's own directory tree (the +**ref root**). URL refs (`file://` included), absolute paths, and `..` or +symlink escapes are refused. Monorepos that share fragments across products +widen the root with `FLUID_REF_ROOT`. See +[Upgrading monorepos that share fragments](#upgrading-monorepos-that-share-fragments) +and the [0.18.0 upgrade notes](../RELEASE_NOTES_0.18.0.md). +::: + +## Example + +```text +orders/ +β”œβ”€β”€ contract.fluid.yaml +β”œβ”€β”€ owner.yaml +β”œβ”€β”€ data/orders.csv +└── parts/ + β”œβ”€β”€ build.yaml + └── policy.yaml +``` + +```yaml +# orders/contract.fluid.yaml +fluidVersion: 0.7.3 +kind: DataProduct +id: sales.orders_v1 +name: Orders +domain: sales +metadata: + layer: Silver + owner: + $ref: ./owner.yaml # the whole file +builds: + - $ref: parts/build.yaml # refs work inside lists +exposes: + - exposeId: orders_summary + kind: table + binding: + platform: local + format: csv + location: + path: out/summary.csv + policy: + $ref: parts/policy.yaml#/gold # file + JSON pointer + contract: + schema: + - name: id + type: INTEGER + - name: doubled + type: INTEGER +``` + +```yaml +# orders/parts/policy.yaml +gold: + classification: Internal + authz: + readers: + - group:sales-analysts +``` + +```console +$ fluid validate contract.fluid.yaml +βœ… Valid FLUID contract (schema v0.7.3) +Validation completed in 0.001s +``` + +[`fluid bundle`](../cli/bundle.md) writes the single resolved document. The +ref nodes are replaced by their targets: + +```console +$ fluid bundle contract.fluid.yaml +fluidVersion: 0.7.3 +kind: DataProduct +id: sales.orders_v1 +name: Orders +domain: sales +metadata: + layer: Silver + owner: + team: sales-analytics + email: sales-analytics@example.com +builds: +- id: summarise + pattern: embedded-logic + engine: sql + properties: + sql: SELECT id, amount * 2 AS doubled FROM read_csv('data/orders.csv') +exposes: +- exposeId: orders_summary + ... + policy: + classification: Internal + authz: + readers: + - group:sales-analysts + ... +``` + +## How a ref is read + +- A `$ref` node is an object whose only key is `$ref`. +- A ref is resolved relative to the file that contains it, so + `parts/build.yaml` may itself say `$ref: ./policy.yaml`. +- `file.yaml#/a/b` selects the object at [JSON pointer](https://www.rfc-editor.org/rfc/rfc6901) + `/a/b` inside the file. Without a `#`, the whole file is used. +- Same-document refs (`$ref: "#/definitions/x"`) are left in place as written. + +## The ref root + +Every ref must name a file inside the **ref root**. By default the ref root is +the directory that holds the root contract file (`orders/` above). It applies +to every ref, including refs inside fragments: a fragment in `orders/parts/` +can reach anything under `orders/` and nothing outside it. + +The check runs after `..` segments and symlinks are resolved, so neither can +be used to get out. It also runs before the target is opened, so the error is +the same whether or not the target exists. + +| `$ref` written in `orders/contract.fluid.yaml` | Result | +|---|---| +| `./owner.yaml`, `parts/policy.yaml#/gold` | resolved | +| `../shared/policy.yaml` | refused: escapes the ref root | +| `./link.yaml`, where `link.yaml` is a symlink to a file outside `orders/` | refused: escapes the ref root | +| `/etc/hosts`, `C:\x.yaml`, `\\server\share\x.yaml` | refused: must be a relative path | +| `file:///etc/hosts`, `https://…`, `s3://…`, `//host/…` | refused: remote refs are not supported | + +### What a refused ref looks like + +`fluid validate` reports it as `contract_load_failed` and exits `1`: + +```console +$ fluid validate contract.fluid.yaml +❌ Validation error: contract_load_failed + error: $ref '../shared/policy.yaml#/gold' at JSON pointer '/exposes/0/policy' in + /work/orders/contract.fluid.yaml escapes the ref root /work/orders (after resolving + '..' and symlinks); refs may only name files inside it. To compose fragments from a + wider tree (e.g. a monorepo's shared/ directory), set FLUID_REF_ROOT to that + directory or pass ref_root= to the loader; see docs/contract-refs.md. + [ERR_CONTRACT_LOAD_FAILED] +``` + +`fluid bundle` prints the same message and exits `2`. The other refusals name +their reason: + +```console +$ fluid bundle contract.fluid.yaml # with $ref: /etc/hosts +❌ $ref resolution error: $ref '/etc/hosts' at JSON pointer '/exposes/0/policy' in +/work/orders/contract.fluid.yaml must be a relative path, got absolute β€” use a path +relative to the file that contains the ref + +$ fluid bundle contract.fluid.yaml # with $ref: file:///etc/hosts +❌ $ref resolution error: $ref 'file:///etc/hosts' at JSON pointer '/exposes/0/policy' +in /work/orders/contract.fluid.yaml is a URL; remote refs (including file://) are not +supported β€” use a relative path to a file inside the ref root +``` + +Every message names the ref, the JSON pointer of the `$ref` node, and the file +it was written in. + +::: tip Why the root exists +Contracts are often untrusted input. A platform such as the FLUID Command +Center runs `fluid validate` and `fluid bundle` on contracts its users upload. +Without a root, a contract could compose any YAML or JSON file the process can +read into itself, and `fluid bundle` would print it back. +::: + +## Widening the root for a monorepo + +To share fragments between products, set the ref root to a directory that +contains both the contracts and the shared fragments: + +```text +repo/ +β”œβ”€β”€ shared/policy.yaml +└── products/orders/contract.fluid.yaml # $ref: ../../shared/policy.yaml#/gold +``` + +```console +$ cd repo +$ FLUID_REF_ROOT=. fluid validate products/orders/contract.fluid.yaml +βœ… Valid FLUID contract (schema v0.7.3) +Validation completed in 0.001s +``` + +Without `FLUID_REF_ROOT`, the same command fails with `escapes the ref root`. + +From Python, pass `ref_root=` to the engine's loader. `load_contract`, +`load_with_overlay` and `compile_contract` in `fluid_build.loader` all accept +it, and it wins over `FLUID_REF_ROOT`: + +```python +from fluid_build.loader import load_contract + +contract = load_contract("repo/products/orders/contract.fluid.yaml", ref_root="repo") +print(contract["exposes"][0]["policy"]["classification"]) +# Internal +``` + +The [public contract-loading API](../advanced/contract-loading-api.md) +(`fluid_build.api.load_contract`) has no `ref_root` argument. It goes through +the same loader, so it honours `FLUID_REF_ROOT`. + +### Rules for the wider root + +- It widens the root only for a contract inside it, and only if it is an + existing directory. +- **`FLUID_REF_ROOT` is ignored for a contract outside it**, and when it is not + a directory or cannot be resolved. That contract gets the default root, its + own directory, and a `ref_root_env_ignored` warning says so. Refs that leave + the directory then fail, and the error says the variable was ignored and why: + + ```console + $ cd repo + $ FLUID_REF_ROOT=. fluid validate /work/upload-1234/contract.fluid.yaml + ref_root_env_ignored: contract /work/upload-1234/contract.fluid.yaml is outside + FLUID_REF_ROOT='.' (resolved to /work/repo); the ref root must contain the contract. + Ignoring it for this contract: its $refs are confined to the contract's own directory + /work/upload-1234 (the default). Logged once per value in this process; a later + contract it is ignored for gets no warning, but its escape errors say the variable + was ignored and why. + ❌ Validation error: contract_load_failed + error: $ref '../shared/policy.yaml#/gold' at JSON pointer '/exposes/0/policy' in + /work/upload-1234/contract.fluid.yaml escapes the ref root /work/upload-1234 (after + resolving '..' and symlinks); refs may only name files inside it. FLUID_REF_ROOT is + set but was ignored for this contract: ... + ``` + +- **`ref_root=` is strict.** The caller sets it for one contract, so a + `ref_root` that cannot be resolved, is not a directory, or does not contain + the contract raises `RefResolutionError`: + + ```python + load_contract("repo/products/orders/contract.fluid.yaml", ref_root="orders") + # RefResolutionError: contract repo/products/orders/contract.fluid.yaml is outside + # ref_root='orders' (resolved to /work/orders); the ref root must contain the contract + ``` + +- A blank `FLUID_REF_ROOT` counts as unset. +- It is only consulted when the contract has a ref to another file. +- It widens the root and nothing else. URLs and absolute paths are still + refused, and refs that resolve into system directories (`/etc`, `/proc`, + `/private/etc` on macOS, and so on) are still blocked. + +::: danger Scope it to one command +Every contract inside `FLUID_REF_ROOT` is widened, with no warning. A value +left exported in your shell widens every contract under it that you load +later, and a service whose upload directory sits under its `FLUID_REF_ROOT` +widens every uploaded contract. Set it per command, as in the examples, and to +the narrowest directory that works. Setting it to `/` turns the confinement +off. +::: + +## Upgrading monorepos that share fragments + +Before 0.18.0 a relative ref could climb out of the contract's directory, so +monorepos shared fragments with `$ref: ../other-product/…`. Those refs now +fail with `escapes the ref root` until the root is widened. Set +`FLUID_REF_ROOT` (or `ref_root=`) to the repository root, or to the narrowest +directory that holds the products and their shared fragments: + +```bash +FLUID_REF_ROOT="$(git rev-parse --show-toplevel)" \ + fluid validate products/orders/contract.fluid.yaml +``` + +In CI, set it on the step that runs `fluid`, not for the whole job. + +Contracts whose refs stay inside their own directory need no change. + +## Catching the error in Python + +`RefConfinementError` is a subclass of `RefResolutionError`, so existing +`except RefResolutionError` handlers keep working. It carries the details as +attributes: + +```python +from fluid_build.loader import RefConfinementError, load_contract + +try: + load_contract("repo/products/orders/contract.fluid.yaml") +except RefConfinementError as err: + print(err.ref) # ../../shared/policy.yaml#/gold + print(err.pointer) # /exposes/0/policy + print(err.source) # /work/repo/products/orders/contract.fluid.yaml + print(err.root) # /work/repo/products/orders + print(err.ignored_ref_root_env) # None +``` + +`ignored_ref_root_env` holds the `FLUID_REF_ROOT` value when it was set but did +not apply to this contract. + +The public API raises `ContractLoadError` with +`event == "contract_ref_unresolved"` for the same refusal; see +[Contract loading API](../advanced/contract-loading-api.md#contractloaderror). + +## OpenAPI fragments inside a bundle + +When `fluid validate` checks a `fluid bundle --format tgz` archive, OpenAPI +documents extracted into `sources/openapi/` have no directory of their own, so +the same check applies with **no** root: only same-document refs +(`$ref: "#/components/schemas/Order"`) are allowed. Any other `$ref` is +reported as an `OAS-REF-EXTERNAL` error, and openapi-spec-validator is not run +on that fragment (it would follow `file://` and `http(s)://` refs). Inline the +referenced schemas under `components` instead. + +This applies to a `$ref` key anywhere in the fragment, including inside +`example`, `examples.*.value` and `x-*` payloads. If an example payload has to +contain a `$ref`, rename the key in the payload (for example `ref`) or drop the +example. + +The check runs where openapi-spec-validator would: when it is not installed, +the fragment is reported as not validated instead. + +## Related + +- [`fluid bundle`](../cli/bundle.md): write the resolved document or a `.tgz` bundle +- [Contract loading API](../advanced/contract-loading-api.md): load a contract from Python exactly as `fluid plan` sees it +- [DuckDB sandbox](../advanced/duckdb-sandbox.md): the matching confinement for SQL inside a contract +- [Upgrading to 0.18.0](../RELEASE_NOTES_0.18.0.md) + +::: tip About the examples +Output on this page was produced with the 0.18.0 code (forge-cli +[#687](https://github.com/Agenticstiger/forge-cli/pull/687)). Long temporary +paths are shortened to `/work`, and long lines are re-wrapped. +::: From d9f2b1e7b0c4a379193c664a2d1684513155f07b Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Fri, 2 Oct 2026 19:50:30 +0200 Subject: [PATCH 2/5] =?UTF-8?q?docs:=200.18.0=20review=20fixes=20=E2=80=94?= =?UTF-8?q?=20in-memory=20ref=20root,=20non-s3=20URLs,=20OAS-REF-EXTERNAL,?= =?UTF-8?q?=20allowlist=20scope?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - contract-loading-api: FLUID_REF_ROOT reaches load_contract only; the in-memory forms always confine refs to base_dir. - release notes + duckdb-sandbox: contract SQL can no longer read http(s)/gs/Azure URLs, declared or not; only declared s3:// is reachable. New upgrade-table rows and migration sections for that and for OAS-REF-EXTERNAL. - duckdb-sandbox: split the INSTALL/LOAD refusal rows, scope the declared roots list to embedded-SQL builds (acquisition builds are narrower), note cwd-relative SQL paths, and warn that an allowed directory is readable in full by any contract that declares a glob in it. --- docs/RELEASE_NOTES_0.18.0.md | 62 +++++++++++++++++++++++---- docs/advanced/contract-loading-api.md | 32 ++++++++++---- docs/advanced/duckdb-sandbox.md | 50 ++++++++++++++++----- docs/concepts/contract-refs.md | 4 +- 4 files changed, 121 insertions(+), 27 deletions(-) diff --git a/docs/RELEASE_NOTES_0.18.0.md b/docs/RELEASE_NOTES_0.18.0.md index c9713c3..7db5388 100644 --- a/docs/RELEASE_NOTES_0.18.0.md +++ b/docs/RELEASE_NOTES_0.18.0.md @@ -30,8 +30,10 @@ service, CI job or shared host that runs contracts its operator did not write. | use `$ref` only to files under the contract's own directory | nothing | | use `$ref: ../…` to reach another product's or a shared directory | [widen the ref root](#ref-to-files-outside-the-contract-directory) | | use a `$ref` with a URL (`file://`, `https://`) or an absolute path | [copy the fragment in](#ref-to-files-outside-the-contract-directory) | +| ship OpenAPI documents in a bundle that `$ref` another file or URL, or carry a `$ref` key in an example or `x-*` payload | [inline the schemas or rename the key](#openapi-fragments-with-an-external-ref) | | run SQL only on files under the contract's directory or workspace | nothing | | declare inputs, outputs or acquisition sources at absolute paths elsewhere (`/data/landing/*.csv`) | [allow the directory](#sql-or-declarations-that-read-outside-the-contract-directory) | +| read an `http(s)://`, `gs://` or Azure URL directly in contract SQL (`read_csv('https://…')`) | [land the data first](#urls-other-than-s3-in-contract-sql) | | call `read_xlsx`, `sqlite_scan`, `ST_Read`, `delta_scan` or `iceberg_scan` in contract SQL | [convert or land the data](#functions-duckdb-used-to-autoload) | | install DuckDB yourself at a version below 1.5.0 | [upgrade DuckDB](#upgrade-duckdb) | @@ -78,15 +80,21 @@ Full reference: [Composing a contract with `$ref`](./concepts/contract-refs.md). ### SQL or declarations that read outside the contract directory -SQL in a contract can read and write only: +SQL in an embedded-SQL build on the local provider can read and write only: - the contract's directory and its FLUID workspace; - `./runtime` and the run's scratch directory; -- the locations the contract declares, and only those inside the same roots; -- any directory the operator lists in `FLUID_DUCKDB_ALLOWED_DIRS`. +- the locations the contract declares, and only those inside the roots above, + the upstream roots in `FLUID_UPSTREAM_CONTRACTS`, or a directory the + operator lists in `FLUID_DUCKDB_ALLOWED_DIRS`. -A declared input, output or acquisition source outside those roots is refused -before any SQL runs: +A DuckDB acquisition build is narrower: its declared sources and landings may +sit only in the contract's directory, its workspace, or a +`FLUID_DUCKDB_ALLOWED_DIRS` directory (not `./runtime`, the scratch directory +or a `FLUID_UPSTREAM_CONTRACTS` root). + +A declared input, output or acquisition source outside its allowed roots is +refused before any SQL runs: ```console $ fluid apply contract.fluid.yaml --mode amend-and-build --yes @@ -107,9 +115,10 @@ FLUID_DUCKDB_ALLOWED_DIRS=/work/reference \ fluid apply contract.fluid.yaml --mode amend-and-build --yes ``` -SQL that reads a path the contract does not declare (`read_csv('/etc/passwd')`, -`../`, an undeclared URL) is refused by DuckDB itself. Declare the location as -an input. +SQL that reads a local path the contract does not declare +(`read_csv('/etc/passwd')`, `../`) is refused by DuckDB itself. Declare the +location as an input. For a URL, declaring it helps only for `s3://`; see +[URLs other than `s3://`](#urls-other-than-s3-in-contract-sql). A declared glob grants the directory above its first wildcard, because DuckDB expands the glob again when the SQL runs; that directory must itself be @@ -117,6 +126,43 @@ inside the allowed roots. Full reference: [DuckDB sandbox for contract SQL](./advanced/duckdb-sandbox.md). +### URLs other than `s3://` in contract SQL + +Before 0.18.0, `read_csv('https://…')` in an embedded-SQL build worked +because DuckDB autoloaded `httpfs`. With autoloading off it fails: + +```console + ❌ Failed: 1 action(s) failed + File https://… requires the extension httpfs to be loaded +``` + +Declaring the URL does not fix it. The local provider treats only `s3://` +locations as remote: for those it loads `httpfs` and creates a credential +secret before the sandbox closes. Any other declared URL is taken as a local +path, so an `https://` input fails with `Input file not found`. Embedded SQL +can reach declared `s3://` locations only, and no contract setting brings back +`http(s)://`, `gs://` or Azure reads. + +**Migrate** by landing the data with a DuckDB acquisition build first +(`pattern: acquisition`, `engine: duckdb`, `source.kind: http` with +`source.connection.uri`), then reading the landed file. Data in GCS or Azure +can be copied to the contract's directory, its workspace, or an `s3://` +location. + +### OpenAPI fragments with an external `$ref` + +`fluid validate` on a bundle (`fluid bundle --format tgz`) now reports an +`OAS-REF-EXTERNAL` error for any `$ref` in a bundled OpenAPI document that is +not a same-document `#/…` pointer. That includes relative refs such as +`./schemas.yaml#/Order`, which openapi-spec-validator used to follow, and a +`$ref` key inside an `example`, `examples.*.value` or `x-*` payload. A bundle +that validated before can now fail. + +**Migrate** by inlining the referenced schemas under `components`, and by +renaming a `$ref` key in an example payload (for example to `ref`) or +dropping the example. See +[OpenAPI fragments inside a bundle](./concepts/contract-refs.md#openapi-fragments-inside-a-bundle). + ### Functions DuckDB used to autoload `sqlite_scan`, `read_xlsx`, `ST_Read`, `delta_scan` and `iceberg_scan` no diff --git a/docs/advanced/contract-loading-api.md b/docs/advanced/contract-loading-api.md index e4a6406..e06dd35 100644 --- a/docs/advanced/contract-loading-api.md +++ b/docs/advanced/contract-loading-api.md @@ -130,8 +130,10 @@ submitted.unresolved_refs # ('./owner.yaml', 'parts/build.yaml', 'parts/policy. ### Text with its fragments and an overlay -Pass `base_dir` to resolve file refs against a directory, under the same -[ref root rules](../concepts/contract-refs.md#the-ref-root) as a file load: +Pass `base_dir` to resolve file refs against a directory. The refs are +confined to `base_dir`, as the +[ref root](../concepts/contract-refs.md#the-ref-root); `FLUID_REF_ROOT` is not +consulted on this path (see [below](#fluid-ref-root)): ```python loaded = load_contract_from_text(text, base_dir="orders") @@ -150,8 +152,9 @@ loaded = load_contract_from_text( With `base_dir` set to a contract's directory and `overlay` set to the parsed overlay file `--env` would select, the result equals -`load_contract(that_file, env=...)`. Without `base_dir`, passing `overlay` for -a document that holds file `$ref` values raises +`load_contract(that_file, env=...)`, provided no ref needs a root wider than +`base_dir` (see [`FLUID_REF_ROOT`](#fluid-ref-root)). Without `base_dir`, +passing `overlay` for a document that holds file `$ref` values raises `contract_overlay_needs_base_dir`, because whether the engine would apply the overlay depends on what the fragments hold. @@ -195,10 +198,23 @@ contract that has overlays. ### `FLUID_REF_ROOT` -These functions have no `ref_root` argument. They go through the engine's -loader, so `FLUID_REF_ROOT` in the process environment applies to them as it -does to the CLI. To widen the root for one call, use -`fluid_build.loader.load_contract(path, ref_root=...)`; see +These functions have no `ref_root` argument, and the environment variable +reaches only one of them. + +- **`load_contract(path)`**, the file form, goes through the engine's file + loader, so `FLUID_REF_ROOT` in the process environment applies to it as it + does to the CLI. +- **`load_contract_from_text` and `load_contract_from_dict`** with `base_dir` + always use `base_dir` as the ref root. `FLUID_REF_ROOT` is not consulted, and + a ref that leaves `base_dir` is refused with `ContractLoadError`, even when + the same contract loads from its file with `FLUID_REF_ROOT` set. The error + text still suggests setting `FLUID_REF_ROOT`; on this path that has no effect. + +To compose a monorepo fragment in memory, pass the wider directory as +`base_dir` and write the refs relative to it (`./shared/policy.yaml#/gold` +with `base_dir` at the repository root, not `../shared/...` with `base_dir` at +the product). Otherwise write the contract to a file and use `load_contract`, +or call `fluid_build.loader.load_contract(path, ref_root=...)`; see [Widening the root for a monorepo](../concepts/contract-refs.md#widening-the-root-for-a-monorepo). ## Reference diff --git a/docs/advanced/duckdb-sandbox.md b/docs/advanced/duckdb-sandbox.md index 29c4997..fa1ef15 100644 --- a/docs/advanced/duckdb-sandbox.md +++ b/docs/advanced/duckdb-sandbox.md @@ -8,8 +8,8 @@ description: What SQL inside a contract can read and write on the local DuckDB e From CLI 0.18.0, every DuckDB connection the engine opens runs inside DuckDB's own sandbox. SQL in a contract can read and write the contract's directory and the locations the contract declares. It cannot read the rest of -the host, fetch a URL the contract does not declare, attach another database, -install an extension, or change those settings. +the host, fetch a URL other than a declared `s3://` location, attach another +database, install an extension, or change those settings. ::: warning Breaking in 0.18.0 - A declared input, output or acquisition source **outside the allowed @@ -18,6 +18,9 @@ install an extension, or change those settings. [`FLUID_DUCKDB_ALLOWED_DIRS`](#reading-a-file-from-another-directory). - **Functions DuckDB used to autoload no longer work in contract SQL**: `sqlite_scan`, `read_xlsx`, `ST_Read`, `delta_scan`, `iceberg_scan`. +- **Contract SQL can no longer read `http(s)://`, `gs://` or Azure URLs**, + declared or not. Only declared `s3://` locations are reachable; land other + remote data with an acquisition build first. - The `local` extra requires **`duckdb>=1.5.0`**, and the engine refuses to open DuckDB on anything older. @@ -66,6 +69,13 @@ The list in parentheses is exactly what that build's SQL may touch: the contract's directory, `./runtime`, the run's scratch directory, and the declared output. +A relative path in SQL is opened relative to the working directory, and +`./runtime` is the working directory's `runtime/`. Run `fluid` from the +contract's directory, as above, or write paths that resolve inside it: the +same build run as `fluid apply orders/contract.fluid.yaml` from the parent +directory is refused (`Cannot access file "data/orders.csv"`), and its list +shows `/runtime` rather than `orders/runtime`. + ## What SQL is refused The same message (`DuckDB refused it: …`) follows each of these: @@ -74,8 +84,9 @@ The same message (`DuckDB refused it: …`) follows each of these: |---|---| | `read_csv('/etc/passwd')`, `read_text(...)`, `read_blob(...)`, `read_parquet(...)`, `read_json(...)`, `glob('/etc/*')` on a path outside the list | `Permission Error: Cannot access file …` | | `read_csv('../shared/x.csv')`, `/./../`, a symlink that points outside | `Permission Error: Cannot access file …` | -| `read_csv('https://example.com/x.csv')`, or any URL the contract does not declare | `File https://… requires the extension httpfs to be loaded` | -| `INSTALL httpfs`, `LOAD …` | `Permission Error: Cannot access directory "~/.duckdb/extensions/…"` | +| `read_csv('https://example.com/x.csv')`, or any `http(s)://`, `gs://` or Azure URL, declared or not (only declared `s3://` locations are readable) | `File https://… requires the extension httpfs to be loaded` | +| `INSTALL httpfs` | `Permission Error: Cannot access directory "~/.duckdb/extensions/…"` | +| `LOAD httpfs`, or `LOAD` of any extension that is not built in (a built-in one such as `json` still loads) | `Permission Error: Loading external extensions is disabled through configuration` | | `SET enable_external_access = true`, or any other `SET` | `Cannot change configuration option … - the configuration has been locked` | | `read_xlsx(...)`, `sqlite_scan(...)`, `ST_Read(...)`, `delta_scan(...)`, `iceberg_scan(...)` | `Catalog Error: Table Function with name "read_xlsx" is not in the catalog` | @@ -119,16 +130,24 @@ acquisition build first. ## Declared locations stay inside the allowed directories Each declared input and output is granted to the build's SQL, and whoever -writes the contract writes the declarations. So a declaration grants a local -path only inside these directories: +writes the contract writes the declarations. So, in an embedded-SQL build on +the local provider, a declaration grants a local path only inside these +directories: - the contract's directory and the FLUID workspace it sits in; - `./runtime` and the run's scratch directory; - the upstream roots in `FLUID_UPSTREAM_CONTRACTS`; - the directories the operator lists in `FLUID_DUCKDB_ALLOWED_DIRS`. -Anything else is refused before any SQL runs. Declaring an innocuous glob in -`$HOME` does not make `~/.aws/credentials` readable: +A DuckDB acquisition build (`pattern: acquisition`, `engine: duckdb`) is +narrower: its sources and landings may sit only in the contract's directory, +its workspace, or a `FLUID_DUCKDB_ALLOWED_DIRS` directory. A source or landing +under `./runtime`, the scratch directory or a `FLUID_UPSTREAM_CONTRACTS` root +is refused there. + +Anything else is refused before any SQL runs. Unless the operator has allowed +`$HOME`, declaring an innocuous glob in it does not make `~/.aws/credentials` +readable: ```yaml properties: @@ -259,8 +278,19 @@ message now includes the declared file: FLUID_DUCKDB_ALLOWED_DIRS entry '/' is the filesystem root ``` -- It lets a contract *declare* a location in that directory. It does not make - the directory readable to SQL that has not declared it. +- It lets a contract *declare* a location in that directory; the directory is + not readable to SQL that declares nothing in it. But a declared glob grants + the whole directory above its first wildcard, and a declared directory its + subtree, and the contract author writes those declarations. + +::: warning Allow the narrowest directory +Allowing a directory makes its whole subtree readable and writable to any +contract that process runs, because a contract can declare `/*` or the +directory itself. With `FLUID_DUCKDB_ALLOWED_DIRS=$HOME`, a contract that +declares `$HOME/*.csv` can read `~/.aws/credentials`. Allow the narrowest +directory that holds the data, never `$HOME` or a directory that holds +credentials. +::: A product inside a FLUID workspace can also read its sibling products' files by path, because the workspace is an allowed directory, and a `consumes[]` diff --git a/docs/concepts/contract-refs.md b/docs/concepts/contract-refs.md index cea5c33..2f908e3 100644 --- a/docs/concepts/contract-refs.md +++ b/docs/concepts/contract-refs.md @@ -211,7 +211,9 @@ print(contract["exposes"][0]["policy"]["classification"]) The [public contract-loading API](../advanced/contract-loading-api.md) (`fluid_build.api.load_contract`) has no `ref_root` argument. It goes through -the same loader, so it honours `FLUID_REF_ROOT`. +the same loader, so it honours `FLUID_REF_ROOT`. Its in-memory forms +(`load_contract_from_text`, `load_contract_from_dict`) do not: their ref root +is always the `base_dir` you pass. ### Rules for the wider root From 98fc18b5dc26c5354b09758cc200b6033bef4868 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Fri, 2 Oct 2026 20:13:41 +0200 Subject: [PATCH 3/5] =?UTF-8?q?docs:=200.18.0=20review=20fixes=20=E2=80=94?= =?UTF-8?q?=20SET/PRAGMA=20lock,=20contract-tests=20confinement,=20OAS=20r?= =?UTF-8?q?ef=20history,=20api=201.1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Release notes: OAS-REF-EXTERNAL history stated per ref kind (file:// and http(s):// were followed; relative refs were already OAS001; a $ref key in an example or x-* payload is the new failure); SET/PRAGMA of a DuckDB setting is a breaking change; contract-tests local actions are confined to the working directory or FLUID_DUCKDB_ALLOWED_DIRS; the unshipped 'configuration has been locked' cascade fix is dropped, kept as a sandbox property (each action's connection is closed even when it fails). - DuckDB sandbox: SET/PRAGMA in the breaking box and the refusal table; both real errors for an https:// URL (with and without a declared s3:// location); contract-tests row names the confinement. - api-stability: fluid_build.api is at 1.1 on 0.18.0, which added the contract-loading API. - contract-refs: the ref/pointer/file triple is scoped to confinement refusals. - contract-loading-api: real loaded.files value and env refusal message. - Threat model stated generically on every page. --- docs/RELEASE_NOTES_0.18.0.md | 69 ++++++++++++++++++++++----- docs/advanced/api-stability.md | 4 +- docs/advanced/contract-loading-api.md | 16 +++++-- docs/advanced/duckdb-sandbox.md | 15 +++--- docs/concepts/contract-refs.md | 15 +++--- 5 files changed, 90 insertions(+), 29 deletions(-) diff --git a/docs/RELEASE_NOTES_0.18.0.md b/docs/RELEASE_NOTES_0.18.0.md index 7db5388..e82b3c7 100644 --- a/docs/RELEASE_NOTES_0.18.0.md +++ b/docs/RELEASE_NOTES_0.18.0.md @@ -19,9 +19,9 @@ A contract can no longer make the engine read the machine it runs on. - **A new public API loads a contract exactly as `fluid plan` sees it.** ([forge-cli #688](https://github.com/Agenticstiger/forge-cli/pull/688)) -These changes come from a security review of the FLUID Command Center, which -runs the engine on user-supplied contracts. The same holes apply to any -service, CI job or shared host that runs contracts its operator did not write. +These changes matter most to services, CI jobs or shared hosts that run +contracts other people wrote: before 0.18.0, such a contract could make the +engine read files and URLs its operator never meant to expose. ## Do I need to change anything? @@ -34,6 +34,8 @@ service, CI job or shared host that runs contracts its operator did not write. | run SQL only on files under the contract's directory or workspace | nothing | | declare inputs, outputs or acquisition sources at absolute paths elsewhere (`/data/landing/*.csv`) | [allow the directory](#sql-or-declarations-that-read-outside-the-contract-directory) | | read an `http(s)://`, `gs://` or Azure URL directly in contract SQL (`read_csv('https://…')`) | [land the data first](#urls-other-than-s3-in-contract-sql) | +| `SET` or `PRAGMA` a DuckDB setting in contract SQL (`memory_limit`, `threads`, `TimeZone`, …) | [remove the statement](#set-and-pragma-in-contract-sql) | +| declare `fluid contract-tests` local action inputs or outputs outside the working directory | [allow the directory](#fluid-contract-tests-local-actions) | | call `read_xlsx`, `sqlite_scan`, `ST_Read`, `delta_scan` or `iceberg_scan` in contract SQL | [convert or land the data](#functions-duckdb-used-to-autoload) | | install DuckDB yourself at a version below 1.5.0 | [upgrade DuckDB](#upgrade-duckdb) | @@ -136,6 +138,11 @@ because DuckDB autoloaded `httpfs`. With autoloading off it fails: File https://… requires the extension httpfs to be loaded ``` +That is the error when the build declares no `s3://` location. When it +declares one, `httpfs` is loaded and DuckDB refuses the URL as outside +`allowed_directories` instead: +`Permission Error: Cannot access file "https://…" - file system operations are disabled by configuration`. + Declaring the URL does not fix it. The local provider treats only `s3://` locations as remote: for those it loads `httpfs` and creates a credential secret before the sandbox closes. Any other declared URL is taken as a local @@ -149,14 +156,55 @@ can reach declared `s3://` locations only, and no contract setting brings back can be copied to the contract's directory, its workspace, or an `s3://` location. +### `SET` and `PRAGMA` in contract SQL + +The sandbox locks DuckDB's configuration before contract SQL runs, so a `SET` +or `PRAGMA` statement that changes a setting (`memory_limit`, `threads`, +`TimeZone`, …) now fails the action: + +```console +$ fluid apply contract.fluid.yaml --mode amend-and-build --yes +πŸ”· Build 'summarise' (embedded-SQL / local DuckDB) + ❌ Failed: 1 action(s) failed + Invalid Input Error: Cannot change configuration option "memory_limit" - the + configuration has been locked ... +``` + +The SQL was `SET memory_limit='1GB'; SELECT …`. `PRAGMA threads=2` and +`SET TimeZone='UTC'` fail the same way, naming `threads` and `TimeZone`. +**Migrate** by removing the statement from the contract SQL. Write timestamps +with an explicit offset (or convert with `AT TIME ZONE`) instead of setting +`TimeZone`. + +### `fluid contract-tests` local actions + +The DuckDB connection of each `fluid contract-tests` local action can reach +only the input and output files that action declares, and each of those must +sit under the working directory or a `FLUID_DUCKDB_ALLOWED_DIRS` directory. +A file declared elsewhere is refused: + +```console +The contract declares '/etc/hosts' (/private/etc/hosts), outside the directories it +may read and write (/work/ct). The operator can allow a directory with +FLUID_DUCKDB_ALLOWED_DIRS. +``` + +**Migrate** by running from a directory that holds the files, or by allowing +their directory with `FLUID_DUCKDB_ALLOWED_DIRS`. + ### OpenAPI fragments with an external `$ref` `fluid validate` on a bundle (`fluid bundle --format tgz`) now reports an `OAS-REF-EXTERNAL` error for any `$ref` in a bundled OpenAPI document that is -not a same-document `#/…` pointer. That includes relative refs such as -`./schemas.yaml#/Order`, which openapi-spec-validator used to follow, and a -`$ref` key inside an `example`, `examples.*.value` or `x-*` payload. A bundle -that validated before can now fail. +not a same-document `#/…` pointer: + +- `file://` and `http(s)://` refs used to be followed by + openapi-spec-validator. They are now reported, not followed. +- Relative refs such as `./schemas.yaml#/Order` were already reported as + unresolvable (`OAS001`); they are now reported as `OAS-REF-EXTERNAL`. +- A `$ref` key inside an `example`, `examples.*.value` or `x-*` payload is now + reported too. This is the case where a bundle that validated before can now + fail. **Migrate** by inlining the referenced schemas under `components`, and by renaming a `$ref` key in an example payload (for example to `ref`) or @@ -204,6 +252,8 @@ object-store builds use the credential-chain secret the engine creates. extensions, and `lock_configuration = true`. Contract SQL can no longer read host files, URLs or other databases, or change those settings. A guard test fails if a new `duckdb.connect` bypasses the helper. +- Each action's DuckDB connection is closed when the action ends, including + when it fails (#689). ## Added @@ -215,11 +265,6 @@ object-store builds use the credential-chain secret the engine creates. a typed `ContractLoadError`. The `fluid_build.api` version is now `1.1`. See [Contract loading API](./advanced/contract-loading-api.md). -## Fixed - -- One failed SQL action no longer makes every later action of the same local - apply fail with "configuration has been locked" (#689). - ## See also - [Composing a contract with `$ref`](./concepts/contract-refs.md) diff --git a/docs/advanced/api-stability.md b/docs/advanced/api-stability.md index 526acd2..9240595 100644 --- a/docs/advanced/api-stability.md +++ b/docs/advanced/api-stability.md @@ -3,7 +3,7 @@ The `fluid_build.api` package is the **stable extension surface** that out-of-tree runners, providers, catalog registrars, lineage emitters, and pre-land hooks target. Anything outside this package is internal and may change without notice. ::: tip Where this fits -The public API shipped alongside the source-aligned acquisition stack (schema `0.7.3`) and is current. It's pinned at version `1.0` (`__api_version__ = "1.0"`). +The public API shipped alongside the source-aligned acquisition stack (schema `0.7.3`) at version `1.0`. As of CLI `0.18.0` it is at version `1.1` (`__api_version__ = "1.1"`): 1.1 added the [contract-loading API](./contract-loading-api.md) (`load_contract`, `load_contract_from_text`, `load_contract_from_dict`). ::: ## SemVer policy @@ -11,7 +11,7 @@ The public API shipped alongside the source-aligned acquisition stack (schema `0 ```python import fluid_build.api print(fluid_build.api.__api_version__) -# "1.0" +# 1.1 ``` The API version is declared in `fluid_build/api/__init__.py`. SemVer applies: diff --git a/docs/advanced/contract-loading-api.md b/docs/advanced/contract-loading-api.md index e06dd35..881dd49 100644 --- a/docs/advanced/contract-loading-api.md +++ b/docs/advanced/contract-loading-api.md @@ -34,7 +34,9 @@ loaded = load_contract("orders/contract.fluid.yaml") loaded.origin # 'file' loaded.digest # 'sha256:ffb903222a5ab79af24df49dcf2729924b7756dc11e0ab221775df93df6dd391' -loaded.files # contract.fluid.yaml, owner.yaml, parts/build.yaml, parts/policy.yaml +loaded.files +# (PosixPath('/work/orders/contract.fluid.yaml'), PosixPath('/work/orders/owner.yaml'), +# PosixPath('/work/orders/parts/build.yaml'), PosixPath('/work/orders/parts/policy.yaml')) loaded.unresolved_refs # () loaded.contract["exposes"][0]["policy"] # {'classification': 'Internal', 'authz': {'readers': ['group:sales-analysts']}} @@ -56,8 +58,16 @@ or an absolute path would merge whichever file it named into the contract. An env that is not a single path component is refused before a file is read: ```python -load_contract("contract.fluid.yaml", env="../../etc/x") -# ContractLoadError: contract_env_invalid +from fluid_build.api import ContractLoadError + +try: + load_contract("contract.fluid.yaml", env="../../etc/x") +except ContractLoadError as err: + err.event # 'contract_env_invalid' + str(err) + # "env '../../etc/x' is not an environment name: an env names an overlay file + # next to the contract, so it must be one path component: not empty, not '.' or + # '..', no '/', '\' or NUL, not drive-qualified ('C:prod')" ``` Refused: `""` (not read as `None`), `.`, `..`, anything holding `/`, `\` or a diff --git a/docs/advanced/duckdb-sandbox.md b/docs/advanced/duckdb-sandbox.md index fa1ef15..9a30be6 100644 --- a/docs/advanced/duckdb-sandbox.md +++ b/docs/advanced/duckdb-sandbox.md @@ -21,6 +21,10 @@ database, install an extension, or change those settings. - **Contract SQL can no longer read `http(s)://`, `gs://` or Azure URLs**, declared or not. Only declared `s3://` locations are reachable; land other remote data with an acquisition build first. +- **`SET` and `PRAGMA` statements that change a DuckDB setting** + (`memory_limit`, `threads`, `TimeZone`, …) in contract SQL now fail the + action with `Cannot change configuration option "memory_limit" - the + configuration has been locked`. Remove them from the SQL. - The `local` extra requires **`duckdb>=1.5.0`**, and the engine refuses to open DuckDB on anything older. @@ -84,10 +88,10 @@ The same message (`DuckDB refused it: …`) follows each of these: |---|---| | `read_csv('/etc/passwd')`, `read_text(...)`, `read_blob(...)`, `read_parquet(...)`, `read_json(...)`, `glob('/etc/*')` on a path outside the list | `Permission Error: Cannot access file …` | | `read_csv('../shared/x.csv')`, `/./../`, a symlink that points outside | `Permission Error: Cannot access file …` | -| `read_csv('https://example.com/x.csv')`, or any `http(s)://`, `gs://` or Azure URL, declared or not (only declared `s3://` locations are readable) | `File https://… requires the extension httpfs to be loaded` | +| `read_csv('https://example.com/x.csv')`, or any `http(s)://`, `gs://` or Azure URL, declared or not (only declared `s3://` locations are readable) | When the build declares no `s3://` location: `File https://example.com/x.csv requires the extension httpfs to be loaded`. When it declares one (so `httpfs` is loaded): `Permission Error: Cannot access file "https://example.com/x.csv" - file system operations are disabled by configuration`, because the URL is outside `allowed_directories` | | `INSTALL httpfs` | `Permission Error: Cannot access directory "~/.duckdb/extensions/…"` | | `LOAD httpfs`, or `LOAD` of any extension that is not built in (a built-in one such as `json` still loads) | `Permission Error: Loading external extensions is disabled through configuration` | -| `SET enable_external_access = true`, or any other `SET` | `Cannot change configuration option … - the configuration has been locked` | +| `SET enable_external_access = true`, or any other `SET` or `PRAGMA` that changes a setting (`SET memory_limit='1GB'`, `PRAGMA threads=2`, `SET TimeZone='UTC'`) | `Invalid Input Error: Cannot change configuration option "memory_limit" - the configuration has been locked` | | `read_xlsx(...)`, `sqlite_scan(...)`, `ST_Read(...)`, `delta_scan(...)`, `iceberg_scan(...)` | `Catalog Error: Table Function with name "read_xlsx" is not in the catalog` | `ATTACH` of another database file and `COPY … TO` / `COPY … FROM` a location @@ -123,7 +127,7 @@ acquisition build first. | Embedded-SQL build on the local DuckDB engine (`builds[].properties.sql`) | the contract's directory, the FLUID workspace it sits in (`fluid.workspace.yaml`), `./runtime`, the run's scratch directory, each declared `parameters.inputs[].path`, each resolved `consumes[]` upstream, the expose's landing path, and the `s3://` prefixes those name. A declared local path counts only [inside the allowed directories](#declared-locations-stay-inside-the-allowed-directories). | | DuckDB acquisition build (`pattern: acquisition`, `engine: duckdb`) | the contract's directory, the declared `source.connection.uri` (or stream paths), and each stream's landing file, each inside the allowed directories. A `mysql` source is attached before the sandbox closes; so is a `sqlite` source, and its file must also be inside the allowed directories. | | `fluid validate` quality rules, `fluid verify`, `fluid diff` | the one file being checked | -| `fluid contract-tests` local actions | each declared input file and each output file | +| `fluid contract-tests` local actions | each declared input file and each output file. Each must sit under the working directory or a `FLUID_DUCKDB_ALLOWED_DIRS` directory, or the action is refused before its SQL runs (new in 0.18.0). | | Discovery (`fluid forge data-model from-source`, `discover`) | the one file or URL being introspected; a JDBC source is attached first | | MCP output port (DuckDB driver) | the bound file | @@ -349,9 +353,8 @@ guide gives. `~/.duckdb/stored_secrets` are no longer loaded; object-store builds use the credential-chain secret the engine creates. - **Defense in depth, not isolation.** DuckDB describes these settings as not a - substitute for proper sandboxing. A service that runs other people's - contracts (the Command Center, a shared CI runner) should still run each one - in its own container. + substitute for proper sandboxing. Services, CI jobs or shared hosts that run + contracts other people wrote should still run each one in its own container. ## Related diff --git a/docs/concepts/contract-refs.md b/docs/concepts/contract-refs.md index 2f908e3..7591047 100644 --- a/docs/concepts/contract-refs.md +++ b/docs/concepts/contract-refs.md @@ -167,14 +167,17 @@ in /work/orders/contract.fluid.yaml is a URL; remote refs (including file://) ar supported β€” use a relative path to a file inside the ref root ``` -Every message names the ref, the JSON pointer of the `$ref` node, and the file -it was written in. +Each of these confinement refusals (a URL, an absolute path, an escape from +the ref root) names the ref, the JSON pointer of the `$ref` node, and the file +it was written in. Other ref errors, such as a ref blocked because it resolves +into a system directory, a missing target file, or a JSON pointer that does not +resolve, do not carry all three. ::: tip Why the root exists -Contracts are often untrusted input. A platform such as the FLUID Command -Center runs `fluid validate` and `fluid bundle` on contracts its users upload. -Without a root, a contract could compose any YAML or JSON file the process can -read into itself, and `fluid bundle` would print it back. +Contracts are often untrusted input: services, CI jobs or shared hosts run +`fluid validate` and `fluid bundle` on contracts other people wrote. Without a +root, a contract could compose any YAML or JSON file the process can read into +itself, and `fluid bundle` would print it back. ::: ## Widening the root for a monorepo From 16e5526f98f527881045e21986e9a093b0f0ba66 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Fri, 2 Oct 2026 20:22:57 +0200 Subject: [PATCH 4/5] =?UTF-8?q?docs:=200.18.0=20review=20fixes=20=E2=80=94?= =?UTF-8?q?=20drop=20contract-tests=20migration,=20OAS=20newly-failing=20c?= =?UTF-8?q?ases,=20print(err)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fluid contract-tests never opens DuckDB, so the migration row and section for its local actions described a change no user can hit. Mention the confinement only as hardening of the legacy fluid_build.contract_tests module. Name every case where a previously valid bundle now fails OAS-REF-EXTERNAL, and show the env-refusal message via print(err) so the backslash is literal. --- docs/RELEASE_NOTES_0.18.0.md | 28 +++++++++------------------ docs/advanced/contract-loading-api.md | 6 +++--- docs/advanced/duckdb-sandbox.md | 2 +- 3 files changed, 13 insertions(+), 23 deletions(-) diff --git a/docs/RELEASE_NOTES_0.18.0.md b/docs/RELEASE_NOTES_0.18.0.md index e82b3c7..966770d 100644 --- a/docs/RELEASE_NOTES_0.18.0.md +++ b/docs/RELEASE_NOTES_0.18.0.md @@ -35,7 +35,6 @@ engine read files and URLs its operator never meant to expose. | declare inputs, outputs or acquisition sources at absolute paths elsewhere (`/data/landing/*.csv`) | [allow the directory](#sql-or-declarations-that-read-outside-the-contract-directory) | | read an `http(s)://`, `gs://` or Azure URL directly in contract SQL (`read_csv('https://…')`) | [land the data first](#urls-other-than-s3-in-contract-sql) | | `SET` or `PRAGMA` a DuckDB setting in contract SQL (`memory_limit`, `threads`, `TimeZone`, …) | [remove the statement](#set-and-pragma-in-contract-sql) | -| declare `fluid contract-tests` local action inputs or outputs outside the working directory | [allow the directory](#fluid-contract-tests-local-actions) | | call `read_xlsx`, `sqlite_scan`, `ST_Read`, `delta_scan` or `iceberg_scan` in contract SQL | [convert or land the data](#functions-duckdb-used-to-autoload) | | install DuckDB yourself at a version below 1.5.0 | [upgrade DuckDB](#upgrade-duckdb) | @@ -176,22 +175,6 @@ The SQL was `SET memory_limit='1GB'; SELECT …`. `PRAGMA threads=2` and with an explicit offset (or convert with `AT TIME ZONE`) instead of setting `TimeZone`. -### `fluid contract-tests` local actions - -The DuckDB connection of each `fluid contract-tests` local action can reach -only the input and output files that action declares, and each of those must -sit under the working directory or a `FLUID_DUCKDB_ALLOWED_DIRS` directory. -A file declared elsewhere is refused: - -```console -The contract declares '/etc/hosts' (/private/etc/hosts), outside the directories it -may read and write (/work/ct). The operator can allow a directory with -FLUID_DUCKDB_ALLOWED_DIRS. -``` - -**Migrate** by running from a directory that holds the files, or by allowing -their directory with `FLUID_DUCKDB_ALLOWED_DIRS`. - ### OpenAPI fragments with an external `$ref` `fluid validate` on a bundle (`fluid bundle --format tgz`) now reports an @@ -203,8 +186,10 @@ not a same-document `#/…` pointer: - Relative refs such as `./schemas.yaml#/Order` were already reported as unresolvable (`OAS001`); they are now reported as `OAS-REF-EXTERNAL`. - A `$ref` key inside an `example`, `examples.*.value` or `x-*` payload is now - reported too. This is the case where a bundle that validated before can now - fail. + reported too. + +A bundle that validated before can now fail if it holds a resolvable `file://` +or `http(s)://` ref, or a `$ref` key in an example or `x-*` payload. **Migrate** by inlining the referenced schemas under `components`, and by renaming a `$ref` key in an example payload (for example to `ref`) or @@ -254,6 +239,11 @@ object-store builds use the credential-chain secret the engine creates. test fails if a new `duckdb.connect` bypasses the helper. - Each action's DuckDB connection is closed when the action ends, including when it fails (#689). +- The legacy local-provider module `fluid_build.contract_tests`, which no + `fluid` command uses, confines each action's DuckDB connection to the files + that action declares, under the working directory or a + `FLUID_DUCKDB_ALLOWED_DIRS` directory (#689). `fluid contract-tests` does not + run DuckDB and is unchanged. ## Added diff --git a/docs/advanced/contract-loading-api.md b/docs/advanced/contract-loading-api.md index 881dd49..36653e5 100644 --- a/docs/advanced/contract-loading-api.md +++ b/docs/advanced/contract-loading-api.md @@ -64,10 +64,10 @@ try: load_contract("contract.fluid.yaml", env="../../etc/x") except ContractLoadError as err: err.event # 'contract_env_invalid' - str(err) - # "env '../../etc/x' is not an environment name: an env names an overlay file + print(err) + # env '../../etc/x' is not an environment name: an env names an overlay file # next to the contract, so it must be one path component: not empty, not '.' or - # '..', no '/', '\' or NUL, not drive-qualified ('C:prod')" + # '..', no '/', '\' or NUL, not drive-qualified ('C:prod') ``` Refused: `""` (not read as `None`), `.`, `..`, anything holding `/`, `\` or a diff --git a/docs/advanced/duckdb-sandbox.md b/docs/advanced/duckdb-sandbox.md index 9a30be6..59fa673 100644 --- a/docs/advanced/duckdb-sandbox.md +++ b/docs/advanced/duckdb-sandbox.md @@ -127,7 +127,7 @@ acquisition build first. | Embedded-SQL build on the local DuckDB engine (`builds[].properties.sql`) | the contract's directory, the FLUID workspace it sits in (`fluid.workspace.yaml`), `./runtime`, the run's scratch directory, each declared `parameters.inputs[].path`, each resolved `consumes[]` upstream, the expose's landing path, and the `s3://` prefixes those name. A declared local path counts only [inside the allowed directories](#declared-locations-stay-inside-the-allowed-directories). | | DuckDB acquisition build (`pattern: acquisition`, `engine: duckdb`) | the contract's directory, the declared `source.connection.uri` (or stream paths), and each stream's landing file, each inside the allowed directories. A `mysql` source is attached before the sandbox closes; so is a `sqlite` source, and its file must also be inside the allowed directories. | | `fluid validate` quality rules, `fluid verify`, `fluid diff` | the one file being checked | -| `fluid contract-tests` local actions | each declared input file and each output file. Each must sit under the working directory or a `FLUID_DUCKDB_ALLOWED_DIRS` directory, or the action is refused before its SQL runs (new in 0.18.0). | +| `fluid contract-tests` local actions | each declared input file and each output file | | Discovery (`fluid forge data-model from-source`, `discover`) | the one file or URL being introspected; a JDBC source is attached first | | MCP output port (DuckDB driver) | the bound file | From dec7dfad26698b7ab8bad9775c6b6076d7f843f3 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Fri, 2 Oct 2026 20:26:07 +0200 Subject: [PATCH 5/5] =?UTF-8?q?docs:=200.18.0=20review=20fixes=20=E2=80=94?= =?UTF-8?q?=20DuckDB=20reach=20table=20names=20the=20legacy=20contract=5Ft?= =?UTF-8?q?ests=20module,=20not=20fluid=20contract-tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/advanced/duckdb-sandbox.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/advanced/duckdb-sandbox.md b/docs/advanced/duckdb-sandbox.md index 59fa673..bc78091 100644 --- a/docs/advanced/duckdb-sandbox.md +++ b/docs/advanced/duckdb-sandbox.md @@ -127,7 +127,7 @@ acquisition build first. | Embedded-SQL build on the local DuckDB engine (`builds[].properties.sql`) | the contract's directory, the FLUID workspace it sits in (`fluid.workspace.yaml`), `./runtime`, the run's scratch directory, each declared `parameters.inputs[].path`, each resolved `consumes[]` upstream, the expose's landing path, and the `s3://` prefixes those name. A declared local path counts only [inside the allowed directories](#declared-locations-stay-inside-the-allowed-directories). | | DuckDB acquisition build (`pattern: acquisition`, `engine: duckdb`) | the contract's directory, the declared `source.connection.uri` (or stream paths), and each stream's landing file, each inside the allowed directories. A `mysql` source is attached before the sandbox closes; so is a `sqlite` source, and its file must also be inside the allowed directories. | | `fluid validate` quality rules, `fluid verify`, `fluid diff` | the one file being checked | -| `fluid contract-tests` local actions | each declared input file and each output file | +| Legacy local-provider module `fluid_build.contract_tests` (no `fluid` command uses it) | each declared input file and each output file, each under the working directory or a `FLUID_DUCKDB_ALLOWED_DIRS` directory | | Discovery (`fluid forge data-model from-source`, `discover`) | the one file or URL being introspected; a JDBC source is attached first | | MCP output port (DuckDB driver) | the bound file |