fix(security): contract SQL runs in DuckDB's sandbox, confined to what the contract may declare

[fas89]

One of three engine-side fixes requested by the maintainers (2026-10-02) for services that run contracts other people wrote: $ref confinement in the loader, a public contract-load API, and a DuckDB sandbox for contract SQL. The three (#687 $ref confinement, #688 public load API, #689 DuckDB sandbox) touch disjoint files and merge in any order. Each was built, adversarially reviewed (every finding judged by 3 independent skeptics), fixed and re-reviewed. Gates: ruff, black 24.10.0, and the full pytest suite on Python 3.12 and 3.10.

What and why

A contract's SQL (builds[].properties.sql, stage SQL, quality and masking checks) ran on DuckDB with the full privileges of the process. read_csv('/etc/passwd'), read_text('~/.aws/credentials'), glob('/'), http(s) URLs, ATTACH of any database file, and COPY ... TO anywhere all worked. Any downstream service that runs a user's contract handed that user its host.

This PR routes every DuckDB connection in the engine through fluid_build/providers/_duckdb_sandbox.py::secure_duckdb_connect. That helper applies DuckDB's own sandbox in the order "Securing DuckDB" gives:

1. Turn off persistent secrets and community extensions.
2. Load only what the call site needs.
3. Turn off extension autoinstall and autoload.
4. Pin home_directory to $HOME (NewVuln24: home_directory Bypasses allowed_directories Restriction via struct-VARIANT duckdb/duckdb#26064).
5. Set allowed_directories / allowed_paths.
6. Set enable_external_access=false.
7. Set lock_configuration=true.

A test fails if any module opens or queries DuckDB without the helper.

A declaration is not a grant of the host (review fix)

Declared inputs, outputs and sources are granted to the SQL, and the contract's author writes them. They are now confined. A declared local path is granted only if its realpath is inside one of these:

• the contract's directory or its FLUID workspace
• ./runtime or the run's scratch directory
• the FLUID_UPSTREAM_CONTRACTS roots
• a directory the operator lists in FLUID_DUCKDB_ALLOWED_DIRS (absolute paths only; / is refused; no contract field can set it)

Everything else fails before any SQL runs:

$ fluid apply contract.fluid.yaml --mode amend-and-build --yes   # inputs: [{name: d, path: /home/me/*.csv}]
🔷 Build 'summarise' (embedded-SQL / local DuckDB)
   ❌ Failed: 1 action(s) failed
      The contract declares '/home/me/*.csv' (/home/me), outside the directories it
      may read and write (/work/orders, /work/orders/runtime, /tmp/fluid_yc3ryr57).
      The operator can allow a directory with FLUID_DUCKDB_ALLOWED_DIRS.

What changed in how declarations are granted:

• A glob is granted as its pattern plus the files it matches, each one confined. It no longer grants the directory above its first wildcard.
• A matched symlink that points out of the roots is refused.
• An in-repo symlink to / is refused.
• <root>/*/../../x is normalised before the check.
• ./*.py resolves where DuckDB opens it and is confined, so it cannot grant a server's working directory.

This applies to the local provider, the DuckDB acquisition runner (source and landing) and fluid contract-tests actions.

One failed action no longer fails the rest of the run (review fix)

All actions of one apply share a session database. DuckDB shares one instance per file per process, and the lock is instance-wide. A connection left open by a failure (kept alive by an exception-to-frame cycle) made every later action fail with configuration has been locked.

Every local-provider and contract-tests connection is now closed in a finally, and the refusal is raised without that cycle. A second connection to an already-open file DB now raises a clear DuckDBSandboxError ("already open in this process").

Extensions (review fixes)

• Functions DuckDB used to autoload are no longer available to contract SQL. Their catalog error ("but it exists in the ... extension") is now classified as a sandbox refusal: it is not retried, and its hint explains the extension is unavailable instead of advising a SET the lock refuses.
• The sqlite / postgres scanners open files and sockets through their own libraries, so the allowlist does not bound them. This is now documented and pinned by tests, and a guard test fails if contract SQL ever runs on a connection that has a database scanner loaded.

Breaking changes (release notes)

• duckdb>=1.5.0 is required by the local extra. Older versions are refused: <dir>/./../ escaped the allowlist up to 1.4.3, and symlinks escaped it through 1.4.x.
• Contract SQL reads only the contract's directory, its workspace, ./runtime, the run's scratch directory and the locations it declares.
• A declared input, output or source outside those directories is refused unless the operator sets FLUID_DUCKDB_ALLOWED_DIRS=/dir1:/dir2. This includes absolute landing paths and acquisition sources such as /data/landing/*.csv.
• A glob declaration grants the files it matches when the run starts, not their directory.
• Autoloaded extension functions no longer work in contract SQL, even on files in the contract's own directory: sqlite_scan, read_xlsx, ST_Read, delta_scan, iceberg_scan. Read CSV, Parquet or JSON instead, or land the data with an acquisition build.
• Persistent DuckDB secrets (~/.duckdb/stored_secrets) are no longer loaded.
• One connection per DuckDB database file per process. Two MCP DuckDB drivers bound to the same .duckdb file, or two concurrent persist=True local runs, now get a clear error.

Docs

docs/duckdb-sandbox.md covers what SQL can reach, declared-location confinement with examples, the operator opt-in, what each kind of declaration grants, how it works, and the limits (remote prefixes bound the bucket, loaded scanners are unbounded, one connection per file, defence in depth rather than isolation).

Tests

tests/providers/test_duckdb_sandbox.py has 85 tests:

• 22 SQL attacks through the real embedded-SQL build.
• Declared-location attacks: an outside glob, the file itself, $HOME, an outside directory, .. after a glob, relative .., ./*.csv from a foreign working directory, a symlink to /, an outside output, and a glob match that is a symlink.
• The operator opt-in.
• Failure-cascade regressions: one apply, sequential persist-mode applies, a contract-tests file DB, and a second connection.
• The autoload refusal.
• The scanner limits and the guard that contract SQL never sees a scanner.
• The bypass detector.

Every fix has a negative control: the fix is reverted in place, the pinning tests go red, and the file is restored byte-identical. Against the previous commit's code, 19 of the 20 new tests that exercise it fail. The 20th was then strengthened to hold the exception like a real caller, and it fails too.

Gates:

• ruff check fluid_build/ tests/ is clean.
• black==24.10.0 --check fluid_build/ tests/ is clean.
• The full suite passes on Python 3.12 with DuckDB 1.5.6: 19785 passed, 637 skipped, 0 failed.
• The touched test files pass on Python 3.10.

Tested live

Engine-level reproductions through _execute_embedded_sql_build and LocalProvider.apply:

• With HOME=<fake> and a declared <fake>/*.csv, read_text('~/.aws/credentials') now exits 1 with the confinement error above. On the previous commit it exited 0 and wrote the credentials to the output.
• The reviewers' 6-action apply now gives ok, ok, error, ok, error, ok, the same as main; on the previous commit it gave ok, ok, error, error, error, error.
• sqlite_scan('data/app.sqlite','t') exits 1 with the extension hint.

Refs:

• https://duckdb.org/docs/current/operations_manual/securing_duckdb/overview.html
• NewVuln24: home_directory Bypasses allowed_directories Restriction via struct-VARIANT duckdb/duckdb#26064
• fix(core): confine DuckDB-backed sources to their own location bordumb/dataing#176

Decisions for the reviewer

• Relative declared paths are resolved against the working directory, not anchor_dir as the fix notes suggested. Shipped examples name inputs relative to the repo root (for example examples/02-csv-to-data-product/contract.fluid.yaml path: "examples/02-csv-to-data-product/customers.csv"), and DuckDB opens relative paths at the working directory. Granting anchor_dir/ would grant a different file from the one DuckDB reads and would break every example. Confinement (realpath inside the roots) is what closes the ./*.py attack, and a test pins it. Making provider inputs anchor-relative would be a separate semantic change.
• Breaking change for local CLI users: a contract that declares an input, output or acquisition source outside its directory or workspace now needs FLUID_DUCKDB_ALLOWED_DIRS. The behaviour is an env var only; a --allow-dir CLI flag would need changes in shared CLI files and was left out to keep the file set narrow. Owner call: is the env var enough, or do we add the flag in a follow-up?
• When LocalProvider runs without an anchor_dir (a bare apply of actions), the working directory is the confinement root. That matches where relative paths resolve, but on a server it means the server's working directory. The embedded-SQL build path always sets anchor_dir.
• Behaviour kept from 14f89fa, outside this review's scope: ./runtime under the process working directory is granted as a whole directory to every contract. On a multi-tenant host whose working directory is shared, one contract could read another's runtime/out previews. A follow-up could move previews into the per-run scratch directory.
• F3: no declared-extensions property was added (properties.duckdb.extensions with an engine allowlist). The loss of the autoloaded readers is documented as a breaking change instead, because loading sqlite before the lock would reopen F4. Owner decision needed if read_xlsx, delta_scan or iceberg_scan support is wanted.
• F4: an acquisition build with a sqlite or postgres source runs engine-built SQL (quality predicates, incremental filter, masking projection) on a connection that has the scanner loaded. Those SQL fragments are built from validated identifiers and quoted literals, and no exploit path was found. The suggested hardening (ATTACH plus postgres_query instead of a callable postgres_scan) was not done.
• One connection per file DB: two MCP DuckDB drivers bound to the same .duckdb file in one process now fail with a clear error instead of sharing the instance, as they did on main. The behaviour is documented. Joining an already-locked instance whose allowlist matches could be supported later.
• The tests that load real extensions (test_a_loaded_sqlite_scanner_, test_a_loaded_postgres_scanner_) install sqlite and postgres into the real ~/.duckdb when online, and skip when offline.
• Declared-path policy: the sandbox lets SQL read exactly what the contract declares (parameters.inputs, acquisition source.connection.uri, the expose's landing path), wherever that is, including /etc/hosts if a contract declares it. That keeps every existing flow working. If a downstream service needs declared paths confined to the contract or workspace dir too, that is a separate policy check on the declared paths before connect; it is not in this PR.
• Workspace-root grant: the local provider grants the nearest ancestor holding fluid.workspace.yaml, the same scope consumes[] resolution already walks, so the 'read upstream by path' workaround keeps working. If a downstream service keeps several tenants' contracts under one directory that has a fluid.workspace.yaml above them, contract SQL in one tenant could read another tenant's files. The downstream service should give each tenant its own workspace root, or say so and this grant can be dropped.
• Windows: DuckDB's SanitizeAllowedPath turns backslashes into '/' on both sides (read from the v1.4.4 source), and _static_prefix handles both separators, but the windows-latest DuckDB e2e leg has not been run. Watch that CI leg.
• Docs home: CONTRIBUTING.md says user-facing docs belong in the separate forge_docs repo. This PR adds docs/duckdb-sandbox.md in-repo; a companion forge_docs page (and a 'Docs TODO / link' tick in the PR template) is still owed.
• pyproject.toml is touched by exactly one line (the duckdb floor plus a comment in the local extra). That should not conflict with the load-API or $ref PRs unless one of them also edits the local extra.
• Pinning home_directory to $HOME cannot be proven by a test on DuckDB 1.5.6 (NC8 stays green). It guards duckdb#26064, which reproduces on 2.0-dev per the issue.

Changes after review

These changes came from adversarial review rounds; each finding was upheld by at least 2 of 3 independent skeptics before it was fixed. A final re-review of the branch found nothing further.

R1 — f6b21c0

Follow-up: four review findings on the confinement commit

• A symlinked ./runtime no longer grants where it points. The local provider and ducksql.apply_sql grant ./runtime by convention, and it lives in the working directory, which is usually the contract's own. A repository that shipped runtime -> ../../.. had $HOME granted, because DuckDB realpaths each allowlist entry. The new unaliased_dir() grants runtime only when its realpath is where its name says it is, or inside the contract's directory or workspace. Otherwise runtime is left out of the allowlist and of the confinement roots, and the provider logs a local_runtime_not_granted warning.
• A sqlite source is confined. The sqlite scanner opens files through its own library, so allowed_directories never bounded the ATTACH of connection.uri|path|database. Any SQLite file on the host could be landed. The path now goes through confine_declared(): it must be inside the contract's directory, its workspace or FLUID_DUCKDB_ALLOWED_DIRS, or it is refused with a DuckDBSandboxError that names that variable. The path that was checked is the path that gets attached.
• A glob grants its directory, not a snapshot of its matches. DuckDB's own glob expansion includes dotfiles (macOS ._x.csv on exFAT/SMB) and files that land after the grant, and DuckDB refuses the whole read if any one is not granted. That broke operator-allowed landing directories. A glob now grants the directory above its first wildcard. With confinement roots, that directory must itself resolve inside them, so the grant is no wider than what the contract could already declare. DuckDB still refuses a matched symlink that leads out, at read time.
• Allowlist building is linear. There is no per-match expansion any more, and _new de-duplicates with a set. Building _allowlist over a 20,000-file glob went from 14.6 s to 0.02 s. Before the fix that cost was paid on every action's connection.

Each fix has a negative-controlled test in tests/providers/test_duckdb_sandbox.py: reverting just that source change makes the test fail. docs/duckdb-sandbox.md is updated: the glob row, the runtime symlink and sqlite confinement.

R2 — f974f17

Wildcard characters in a directory name (review follow-up). A contract in a directory whose name contains [, ? or * (for example Proj [old]/) had every declared input and output refused. The callers join the contract's directory, the working directory or $HOME in front of a declared path, and the allowlist read the [ in that name as a wildcard. It cut the path to the directory above it and refused that directory as outside the contract. The error named the parent directory as the resolved path. On main the same contract builds.

Now a directory component that exists under its literal name is treated as a directory, and wildcards are looked for only after it. The last component is still treated as a pattern, because DuckDB reads what it matches. Read literally, a path grants less than the cut-off directory did, never more, and confinement still resolves it with realpath. If such a name also matches a sibling as a pattern (p x/ for p [x]/), DuckDB reads the sibling. The sibling is not granted, so the read is refused rather than widened, and a test pins this.

New tests:

• builds through the embedded-SQL path, the local provider's apply, and the real CLI (examples/02 applied as its README says, from repo [old]/);
• grant checks covering ? and a [ in $HOME.

All 15 new cases fail with the old prefix computation restored. docs/duckdb-sandbox.md notes the rule.

Documentation

Companion docs PR: Agenticstiger/forge_docs#139 (new pages for $ref confinement, the DuckDB sandbox and the contract-loading API, plus 0.18.0 release notes). It merges after 0.18.0 is on PyPI.

[github-actions]

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

• Link a docs PR — check the "Docs PR linked" box and paste the URL
• Mark as no docs needed — check "No docs needed" with a justification
• Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.