fix(loader): confine $ref to the contract's directory tree

[fas89]

One of three engine-side fixes requested by the maintainers (2026-10-02) for services that run contracts other people wrote: $ref confinement in the loader, a public contract-load API, and a DuckDB sandbox for contract SQL. The three (#687 $ref confinement, #688 public load API, #689 DuckDB sandbox) touch disjoint files and merge in any order. Each was built, adversarially reviewed (every finding judged by 3 independent skeptics), fixed and re-reviewed. Gates: ruff, black 24.10.0, and the full pytest suite on Python 3.12 and 3.10.

Summary

A relative $ref was joined to the contract's directory with ../ honoured. The only guards were "not absolute" and the system-directory deny list, so a contract could compose any dict-rooted YAML/JSON file the process can read into itself, and fluid bundle printed it back. The downstream service runs fluid validate / bundle on contracts its users upload.

Every external $ref now goes through one check, fluid_build/util/ref_confinement.py::confine_ref:

• A URL scheme (file:// included) or a //host form is treated as remote and refused.
• An absolute path (POSIX, Windows drive, UNC, rooted) is refused.
• The target is Path.resolve()-d (.. and symlinks) and must be is_relative_to(root).
• The root is the root contract's directory, and nested refs are held to that same root.
• The check runs before the target's existence is tested, so a refused ref cannot be used to probe whether a file exists.

The error is RefConfinementError, a RefResolutionError subclass, with .ref / .pointer / .source / .root.

Widening the root is opt-in: pass ref_root= to load_contract / load_with_overlay / compile_contract, or set FLUID_REF_ROOT for the CLI.

• FLUID_REF_ROOT is process-wide. If it is not a directory or does not contain the contract, it is ignored for that contract with a ref_root_env_ignored WARNING, and the contract gets its own directory as the root (the same root as with the variable unset). So a variable set once in a shell or a service container never breaks or widens an unrelated contract, such as an upload materialised in a temp directory.
• An unusable ref_root= raises.
• URLs stay refused under a widened root, and the SecurePathValidator deny list stays as a second layer.

Bundled OpenAPI fragments (sources/openapi/*) go through the same check with no root. Any non-#/ $ref anywhere in the fragment, including example and x-* payloads, is an OAS-REF-EXTERNAL error, and openapi-spec-validator is not run on that fragment. Its default handlers follow file:// and http(s)://. Keys are not skipped by name, because properties: {example: {$ref: …}} is a schema whose $ref the validator does follow (checked against openapi-spec-validator 0.9.0 with an audit hook on open()).

⚠️ BREAKING: default $ref confinement

Monorepos that shared fragments between products with $ref: ../other-product/… (the old loader comment explicitly allowed ../sibling/) now fail with escapes the ref root.

Migration: set the ref root to a directory that holds the products and their shared fragments:

export FLUID_REF_ROOT="$(git rev-parse --show-toplevel)"   # or the narrowest common parent
fluid validate products/orders/contract.fluid.yaml

From Python: load_contract(path, ref_root="<repo root>"). Contracts whose refs stay inside their own directory need no change. See docs/contract-refs.md#widening-the-root-monorepos and the new "Upgrading" subsection.

Release notes (for CHANGELOG)

• Security / BREAKING: $ref targets are confined to the root contract's directory tree. URL and absolute refs are refused, and .. and symlink escapes are refused after resolution. Widen with FLUID_REF_ROOT / ref_root=.
• FLUID_REF_ROOT is ignored, with a ref_root_env_ignored warning, for a contract outside it. An unusable ref_root= raises.
• fluid validate <bundle>.tgz: a bundled OpenAPI fragment with any external $ref (example payloads included) gets OAS-REF-EXTERNAL instead of being handed to openapi-spec-validator.

Review follow-ups addressed (second commit)

• F1: a stale FLUID_REF_ROOT hard-failed every unrelated contract with a file $ref, contradicting the docs. It now falls back as described above, and ref_root= stays strict.
• F2: the bitcoin-multifile README advertised cross-product $ref reuse. It now gives the FLUID_REF_ROOT step and has a worked sibling-product section, checked against the CLI. The change is marked BREAKING above.
• F3: the $ref-in-example rejection is documented and pinned by tests rather than relaxed, because relaxing it by key name would let schema-position refs through.

Docs

• docs/contract-refs.md: new page.
• The SECURITY.md and AGENTS.md invariants are updated.
• examples/0.7.1/bitcoin-multifile/README.md has a new section, "Sharing a fragment with another product".

Borrowed, not built

The containment shape comes from:

• datamodel-code-generator ≥ 0.62.0's fix for CVE-2026-55389: resolve, then is_relative_to(base_path); file:// is treated as remote; widening is an explicit opt-in.
• python-jsonschema/referencing's prefix-checked filesystem retriever (raises NoSuchResource).

Neither library composes YAML fragments with a JSON-pointer suffix, so the policy is borrowed and the resolver kept.

Tests

• tests/test_loader_ref_confinement.py:
  • Escapes are refused through every entry point: .., absolute, Windows-shaped, file://, remote, symlinked file and directory, nested refs, and no existence probe.
  • The opt-in: ref_root=-strict, and the env fallback for elsewhere and missing roots across all 3 entry points, which warns once and is never wider than the default.
  • Real-CLI tests: validate refuses an escape and leaks no secret; bundle exits 2; validate passes under the opt-in; validate passes with FLUID_REF_ROOT pointing elsewhere.
• tests/forge/test_openapi_external_refs.py: OAS-REF-EXTERNAL for URL, absolute and relative refs, for refs in data and schema positions, and against the real validator.
• tests/util/test_ref_confinement.py: unit tests for confine_ref.
• Negative controls: every new test was seen red with the fix reverted in place, and the files were restored byte-identical (SHA checked).
• Full suite (pytest -n auto --dist loadscope --ignore=tests/perf, Python 3.12): 19728 passed, 637 skipped, 11 xfailed, 15 xpassed.
• ruff check fluid_build/ tests/ passes, and black==24.10.0 --check fluid_build/ tests/ passes.

Tested live

$ FLUID_REF_ROOT=/some/other/dir fluid validate examples/0.7.1/bitcoin-multifile/contract.fluid.yaml
ref_root_env_ignored: contract …/bitcoin-multifile/contract.fluid.yaml is outside FLUID_REF_ROOT='/some/other/dir' (…); the ref root must contain the contract. Ignoring it for this contract: its $refs are confined to the contract's own directory …/bitcoin-multifile (the default).
✅ Valid FLUID contract (schema v0.7.1)            # exit 0 (was: contract_load_failed)

$ fluid validate examples/0.7.1/eu-orders/contract.fluid.yaml     # $ref: ../bitcoin-multifile/fragments/…
❌ Validation error: contract_load_failed
   error: $ref '../bitcoin-multifile/fragments/sovereignty.yaml' at JSON pointer '/sovereignty' in …/eu-orders/contract.fluid.yaml escapes the ref root …/eu-orders …   # exit 1

$ FLUID_REF_ROOT=examples/0.7.1 fluid validate examples/0.7.1/eu-orders/contract.fluid.yaml
✅ Valid FLUID contract (schema v0.7.1)            # exit 0

Decisions for the reviewer

• The branch has two commits (bbd5aa1, then 4b0ee86, which fixes the review findings). Should they be squash-merged as one PR? The PR title and body are written for the combined change.
• The load-API PR is supposed to be additive, but if it adds an __all__ export line in fluid_build/loader.py it will touch the same file as this PR. This PR changes only _effective_ref_root and adds two module-level names next to it, which is a separate region, so a 3-way merge should resolve cleanly. The orchestrator should still check whether the load-API PR edits loader.py's __all__.
• The ref_root_env_ignored warning is logged once per (contract directory, value) per process, through _NOTED_REF_ROOT_ENV_IGNORED. A long-running service such as a downstream service backend sees a new temp directory for every upload, so it logs one WARNING per uploaded contract with a file $ref while FLUID_REF_ROOT is set. That is the intended signal, but operators who set the variable for a shared-fragments checkout will see it on every upload.
• Pre-existing bug found while tracing ref paths (out of scope, not fixed here). _contract_loader._auto_bundle_if_needed re-runs compile_contract(path) on the base contract whenever the loaded document still has any $ref — including a same-document #/... ref that load_with_overlay intentionally leaves in place. The result replaces the overlaid contract, so the CLI path silently drops --env overlays for such contracts. Reproduced: load_with_overlay('c.yaml','prod')['name'] gives 'prod', but _contract_loader.load_contract_with_overlay('c.yaml','prod', log)['name'] gives 'base'. The function also swallows compile errors at DEBUG level. Recommend a separate PR.
• Should there be a --ref-root CLI flag as well as FLUID_REF_ROOT? I left it out to stay inside the files this fix needs; adding it would touch validate/plan/apply/bundle argument parsing.
• Should fluid_build/util/ref_confinement.py be added to the [[tool.mypy.overrides]] strict = true hotspot allowlist? It passes mypy --strict today. I did not add it because that means editing the shared pyproject.toml.
• Overlay files are still loaded without resolving refs (unchanged). An overlay-introduced $ref stays a literal dict, so it is not an escape vector, but overlays cannot compose fragments.
• The forge_docs companion page is still to be written (CONTRIBUTING.md docs policy). The in-repo docs/contract-refs.md is ready to port.

Changes after review

These changes came from adversarial review rounds; each finding was upheld by at least 2 of 3 independent skeptics before it was fixed. A final re-review of the branch found nothing further.

R1 — 38d3224

Follow-up: an escape after an ignored FLUID_REF_ROOT explains itself

• Escape errors say when FLUID_REF_ROOT was ignored. If FLUID_REF_ROOT is not a directory, or does not contain the contract, the loader ignores it for that contract. The ref_root_env_ignored warning is logged only once per process. Before this change, the escape error that followed told the user to "set FLUID_REF_ROOT to that directory", even though it was already set. Repeat loads in a long-running process, such as a downstream service worker, gave that advice with no warning at all. Now every escape error in this case says FLUID_REF_ROOT is set but was ignored for this contract: <reason>. RefConfinementError also gains ignored_ref_root_env: the ignored value, or None. A library caller can check it without parsing the message.
• Docs. docs/contract-refs.md no longer says a FLUID_REF_ROOT left in the shell "never widens another contract". It now says the variable is ignored for a contract outside it, and that every contract inside it is widened. The upgrade note sets the variable for a single command (FLUID_REF_ROOT=... fluid validate ...) instead of using export.
• Tests. New tests load the same contract twice through load_contract, compile_contract and load_with_overlay, with the escape in a nested fragment. They check that both errors explain the ignored variable and that the warning is logged only once. The tests fail on the previous commit and against a mutant that explains only on the first load. A separate test checks that the generic hint is unchanged when the variable is unset or applied.

R2 — 7d2cd0c

Follow-up: a FLUID_REF_ROOT that cannot be resolved, and warning dedupe

A broken value now falls back too. Before this, the FLUID_REF_ROOT fallback only covered a value that resolved to a path that is not a directory, or to a directory that does not contain the contract. Three kinds of stale value raised before the fallback was reached:

• ~olduser/repo for a user that no longer exists raised RuntimeError.
• A symlink loop raised RuntimeError.
• A root under a directory the process cannot enter raised PermissionError from is_dir().

The raw exception failed every contract that has a file $ref. There was no ref_root_env_ignored warning, the error never named FLUID_REF_ROOT, and except RefResolutionError handlers did not catch it. ref_root= raised the same raw errors instead of the documented RefResolutionError.

Now the root is resolved and checked inside except (OSError, RuntimeError, ValueError). A failure becomes <origin>=<value> cannot be resolved: <reason> and takes the existing path:

• ref_root= raises RefResolutionError, naming ref_root.
• For FLUID_REF_ROOT, the value is ignored for this contract with the warning, the contract gets the default root, and every escape error says the variable was ignored and why.

The warning is now deduped on the value alone. The ref_root_env_ignored dedupe was keyed on (contract directory, value). A service that sets FLUID_REF_ROOT once and loads each upload from a fresh temp directory added one permanent entry and logged one warning per upload. It now logs once per value per process and holds one key per value. A later contract's escape errors still carry their own explanation.

docs/contract-refs.md is updated to match.

Tests. New parametrized tests cover an unknown ~user, a symlink loop and a chmod-000 parent (skipped on Windows and as root), through all three loader entry points, plus ref_root=. A five-upload test checks there is one warning and one dedupe key. All 10 new cases fail against the previous loader, with RuntimeError, PermissionError, or assert 5 == 1, and pass with this change on Python 3.10 and 3.12.

Tested live. With FLUID_REF_ROOT set to each of the three broken values, fluid validate examples/0.7.1/bitcoin-multifile/contract.fluid.yaml now prints the "cannot be resolved" warning and exits 0. Before this change it exited 1 with the raw error.

Documentation

Companion docs PR: Agenticstiger/forge_docs#139 (new pages for $ref confinement, the DuckDB sandbox and the contract-loading API, plus 0.18.0 release notes). It merges after 0.18.0 is on PyPI.

[github-actions]

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

• Link a docs PR — check the "Docs PR linked" box and paste the URL
• Mark as no docs needed — check "No docs needed" with a justification
• Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.