feat(conformance): make FLUID's claims executable

[fas89]

FLUID is published as "the open, declarative standard for Data Products". Until this PR it shipped twelve schema files, zero tests, zero conformance vectors, and a backward-compatibility promise — repeated in the README and every release note — that nothing checked. Its sibling FLUX already ships enforcement vectors, a regression suite and a meta-test. This closes that gap on the flagship.

It is not only about third parties. The FLUID contract already has more than one first-party reader — FluidSchemaManager in forge-cli, the FLUX engine's seam/ (which resolves and compiles a referenced contract with its own type parser and its own FLX-#### codes), and the Command Center — and nothing has ever proved they agree about what a contract means. A shared corpus is how they get held to one answer.

What's here

Path	What it is
tests/0.7.5/	267 vectors across 11 groups
tests/optional/0.7.5/	2 vectors for behaviour the spec permits but does not require
conformance/run.py	The runner — --version, --group, --junit, --list
conformance/check_reference.py	Runs the corpus against forge-cli: implementation #1 under test, never the referee
conformance/mutation_coverage.py	Deletes one schema constraint at a time, reports which no case would catch
conformance/known-failures.txt	Sorted failure list; listed cases still execute
scripts/check-compat.py	The compatibility gate: static narrowing diff + empirical replay
scripts/compat-waivers.txt	Three waivers, each with the evidence that settled it
tests/meta_test.py	19 checks proving both gates can actually fail
.github/workflows/conformance.yml	meta → conformance + compat

Assertions are on JSON Schema keyword + RFC 6901 pointer, never message text, so the corpus ports to ajv, check-jsonschema, or an implementation in a language nobody has written yet. An invalid case that declares no reason is refused by the loader — it would pass for any reason at all.

What the gates found on their first run

0.7.2 is not backward compatible with 0.7.1, contrary to docs/releases/0.7.2.md ("100% backward compatible", "No breaking changes"). $defs/notification gained additionalProperties: false, so a contract carrying any extra member on a build.execution.notifications[] entry validated under 0.7.1 and is rejected by 0.7.2.

Proven with a document, plus a negative control isolating the cause:

build.execution.notifications[0].retryOnFailure
  0.7.1: VALID     0.7.2: REJECTED (additionalProperties)
same document without that member
  0.7.1: VALID     0.7.2: VALID

Waived, because it cannot be un-shipped — but the waiver records it as a real break, and the release note still needs correcting. The other two flags were checked and are not bugs: the identifier pattern change is a strict widening (verified exhaustively over 11,110 generated strings, zero counterexamples) and the fluidVersion enum is the version gate working as intended.

FLUID never says whether format is assertive. JSON Schema 2020-12 makes it an annotation by default and the reference implementation does not assert it — so a contract can validate with a malformed metadata.owner.email. The core tier follows the weak reading, format cases live in optional/, and the open question is written into tests/README.md rather than silently assumed. This wants a decision.

forge-cli validates a 2020-12 schema with Draft7Validator. Latent today — the schemas use only $defs from the 2020-12-only keyword set, which resolves under Draft 7 as ordinary JSON pointers. It becomes a silent-acceptance bug the moment a schema uses prefixItems, unevaluatedProperties, dependentRequired, minContains or maxContains.

The number to hold this to

Mutation coverage is 16.8% — of 1,140 constraints in the 0.7.5 schema, only 191 have a case that goes red if you delete them. 269 cases sound like more coverage than they buy. Most of the gap is type constraints (701), but 84 enums, 67 closed objects and 56 required members are unpinned, and exposePolicy and semanticModel have no cases at all. That is the honest baseline to ratchet up, not a result to celebrate.

Tested

Every gate run in a clean venv built from scratch (Python 3.14, jsonschema[format]>=4.22 only), not just the dev environment:

meta-test              exit=0   (19/19 checks)
corpus                 exit=0   (269 cases, 269 passed)
compat 0.7.1 -> 0.7.2  exit=0
compat 0.7.2 -> 0.7.3  exit=0
compat 0.7.3 -> 0.7.4  exit=0
compat 0.7.4 -> 0.7.5  exit=0
reference impl         263 agree, 0 disagree, 2 optional not implemented

That clean-venv run caught a real "passes locally, fails CI" bug before push: jsonschema only registers a date-time checker when rfc3339-validator is present, which the dev environment had transitively and CI would not. Reproduced red with bare jsonschema, fixed by requiring the [format] extra, re-verified green. Hence the extra is pinned in all three jobs, with a comment saying why.

Notes for review

• Purely additive — every file is new except a Python section added to .gitignore. No overlap with the two open PRs (Add FLUID schema 0.7.2 (Semantic Truth Engine) #21 touches schema/, specs/, schema-diffs/; Enhance FLUID Documentation and Structure for Better Accessibility and Adoption #1 touches root docs). Nothing here modifies schema/, so schema-sync is unaffected, and link-check is path-filtered to docs/**.
• The conformance job installs data-product-forge unpinned and is allowed to fail the build if the engine disagrees with the corpus. That is the point of the check, not an oversight.
• Security-reviewed as a public-repo publish: no local paths, usernames, credentials or real emails in the 269 published vectors; workflow is pull_request (not pull_request_target) with contents: read, no secrets and no ${{ }} interpolation; meta_test.py was confirmed non-destructive by hashing all 1,690 repo files before and after a run.

Borrowed

Corpus shape and core/optional split from JSON-Schema-Test-Suite (MIT) · failure-list file from protobuf (BSD-3-Clause) · still-execute-known-failures from connectrpc/conformance (Apache-2.0) · expected-error-code assertions from w3c/json-ld-api · vectors envelope and meta-test discipline from flux-spec. Reasoning is recorded in each file's docstring.

🤖 Generated with Claude Code