feat(conformance): close 206 uncovered constraints; coverage 18.8% → 40.7% - #45
Merged
Merged
Conversation
… 40.7% The first wave shipped 269 cases and 16.8% mutation coverage. Case count is not coverage: the question that matters is whether deleting a constraint from the schema turns anything red, and for most of the schema the answer was no. exposePolicy and semanticModel -- both governance-critical -- had no cases at all. Seven new groups, each written against a measured list of constraints that nothing pinned: expose-policy 14 cases 13/13 targets closed semantic-model 22 cases 21/21 orchestration 47 cases 51/56 acquisition 71 cases 69/69 lineage-ml 27 cases 26/26 binding-governance 18 cases 17/17 expose-mcp 11 cases 9/9 206 of 211 targeted constraints closed. The five left open are airflowTask `allOf` branches that are genuinely unreachable -- allOf[0]'s `then` carries only a description, so no document can make it fail -- and they are reported as unreachable rather than papered over with a case that proves nothing. Coverage 18.8% -> 40.7% (192 -> 415 of 1020 falsifiable constraints). The constraints that carry real meaning -- enums, required members, closed objects, bounds -- went from 248 unpinned to 38. What remains is dominated by `type` deletions, which need a wrong-typed value for every property in the schema and are individually low-value. Also in this change: * mutation_coverage.py learns to tell "uncovered" from "redundant". A constraint sitting beside an `enum` that already restricts the instance to values satisfying it is unfalsifiable -- no case could ever catch its removal. fluidVersion is the clearest example: `pattern`, `type: string` and an enum of three version strings, so two of the three are dead weight. Counting those as gaps blames the corpus for something it cannot fix. 120 such constraints are now excluded from the denominator and reported as schema findings instead. * envelope.json gains the four missing required-member cases (kind, id, name, exposes) and the empty-name case. The mutation report found those; the original corpus tested only two of the six required members going missing. Every group was written by one agent and then adversarially reviewed by another that independently re-measured the coverage claim rather than trusting it, checked every invalid case raises exactly one error, and confirmed that reversing only the declared defect makes the document valid. All seven came back sound with the claim confirmed. Verified in a clean venv built from scratch (Python 3.14, jsonschema[format] only): 480/480 cases, meta-test 19/19, all four compat pairs clean, and the reference implementation agrees on 478 with 0 disagreements. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #44, which shipped 269 cases at 16.8% mutation coverage. Case count is not coverage: the question that matters is whether deleting a constraint from the schema turns anything red, and for most of the schema the answer was no.
exposePolicyandsemanticModel— both governance-critical — had no cases at all.Seven new groups, written against measured gaps
Each agent was given the exact list of constraints that nothing pinned, not a vague brief.
expose-policysemantic-modelorchestrationacquisitionlineage-mlbinding-governanceexpose-mcp206 of 211 closed. The five left open are
airflowTaskallOfbranches that are genuinely unreachable —allOf[0]'sthencarries only adescription, so no document can make it fail. They are reported as unreachable rather than papered over with a case that proves nothing.The number
18.8% → 40.7% (192 → 415 of 1020 falsifiable constraints).
Better framed by what kind of constraint: the ones that carry real meaning — enums, required members, closed objects, bounds — went from 248 unpinned to 38. What remains is dominated by
typedeletions (567), which need a wrong-typed value for every property in the schema and are individually low-value.The metric got more honest, too
mutation_coverage.pynow distinguishes uncovered from redundant. A constraint sitting beside anenumthat already restricts the instance to values satisfying it is unfalsifiable — no case could ever catch its removal, so counting it as a gap blames the corpus for something it cannot fix.fluidVersionis the clearest example: it carriespattern,type: stringand anenumof three version strings. Two of the three are dead weight. 120 such constraints are now excluded from the denominator and reported as findings about the schema instead.envelope.jsonalso gains the four missing required-member cases (kind,id,name,exposes) plus an empty-namecase. The mutation report caught that the original corpus tested only two of the six required members going missing — a gap in the file I wrote by hand, found by the tool.How it was checked
Every group was written by one agent and then adversarially reviewed by a second that independently re-measured the coverage claim rather than trusting it — checking that every invalid case raises exactly one error, and that reversing only the declared defect makes the document valid. All seven came back sound with the author's claim confirmed. (That step exists because in the first wave a verifier caught an author reporting "18 passed, exit 0" for a file that actually failed.)
Tested
Clean venv built from scratch, Python 3.14,
jsonschema[format]only — not the dev environment:Notes
envelope.json,mutation_coverage.pyandtests/README.md. No schema changes, soschema-syncis unaffected;link-checkis path-filtered todocs/**.required_status_checksonmainis empty, so these gates are advisory at merge time.🤖 Generated with Claude Code