Skip to content

feat(conformance): close 206 uncovered constraints; coverage 18.8% → 40.7% - #45

Merged
fas89 merged 1 commit into
mainfrom
feat/conformance-corpus-wave2
Sep 7, 2026
Merged

fas89 merged 1 commit into
mainfrom
feat/conformance-corpus-wave2

Conversation

@fas89

@fas89 fas89 commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #44, which shipped 269 cases at 16.8% mutation coverage. Case count is not coverage: the question that matters is whether deleting a constraint from the schema turns anything red, and for most of the schema the answer was no. exposePolicy and semanticModel — both governance-critical — had no cases at all.

Seven new groups, written against measured gaps

Each agent was given the exact list of constraints that nothing pinned, not a vague brief.

Group Cases Targets closed
expose-policy 14 13 / 13
semantic-model 22 21 / 21
orchestration 47 51 / 56
acquisition 71 69 / 69
lineage-ml 27 26 / 26
binding-governance 18 17 / 17
expose-mcp 11 9 / 9

206 of 211 closed. The five left open are airflowTask allOf branches that are genuinely unreachable — allOf[0]'s then carries only a description, so no document can make it fail. They are reported as unreachable rather than papered over with a case that proves nothing.

The number

18.8% → 40.7% (192 → 415 of 1020 falsifiable constraints).

Better framed by what kind of constraint: the ones that carry real meaning — enums, required members, closed objects, bounds — went from 248 unpinned to 38. What remains is dominated by type deletions (567), which need a wrong-typed value for every property in the schema and are individually low-value.

The metric got more honest, too

mutation_coverage.py now distinguishes uncovered from redundant. A constraint sitting beside an enum that already restricts the instance to values satisfying it is unfalsifiable — no case could ever catch its removal, so counting it as a gap blames the corpus for something it cannot fix.

fluidVersion is the clearest example: it carries pattern, type: string and an enum of three version strings. Two of the three are dead weight. 120 such constraints are now excluded from the denominator and reported as findings about the schema instead.

envelope.json also gains the four missing required-member cases (kind, id, name, exposes) plus an empty-name case. The mutation report caught that the original corpus tested only two of the six required members going missing — a gap in the file I wrote by hand, found by the tool.

How it was checked

Every group was written by one agent and then adversarially reviewed by a second that independently re-measured the coverage claim rather than trusting it — checking that every invalid case raises exactly one error, and that reversing only the declared defect makes the document valid. All seven came back sound with the author's claim confirmed. (That step exists because in the first wave a verifier caught an author reporting "18 passed, exit 0" for a file that actually failed.)

Tested

Clean venv built from scratch, Python 3.14, jsonschema[format] only — not the dev environment:

corpus                 exit=0   480 cases, 480 passed
meta-test              exit=0   19/19 checks
compat 0.7.1 -> 0.7.2  exit=0
compat 0.7.2 -> 0.7.3  exit=0
compat 0.7.3 -> 0.7.4  exit=0
compat 0.7.4 -> 0.7.5  exit=0
reference impl         478 agree, 0 disagree, 2 optional not implemented

Notes

  • Additive: seven new files, plus envelope.json, mutation_coverage.py and tests/README.md. No schema changes, so schema-sync is unaffected; link-check is path-filtered to docs/**.
  • No local paths, credentials or real email addresses in the 211 newly published vectors.
  • Still worth doing separately: required_status_checks on main is empty, so these gates are advisory at merge time.

🤖 Generated with Claude Code

… 40.7%

The first wave shipped 269 cases and 16.8% mutation coverage. Case count is
not coverage: the question that matters is whether deleting a constraint from
the schema turns anything red, and for most of the schema the answer was no.
exposePolicy and semanticModel -- both governance-critical -- had no cases at
all.

Seven new groups, each written against a measured list of constraints that
nothing pinned:

  expose-policy       14 cases   13/13 targets closed
  semantic-model      22 cases   21/21
  orchestration       47 cases   51/56
  acquisition         71 cases   69/69
  lineage-ml          27 cases   26/26
  binding-governance  18 cases   17/17
  expose-mcp          11 cases    9/9

206 of 211 targeted constraints closed. The five left open are airflowTask
`allOf` branches that are genuinely unreachable -- allOf[0]'s `then` carries
only a description, so no document can make it fail -- and they are reported
as unreachable rather than papered over with a case that proves nothing.

Coverage 18.8% -> 40.7% (192 -> 415 of 1020 falsifiable constraints). The
constraints that carry real meaning -- enums, required members, closed
objects, bounds -- went from 248 unpinned to 38. What remains is dominated by
`type` deletions, which need a wrong-typed value for every property in the
schema and are individually low-value.

Also in this change:

* mutation_coverage.py learns to tell "uncovered" from "redundant". A
  constraint sitting beside an `enum` that already restricts the instance to
  values satisfying it is unfalsifiable -- no case could ever catch its
  removal. fluidVersion is the clearest example: `pattern`, `type: string` and
  an enum of three version strings, so two of the three are dead weight.
  Counting those as gaps blames the corpus for something it cannot fix. 120
  such constraints are now excluded from the denominator and reported as
  schema findings instead.

* envelope.json gains the four missing required-member cases (kind, id, name,
  exposes) and the empty-name case. The mutation report found those; the
  original corpus tested only two of the six required members going missing.

Every group was written by one agent and then adversarially reviewed by
another that independently re-measured the coverage claim rather than trusting
it, checked every invalid case raises exactly one error, and confirmed that
reversing only the declared defect makes the document valid. All seven came
back sound with the claim confirmed.

Verified in a clean venv built from scratch (Python 3.14, jsonschema[format]
only): 480/480 cases, meta-test 19/19, all four compat pairs clean, and the
reference implementation agrees on 478 with 0 disagreements.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fas89
fas89 merged commit 9da53c7 into main Sep 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant