Skip to content

Make generated filter schemas a narrowing again - #10

Merged
christosgkoros merged 1 commit into
mainfrom
fix/generated-schema-dependency-keywords
Sep 10, 2026
Merged

christosgkoros merged 1 commit into
mainfrom
fix/generated-schema-dependency-keywords

Conversation

@christosgkoros

Copy link
Copy Markdown
Collaborator

Fixes #8.

The leak

tools/generate-filter-schema.mjs restated the published constraint object's dependency rules instead of reading them, and had one of the two: dependentRequired ($flags → $regex) was carried, dependentSchemas ($unknownAs needs something to modify) was not. So the generated schema accepted a filter the grammar rejects:

{"microchip":{"$unknownAs":false}}   grammar: false   generated: true

That is the one direction generation may never go. A lone $unknownAs has no predicate under it, so a server on the documented path — generated schema as the MCP inputSchema, published semantics behind it — had nothing to compile: experiments/filter-to-sql emitted SELECT id FROM pets WHERE coalesce((), FALSE) and SQLite answered with a syntax error, i.e. a 500 for a filter the tool's own schema had declared valid.

The fix

  • tools/generate-filter-schema.mjs — both dependency keywords are now read off $defs/ConstraintObject and carried onto every generated constraint object whose trigger operator survived, so a rule added to the published definition reaches generated schemas with the version that introduces it. The $comment justifying a rule is deliberately not copied: a validator never reads it, and one copy per field is charged by the token to whoever inlines the schema in a tool definition.
  • examples/pet.filter.json — regenerated. 12 of its 18 constraint objects gain the rule: exactly the fields that can be absent or null, which are the ones offering $unknownAs.
  • No change to query-language-schema.json. The grammar was right; the generator disagreed with it.

Testing the property instead of listing cases

tests/generator.test.mjs asserted narrowing over a hand-written list of fifteen filters. The property is quantified over every filter, so a list can only re-check the leaks someone already thought of — and it did not contain this one, for as long as it existed.

  • tests/fuzz.mjs (new) samples filters out of a generated schema's own vocabulary: mostly conformant, because a random JSON value is rejected by both schemas and tells you nothing, but deliberately blind to the keywords that bound the shape of a value rather than its type — minProperties, minItems/maxItems, uniqueItems, and the dependency keywords. Seeded, so a failure reproduces.
  • The soundness test runs 5000 samples against three generated schemas (pet, pet with core only, a recursive resource) and asserts every filter the generated schema accepts is valid JQL. It also asserts the run was not vacuous: a fraction of samples must be accepted, and every operator the schema offers must be reached by an accepted one.
  • Against the schema this PR replaces, that test reports 205 leaks; against the fixed one, 0.
  • A second test pins what the generator does with each instance-constraining keyword of $defs/ConstraintObject, so a keyword added there fails the suite until it is handled deliberately. This is what the issue asked be checked — dependentSchemas was the only one missing.

Downstream

experiments/filter-to-sql now rejects a constraint object with no predicate in it as malformed-query, with a pointer at the clause, rather than compiling an empty expression. Both schemas reject those now, so this only matters for a compiler reached another way — but the failure it replaces was a database syntax error. Case f05b covers it, and the counts in the experiment's README are brought back in line with what --metrics prints (some were already stale by one corpus).

npm test — 119 pass. npm run test:experiment — 152 pass. npm run example:mcp:demo unchanged: the server reads pet.filter.json at startup, so it picks the rule up.

Generated with Claude Code

The generator restated the published constraint object's dependency rules
instead of reading them, and had only one of the two: $flags → $regex was
carried, $unknownAs → "needs something to modify" was not. So a generated
schema accepted {"microchip": {"$unknownAs": false}} — a modifier with no
predicate under it — where query-language-schema.json rejects it, which is
the one direction generation may never go. A server on the documented path
then had to evaluate a filter with nothing in it; the SQL compiler emitted
coalesce((), FALSE) and the database answered 500 to a filter its own tool
schema had declared valid.

Both keywords now come off $defs/ConstraintObject, so a rule added there
arrives in generated schemas with the version that introduces it. The
$comment is left behind: a validator never reads it, and one copy per field
is charged by the token to whoever inlines the schema in a tool definition.

The property this breaks is universally quantified, and it sat under a list
of fifteen filters — which can only re-check the leaks someone already
thought of, and did not contain this one. tests/fuzz.mjs samples filters out
of a generated schema's own vocabulary, deliberately blind to the keywords
that bound the shape of a value rather than its type, and the soundness test
now runs 5000 samples against three generated schemas: every one the
generated schema accepts must be valid JQL. It also asserts the run was not
vacuous — a fraction of samples accepted, every offered operator reached.
Against the schema this commit replaces it reports 205 leaks. A second test
pins what the generator does with each instance-constraining keyword of
$defs/ConstraintObject, so adding one there fails the suite until it is
handled deliberately.

experiments/filter-to-sql rejects a constraint object with no predicate in
it as malformed-query, pointing at the clause, rather than compiling an
empty expression. Both schemas reject those now, so this only matters for a
compiler reached another way — but the failure it replaces was a database
syntax error.

Fixes #8
@christosgkoros
christosgkoros merged commit ac18328 into main Sep 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generated filter schemas are not a narrowing: a lone $unknownAs is accepted, and compiles to invalid SQL

1 participant