From 4509e1f84dc0390958b2953e96c65ce24dac08c3 Mon Sep 17 00:00:00 2001 From: Christos Gkoros Date: Sun, 13 Sep 2026 14:20:54 +0300 Subject: [PATCH 1/2] Let a provider select the operators it serves, and ship the generator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The generator narrowed the value side of a filter well and left the feature side all-or-nothing per profile. Internally it was already operator-granular — every emission site gates on ctx.enabled — but the only thing that could write to that set was the union of whole profiles, so a backend with LIKE and no POSITION could not offer $like without also advertising $contains, and a store with no notion of key absence could not decline $exists at all. Six knobs reach the set and the emitted shape, each a flag and a JS API option: --operators, --drop-operators, --no-shorthand, --max-filter-depth, --limits, and --config to hold the combination in a file checked in beside the resource schema. What a provider declines is absent from the generated schema, so a client learns it from validation rather than from a runtime error — or worse, from an empty result set. --max-filter-depth is the one that restructures the output. JSON Schema cannot count how deep an instance already is, so a bounded filter is emitted as a chain: level i offers the logical operators over level i+1 and the last offers none. Levels share the operand $defs, so the cost is n copies of a map of $refs. Unbounded stays the single self-referential object it has always been, and every default is byte-identical to before. The capability document stops echoing the request. SPEC §2.1 makes a partial profile one an implementation may not advertise, so `profiles` now lists only the profiles the final operator set covers in full, the per-field `operators` lists carry the rest, and the demotion is warned about on stderr naming the operator responsible. Declining a core operator says outright that the result is not conforming. §2.2 gains a table for the top-level members, two of which the generator already emitted and nothing described. The tool also travels with the package for the first time. tools/ was not in package.json `files` and there was no bin entry, so the generator the README points readers at could not leave the repository. It is now a bin named jql-generate — reachable from a clone or a git install, since this repository still publishes nothing to a registry, but present in `npm pack` and ready for the day it does. Two bugs surfaced on the way. --include pruned only the root filter, so with a depth cap every excluded path walked back in one $and down. And an operator whose dependency was dropped stayed: --drop-operators $regex left $flags with a dependentRequired rule naming a member additionalProperties: false forbids. The closure is read off $defs/ConstraintObject so a later dependency is handled by construction. --max-depth also stopped coercing a non-number to NaN in silence. The narrowing property is what covers the new shapes: the seeded sampler in tests/fuzz.mjs now runs over dropped operators, no-shorthand and a capped depth, because a list of filters can only re-check the leaks someone already thought of. Closes #11. Co-Authored-By: Claude --- CHANGELOG.md | 59 ++++ README.md | 47 ++- SPEC.md | 16 +- examples/pet.jql.config.json | 9 + package.json | 9 +- tests/generator.test.mjs | 226 ++++++++++++++- tools/generate-filter-schema.mjs | 476 +++++++++++++++++++++++++------ 7 files changed, 750 insertions(+), 92 deletions(-) create mode 100644 examples/pet.jql.config.json mode change 100644 => 100755 tools/generate-filter-schema.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 69e199a..5cd978e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,49 @@ No change to the grammar or to the semantics of evaluation. The only edit to ### Added +- **The generator selects capabilities, not just profiles** + ([#11](https://github.com/christosgkoros/json-query-language/issues/11)). Profiles are the unit + a server advertises, but three shapes do not fit inside one: a backend with `LIKE` and no + `POSITION` supports `$like` and not `$contains`; a key-value store cannot implement `$exists` at + all; a provider compiling to a flat conjunctive index wants one AND level and no shorthand. Six + new knobs, each available as a flag and as a JS API option — `--operators`, `--drop-operators`, + `--no-shorthand`, `--max-filter-depth`, `--limits`, and `--config` to hold the combination. What + is declined is absent from the generated schema, so a client learns it from validation rather + than from an `unsupported-operator` at runtime. Defaults are unchanged: with none of them given + the output is byte-identical to before. +- **`--config `, and `examples/pet.jql.config.json`.** The capability selection is a + decision about the endpoint, not a shell invocation, so it goes in a JSON file checked in beside + the resource schema and regenerated from. Its keys are the JS API's option names plus `resource`, + `out` and `capabilities`; relative paths in it resolve against its own directory, an explicit + flag beats it, and an unrecognised key is refused rather than ignored — a misspelled key is a + capability that silently did not apply. `npm run generate:example` now runs through one. +- **`--max-filter-depth `** caps how deep `$and`/`$or`/`$nor`/`$not` may nest: `1` is a flat + filter offering no logical operators at all, `2` permits one level of them. JSON Schema + cannot count how deep an instance already is, so the filter is emitted as a chain of levels: + level *i* offers the logical operators over level *i+1* and the last level does not offer them + at all. Every level shares the operand `$defs`, so the cost is *n* copies of a map of `$ref`s. + Field-level `$not` is bounded to a single application by the same flag — under Kleene logic + `¬¬X ≡ X` even for UNKNOWN, so a negated negation says nothing the plain constraint does not. +- **`--limits `** puts the SPEC §7 numbers a provider actually enforces into the + capability document. They were emitted unconditionally, so every document generated from the CLI + claimed `maxDepth: 10, maxClauses: 100, maxSetLength: 1000` whether or not that was true. Where + `--max-filter-depth` is given and `maxDepth` is not, the enforced bound is published. +- **The generator is part of the package.** `tools/` was not in `package.json` `files` and there + was no `bin` entry, so the tool the README points readers at could not travel with the package + at all. It is now a `bin` named `jql-generate`, with `json-query-language/generate` exporting + `generateFilterSchema` for programmatic use. This repository still publishes no artifacts + ([RELEASING.md](./RELEASING.md)), so the command is reachable from a clone or a git install and + not from npmjs; what changed is that it is ready to be, and `npm pack` now contains it. +- **[SPEC.md §2.2](./SPEC.md#22-capability-discovery) documents the capability document's + top-level members** — `queryLanguage`, `profiles`, `fields`, `limits` and `filterSchema` — in a + table beside the existing per-field one. `limits` appeared in the example and in no table, and + `filterSchema`, `itemValues` and `nullable` were emitted by the generator and described nowhere. + No normative change to what the members mean. +- **[SPEC.md §2.1](./SPEC.md#21-profiles) says what a partial profile may and may not do.** The + rule was already there — a profile other than `core` is implemented in full or not at all — but + it read as a prohibition on the implementation rather than on the advertisement. An endpoint + accepting part of a profile is not prohibited from existing; it states what it accepts per path + and omits the incomplete profile from `profiles`. - **`examples/mcp-server/`** — a runnable MCP server whose one tool, `search_pets`, takes a filter as its `filter` argument and nothing else. The tool's `inputSchema` is `examples/pet.filter.json` inlined verbatim; validation is ajv against that same file, and @@ -27,6 +70,14 @@ No change to the grammar or to the semantics of evaluation. The only edit to ### Changed +- **The capability document's `profiles` reports coverage rather than the request.** It echoed + whatever `--profiles` was given; it now lists only the profiles the final operator set covers in + full, because [SPEC.md §2.1](./SPEC.md#21-profiles) makes a partial profile one an + implementation may not advertise. `--drop-operators '$contains'` therefore costs the `strings` + claim, and the per-field `operators` lists carry what is on offer instead — with a warning on + stderr naming the operator responsible. Declining a `core` operator drops `core` too, and warns + that the result is not a conforming implementation. Nothing changes for a selection that is + whole profiles, which is every invocation before this release. - **Positioned as one JSON-Schema-described query language with two integration points**, rather than as an agent interface. An earlier revision in this same unreleased window led with the MCP tool definition and moved §*Exposing search to an agent* ahead of the OpenAPI and generator @@ -51,6 +102,14 @@ No change to the grammar or to the semantics of evaluation. The only edit to ### Fixed +- **`--max-depth` accepted a value that was not a number.** It was coerced with `Number()` and + never checked, so `--max-depth deep` became `NaN` and silently stopped the walk at the first + nested object. It and `--max-filter-depth` are both validated now. +- **An operator whose dependency was dropped is dropped with it.** `$flags` carries + `dependentRequired: ["$regex"]` out of the grammar, so `--drop-operators '$regex'` would have + left `$flags` in `properties` with a rule naming a member `additionalProperties: false` forbids + — present in the schema and impossible to use. The closure is read off + `$defs/ConstraintObject`, so a dependency added later is handled by construction. - **Generated filter schemas were not a narrowing** ([#8](https://github.com/christosgkoros/json-query-language/issues/8)). `tools/generate-filter-schema.mjs` carried the published constraint object's `dependentRequired` rule but not its `dependentSchemas` one, so `{"microchip": {"$unknownAs": diff --git a/README.md b/README.md index 51fb88d..7607f62 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ One schema, two integration points, because JSON Schema is what both already spe - **Semantics** — [`SPEC.md`](./SPEC.md) — nulls, paths, coercion, errors, limits - **MCP server** — [`examples/mcp-server/`](./examples/mcp-server) — a search tool with the language as its `inputSchema`, runnable - **OpenAPI documents** — [`examples/`](./examples) — working 3.1 (`POST /…/search`) and 3.2 (`QUERY`) integrations -- **Generator** — [`tools/generate-filter-schema.mjs`](./tools/generate-filter-schema.mjs) — turns a resource's JSON Schema into a per-field filter schema +- **Generator** — [`tools/generate-filter-schema.mjs`](./tools/generate-filter-schema.mjs) — turns a resource's JSON Schema, plus the slice of the language you can serve, into a per-field filter schema - **Compared with GraphQL** — [`COMPARISON.md`](./COMPARISON.md) — what this overlaps with, what it does not, and what a JSON-Schema-native alternative would still need - **Version** — `0.4.0`, and the schema's `$id` names `v0.4.0`. This release is **breaking**: it replaces two array mechanisms with one quantifier family and adds `$unknownAs`. See [`CHANGELOG.md`](./CHANGELOG.md) for the migration, and [`decisions/0001`](./decisions/0001-array-quantifiers-and-unknown-handling.md) for why. @@ -323,7 +323,43 @@ What it decides, and why: - **Prose comes from the grammar**, not from the generator, so operator descriptions stay in one place. `--descriptions brief` (the default) keeps them for the operators people get wrong and drops them for `$eq` and `$gt`, which matters when the output goes into an MCP tool definition. - **Narrowing only.** Every filter the generated schema accepts is also valid against the published grammar, so a server implementing the published semantics evaluates it unchanged. The rule that makes this hold: the constraint object's own keywords are copied from the grammar, and only its *operator set* and *operand schemas* are narrowed. `tests/generator.test.mjs` asserts the property by sampling filters out of each generated schema's vocabulary — a list of examples can only re-check the leaks someone already thought of. -Opt a property out, or override its operators, from the resource schema itself: +### Selecting what you support + +Everything above narrows the *value* side of a filter. The other half is the *feature* side: which operators, and which parts of the grammar, your backend can actually honour. Whatever you decline here the generated schema refuses, so a client finds out from validation rather than from a runtime error — or worse, from an empty result set. + +Profiles are the coarse unit, and three shapes do not fit inside one: + +| Your situation | Say | +| --- | --- | +| `LIKE` but no `POSITION`, so `$like` works and `$contains` does not | `--drop-operators '$contains'` | +| A key-value store with no notion of key absence, so `$exists` is unimplementable | `--drop-operators '$exists'` | +| A flat conjunctive index: one AND level and no shorthand | `--max-filter-depth 2 --no-shorthand` | +| A single-clause lookup: no logical operators at all | `--max-filter-depth 1` | + +`--operators` takes the other direction — exactly these, intersected with `--profiles` — and `--limits` replaces SPEC §7's defaults with your real bounds in the capability document. + +The combinations get long, and they are not a thing to retype, so the same selection goes in a file you check in beside the resource schema and regenerate from: + +```json +{ + "resource": "pet.schema.json", + "id": "https://api.example.com/schemas/pet.filter.json", + "out": "pet.filter.json", + "capabilities": "pet.capabilities.json", + "profiles": ["core", "strings"], + "dropOperators": ["$contains", "$exists"], + "shorthand": false, + "limits": { "maxDepth": 4, "maxClauses": 40, "maxSetLength": 100 } +} +``` + +```bash +node tools/generate-filter-schema.mjs --config jql.config.json # paths resolve against the config file +``` + +One consequence worth knowing before you use it. [SPEC §2.1](./SPEC.md#21-profiles) says a profile other than `core` is implemented in full or not at all, so a profile you have narrowed is no longer one you can advertise: it drops out of the capability document's `profiles`, the per-field `operators` lists carry what you do offer, and the generator says on stderr which operator cost you the claim. Declining a `core` operator costs conformance outright, and it says that too. The *schema* stays legal either way — it accepts strictly fewer filters than the published grammar, which is the only rule generation has. + +Opt a single property out, or override its operators, from the resource schema itself: ```json { "internalNotes": { "type": "string", "x-jql": false } } @@ -332,6 +368,8 @@ Opt a property out, or override its operators, from the resource schema itself: `--include`, `--exclude`, `--max-depth` and `--pointer` do the rest. Run `--help` for the full list. +The tool is part of the package and is installed as `jql-generate`, so once the package is published it is `npx jql-generate` rather than a path. It is not published yet — see [Status](#status) — so today it is either a clone or `npm install github:christosgkoros/json-query-language`. + ## Exposing search to an agent The other integration path, and a working server for it lives in [`examples/mcp-server/`](./examples/mcp-server) — one tool, real records, `node examples/mcp-server/demo.mjs` to watch it answer and reject. @@ -389,10 +427,10 @@ query-language-schema.json the schema — the only file you need to consume SPEC.md normative semantics RELEASING.md how a release is cut (no artifacts are published) COMPARISON.md how this relates to GraphQL, OData and JSON:API -tools/generate-filter-schema.mjs resource schema -> per-field filter schema + capabilities +tools/generate-filter-schema.mjs resource schema + capability selection -> filter schema + capabilities examples/mcp-server/ a runnable MCP server; the language as a tool's inputSchema examples/ working OpenAPI 3.1 and 3.2 documents -examples/pet.schema.json the generator's input, and its committed output beside it +examples/pet.schema.json the generator's input, its config, and its committed output beside them tests/validate.test.mjs meta-validation + fixture runner tests/generator.test.mjs the generator: narrowing, soundness, recursion tests/fixtures/valid/ one per operator; also the docs' example set @@ -417,6 +455,7 @@ experiments/filter-to-sql/ an exercise: compile a filter to SQL, then judge | `$id` / `$ref` — `https://christosgkoros.com/json/query-language/v0.4.0/query-language-schema.json` | Does not resolve. Used throughout [Using it from OpenAPI](#using-it-from-openapi) and in the capability document examples. | | The package names `json-query-language` and `@christosgkoros/json-query-language` | Not published, to npmjs or to GitHub Packages, and the release pipeline no longer tries to. Claiming a name under a working title would burn it. | | The version line at the top, and the version inside the `$id` | May lag the latest tag. `CHANGELOG.md` is authoritative. | +| `npx jql-generate` | Not reachable from a registry, for the reason in the row above. The `bin` entry is real and the tool ships inside the package, so this works from a clone or a git install; it does not work from npmjs. | These will be fixed in one pass once the name is fixed, because fixing them before then means doing it twice. Until then the only fetchable copy of the schema is raw GitHub: diff --git a/SPEC.md b/SPEC.md index 2bda37b..5ba4dbe 100644 --- a/SPEC.md +++ b/SPEC.md @@ -41,6 +41,8 @@ Operators are grouped into profiles so that a server can implement a subset hone A conforming implementation MUST implement `core` in full. Every other profile is OPTIONAL, and MUST be implemented in full or not at all — partial profiles defeat the purpose of advertising them. +An implementation that accepts only part of a profile is not prohibited from existing; it is prohibited from *advertising* that profile. It states what it accepts per path instead, through the `operators` member of §2.2, and omits the incomplete profile from `profiles`. + An implementation MUST reject an operator it does not support with an `unsupported-operator` error (§8). It MUST NOT silently ignore the clause: dropping a predicate from a filter widens the result set, which is the most dangerous possible failure mode for an authorization-adjacent filter. ### 2.2 Capability discovery @@ -72,14 +74,26 @@ An implementation SHOULD publish which profiles and fields it accepts. This spec } ``` +The top-level members: + +| Member | Required | Meaning | +| --- | --- | --- | +| `queryLanguage` | SHOULD | The `$id` of the grammar version this endpoint implements. | +| `profiles` | yes | The profiles implemented **in full**, per §2.1. An endpoint that accepts only part of a profile MUST NOT list it here; the per-path `operators` are where the accepted subset is stated. A client MUST therefore read `operators` rather than deriving a path's operators from `profiles`. | +| `fields` | yes | One member per queryable path, described below. | +| `limits` | SHOULD | The bounds of §7 as this endpoint enforces them. Absent means the §7 defaults apply. | +| `filterSchema` | — | The URL of a per-resource filter schema (§6) carrying the same operators and domains, where one is published. | + Each member of `fields` describes one queryable path: | Member | Required | Meaning | | --- | --- | --- | -| `operators` | yes | The operators accepted on this path. A subset of those implied by `profiles`. | +| `operators` | yes | The operators accepted on this path. A subset of those implied by `profiles`, and the authoritative list where a profile is only partly implemented. | | `type` | SHOULD | The JSON type of the field's value, drawn from the `$type` vocabulary of §5.10. | | `format` | — | A format name constraining a `type: "string"` value: `date`, `date-time`, `uuid`, and so on. | | `values` | SHOULD, where the domain is closed | The complete set of accepted values. | +| `itemValues` | SHOULD, where an array's element domain is closed | The complete set of accepted element values. | +| `nullable` | — | Present and `true` where the field's value may be `null`, which is what makes `$isNull` and `$unknownAs` meaningful on it (§4.6). | | `description` | — | Prose stating what the field holds. | `operators` tells a client what it may write; `type`, `format` and `values` tell it *what to write*. The grammar cannot carry that second half: field names are constrained through `propertyNames`, but every path shares one `Constraint` definition, so per-field operand domains are not expressible in the schema (§6). A filter naming a real field with a value outside that field's domain is therefore well-formed, and matches nothing — the failure surfaces as an empty result set rather than an error (§8). The capability document is the only place the domain can be stated, which is why `values` is RECOMMENDED wherever the domain is closed. diff --git a/examples/pet.jql.config.json b/examples/pet.jql.config.json new file mode 100644 index 0000000..4ba325e --- /dev/null +++ b/examples/pet.jql.config.json @@ -0,0 +1,9 @@ +{ + "resource": "pet.schema.json", + "id": "https://api.example.com/schemas/pet.filter.json", + "out": "pet.filter.json", + "capabilities": "pet.capabilities.json", + "profiles": ["core", "strings", "ranges", "collections"], + "descriptions": "brief", + "limits": { "maxDepth": 10, "maxClauses": 100, "maxSetLength": 1000 } +} diff --git a/package.json b/package.json index 4deaa1e..8bd298a 100644 --- a/package.json +++ b/package.json @@ -26,12 +26,17 @@ "http-query-method" ], "main": "query-language-schema.json", + "bin": { + "jql-generate": "tools/generate-filter-schema.mjs" + }, "exports": { ".": "./query-language-schema.json", - "./query-language-schema.json": "./query-language-schema.json" + "./query-language-schema.json": "./query-language-schema.json", + "./generate": "./tools/generate-filter-schema.mjs" }, "files": [ "query-language-schema.json", + "tools/", "SPEC.md", "README.md", "CHANGELOG.md" @@ -39,7 +44,7 @@ "scripts": { "test": "node --test tests/*.test.mjs", "generate": "node tools/generate-filter-schema.mjs", - "generate:example": "node tools/generate-filter-schema.mjs examples/pet.schema.json --id https://api.example.com/schemas/pet.filter.json --out examples/pet.filter.json --capabilities examples/pet.capabilities.json", + "generate:example": "node tools/generate-filter-schema.mjs --config examples/pet.jql.config.json", "example:mcp": "node examples/mcp-server/server.mjs", "example:mcp:demo": "node examples/mcp-server/demo.mjs", "test:experiment": "node --test experiments/filter-to-sql/compile.test.mjs", diff --git a/tests/generator.test.mjs b/tests/generator.test.mjs index 7ab7987..0f4b1f4 100644 --- a/tests/generator.test.mjs +++ b/tests/generator.test.mjs @@ -1,13 +1,14 @@ import test from "node:test"; import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; import { fileURLToPath } from "node:url"; import { dirname, join } from "node:path"; import _Ajv2020 from "ajv/dist/2020.js"; import _addFormats from "ajv-formats"; -import { generateFilterSchema } from "../tools/generate-filter-schema.mjs"; +import { generateFilterSchema, resolveOptions } from "../tools/generate-filter-schema.mjs"; import { sampler, operatorsIn, operatorsUsed } from "./fuzz.mjs"; const Ajv2020 = _Ajv2020.default ?? _Ajv2020; @@ -113,6 +114,13 @@ test("everything the generated schema accepts, the published grammar also accept ["pet", generated], ["pet, core profile only", generateFilterSchema(pet, { profiles: ["core"] }).schema], ["a recursive resource", generateFilterSchema(RECURSIVE_RESOURCE, { maxDepth: 2 }).schema], + // The three shapes --operators/--drop-operators, --no-shorthand and + // --max-filter-depth produce. Each one edits the emitted structure, so each + // one is a fresh chance to emit something the grammar does not accept — and + // the property, not a case list, is what would notice. + ["pet, operators dropped", generateFilterSchema(pet, { dropOperators: ["$contains", "$exists", "$regex"] }).schema], + ["pet, no shorthand", generateFilterSchema(pet, { shorthand: false }).schema], + ["pet, logical nesting capped", generateFilterSchema(pet, { maxFilterDepth: 2 }).schema], ]; for (const [label, schema] of targets) { @@ -397,3 +405,217 @@ test("$unknownAs is emitted only where UNKNOWN is reachable", () => { "microchip is nullable, so UNKNOWN is reachable and the modifier applies", ); }); + +// --------------------------------------------------------------------------- +// Capability selection (#11): the operator set, the emitted shape, and the +// claims the capability document is allowed to make about them. +// --------------------------------------------------------------------------- + +/** The constraint object one path resolves to, whichever form the property takes. */ +function constraintOf(schema, path) { + const node = schema.properties[path]; + const ref = node.$ref ?? node.anyOf.at(-1).$ref; + return schema.$defs[ref.replace("#/$defs/", "")]; +} + +test("--operators selects within the profiles rather than beyond them", () => { + const { schema, warnings: w } = generateFilterSchema(pet, { + profiles: ["core", "strings"], + operators: ["$eq", "$in", "$like", "$between"], + }); + assert.deepEqual(Object.keys(constraintOf(schema, "name").properties), ["$eq", "$in", "$like"]); + // $between is in `ranges`, which was not requested: naming it cannot widen + // the selection, so the intersection drops it and says why. + assert.ok(w.some((m) => m.includes("$between") && m.includes("ranges")), w.join("\n")); + assert.ok(!JSON.stringify(schema).includes('"$between"')); +}); + +test("an unknown operator is refused rather than silently dropped", () => { + // The same treatment --profiles gives an unknown profile: a typo in a + // capability selection is a capability that silently did not apply. + assert.throws(() => generateFilterSchema(pet, { operators: ["$eq", "$matches"] }), /unknown operator "\$matches"/); + assert.throws(() => generateFilterSchema(pet, { dropOperators: ["$liek"] }), /unknown operator "\$liek"/); + assert.throws(() => generateFilterSchema(pet, { operators: [] }), /no operators enabled/); +}); + +test("--drop-operators reaches the schema and the capability document alike", () => { + // Issue #11's first shape: a backend with LIKE but no POSITION offers $like + // and not $contains, which no whole-profile choice can express. + const { schema, capabilities: caps } = generateFilterSchema(pet, { + profiles: ["core", "strings"], + dropOperators: ["$contains"], + }); + const ops = Object.keys(constraintOf(schema, "name").properties); + assert.ok(ops.includes("$like") && !ops.includes("$contains")); + assert.ok(!JSON.stringify(schema).includes('"$contains"')); + for (const [path, field] of Object.entries(caps.fields)) { + assert.ok(!field.operators.includes("$contains"), `${path} still advertises $contains`); + } +}); + +test("an operator whose dependency was dropped goes with it", () => { + // $flags carries dependentRequired: ["$regex"] out of the grammar, so keeping + // it beside a dropped $regex would emit a member that additionalProperties: + // false forbids — present in the schema and impossible to use. + const { schema, warnings: w } = generateFilterSchema(pet, { + profiles: ["core", "regex"], + dropOperators: ["$regex"], + }); + const json = JSON.stringify(schema); + assert.ok(!json.includes('"$flags"'), "$flags outlived the $regex it depends on"); + assert.ok(w.some((m) => m.includes("$flags") && m.includes("$regex")), w.join("\n")); +}); + +test("the capability document claims only the profiles offered in full", () => { + // SPEC §2.1: a profile other than core is implemented in full or not at all. + // Narrowing the *schema* is always legal — it accepts fewer filters than the + // grammar — but the profile claim is not survivable, and the per-field + // operator lists are where the truth goes instead (§2.2). + const partial = generateFilterSchema(pet, { + profiles: ["core", "strings", "ranges"], + dropOperators: ["$contains"], + }); + assert.deepEqual(partial.capabilities.profiles, ["core", "ranges"]); + assert.ok(partial.capabilities.fields.name.operators.includes("$like")); + assert.ok(partial.warnings.some((m) => m.includes('"strings"') && m.includes("§2.1"))); + + // Issue #11's second shape: a store that cannot implement $exists at all. + // core is mandatory, so what is left is not a conforming implementation, and + // the document must not say otherwise. + const noCore = generateFilterSchema(pet, { dropOperators: ["$exists"] }); + assert.ok(!noCore.capabilities.profiles.includes("core")); + assert.ok(noCore.warnings.some((m) => m.includes("not a conforming implementation"))); +}); + +test("--no-shorthand gives scalars the object-only form", () => { + const { schema } = generateFilterSchema(pet, { shorthand: false }); + assert.deepEqual(schema.properties.status, { $ref: "#/$defs/C_status" }); + assert.ok(!("anyOf" in schema.properties.status)); + // The root description teaches the rules a reader would otherwise get wrong. + // Leaving the shorthand among them would advertise a form this schema rejects. + assert.ok(!schema.description.includes("bare scalar")); + assert.ok(generateFilterSchema(pet, {}).schema.description.includes("bare scalar")); + + const check = makeAjv().compile(schema); + assert.equal(check({ status: "available" }), false, "the shorthand should be gone"); + assert.ok(check({ status: { $eq: "available" } }), errs(check)); +}); + +test("--max-filter-depth bounds how deep the logical operators nest", () => { + // Issue #11's third shape: a provider compiling to a flat conjunctive index + // wants one AND level and no more. JSON Schema cannot count how deep an + // instance already is, so the filter is unrolled into a chain of levels — + // depth 1 being the flat filter that offers no logical operators at all. + const flat = generateFilterSchema(pet, { maxFilterDepth: 1 }).schema; + assert.deepEqual(Object.keys(flat.properties).filter((k) => k.startsWith("$")), []); + + const { schema } = generateFilterSchema(pet, { maxFilterDepth: 2 }); + const deeper = schema.$defs[schema.properties.$and.items.$ref.replace("#/$defs/", "")]; + assert.deepEqual(Object.keys(deeper.properties).filter((k) => k.startsWith("$")), []); + assert.ok("status" in deeper.properties, "every level offers the same fields"); + assert.equal(deeper.properties.status.$ref, schema.properties.status.$ref, "levels share the operand defs"); + + const check = makeAjv().compile(schema); + assert.ok(check({ $and: [{ status: "available" }, { name: { $like: "F%" } }] }), errs(check)); + assert.equal(check({ $and: [{ $and: [{ status: "available" }] }] }), false, "two AND levels"); + assert.equal(check({ $and: [{ $or: [{ status: "available" }] }] }), false, "OR inside AND"); + // Field-level $not is self-referential too, so it is bounded to a single + // application: under Kleene logic ¬¬X ≡ X even for UNKNOWN. + assert.ok(check({ status: { $not: { $eq: "available" } } }), errs(check)); + assert.equal(check({ status: { $not: { $not: { $eq: "available" } } } }), false, "negated negation"); + + assert.throws(() => generateFilterSchema(pet, { maxFilterDepth: 0 }), /--max-filter-depth/); +}); + +test("--include reaches every level of a capped filter", () => { + // The levels repeat the root's field properties, so pruning only the root left + // every excluded path reachable one $and down. + const { schema } = generateFilterSchema(pet, { include: ["status"], maxFilterDepth: 2 }); + const deeper = schema.$defs[schema.properties.$and.items.$ref.replace("#/$defs/", "")]; + assert.deepEqual(Object.keys(deeper.properties), ["status"]); + + const check = makeAjv().compile(schema); + assert.ok(check({ $and: [{ status: "available" }] }), errs(check)); + assert.equal(check({ $and: [{ name: "Fido" }] }), false, "name is not in --include"); +}); + +test("the default is still one self-referential filter, not a chain", () => { + // The chain is what --max-filter-depth costs; nobody who did not ask for a + // bound should pay it. + assert.equal(generated.properties.$and.items.$ref, "#"); + assert.ok(!Object.keys(generated.$defs).some((name) => name.includes("_depth_"))); +}); + +test("--limits publishes the provider's real bounds", () => { + // SPEC §7's numbers were emitted unconditionally, so every generated document + // claimed them whether or not they were true. + const { capabilities: caps } = generateFilterSchema(pet, { limits: { maxClauses: 40 } }); + assert.deepEqual(caps.limits, { maxDepth: 10, maxClauses: 40, maxSetLength: 1000 }); + + assert.throws(() => generateFilterSchema(pet, { limits: { maxNesting: 4 } }), /unknown limit "maxNesting"/); + assert.throws(() => generateFilterSchema(pet, { limits: { maxDepth: 0 } }), /must be a positive integer/); + + // A schema that refuses nesting past n and a document claiming a deeper + // maxDepth would contradict each other, so the enforced bound is published. + const capped = generateFilterSchema(pet, { maxFilterDepth: 3 }); + assert.equal(capped.capabilities.limits.maxDepth, 3); + const both = generateFilterSchema(pet, { maxFilterDepth: 3, limits: { maxDepth: 9 } }); + assert.equal(both.capabilities.limits.maxDepth, 9); + assert.ok(both.warnings.some((m) => m.includes("maxDepth"))); +}); + +test("a config file is the selection, and an explicit flag still beats it", () => { + // The config file is what a provider checks in beside the resource schema and + // regenerates from, so its paths are relative to itself rather than to + // whatever directory the command happened to run in. + const dir = mkdtempSync(join(tmpdir(), "jql-config-")); + writeFileSync(join(dir, "pet.schema.json"), readFileSync(join(repo, "examples", "pet.schema.json"))); + const file = join(dir, "jql.config.json"); + writeFileSync(file, JSON.stringify({ + resource: "pet.schema.json", + profiles: ["core", "strings"], + dropOperators: ["$contains"], + shorthand: false, + limits: { maxDepth: 4 }, + out: "pet.filter.json", + })); + + const fromConfig = resolveOptions({ config: file }, [], repo); + assert.equal(fromConfig.source, join(dir, "pet.schema.json")); + assert.equal(fromConfig.out, join(dir, "pet.filter.json")); + assert.deepEqual(fromConfig.options.dropOperators, ["$contains"]); + assert.equal(fromConfig.options.shorthand, false); + assert.deepEqual(fromConfig.options.limits, { maxDepth: 4 }); + + // A one-off refinement on top of a checked-in config is the point of having + // both, so the flag wins. + const overridden = resolveOptions({ config: file, profiles: "core", include: "status,name" }, [], repo); + assert.deepEqual(overridden.options.profiles, ["core"]); + assert.deepEqual(overridden.options.include, ["status", "name"]); + assert.deepEqual(overridden.options.dropOperators, ["$contains"], "the config still supplies the rest"); + + // The schema the config describes is generated without further arguments. + const { schema } = generateFilterSchema( + JSON.parse(readFileSync(fromConfig.source, "utf8")), + fromConfig.options, + ); + assert.ok(!JSON.stringify(schema).includes('"$contains"')); + assert.deepEqual(schema.properties.status, { $ref: "#/$defs/C_status" }); +}); + +test("a misspelled config key is refused rather than ignored", () => { + const dir = mkdtempSync(join(tmpdir(), "jql-config-")); + const file = join(dir, "jql.config.json"); + writeFileSync(file, JSON.stringify({ profiles: ["core"], dropOperator: ["$exists"] })); + assert.throws(() => resolveOptions({ config: file }, [], repo), /unknown key "dropOperator"/); + + writeFileSync(file, JSON.stringify(["core"])); + assert.throws(() => resolveOptions({ config: file }, [], repo), /must contain a JSON object/); +}); + +test("--max-depth refuses a value that is not a number", () => { + // It was coerced with Number() and never checked, so --max-depth deep became + // NaN and silently stopped the walk at the first nested object. + assert.throws(() => resolveOptions({ "max-depth": "deep" }, ["x.json"], repo), /--max-depth must be an integer/); + assert.equal(resolveOptions({ "max-depth": "0" }, ["x.json"], repo).options.maxDepth, 0); +}); diff --git a/tools/generate-filter-schema.mjs b/tools/generate-filter-schema.mjs old mode 100644 new mode 100755 index 13ac486..4b0dc45 --- a/tools/generate-filter-schema.mjs +++ b/tools/generate-filter-schema.mjs @@ -20,21 +20,45 @@ * than restated here, so the prose an agent reads stays in one place. * * Usage: - * node tools/generate-filter-schema.mjs [options] + * jql-generate [options] + * jql-generate --config * + * `jql-generate` is the installed name; from a clone it is + * `node tools/generate-filter-schema.mjs`, with the same arguments. + * + * What the schema describes: * --id $id for the generated schema (recommended) * --title title for the generated schema - * --profiles comma-separated; default core,strings,ranges,collections * --pointer subschema of the input file to treat as the resource * --max-depth how far to descend into nested objects (default 3) + * --include comma-separated paths; omit for "everything found" + * --exclude comma-separated paths or path prefixes ending in * + * + * What the endpoint can actually serve — the filter schema is a narrowing, so + * anything declined here is rejected by validation instead of at runtime: + * --profiles comma-separated; default core,strings,ranges,collections + * --operators exactly these operators, intersected with --profiles + * --drop-operators everything --profiles implies, minus these + * --no-shorthand drop the bare-scalar form; every constraint takes + * the object form + * --max-filter-depth the deepest filter the schema accepts. 1 is a flat + * filter with no logical operators at all, 2 permits + * one $and/$or/$nor/$not level. Unbounded by default + * --limits maxDepth, maxClauses and maxSetLength for the + * capability document, merged over the defaults + * + * Output: * --descriptions all | brief | none. Default brief: the operators * whose semantics surprise people keep their prose, * the self-evident ones keep only a title. - * --include comma-separated paths; omit for "everything found" - * --exclude comma-separated paths or path prefixes ending in * * --out write the schema here instead of stdout * --capabilities also write a SPEC.md §2.2 capability document * --grammar path to query-language-schema.json + * --config read these options from JSON, using the camelCase + * names of the JS API plus "resource", "out" and + * "capabilities". Relative paths in it resolve + * against its own directory, and an explicit flag + * always beats it. This is the file to check in. * --quiet suppress warnings on stderr * * A property may also opt out or override in the resource schema itself: @@ -45,12 +69,14 @@ import { readFileSync, writeFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; -import { dirname, join } from "node:path"; +import { dirname, join, resolve } from "node:path"; import { parseArgs } from "node:util"; const here = dirname(fileURLToPath(import.meta.url)); const DEFAULT_GRAMMAR = join(here, "..", "query-language-schema.json"); const DEFAULT_PROFILES = ["core", "strings", "ranges", "collections"]; +/** SPEC §7's RECOMMENDED defaults, published through the capability document (§2.2). */ +const DEFAULT_LIMITS = { maxDepth: 10, maxClauses: 100, maxSetLength: 1000 }; /** Formats whose values are opaque tokens: substring matching on them is noise. */ const OPAQUE_FORMATS = new Set(["uuid", "uri", "iri", "email", "ipv4", "ipv6", "duration"]); @@ -473,6 +499,9 @@ function dependencyRules(ctx, props) { function emitConstraint(ctx, field, prefix) { const ops = operatorsFor(field, ctx); if (ops.length === 0) return null; + // A bounded $not points at a copy of this constraint object with $not removed, + // so it needs at least one other operator to negate. + if (ctx.opts.maxFilterDepth !== undefined && ops.length === 1 && ops[0] === "$not") return null; const name = defName(ctx, `${prefix}C_`, field.nameHint ?? field.path); const props = {}; @@ -492,6 +521,16 @@ function emitConstraint(ctx, field, prefix) { } return { $ref: `#/$defs/${field.orderedDef}` }; }; + // Unbounded, $not negates this very constraint object, so {"$not": {"$not": …}} + // nests forever. Where a depth is set that has to stop somewhere, and one + // level is the natural place: under Kleene logic ¬¬X ≡ X even for UNKNOWN, so + // a negated negation says nothing the plain constraint does not. + let negName = null; + const notRef = () => { + if (ctx.opts.maxFilterDepth === undefined) return `#/$defs/${name}`; + negName ??= defName(ctx, `${prefix}N_`, hint); + return `#/$defs/${negName}`; + }; const itemRef = () => { if (!field.itemDef) { field.itemDef = defName(ctx, `${prefix}I_`, hint); @@ -530,7 +569,7 @@ function emitConstraint(ctx, field, prefix) { break; } case "$not": - props[op] = annotate(ctx, op, { $ref: `#/$defs/${name}` }); + props[op] = annotate(ctx, op, { $ref: notRef() }); break; case "$flags": props[op] = annotate(ctx, op, { type: "string", pattern: "^[ims]{0,3}$" }); @@ -545,6 +584,21 @@ function emitConstraint(ctx, field, prefix) { } } + if (negName) { + // The same constraint object minus $not, so one negation is expressible and + // two are not. Reachable only through $not, whose own description explains + // negation, so the field's prose is not repeated into it. + const { $not: dropped, ...rest } = props; + ctx.defs[negName] = { + title: `${field.path}, negated`, + type: "object", + minProperties: 1, + ...dependencyRules(ctx, rest), + properties: rest, + additionalProperties: false, + }; + } + const constraint = { title: field.path, ...(field.schema.description ? { description: field.schema.description } : {}), @@ -560,7 +614,10 @@ function emitConstraint(ctx, field, prefix) { field.operators = ops; // Scalars keep the shorthand: {"status": "open"} is {"status": {"$eq": "open"}}. - if (field.kind === "scalar") { + // --no-shorthand gives them the object-only form arrays and objects already + // have, which costs a provider nothing to accept and buys a validator that + // reports where a constraint went wrong instead of an anyOf that failed. + if (field.kind === "scalar" && ctx.opts.shorthand) { return { anyOf: [valueRef(), { $ref: `#/$defs/${name}` }] }; } return { $ref: `#/$defs/${name}` }; @@ -608,7 +665,6 @@ function emitQuantifierTarget(ctx, field, prefix) { */ function emitFilter(ctx, node, name, state) { if (name !== "__root__") ctx.reserved.add(name); - const selfRef = state.selfRef ?? "#"; const fields = collectFields(node, ctx, state); if (fields.length === 0) { ctx.warn(`no queryable fields found${name === "__root__" ? "" : ` for ${name}`}.`); @@ -616,31 +672,63 @@ function emitFilter(ctx, node, name, state) { } const prefix = name === "__root__" ? "" : `${sanitize(name)}_`; - const properties = {}; - for (const op of ["$and", "$or", "$nor"]) { - if (!ctx.enabled.has(op)) continue; - properties[op] = annotate(ctx, op, { type: "array", minItems: 1, items: { $ref: selfRef } }, "filter"); - } - if (ctx.enabled.has("$not")) properties.$not = annotate(ctx, "$not", { $ref: selfRef }, "filter"); - + const fieldProps = {}; const emitted = []; for (const field of fields) { const ref = emitConstraint(ctx, field, prefix); if (!ref) continue; - properties[field.path] = ref; + fieldProps[field.path] = ref; emitted.push(field); } + if (emitted.length === 0) { + // Logical operators over no field predicates is an infinite regress with no + // base case: every instance of it is unsatisfiable. + ctx.warn(`no constraints could be emitted${name === "__root__" ? "" : ` for ${name}`}.`); + return null; + } - const filter = { - type: "object", - minProperties: 1, - properties, - additionalProperties: false, + // Unbounded nesting is the single self-referential object it has always been. + // A bounded one has to be a chain, because JSON Schema cannot count how deep + // an instance already is: level i offers the logical operators over level + // i+1, and the last level does not offer them at all. Every level shares the + // field properties, so the cost is n copies of a map of $refs, not n copies + // of the operand schemas. + const depth = ctx.opts.maxFilterDepth ?? 1; + const hint = name === "__root__" ? "Filter" : name; + const deeper = []; + for (let i = 2; i <= depth; i++) deeper.push(defName(ctx, "", `${hint}_depth_${i}`)); + + const level = (i) => { + const properties = {}; + const next = + ctx.opts.maxFilterDepth === undefined ? (state.selfRef ?? "#") + : i < depth ? `#/$defs/${deeper[i - 1]}` + : null; + if (next) { + for (const op of ["$and", "$or", "$nor"]) { + if (!ctx.enabled.has(op)) continue; + properties[op] = annotate(ctx, op, { type: "array", minItems: 1, items: { $ref: next } }, "filter"); + } + if (ctx.enabled.has("$not")) properties.$not = annotate(ctx, "$not", { $ref: next }, "filter"); + } + return { + type: "object", + minProperties: 1, + properties: { ...properties, ...fieldProps }, + additionalProperties: false, + }; }; + for (let i = 2; i <= depth; i++) ctx.defs[deeper[i - 2]] = level(i); + const filter = level(1); + if (name === "__root__") { ctx.rootFilter = filter; ctx.rootFields = emitted; + // The deeper levels repeat the root's field properties, so --include has to + // reach them too — otherwise a path pruned from the root walks back in + // under an $and. + ctx.rootLevels = deeper.map((level) => ctx.defs[level]); } else { ctx.defs[name] = filter; } @@ -651,9 +739,12 @@ function emitFilter(ctx, node, name, state) { // Public API // --------------------------------------------------------------------------- +const SHORTHAND_RULE = + "A bare scalar is equality: {\"status\": \"open\"} is {\"status\": {\"$eq\": \"open\"}}."; + const SILENT_RULES = [ "Sibling members are combined with implicit AND, at every level.", - "A bare scalar is equality: {\"status\": \"open\"} is {\"status\": {\"$eq\": \"open\"}}.", + SHORTHAND_RULE, "Comparisons use three-valued logic: $ne and $not do NOT match records where the field is null or absent. To include those records, add \"$unknownAs\": true to the same constraint.", "$in compares the whole field value; it is not array membership. To say something about the elements of an array, quantify: {\"tags\": {\"$some\": {\"$in\": [\"a\"]}}}.", ]; @@ -671,11 +762,12 @@ export function generateFilterSchema(resource, options = {}) { title, description: [ `A filter over ${ctx.resource.title ?? "this resource"}, in the JSON Query Language.`, - ...SILENT_RULES, + ...SILENT_RULES.filter((rule) => ctx.opts.shorthand || rule !== SHORTHAND_RULE), ].join(" "), $comment: `Generated by tools/generate-filter-schema.mjs from ${ctx.resource.$id ?? options.source ?? "a resource schema"} ` + - `against ${ctx.grammar.$id}. Profiles: ${ctx.opts.profiles.join(", ")}. Do not edit by hand.`, + `against ${ctx.grammar.$id}. Profiles: ${ctx.opts.profiles.join(", ")}` + + `${ctx.opts.excluded.length ? ` (minus ${ctx.opts.excluded.join(", ")})` : ""}. Do not edit by hand.`, ...ctx.rootFilter, $defs: ctx.defs, }; @@ -692,8 +784,10 @@ export function generateCapabilities(resource, options = {}) { */ function pruneToIncluded(ctx) { const keep = new Set(ctx.opts.include); - for (const path of Object.keys(ctx.rootFilter.properties)) { - if (!path.startsWith("$") && !keep.has(path)) delete ctx.rootFilter.properties[path]; + for (const filter of [ctx.rootFilter, ...ctx.rootLevels]) { + for (const path of Object.keys(filter.properties)) { + if (!path.startsWith("$") && !keep.has(path)) delete filter.properties[path]; + } } for (const path of ctx.opts.include) { if (!(path in ctx.rootFilter.properties)) ctx.warn(`--include names "${path}", which was not found.`); @@ -722,10 +816,109 @@ function buildCapabilities(ctx, options) { return { queryLanguage: ctx.grammar.$id, ...(options.id ? { filterSchema: options.id } : {}), - profiles: ctx.opts.profiles, + profiles: ctx.opts.complete, fields, - limits: options.limits ?? { maxDepth: 10, maxClauses: 100, maxSetLength: 1000 }, + limits: ctx.opts.limits, + }; +} + +/** + * The operator set the generated schema will offer: profiles first, because + * they are the coarse unit a server advertises, then the operator-level refinement + * for the cases a profile boundary does not fit — a backend with LIKE but no + * POSITION supports $like and not $contains, and a key-value store cannot + * implement $exists at all. + * + * Narrowing the set is always safe: the emitted schema accepts fewer filters + * than the grammar, never more. What it costs is the profile *claim*, which is + * why the demotion is warned about here and reflected in the capability + * document rather than silently kept (SPEC §2.1). + */ +function selectOperators(grammar, profiles, options, warn) { + const byProfile = grammar["x-profiles"]; + const defined = new Set(Object.values(byProfile).flat()); + const profileOf = (op) => Object.keys(byProfile).find((p) => byProfile[p].includes(op)); + + const check = (names, flag) => { + for (const op of names) { + if (!defined.has(op)) { + throw new Error(`unknown operator "${op}" in ${flag}; the grammar defines ${[...defined].join(", ")}`); + } + } }; + check(options.operators ?? [], "--operators"); + check(options.dropOperators ?? [], "--drop-operators"); + + let enabled = new Set(profiles.flatMap((p) => byProfile[p])); + + if (options.operators) { + for (const op of options.operators) { + if (!enabled.has(op)) { + warn(`--operators names "${op}", which the "${profileOf(op)}" profile supplies; add it to --profiles for the operator to take effect.`); + } + } + enabled = new Set(options.operators.filter((op) => enabled.has(op))); + } + for (const op of options.dropOperators ?? []) enabled.delete(op); + + // A kept operator whose dependency was dropped would be emitted with a + // `dependentRequired` rule naming a member `additionalProperties: false` + // forbids — present in the schema and impossible to use. The rule is read off + // the grammar (today: $flags needs $regex) so a dependency added later is + // handled by construction rather than by a second list here. + for (const [trigger, requires] of Object.entries(grammar.$defs.ConstraintObject.dependentRequired ?? {})) { + if (!enabled.has(trigger)) continue; + const missing = requires.filter((op) => !enabled.has(op)); + if (missing.length === 0) continue; + enabled.delete(trigger); + warn(`"${trigger}" requires ${missing.join(", ")} beside it, so it is excluded too.`); + } + + if (enabled.size === 0) throw new Error("the selection leaves no operators enabled"); + + // Which profiles survive as *claims*. SPEC §2.1: a profile other than core is + // implemented in full or not at all, so a profile missing an operator is no + // longer on offer — the per-field `operators` lists carry what is. + const requested = new Set(profiles.flatMap((p) => byProfile[p])); + const excluded = [...requested].filter((op) => !enabled.has(op)); + const complete = profiles.filter((p) => byProfile[p].every((op) => enabled.has(op))); + for (const p of profiles) { + if (complete.includes(p)) continue; + const missing = byProfile[p].filter((op) => !enabled.has(op)); + warn( + p === "core" + ? `the "core" profile is incomplete (${missing.join(", ")} excluded). SPEC §2.1 requires core in full, so this endpoint is not a conforming implementation; the capability document will not claim core.` + : `profile "${p}" is not offered in full (${missing.join(", ")} excluded), so it is omitted from the capability document's profiles (SPEC §2.1).`, + ); + } + + return { enabled, complete, excluded }; +} + +/** + * SPEC §7's limits are a claim published through the capability document, not + * something the schema enforces — except `maxDepth`, which `--max-filter-depth` + * does enforce. Where both are given they should agree, so a disagreement is + * worth saying out loud. + */ +function resolveLimits(options, warn) { + const given = options.limits ?? {}; + for (const [key, value] of Object.entries(given)) { + if (!(key in DEFAULT_LIMITS)) { + throw new Error(`unknown limit "${key}"; --limits takes ${Object.keys(DEFAULT_LIMITS).join(", ")}`); + } + if (!Number.isInteger(value) || value < 1) { + throw new Error(`limit "${key}" must be a positive integer, got ${JSON.stringify(value)}`); + } + } + const limits = { ...DEFAULT_LIMITS, ...given }; + if (options.maxFilterDepth !== undefined) { + if (given.maxDepth === undefined) limits.maxDepth = options.maxFilterDepth; + else if (given.maxDepth !== options.maxFilterDepth) { + warn(`--limits maxDepth is ${given.maxDepth} but the schema refuses logical nesting past ${options.maxFilterDepth}; the published limit is the looser claim.`); + } + } + return limits; } function prepare(resource, options) { @@ -743,7 +936,15 @@ function prepare(resource, options) { } if (!profiles.includes("core")) throw new Error(`the "core" profile is mandatory (SPEC §2.1)`); - const enabled = new Set(profiles.flatMap((p) => grammar["x-profiles"][p])); + if (options.maxFilterDepth !== undefined + && (!Number.isInteger(options.maxFilterDepth) || options.maxFilterDepth < 1)) { + throw new Error(`--max-filter-depth must be an integer of 1 or more, got ${JSON.stringify(options.maxFilterDepth)}`); + } + + const warnings = []; + const warn = (m) => warnings.push(m); + const { enabled, complete, excluded } = selectOperators(grammar, profiles, options, warn); + const limits = resolveLimits(options, warn); const root = resource; const deref = makeDeref(root); @@ -751,7 +952,6 @@ function prepare(resource, options) { if (entry === undefined) throw new Error(`--pointer "${options.pointer}" does not resolve`); const resolved = flatten(entry, deref, new Set()); - const warnings = []; return { grammar, deref, @@ -759,11 +959,16 @@ function prepare(resource, options) { defs: {}, reserved: new Set(["Size"]), warnings, - warn: (m) => warnings.push(m), + warn, resource: resolved, opts: { profiles, + complete, + excluded, + limits, maxDepth: options.maxDepth ?? 3, + maxFilterDepth: options.maxFilterDepth, + shorthand: options.shorthand !== false, descriptions: options.descriptions ?? "brief", include: options.include ?? null, exclude: (options.exclude ?? []).map(globToRegExp), @@ -775,65 +980,170 @@ function prepare(resource, options) { // CLI // --------------------------------------------------------------------------- -function main(argv) { - const { values, positionals } = parseArgs({ - args: argv, - allowPositionals: true, +const FLAGS = { + id: { type: "string" }, + title: { type: "string" }, + profiles: { type: "string" }, + operators: { type: "string" }, + "drop-operators": { type: "string" }, + "no-shorthand": { type: "boolean" }, + "max-filter-depth": { type: "string" }, + limits: { type: "string" }, + pointer: { type: "string" }, + "max-depth": { type: "string" }, + include: { type: "string" }, + exclude: { type: "string" }, + capabilities: { type: "string" }, + descriptions: { type: "string" }, + grammar: { type: "string" }, + out: { type: "string" }, + config: { type: "string" }, + quiet: { type: "boolean" }, + help: { type: "boolean", short: "h" }, +}; + +/** Config keys that name a file, and so resolve against the config's own directory. */ +const CONFIG_PATHS = new Set(["resource", "out", "capabilities", "grammar"]); + +/** The config file's vocabulary: the JS API's option names, plus where things go. */ +const CONFIG_KEYS = [ + "resource", "id", "title", "profiles", "operators", "dropOperators", "shorthand", + "maxFilterDepth", "limits", "pointer", "maxDepth", "include", "exclude", + "descriptions", "out", "capabilities", "grammar", "quiet", +]; + +/** + * Merges a config file with the command line into one options object. + * + * The config file is the artifact a provider checks in beside their resource + * schema and regenerates from, so the capability selection lives in version + * control rather than in whoever's shell history. A flag given explicitly wins + * over it, which is what makes a one-off `--include` on top of a checked-in + * config work. + * + * Exported so the merge can be tested without spawning a process. + */ +export function resolveOptions(values, positionals, cwd = process.cwd()) { + if (positionals.length > 1) { + throw new Error(`expected one resource schema, got ${positionals.length}: ${positionals.join(", ")}`); + } + + let config = {}; + let base = cwd; + if (values.config !== undefined) { + const file = resolve(cwd, values.config); + base = dirname(file); + config = JSON.parse(readFileSync(file, "utf8")); + if (!config || typeof config !== "object" || Array.isArray(config)) { + throw new Error(`${values.config} must contain a JSON object`); + } + // Refused rather than ignored, for the same reason parseArgs refuses an + // unknown flag: a misspelled key is a selection that silently did not apply. + const unknown = Object.keys(config).filter((k) => !CONFIG_KEYS.includes(k)); + if (unknown.length > 0) { + throw new Error( + `unknown key${unknown.length > 1 ? "s" : ""} ${unknown.map((k) => `"${k}"`).join(", ")} in ` + + `${values.config}; it takes ${CONFIG_KEYS.join(", ")}`, + ); + } + } + + // Paths in a config file are relative to the config file. Paths on the command + // line are relative to the cwd, and are left as written so that a generated + // $comment names the schema the way the reader would. + const configValue = (key) => (CONFIG_PATHS.has(key) ? resolve(base, config[key]) : config[key]); + const pick = (flag, key) => + values[flag] !== undefined ? values[flag] + : config[key] !== undefined ? configValue(key) + : undefined; + + const list = (v) => { + if (typeof v !== "string") return v; + const items = v.split(",").map((s) => s.trim()).filter(Boolean); + return items.length > 0 ? items : undefined; + }; + const integer = (v, flag, min) => { + if (v === undefined) return undefined; + const n = Number(v); + if (!Number.isInteger(n) || n < min) { + throw new Error(`${flag} must be an integer of ${min} or more, got ${JSON.stringify(v)}`); + } + return n; + }; + const limits = () => { + const raw = values.limits; + if (raw === undefined) return config.limits; + const text = raw.startsWith("@") ? readFileSync(resolve(cwd, raw.slice(1)), "utf8") : raw; + try { + return JSON.parse(text); + } catch (error) { + throw new Error(`--limits is not valid JSON: ${error.message}`); + } + }; + + const source = positionals[0] ?? (config.resource !== undefined ? resolve(base, config.resource) : undefined); + + return { + source, + out: pick("out", "out"), + capabilities: pick("capabilities", "capabilities"), + quiet: values.quiet ?? config.quiet ?? false, options: { - id: { type: "string" }, - title: { type: "string" }, - profiles: { type: "string" }, - pointer: { type: "string" }, - "max-depth": { type: "string" }, - include: { type: "string" }, - exclude: { type: "string" }, - capabilities: { type: "string" }, - descriptions: { type: "string" }, - grammar: { type: "string" }, - out: { type: "string" }, - quiet: { type: "boolean" }, - help: { type: "boolean", short: "h" }, + id: pick("id", "id"), + title: pick("title", "title"), + source, + profiles: list(pick("profiles", "profiles")), + operators: list(pick("operators", "operators")), + dropOperators: list(pick("drop-operators", "dropOperators")), + shorthand: values["no-shorthand"] ? false : config.shorthand, + maxFilterDepth: integer(pick("max-filter-depth", "maxFilterDepth"), "--max-filter-depth", 1), + limits: limits(), + pointer: pick("pointer", "pointer"), + maxDepth: integer(pick("max-depth", "maxDepth"), "--max-depth", 0), + descriptions: pick("descriptions", "descriptions"), + include: list(pick("include", "include")), + exclude: list(pick("exclude", "exclude")), + grammarPath: pick("grammar", "grammar"), }, - }); - - if (values.help || positionals.length !== 1) { - // The header comment is the help text; keeping one copy avoids the usual drift. - const header = readFileSync(fileURLToPath(import.meta.url), "utf8") - .split("*/")[0] - .replace(/^#!.*\n/, "") - .replace(/^\/\*\*?\n?/, "") - .replace(/^ \*\/?/gm, "") - .replace(/^ /gm, "") - .trimEnd(); - process.stdout.write(`${header}\n`); - process.exit(values.help ? 0 : 1); - } - - const source = positionals[0]; - const resource = JSON.parse(readFileSync(source, "utf8")); - const list = (v) => (v ? v.split(",").map((s) => s.trim()).filter(Boolean) : undefined); - - const { schema, capabilities, warnings } = generateFilterSchema(resource, { - id: values.id, - title: values.title, - source, - profiles: list(values.profiles), - pointer: values.pointer, - maxDepth: values["max-depth"] ? Number(values["max-depth"]) : undefined, - descriptions: values.descriptions, - include: list(values.include), - exclude: list(values.exclude), - grammarPath: values.grammar, - }); + }; +} + +function usage() { + // The header comment is the help text; keeping one copy avoids the usual drift. + return readFileSync(fileURLToPath(import.meta.url), "utf8") + .split("*/")[0] + .replace(/^#!.*\n/, "") + .replace(/^\/\*\*?\n?/, "") + .replace(/^ \*\/?/gm, "") + .replace(/^ /gm, "") + .trimEnd(); +} + +function main(argv) { + const { values, positionals } = parseArgs({ args: argv, allowPositionals: true, options: FLAGS }); + + if (values.help) { + process.stdout.write(`${usage()}\n`); + process.exit(0); + } + + const resolved = resolveOptions(values, positionals); + if (!resolved.source) { + process.stdout.write(`${usage()}\n`); + process.exit(1); + } + + const resource = JSON.parse(readFileSync(resolved.source, "utf8")); + const { schema, capabilities, warnings } = generateFilterSchema(resource, resolved.options); const json = `${JSON.stringify(schema, null, 2)}\n`; - if (values.out) writeFileSync(values.out, json); + if (resolved.out) writeFileSync(resolved.out, json); else process.stdout.write(json); - if (values.capabilities) { - writeFileSync(values.capabilities, `${JSON.stringify(capabilities, null, 2)}\n`); + if (resolved.capabilities) { + writeFileSync(resolved.capabilities, `${JSON.stringify(capabilities, null, 2)}\n`); } - if (!values.quiet) { + if (!resolved.quiet) { for (const w of warnings) process.stderr.write(`warning: ${w}\n`); } } From afab7b25739fcee94f2a6e53e448b05ab3c9659f Mon Sep 17 00:00:00 2001 From: Christos Gkoros Date: Sun, 13 Sep 2026 14:23:53 +0300 Subject: [PATCH 2/2] v0.5.0: a generator a provider can point at their own API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cuts the unreleased window as 0.5.0. Minor, and not breaking: no filter valid under 0.4.0 becomes invalid. The schema's $id stays at v0.4.0, as it did across 0.3.1, because the $id tracks the grammar and the grammar did not move — query-language-schema.json is byte-identical apart from the root description already in this window. Consumers pinning that $id have nothing to do. SPEC.md's version line stays at 0.4.0 for the same reason. Co-Authored-By: Claude --- CHANGELOG.md | 21 +++++++++++++++++++-- README.md | 2 +- package.json | 2 +- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5cd978e..a595840 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,24 @@ minor release may break compatibility, in which case the break is spelled out be ## [Unreleased] -No change to the grammar or to the semantics of evaluation. The only edit to -`query-language-schema.json` is its root `description`, and its `$id` still names `v0.4.0`. +## [0.5.0] — 2026-09-13 + +**A tooling release.** No change to the grammar or to the semantics of evaluation: +`query-language-schema.json` is byte-identical to 0.4.0 apart from its root `description`, and its +`$id` still names `v0.4.0`, because the `$id` version tracks the grammar and the grammar did not +move. Consumers pinning that `$id` have nothing to do. + +What did move is the generator, which is now the thing an API provider uses when implementing +search: point it at the resource schema, state the slice of the language you can actually serve, +and get back a filter schema that permits exactly that slice plus a capability document that +describes it honestly. It also travels with the package for the first time, as a `bin` named +`jql-generate`, rather than being a file inside a repository nobody installs. + +[SPEC.md](./SPEC.md) gains two clarifications in service of that, both about what an +implementation may *claim* rather than about what a filter means: +[§2.1](./SPEC.md#21-profiles) says out loud that accepting part of a profile is permitted and +advertising it is not, and [§2.2](./SPEC.md#22-capability-discovery) documents the capability +document's top-level members. No filter valid under 0.4.0 becomes invalid. ### Added @@ -418,6 +434,7 @@ Initial research draft: `$and`, `$or`, `$not` over eight leaf condition types (`$eq`, `$ne`, `$in`, `$nin`, `$like`, `$nlike`, `$gt`/`$gte`/`$lt`/`$lte`/`$between`, `$isnull`), laid out as an OpenAPI `components.schemas` fragment. +[0.5.0]: https://github.com/christosgkoros/json-query-language/compare/v0.4.0...v0.5.0 [0.4.0]: https://github.com/christosgkoros/json-query-language/compare/v0.3.1...v0.4.0 [0.3.1]: https://github.com/christosgkoros/json-query-language/compare/v0.3.0...v0.3.1 [0.3.0]: https://github.com/christosgkoros/json-query-language/compare/v0.2.0...v0.3.0 diff --git a/README.md b/README.md index 7607f62..361a8d9 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ One schema, two integration points, because JSON Schema is what both already spe - **OpenAPI documents** — [`examples/`](./examples) — working 3.1 (`POST /…/search`) and 3.2 (`QUERY`) integrations - **Generator** — [`tools/generate-filter-schema.mjs`](./tools/generate-filter-schema.mjs) — turns a resource's JSON Schema, plus the slice of the language you can serve, into a per-field filter schema - **Compared with GraphQL** — [`COMPARISON.md`](./COMPARISON.md) — what this overlaps with, what it does not, and what a JSON-Schema-native alternative would still need -- **Version** — `0.4.0`, and the schema's `$id` names `v0.4.0`. This release is **breaking**: it replaces two array mechanisms with one quantifier family and adds `$unknownAs`. See [`CHANGELOG.md`](./CHANGELOG.md) for the migration, and [`decisions/0001`](./decisions/0001-array-quantifiers-and-unknown-handling.md) for why. +- **Version** — `0.5.0`. The schema's `$id` still names `v0.4.0`: the `$id` tracks the grammar, and 0.5.0 changed only the tooling and what it may claim. See [`CHANGELOG.md`](./CHANGELOG.md) for this release, and [`decisions/0001`](./decisions/0001-array-quantifiers-and-unknown-handling.md) for the v0.4.0 migration. > **Work in progress — including the name.** This is a design published for review, not a distribution you can depend on yet. The artifact's own name is a working title, and every identifier that follows from it — the package names, the schema `$id`, the URLs in the integration examples — is a placeholder. Several do not currently resolve, and getting them right is deliberately not a goal until the name is settled. The grammar and its semantics are the part worth reviewing. See [Status](#status) before you try to install or `$ref` anything. diff --git a/package.json b/package.json index 8bd298a..2f551b9 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "json-query-language", - "version": "0.4.0", + "version": "0.5.0", "description": "A JSON-encoded, SQL-flavoured predicate language described by a single JSON Schema: $ref it from OpenAPI, or inline it into an MCP tool's inputSchema.", "type": "module", "license": "MIT",