[christosgkoros]

experiments/intent-to-filter/ puts sixty-three natural-language questions to six search interfaces and grades by executed row sets. Predictions were committed before the run. Four of seven are refuted, including the headline.

Closes nothing; raises the question of whether README.md's lead should stand. Related: #8.

Verdict

The agent-first positioning is not supported and should be withdrawn or heavily qualified. The language is fine. The comparative claim made about it is what fails, and it fails precisely where it is loudest.

GPT-5.5 — 63 questions x 6 arms x 5 trials, 1,890 calls, no failures

  arm                 n   correct    loud   silent  loud:silent
  jql-raw           295   66% ± 5     24%       4%          6.0
  jql-generated     295   79% ± 5      9%       9%          1.0
  jql-published     315   44% ± 5     39%       6%          6.2
  search-string     265   89% ± 4      2%       5%          0.4
  bespoke-json      315   86% ± 4      0%       7%          0.0
  fixed-params       45  100% ± 4      0%       0%            —

Why the QL is not the right answer for the agent case, as positioned

1. It loses on the axis it is sold on. README.md:3 calls this "a search interface for agents". Six hundred tokens of prose describing a Lucene-style string answered 89% correctly; fourteen thousand tokens of generated JSON Schema answered 79%. Per correct answer: 1,243 tokens against 19,044. Fifteen times the cost, ten points less accuracy. A bespoke JSON dialect with no schema at all, documented only in prose, also beat it at 86%.

2. The mechanism the repository actually argues for is absent. README.md:210 does not claim higher accuracy; it claims that other approaches "fail as an empty result set, which an agent cannot distinguish from 'no such records'" while this one fails loudly and repairs in one round trip. Measured: jql-generated had the highest silent-failure rate of the three working arms (9%, against 5% and 7%), and a repair yield of 4%. bespoke-json posted the worst possible loud:silent ratio, 0.0, and still won.

3. Two of three providers cannot receive it. Gemini's function declarations have no $ref, and the grammar is recursive so it cannot be flattened; flattened anyway, dependentRequired and patternProperties are also missing. Groq rejects the flattened schema as too large (413) and fails tool-call validation on the raw one. OpenAI accepts the raw schema and silently discards $defs — 94% of the file — so it bills 704 input tokens instead of 14,089 and the grammar never arrives. It works only if you dereference it yourself first, which the README does not mention and examples/mcp-server does not do. 1,114 cells were skipped for these reasons.

4. The mistakes it prevents are largely mistakes a strong model no longer makes. $in-as-array-membership, one of the three canonical failures the README is built around, occurred zero times on every arm. On the closed-domain probe the generated schema scored 5/5 — and so did the search string and the prose JSON, because the model normalised Available → available unprompted. The domain in the schema was doing work the model was going to do anyway. Only $ne-drops-nulls was common, at 5–10% across every arm, and no arm's design reduced it.

What survives

The internal argument holds, and by large margins: bare narrowed grammar 44% → shipped raw 66% → generated and flattened 79%. Generation is worth 35 points, so the generator earns its place. The one case where the published grammar fails exactly as predicted — five silent empty result sets on {"status": "Available"} — is the case generation fixes.

Nothing here contradicts the original claim: one grammar across every search endpoint, validated in CI, typed in generated clients. That claim predates the agent framing. The evidence supports going back to it.

Against the predictions

#	Predicted	Measured
H1	lowest silent rate, ≤10% vs ≥25%	9% vs 5% — the highest of the working arms	refuted
H2	more loud failures, but they convert	more loud, 4% conversion	half refuted
H3	repair yield ≥15% for schema arms	4% and 2%	refuted
H4	jql-published silent 2–3× jql-generated	6% vs 9% — published is lower	refuted
H5	$ne mistake not reduced by the schema	5% vs 8% vs 6%; within noise	confirmed
H6	fixed-params ≥40% unanswerable, most accurate	86% unanswerable, 100% on the nine it can put	confirmed
H7	~10–20× tokens; string wins per dollar	15×, and lower accuracy	confirmed, worse than predicted

Two of the three that held were written against the repository's interest. That is why the other four are worth believing.

How it avoids rigging the result

• The answer key predates the experiment. Sixty of the questions reuse experiments/filter-to-sql/cases.mjs, hand-derived from SPEC.md months ago to measure compiler complexity. It cannot have been chosen to flatter the language. Three authored probes are tagged and never averaged in.
• One evaluator for every arm. Each arm translates to a filter; the same compiler runs all of them against the same records. No arm can win on its own null semantics.
• A ceiling is not a failure. Every arm has a hand-written gold query per case, and gold.test.mjs executes all 63 × 6 before anything is paid for. Questions an interface cannot put are counted as its expressivity ceiling and dropped from its denominator, not marked wrong.
• The competitors are built to win. search-string is real Lucene/Elasticsearch syntax with a deliberately generous parser. bespoke-json is near-isomorphic to the grammar, $unknownAs equivalent included, so its ceiling is empty by design — the variable isolated is schema-versus-prose, not expressivity. Neither is Mongo-shaped, to keep model familiarity out of it.
• Right rows is not right filter. Only 44 distinct row sets across 62 cases on ten records, so a second seeded 200-record set checks candidate ≡ gold. That separates all 58.
• The prose budget binds the competitors, not the schema arms, and a test asserts the schema arms' descriptions stay under 100 tokens and never mention $unknownAs, $some, three-valued logic or nulls.

Limits

One model completed. Groq exhausted its rate limit and Gemini its free-tier daily quota, so the cross-provider accuracy comparison the design was built for does not exist — only the deliverability findings, which are complete. Sixty-three questions, one resource, ten records. And the competing arms were written by this experiment: carefully, but not disinterestedly.

Recommended follow-ups, not in this PR

• Withdraw "A search interface for agents" as the lead, or attach the numbers to it.
• Replace README.md:210's loud-versus-silent argument, which is contradicted as written.
• Document the deliverability problem in §"Exposing search to an agent".
• Give COMPARISON.md:107-110 a citation or delete it.
• Fix Generated filter schemas are not a narrowing: a lone $unknownAs is accepted, and compiles to invalid SQL #8.

Verifying this PR

npm test                                                 # 117/117, unchanged
npm run test:intent                                      # 14/14 pre-flight checks
node experiments/intent-to-filter/report.mjs \
  --file experiments/intent-to-filter/results/full.jsonl  # regenerates every number above

Raw responses are committed as JSONL, so the tables can be rechecked without re-running or spending anything. The experiment is not in CI: it costs money and needs three API keys.

Generated with Claude Code