Skip to content

fix: preserve full Groq response schemas in evals - #365

Open
cwilmott-groq wants to merge 3 commits into
mainfrom
cwilmott/groq-response-schema-null-description
Open

fix: preserve full Groq response schemas in evals#365
cwilmott-groq wants to merge 3 commits into
mainfrom
cwilmott/groq-response-schema-null-description

Conversation

@cwilmott-groq

Copy link
Copy Markdown

Summary

  • retain the original JSON Schema alongside Inspect typed schemas so Groq receives oneOf, conditionals, patterns, lengths, and other unmodeled assertions
  • validate each schema with its declared dialect instead of forcing every input through Draft 2020-12
  • normalize nullable type arrays for the Inspect compatibility representation
  • omit null response-schema descriptions from Groq requests
  • validate decimal multipleOf exactly and use RFC3339 time semantics in the scorer

Why

Inspect previously discarded assertions before inference while the scorer retained them, creating false constrained-decoding failures. This change makes the sent schema and scored schema agree.

Validation

  • 15 targeted provider, scorer, and JSONSchemaBench dataset tests pass
  • Ruff formatting and lint checks pass

cwilmott-groq and others added 2 commits August 25, 2026 22:34
Registers an exact RFC 3339 full-time format checker: without the
rfc3339-validator extra, python-jsonschema's built-in "time" checker
mis-rejects valid values like "10:00:00Z" and "21:00:00.000Z", which
produced the two false-positive schema_leak results in the 2026-08-26
production run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first pass accepted any ss=60; mirror the API validator
(santhosh-tekuri isTime) instead: a leap second is only valid when the
offset-normalized time is 23:59 UTC. Add permanent accept/reject
regressions for the checker.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant