Skip to content

Foundation: fixture specification and validator #65

Closed

Description

@CompleteDotTech

Goal

One fixture format across sixty recipes, checked by a tool, so fixtures stay readable and honest.

Scope

  • A written specification for fixtures/: inputs, gold labels, and model responses, in JSON or JSON Lines, with stable identifiers linking the three.
  • Split support (validation, test, and others as needed) recorded per example.
  • Required provenance on every response: synthetic or recorded, plus model string and date when recorded.
  • A JSON Schema and a validator: python -m jev_cookbook.fixtures validate recipes/NN-slug.
  • Checks beyond schema: every evaluated input has a label, identifiers are unique, no response without a matching input, and a scan for strings that look like keys or tokens.

Acceptance criteria

  • Validator passes on a good example and fails with a specific message on each kind of bad one, with tests for both
  • The specification is one page in docs/ and is linked from CONTRIBUTING.md
  • The validator exits non-zero on failure so CI can use it
  • Lint and tests pass in CI, or locally with the output pasted in the pull request if CI does not exist yet

Dependencies

References

Activity

  1. added
    type: foundationShared code, tooling, CI, or docs that recipes depend on
    priority: p0Blocks everything else; do first
    status: blockedWaiting on the issues listed under Dependencies
    on Oct 6, 2026
  2. added
    status: in progressClaimed; a branch or pull request exists
    and removed
    status: blockedWaiting on the issues listed under Dependencies
    on Oct 7, 2026
  3. added a commit that references this issue on Oct 7, 2026
    8a1ac9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: p0Blocks everything else; do firstsize: MAbout a daytype: foundationShared code, tooling, CI, or docs that recipes depend on

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions