Recipe 03: Response clarity scoring #3

Recipe 03 of 60 · Level 1 (Beginner) · Language & content · Score

Use case

Score a support response against a clearly defined clarity rubric to identify examples that need editing.

What to build

The folder recipes/03-response-clarity-scoring/ with notebook.ipynb, README.md, and fixtures/, meeting the recipe contract. Start from python tools/new_recipe.py 03 and the template in recipes/_template/.

Build notes

• Write a clarity rubric with every level described in words, lowest to highest.
• Fixtures: support responses with a human rubric level each, spanning the whole scale.
• Python flags responses below a cutoff as needing an edit. The cutoff lives in code.

Decision design

• Score. Describe every rubric level in words, in order. Show the score and the distribution over levels. Mapping, weighting, and arithmetic on scores happen in Python.

Level 1 scope

One bounded judgment over a short prepared input with direct inspection of typed results.

Acceptance criteria

• recipes/03-response-clarity-scoring/notebook.ipynb runs top to bottom offline, with no key and no network, and its outputs are committed
• Live mode is opt-in (JEV_COOKBOOK_LIVE=1 with TYPESAFE_API_KEY) and uses the same question definitions as the offline run
• Fixtures are synthetic, include the hard cases named in the build notes, carry gold labels, and pass the fixture validator
• Every response fixture records its provenance, and the notebook states which mode it ran in
• Evaluation: Exact-level agreement and mean absolute error against the human levels, plus the distribution over levels for a few examples.
• A reader with basic Python can follow it end to end; every new term is explained where it first appears
• At least three individual typed answers are displayed and discussed before any aggregate metric
• No statement about Jev's quality, latency, or cost appears without recorded live inference; in a synthetic run, metrics are labelled as pipeline checks
• recipes/03-response-clarity-scoring/README.md says what the recipe teaches, how to run it, and what was and was not measured live
• The pull request changes only recipes/03-response-clarity-scoring/ and CI is green

Dependencies

• Blocked by Foundation: CI executes every notebook offline and validates fixtures #69 (CI executes every notebook offline and validates fixtures)
• Tracked in Epic: Level 1 · Beginner recipes #75

Sources

Read these before writing the questions.

• S02 · TypeSafe AI: Primitives (Questions): Choice, Noul, Score, independent questions, and dependent follow-up calls.
• S07 · TypeSafe AI: Jev 1.13 jaggedness: Version-specific limitations affecting arithmetic, context selection, adversarial content, and generation.

Out of scope

• Changes to shared code, the catalog, or the root README. If the shared package is missing something, open a foundation issue and link it here.
• Real side effects of any kind. Actions are simulated.