Recipe 32: Regression test selection #32

Recipe 32 of 60 · Level 3 (Intermediate) · Software engineering · Score

Use case

Score candidate tests against the behavior changed by a patch so Python can propose a focused regression suite.

What to build

The folder recipes/32-regression-test-selection/ with notebook.ipynb, README.md, and fixtures/, meeting the recipe contract. Start from python tools/new_recipe.py 32 and the template in recipes/_template/.

Build notes

• One Score per candidate test for relevance to the behaviour the patch changes.
• Python proposes a suite under a size budget.
• Baseline: file-path and lexical overlap between patch and test.

Decision design

• Score. Describe every rubric level in words, in order. Show the score and the distribution over levels. Mapping, weighting, and arithmetic on scores happen in Python.

Level 3 scope

Multiple judgments or stages, structured evidence, deterministic composition, and task-level evaluation.

Acceptance criteria

• recipes/32-regression-test-selection/notebook.ipynb runs top to bottom offline, with no key and no network, and its outputs are committed
• Live mode is opt-in (JEV_COOKBOOK_LIVE=1 with TYPESAFE_API_KEY) and uses the same question definitions as the offline run
• Fixtures are synthetic, include the hard cases named in the build notes, carry gold labels, and pass the fixture validator
• Every response fixture records its provenance, and the notebook states which mode it ran in
• Evaluation: Recall of the gold relevant tests at the budget, against the baseline.
• The deterministic composition lives in a Python function with tests in tests/
• Evaluation is at task level, not only per question
• No statement about Jev's quality, latency, or cost appears without recorded live inference; in a synthetic run, metrics are labelled as pipeline checks
• recipes/32-regression-test-selection/README.md says what the recipe teaches, how to run it, and what was and was not measured live
• The pull request changes only recipes/32-regression-test-selection/ and CI is green

Dependencies

• Blocked by Foundation: CI executes every notebook offline and validates fixtures #69 (CI executes every notebook offline and validates fixtures)
• Builds on Recipe 17: Passage reranking #17 (Passage reranking). Merge those first and reuse their patterns; do not copy their fixtures.
• Tracked in Epic: Level 3 · Intermediate recipes #77

Sources

Read these before writing the questions.

• S02 · TypeSafe AI: Primitives (Questions): Choice, Noul, Score, independent questions, and dependent follow-up calls.
• S03 · TypeSafe AI: Confidence: Distribution-derived confidence and task-specific thresholds informed by observed outcomes.
• S04 · TypeSafe AI: How to build with TypeSafe: Narrow semantic judgments integrated with workflows, deterministic logic, and side effects controlled by code.
• S05 · TypeSafe AI: Example use cases: Application families covering classification, routing, scoring, retrieval, and verification.

Out of scope

• Changes to shared code, the catalog, or the root README. If the shared package is missing something, open a foundation issue and link it here.
• Real side effects of any kind. Actions are simulated.