Recipe 55: Adaptive model routing #55

Recipe 55 of 60 · Level 5 (Expert) · Evaluation & optimization · Choice + Score

Use case

Use Jev judgments to drive a model-routing policy and compare quality, latency, and cost across traffic mixes while Python enforces budgets and fallback rules.

What to build

The folder recipes/55-adaptive-model-routing/ with notebook.ipynb, README.md, and fixtures/, meeting the recipe contract. Start from python tools/new_recipe.py 55 and the template in recipes/_template/.

Build notes

• Jev judgments drive a policy that routes requests among model tiers.
• Compare quality, latency, and cost across traffic mixes. Python enforces budgets and fallback rules.
• Offline, tier costs and latencies are stated assumptions in a table, labelled as such.
• Switching models mid-session invalidates the prompt cache. The assumptions table includes a per-switch cache-miss cost, and the traffic mixes include multi-turn sessions as well as independent requests, so the policy pays for switching.
• The README names routing reasoning effort on a single model as a variant that avoids switching costs, and links reasoning-router as related external work. Its figures come from that project's own evaluation. They are not reported as results of this recipe and are not used as fixtures.

Decision design

• Choice. Options are a fixed set supplied by Python, including the fallback outcome this use case names. Show the choice, the per-option probabilities, and the confidence for individual examples before aggregating.
• Score. Describe every rubric level in words, in order. Show the score and the distribution over levels. Mapping, weighting, and arithmetic on scores happen in Python.
• Composition. Independent questions share one request. A question that depends on an earlier answer goes in a later request, and the notebook makes that boundary visible.

Level 5 scope

Controlled optimization studies or closed-loop simulations requiring budgets, baselines, and held-out or trajectory evaluation.

Acceptance criteria

• recipes/55-adaptive-model-routing/notebook.ipynb runs top to bottom offline, with no key and no network, and its outputs are committed
• Live mode is opt-in (JEV_COOKBOOK_LIVE=1 with TYPESAFE_API_KEY) and uses the same question definitions as the offline run
• Fixtures are synthetic, include the hard cases named in the build notes, carry gold labels, and pass the fixture validator
• Every response fixture records its provenance, and the notebook states which mode it ran in
• Evaluation: Quality, latency, and cost per traffic mix against always-cheapest and always-strongest baselines, every figure labelled assumed or recorded.
• The offline run is presented as a harness check. The recipe README has a section on how to produce real results with a live run, and reports none unless one was recorded
• Budgets, seeds, and frozen configurations are recorded in the notebook output
• A baseline is run under the same conditions, and any held-out split is used exactly once after choices are frozen
• Results are reproducible from the committed seeds and fixtures
• No statement about Jev's quality, latency, or cost appears without recorded live inference; in a synthetic run, metrics are labelled as pipeline checks
• recipes/55-adaptive-model-routing/README.md says what the recipe teaches, how to run it, and what was and was not measured live
• The pull request changes only recipes/55-adaptive-model-routing/ and CI is green

Dependencies

• Blocked by Foundation: CI executes every notebook offline and validates fixtures #69 (CI executes every notebook offline and validates fixtures)
• Blocked by Foundation: opt-in live Jev backend and response recorder #64 (opt-in live Jev backend and response recorder)
• Builds on Recipe 50: Model comparison and calibration #50 (Model comparison and calibration). Merge those first and reuse their patterns; do not copy their fixtures.
• Tracked in Epic: Level 5 · Expert recipes #79

Sources

Read these before writing the questions.

• S02 · TypeSafe AI: Primitives (Questions): Choice, Noul, Score, independent questions, and dependent follow-up calls.
• S03 · TypeSafe AI: Confidence: Distribution-derived confidence and task-specific thresholds informed by observed outcomes.

Out of scope

• Changes to shared code, the catalog, or the root README. If the shared package is missing something, open a foundation issue and link it here.
• Real side effects of any kind. Actions are simulated.