Epic: Level 2 · Easy recipes

Recipes 11 to 23. Context-sensitive matching, scoring, or verification with candidate sets and explicit uncertain or no-match outcomes.

Recipes (13)

Done when

Every recipe above is merged and its issue closed
The README catalog is regenerated (python tools/render_catalog.py) and shows these recipes as published
CI is green on main
A consistency pass across the level is done: section order, terminology, chart style, run-mode header

Dependencies

Scheduled after Epic: Level 1 · Beginner recipes #75. A recipe here may start earlier once its own blockers are closed.
Tracked in Cookbook roadmap: 60 recipes from foundation to release #80

Wave 2 consistency review

Read-only cross-recipe pass over the thirteen Level 2 recipes recipes/11-* through recipes/23-* at origin/main tip 65978eeafb2ed7a8a19d5f02c6ce8087b0f5e943 (the recipe 19 merge, PR #180), with recipes 01 and 08 and recipes/_template/ as references, against CONTRIBUTING.md, docs/recipe-template.md, docs/evaluation.md, docs/notebook-style.md and docs/glossary.md. No notebook was executed and no test suite was run. Correctness of individual numbers is out of scope (each recipe had two to four Opus rounds); this is about whether the thirteen read as one cookbook, and about residual contract slips.

Findings are grouped by route. Each gives recipe, file/cell, the exact text, and the fix.

Route → #163 (recipe-side follow-ups)

R1. Recipe 18 prints {selection} alone on every validation line — 14 lines, 5 cells.
notebook.ipynb cells pythons-part, gate-sweep, evaluation-validation, side-effect. E.g. pythons-part output: chosen confidence gate, frozen from here on: 0.8400 (over 9 real-candidate matches in validation, 1 of them wrong) (a selection step, not a reported result). The other twelve all print {selection}{check}. This is exactly the under-disclosure docs/recipe-template.md warns about ("in a synthetic or scripted run a validation number needs both, so it prints {selection}{check}, never {selection} alone"). Fix: append {check} to all 14 lines.

R2. Validation figure titles take four different forms across the thirteen.

{check} only — 13 evaluation-gate, 19 gate-sweep-code, 20 rule-sweep, 21 eval-curve, 22 gate-sweep
{selection}{check} — 14 sweep-code: title=f"Validation confidence gate, named picks only{selection}{check}"
{selection} only — 15 gate-curve: title=f"Validation confidence gate{selection}"; 15 sweep-chart: title=f"Validation, {len(val_examples)} examples{selection}"; 18 gate-sweep: title=f"Confidence sweep over validation{selection}"
no label at all — 12 gate-sweep: title="Validation confidence sweep"

Fix: one form everywhere ({check} only is the majority and reads best in a title); see #164 T2 to write the convention down.

R3. Recipe 17 prints six metric rows with no disclosure label — the only unlabelled metric lines in the thirteen.
notebook.ipynb cell selective: print(f" gate {t:.4f}: coverage {cov:.4f}, accuracy {acc:.4f}, risk {risk:.4f}"), output e.g. gate 0.5600: coverage 0.9000, accuracy 0.9444, risk 0.0556. These are test numbers in a synthetic run. Fix: append {check} (and to the risk-coverage curve (test), … header line above them).

R4. Recipe 23 is the only confusion matrix in the thirteen plotted with nothing printed.
Cell eval-confusion builds matrix and calls plot_confusion_matrix(...) with no print. 11, 13, 16, 18, 19, 20 and 21 all print the gold-by-predicted table beside the figure (21's eval-confusion is the cleanest model). Related: recipe 14 notstated-code prints print(matrix.labels) then print(matrix.matrix.tolist()) — a raw list dump (['not_stated', 'named'] / [[4, 0], [1, 16]]) rather than a labelled table. Fix: print the labelled table in both.

R5. Recipe 18 exempts no_match from the gate and never prints the counterfactual.
side-effect-md: "no_match is never gated on confidence at all" and "there is no side effect a confidence gate could protect there" (pythons-part-md). There is no ungated or counterfactual computation anywhere in the file. Its structural twin 22 does it properly — side-effect prints counterfactual on test, no_match gated too (not what this notebook ships): / no_match gated too: coverage 0.7000, accuracy 0.9286, risk 0.0714, and side-effect-note reads the trade off it. Fix: copy 22's counterfactual cell shape. 18 is the only exempting recipe of the six with no gated counterfactual.

R6. Recipe 19 uses evaluate_selective for a rule that has a second, unconditional review branch.
eval-md: "Every outcome goes through the same gate, so classify's own accept/review split is exactly confidence >= threshold". But an accepted unknown is written to the ReviewQueue with reason "no option fits" — python-md says so itself: "both land in the queue, but only one of them also clears the gate". So three of the fifteen test builds inside coverage 0.7895 are a person's decision, not an answer. Recipe 21, the same shape (needs_review → ReviewQueue), reports evaluate_outcomes and gets a materially different number (coverage 0.7500 against the gate-only 0.8750) and prints both. 19 also redefines the term: eval-md says "coverage is the fraction of builds the gate accepts", where the other twelve say "answered rather than held for review". Fix: report evaluate_outcomes(accepted, correct) with accepted false for a queued unknown, or print both views as 21 does; align the coverage definition.

R7. test_stored_answers_are_not_all_right is a real guard in eight recipes and a stub in five.
11, 14, 15, 17, 18, 21, 22, 23 recompute the frozen threshold from validation exactly as the notebook does and assert a wrong test answer at or above it (17: assert wrong_and_confident, "expected at least one confidently wrong test answer"). 12, 13, 16, 19, 20 only assert wrong, "the fixtures should contain some wrong answers" over all splits — they pass even if every mistake is caught by the gate, so they do not guard the lesson they are named for. Fix: give 12/13/16/19/20 the 17/21 form.

R8. Five recipes ship the pre-#162 tests/test_build_fixtures.py.
11, 15, 16, 17, 18 have no inputs/labels regeneration assertion (no build_inputs_and_labels, no _jsonl helper); 12, 13, 14, 19, 20, 21, 22, 23 match the current template. Fix: copy recipes/_template/tests/test_build_fixtures.py (the current one, not a merged recipe's).

R9. Internal issue numbers in a shipped recipe file.
recipes/16-discord-moderation-triage/tests/test_build_fixtures.py:1: """build_fixtures.py follows the recipes/_template/build_fixtures.py pattern (#124, #129):. The only occurrence in the thirteen; the template's own docstring carries no numbers. Fix: drop (#124, #129).

R10. "the issue" / "the issue's build note" cited in 11 of the 13.
Counts of the issue / build note: 11×5, 12×3, 13×1, 15×3, 16×6, 17×5, 18×2, 19×3, 20×3, 22×5, 23×3. 14 and 21 are clean. E.g. 11 python-md: "accepts no_clarification_needed as a final, un-gated result whatever its confidence (the issue's build note: options come from a predefined catalog…)"; 22 helpers.py ×3; 17 helpers.py ×2. A reader cannot see a private issue. Fix: rewrite as "this recipe's use case" / "the use case names", the way 14 and 21 already do.

R11. Terminology outside the fixed lexicon.

Recipe 17 uses "business cutoff" where the lexicon says business threshold — and it is baked into reason strings: helpers.py:114 "modal level below the business cutoff", :115 "modal level clears the business cutoff", pinned by tests/test_helpers.py:47,53, repeated in README.md:13 and in the notebook (python-md, and the python-rule output KB-301: level 3 at confidence 0.78 -> match (modal level clears the business cutoff)). 15 and 20, by contrast, say "business threshold". Fix: rename in all four places.
Recipe 21 eval-selective-md: "a response the normaliser settled has no confidence to gate and is accepted with certainty." "Certainty" is a banned synonym. Fix: "is accepted without a confidence check".
"gold answer" for gold label: 12 gold-positions-md ("which position … its gold set occupies" is fine, but the preceding "it actually spreads the gold answer across positions" is not), 17 build_fixtures.py:20 and helpers.py:211.
Recipe 12 questions-md: "and Jev commits to exactly one of them." This is the exact phrasing the lexicon bans for attributing behaviour to Jev. Fix: "a Choice answer names exactly one of them".

R12. Lexicon terms used before they are defined, in four recipes.

gold label is never defined in 12 (2 uses) or 20 (4 uses), with no glossary link either. 13 first uses it in upclose-ambiguous-prose and defines it two cells later in upclose-wrong-prose; 15 first uses it in questions-md and defines it in answer-md.
coverage / accuracy / risk are printed as sweep-table columns before definition in 12 (printed in gate-sweep, defined in selective-md) and 20 (printed in rule-sweep, defined in eval-selective-intro). The other eleven define each in the markdown cell immediately above the first cell that prints it.
selective prediction appears bare in 15, 17, 18, 20, 21, 22; only 11 (outcomes-md) defines it.

R13. Glossary linking is effectively absent and wildly uneven.
Links to docs/glossary.md: 16 has six (#state, #choice, #confidence, #gold-label, #provenance, #run-mode); 13, 14, 20, 23 have one (#gold-label); 18, 22 have one (#confidence); 11, 12, 15, 17, 19, 21 have none. Fix: adopt 16's pattern. (The deeper cause is #164 T1: six of the seven lexicon terms have no glossary entry to link to.)

R14. Stored-answer count missing from the intro in 19, 20, 21, 23.
11 ("The 40 stored answers in fixtures/ are all synthetic"), 15 (52 documents), 16 (41 messages), 17 ("43 stored, synthetic answers"), 18 ("31 hand-written synthetic answers"), 22 ("38 stored, hand-written synthetic answers") and 13 (40) all state it; 12 and 14 state split sizes only; 19, 20, 21, 23 state nothing. The template's own intro says "22 made-up messages". Fix: one clause in each intro.

R15. Recipe 11's README install paragraph does not match the post-#162 template.
11 replaces the pip install -e ".[dev]" line with pip install -e ".[ml]" -c .github/constraints-notebooks.txt as the environment for all three commands, and drops the "Reproducing the committed notebook outputs byte for byte…" paragraph. The other twelve match the template exactly. Fix: restore the template shape.

R16. Recipe 22's README quotes metric numbers in prose.
"Validation's link accuracy (0.95), both baselines (0.70, 0.80) and the rule's own coverage (0.90), accuracy (1.00) and risk (0.00) … link accuracy (0.85) … (0.70, 0.75) … 0.80 and 0.80 (support 5) … coverage (0.85), accuracy (0.88) and risk (0.12) … false-link rate (0.08)". No other README of the thirteen quotes a metric; a README cannot derive its numbers, so these go stale on any fixture edit. Fix: describe the shape, not the digits.

R17. Option-order evidence missing where the recipe claims it.
13 questions-md argues "Fixed positions, not fixed content … Python deliberately varies which position holds the correct rewrite" but never prints the distribution. 12 (gold-positions), 14 (position-code), 22 (rank) and 23 (position-table) all print one. 18 — a ranked-candidate recipe like 22 — prints none either. Fix: add the gold-position table to 13 and 18.

R18. Recipe 16 asserts the evaluate_outcomes/evaluate_selective agreement instead of showing it.
Cells evaluation-outcomes-validation and evaluation-outcomes-test contain assert (val_outcomes.coverage, val_outcomes.accuracy, val_outcomes.risk) == (val_selective.coverage, …) and print only one triple plus (evaluate_outcomes and evaluate_selective agree on all three). Recipe 12 selective prints both triples and a derived the two agree on every field: True. A bare assert in a notebook also hard-fails the Notebook (<recipe>) job on any fixture edit. Separately, the two sides are built from different correctness vectors (test_correct = [r.label == g …] vs test_selective from test_correct_raw = [a.choice == g …]), so the equality is not quite the claim the prose makes. Fix: print both triples, drop the assert, use one correctness vector.

R19. Recipe 21's coverage denominator is the called subset, not the split.
eval-selective: validation, called subset: answered 5 of 8 responses sent to Jev (coverage 0.6250). It is labelled clearly, but it is the only one of the thirteen whose coverage does not denominate over the split's scored examples, so a reader meeting the word for the eleventh time meets a new denominator. Fix: one sentence giving the whole-split figure too (the normaliser settles 11 of 19 validation responses with no request).

R20. Recipe 21 links an unpublished neighbour.
next-md: [36-card-game-action-selection](../36-card-game-action-selection/). The slug matches catalog/recipes.json rank 36, but recipes/36-card-game-action-selection/ does not exist at 65978ee. It is the only dangling Next-steps target in the thirteen. Apply whichever way #164 T8 resolves (the contract currently blesses this link and the Wave-2 rule forbids it).

R21. Seven of the thirteen have no inbound Next-steps link from any published recipe.
Inbound counts over all published recipes: 08 ×10, 01 ×6, 02 ×6, 04 ×6, 05 ×5, 03 ×4, 07 ×4, 10 ×4, 18 ×2, 11/13/15/16/20 ×1 — and zero for 12, 14, 17, 19, 21, 22, 23. Level 2 links overwhelmingly backward into Level 1 (eight of the thirteen link 08-faq-selection), so a reader finishing 23 is sent back to 03/08/10 and never forward through the level. Fix: have each Level-2 recipe link at least one Level-2 neighbour by slug, so no recipe in the band is an orphan.

R22. Recipe 22's fixtures are real products at real versions — CONTRIBUTING §5.
fixtures/inputs.jsonl and build_fixtures.py use real vendors and real products at real, published build numbers: Adobe Systems Incorporated | Acrobat | 11.0.23, Adobe | Acrobat Reader DC | 2023.008.20470, Oracle Corp | Database | 19.3.0.0.0 (19c), Oracle | Java SE | 17.0.9, SAP SE | ECC | 6.0 EHP8, Red Hat, Inc. | RHEL | 8.6, Atlassian | Jira Software | 9.4.1, Atlassian | Confluence | 8.5.4, MS Corp | SQL Server | SQL2019 CU18 (15.0.4261.1), Microsoft | SQL Server | 13.0.6300.2 (2016 SP3), 16.0.1000.6, SQL 2022, plus Salesforce, Zoom, AgileBits/1Password, Docker, GitHub, GitLab, Notion, Figma. helpers.py hard-codes the real legal names as alias tables: "adobe systems incorporated", "salesforce.com, inc.", "red hat, inc.", "sap ag", "oracle corp", "microsoft corporation", "erp central component". §5 is "Fabricated names and numbers only", and recipe 19 — after its r2 review — solves the identical problem the other way, fabricating its packages (acme-sdk==4.2.0, search_index==1.8.3) inside otherwise realistic pytest/pip output. These are the two recipes a reader is most likely to read back to back on this point. Fix: rename to fabricated vendors and products with fabricated build numbers, keeping the look-alike / major-version-sibling / edition-sibling structure that carries the lesson. Lesser instance: 18 build_fixtures.py:194 "The internal Slack integration stopped posting deploy notifications to the team channel."

R23. Recipe 21's cost framing.
eval-baseline-md: "Calling Jev is only worth it if it beats something much cheaper. Both baselines below are…". No figure, but it is a sentence about Jev's cost in a recipe with no recorded run (CONTRIBUTING §2 allows none). Fix: "…only worth it if it beats a rule you could have written in a few lines."

Route → #164 (template / docs / contract)

T1. Six of the seven fixed-lexicon terms have no glossary entry.
docs/glossary.md defines state, question, Choice, Noul, Score, criteria, probabilities, confidence, gold label, threshold, accuracy, replay key, fixture, provenance, run mode — and not business threshold, confidence gate, review, coverage, risk or selective prediction. That is the root cause of R12 and R13: thirteen recipes each define these in their own words, and there is nothing to link. Fix: add the six entries (splitting threshold into business threshold vs confidence gate, and pinning the review outcome value "review" and reason string "confidence below the threshold"), and state in docs/recipe-template.md that each is defined on first prose use with a glossary link.

T2. The template does not settle the label on a figure title. The template has no validation figure, so the thirteen invented four conventions (R2). Decide one ({check} only) and write it beside the existing {selection}{check} bullet in docs/recipe-template.md.

T3. "Print what you plot" is not in the template. recipes/_template/notebook.ipynb cell evaluation-matrix plots the confusion matrix with no printed table — so recipe 23 matches the template, and seven recipes exceed it. Fix: print the table in the template cell and state the rule (it is currently only in the Wave-1 lesson list).

T4. CONTRIBUTING §4's fallback clause gives no test to apply, and the thirteen split on the same premise.
Each recipe's §4 rule, as shipped:

#	fallback	final at any confidence?	goes to review	counterfactual shown
11	`no_clarification_needed` → `proceed`	yes, un-gated	real catalog match below gate	yes (`comparison`: outcomes 0.9474/0.8889/0.1111 vs selective 0.8947/0.9412/0.0588)
12	`keep_original`	no — gated like every option	below gate; foreign option	n/a (prints both helpers; they agree)
13	`no_suitable_rewrite` → `kept_original`	yes, un-gated	real candidate below gate	yes (`evaluation-selective`: test 0.8421/0.8750/0.1250 vs 0.7895/0.9333/0.0667, plus the disagreeing ids)
14	`not_stated`	yes, un-gated	named pick below gate; foreign option	yes (`confonly-code`: 0.6190/0.8462/0.1538 vs 0.4286/0.8889/0.1111)
16	`review_needed` (and `allowed`)	no — every option gated	below gate → `escalate`	argues against the exemption; prices it (val coverage 0.6316 vs 0.6842)
18	`no_match` → `no_duplicate`	yes, un-gated	real candidate below gate; foreign option	none (R5)
19	`unknown` → `manual_triage`	gated, but an accepted `unknown` goes to the `ReviewQueue`	below gate → queue	no (and counts queued `unknown` as coverage — R6)
21	`needs_review`	never final — always queued	Jev's own `needs_review`; below gate	yes (`eval-compare`: test 0.7500/0.8333/0.1667 vs 0.8750/0.8571/0.1429)
22	`no_match`	yes, un-gated	real candidate below gate; foreign option	yes (`side-effect`: `no_match gated too: coverage 0.7000, accuracy 0.9286, risk 0.0714`)
23	`insufficient_evidence`	yes, un-gated	orders disagree; agreed `a`/`b`/`tie` below gate	yes (`eval-selective-vs-outcomes`: 0.7000/0.8571/0.1429 vs 0.9000/0.7222/0.2778, with both disagreement lists)
15, 17, 20	— no fallback option	n/a	gate only	n/a

The contradiction is not in the outcomes but in the argument. Six exempt their fallback because "choosing it triggers no side effect"; two reach the opposite conclusion from the identical premise:

12 python-md: "keep_original here is a real, final answer … it triggers no side effect of its own, and resolve holds it to exactly the same confidence gate as every other option rather than an automatic review".
16 python-md: "allowed resolves to ignore (no side effect, so there is nothing left for a later step to protect)" — followed three sentences later by "allowed gets no exemption: a confidently wrong allowed answer leaves a message standing that a person never sees". That parenthetical contradicts 16's own conclusion.

A real distinguishing test exists — 11/13/14/18/22/23's fallback has no payload to protect and its being wrong leaves nothing standing that a person would otherwise have caught, whereas a wrong allowed leaves a violating message up and a wrong keep_original is a positive, scorable claim — but no recipe states it and none cross-references a neighbour that decides the other way. Recipe 11 comes closest and then undercuts itself: "no queue, no routing, no action is taken on it … in a real pipeline the cost would be the task moving on to its next step without the information it actually needed", which is the same "nothing happens but the harm stands" shape 16 refuses to exempt.
Fix: put the two-part test in §4 and in the template bullet; require the exemption to be shown with gated counterfactual numbers (18 is the one gap today); and have each of the seven arguing recipes name the test in one sentence and point at the neighbour that decides the other way.

T5. Nothing says whether an "answered-but-queued" outcome counts toward coverage. 19 counts an accepted unknown (routed to the ReviewQueue) inside coverage; 21 does not count an accepted-branch needs_review. Fix: settle it in docs/evaluation.md "Selective prediction", and say plainly that a rule with any branch sending an accepted answer to a person is not confidence-only and needs evaluate_outcomes.

T6. evaluate_selective and sentinel confidences — the doc and the best recipe disagree. docs/evaluation.md and docs/recipe-template.md say never hand a sentinel confidence to evaluate_selective. Recipe 14 confonly-code does: confidence_only = [1.0 if a is None else a.confidence for a in test_answers], and confonly-md discloses it and quantifies the distortion ("covers 9 of 21 test documents (0.4286) rather than the 8 of 20 asked documents a gate could actually apply to (0.4000)"). This is the right handling if it is allowed at all. Fix: bless the disclosed, in-range stand-in explicitly, or forbid it and say what to do instead.

T7. Extra sections: ## sibling or ### under Evaluation? docs/recipe-template.md says "Nine sections, in a fixed order … A recipe keeps these headings." 14 (### Where the gold span sits, ### Three baselines with no model at all, ### What the confidence gate actually buys on test, ### A confidence-only view of the same test answers) and 21 and 22 use ### under Evaluation. 11 (## What the rule's own outcomes say, ## Why this is not the same as reapplying one confidence gate), 16 (## Simulating the moderator queue, before Evaluation), 17 (## Candidate sets and the lexical baseline, ## Ranking: nDCG and top-1 accuracy), 18 and 22 (## The simulated side effect(s)) add ## siblings. Fix: pick one and say so.

T8. Forward neighbour links: the contract and the review rule contradict. docs/recipe-template.md: "A folder link 404s on GitHub until that recipe's notebook.ipynb is committed; that is expected, not a defect to fix by removing the link." The Wave-2 rule applied from recipe 22's review: "Next-steps neighbours must exist on main at review time." Only recipe 21 is affected (R20). Fix: resolve once, in one place.

T9. No rule on citing "the issue". Eleven of the thirteen cite a private issue as a source in shipped prose and code (R10); the convention exists only as a one-line lesson from recipe 20's review, and 20 itself still carries three. Fix: state in CONTRIBUTING §6 that recipe files cite only reader-visible sources (the catalog's S numbers, repository docs, the recipe's own files).

T10. The template defines "gold label" in a code comment. recipes/_template/notebook.ipynb cell evaluation-validation: # The metrics below are about the examples that have a gold label (the example's recorded correct answer, from fixtures/labels.jsonl, used only to score an answer after the fact, never shown to Jev); the demo ones do not. CONTRIBUTING §6: "Prose in markdown cells, not in code comments." Recipes copied the placement, which is part of why 12 and 20 never define the term in prose at all. Fix: move it into evaluation-md with the glossary link from T1.

T11. The cell-id suffix convention is unstated. Discussion cells are -note (11, 15, 16, 18, 22), -discuss (12, 17, 19), -prose (13, 14) or -md (20, 21, 23); the last cell is next-md (11-17, 21), next-steps (18, 19, 20, 22) or next-steps-md (23); recipe 20 uses -intro for every section header (setup-intro, state-intro, questions-intro, …) and 19 appends -code to code cells (setup-code, measured-code). Every id is semantic, so this is cosmetic — but it is the first thing anyone diffing two recipes notices. Fix: name the suffixes in the template.

Route → #165 (parametrised guard test)

G1. Fold these into the parametrised suite over published recipes so they cannot drift per recipe:

(a) tests/test_build_fixtures.py matches the current template modulo the slug — catches R8 (five recipes behind today).
(b) no #NNN in any recipe file — catches R9.
(c) test_stored_answers_are_not_all_right exists and recomputes the frozen threshold and asserts a wrong test answer at or above it. A name-only check passes today for 12/13/16/19/20 — catches R7.
(d) every printed line in a validation cell carries {selection}{check}, and every printed metric line carries at least {check}. tests/test_template.py::test_every_metric_line_of_the_evaluation_carries_the_pipeline_check_label already does this for the template only — catches R1 (14 lines) and R3 (6 lines).
(e) every plot_* call has a print of the same data in the same cell, and figure titles follow the single convention T2 settles — catches R4 and R2.
(f) every ../NN-slug/ in a Next-steps cell resolves to a folder containing notebook.ipynb (once T8 is decided) — catches R20; and every published recipe is the target of at least one other recipe's Next steps — catches R21.

Route → #161 / #172 / #175 / #181 (shared package)

S1 → #181. The stable seeded permutation helper has now been written twice, independently, with the same convention: 12 build_fixtures.py:67-70 (hashlib.sha256(f"{SEED}:{item_id}".encode()).hexdigest() → random.Random(int(digest[:16], 16))) and 23 helpers.py:83 (hashlib.sha256(f"{seed}:{comparison_id}".encode()).digest(), first-byte parity). Take both as the interface test, and place the helper where a build_fixtures.py and a helpers.py can both call it (12 keeps it in the builder, 23 in helpers).

S2 → #161. No outcome-based risk–coverage sweep exists, and five recipes each write their own paragraph apologising for it: 13 evaluation-gate, 18 gate-sweep, 19 gate-sweep-code, 21 eval-curve and 22 gate-sweep all plot selective_curve and then explain in prose that the curve is confidence-only and therefore does not describe the rule they just reported. 21 eval-curve-md spends three sentences on it ("No outcome-based sweep helper exists yet in jev_cookbook.evaluation"). An outcome_curve that sweeps the gate through the rule itself would delete five bespoke caveats.

S3 → #161. Seven spellings of the (accepted, correct) pair evaluate_outcomes takes: 11 helpers.outcome_accounting, 12 inline [r.outcome == helpers.ACCEPTED …], 13/14/16 inline [r.outcome != helpers.REVIEW …], 18 and 22 helpers.accepted_and_correct, 21 inline inside selective_report, 23 inline. A shared accepted_and_correct(results, gold, *, accepted_when, label_of) would make the §4 story look identical in every recipe.

S4 → #161. helpers.false_merge_rate (18) and helpers.false_link_rate (22) are the same function under two names: the error rate within the accepted subset that actually carries the side effect. Both notebooks explain in prose that "evaluate_outcomes has no notion of a second, narrower outcome inside 'answered'". That belongs in the shared helper (evaluate_outcomes(..., side_effect=...), or an action_error_rate).

S5 → #172 / #175. No new instances in the thirteen. 15 is the only Noul recipe and its selective_curve shows no duplicate row. No recipe builds a single-option Choice: 22 no_candidate_resolution and 14 helpers.no_candidates both short-circuit before a question is built, as #175 prescribes. Replay keys are unique within every recipe (key slots equal distinct keys in all thirteen: 11/13/19/20 ×40, 12/17 ×43, 14 ×40 of 43 inputs, 15 ×52, 16 ×41, 18 ×31, 21 ×17 of 40, 22 ×38 of 43, 23 ×82), so no cross-split leak.

Verified clean across all thirteen

So the findings above are read against what is already uniform: the run-mode header (every recipe calls run_header(N, title, backend=backend, n_examples=len(scored)) and prints the exact three-line synthetic text); the mode-conditional measured cell (byte-identical to the template in all thirteen, with only additive extra prints in 16, 20 and 23); the nine section headings in the template's order; semantic cell ids throughout; the chart style (no direct matplotlib/plt anywhere — every figure comes from jev_cookbook.style); the review reason string, exactly "confidence below the threshold" in all thirteen helpers.py; the sources lists (README and notebook intro both equal catalog/recipes.json for all thirteen); the live budget note (equal to the stored-answer count in all thirteen, 21 correctly noting 17 is below the default 25); the "What was and was not measured" README sections (all thirteen close with "they say nothing about how Jev performs, how fast it is, or what it costs", and all say "not measured live"); fabricated-data discipline outside R22 (15's PII fixtures use the reserved 555-01xx range and .test email domains throughout); and test_stored_answers_are_not_all_right present in every recipe.

Overall judgement

The thirteen read as one cookbook on the things that were mechanised — the header, the measured cell, the section order, the chart helpers, the review reason string, the sources and budget notes are indistinguishable across all of them, and the best of these recipes (14, 21, 22, 23) teach the evaluate_outcomes-versus-evaluate_selective distinction better than the template does. Where they stop reading as one cookbook is everywhere the convention lived only in a reviewer's head rather than in docs/ or a test: four different labels on a validation figure title, six recipes with no glossary link and six lexicon terms with no glossary entry to link to, five recipes whose test_stored_answers_are_not_all_right is a stub while eight recompute the frozen threshold, five recipes on the pre-#162 fixture-builder test, and eleven recipes citing a private issue at the reader. The two findings that would actually mislead a learner are narrower: recipe 19 reports confidence-only coverage for a rule that routes accepted unknown answers to a person (R6), which contradicts recipe 21's handling of the identical shape two recipes later; and the §4 fallback argument is used to justify both exempting and gating a no-side-effect fallback, with recipe 16's own parenthetical contradicting its own conclusion (T4). Neither is a defect in any single recipe's numbers — both are the cost of §4 and docs/evaluation.md leaving the test unstated. Fix T4, T5 and T1 in the contract and docs, land the #165 guards, and the thirteen converge; leave them, and recipes 24 onward will each re-derive a slightly different answer.

Reviewed read-only at 65978eeafb2ed7a8a19d5f02c6ce8087b0f5e943. No notebook executed, no suite run, no repository file changed.

All thirteen Level 2 recipes merged and their issues closed; README catalog regenerated (23 of 60); CI green on main 65978ee; consistency pass posted above. Close-out report on #80.

Recipe 22: fabricate vendor, product and version names (#182)

Foundation: parametrised recipe guard with a known-failure allowlist (#…

Navigation Menu

Repository navigation

Epic: Level 2 · Easy recipes #76

Recipes (13)

Done when

Dependencies

Wave 2 consistency review

Route → #163 (recipe-side follow-ups)

Route → #164 (template / docs / contract)

Route → #165 (parametrised guard test)

Route → #161 / #172 / #175 / #181 (shared package)

Verified clean across all thirteen

Overall judgement

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Epic: Level 2 · Easy recipes #76

Description

Recipes (13)

Done when

Dependencies

Activity

CompleteDotTech commented on Oct 9, 2026

Wave 2 consistency review

Route → #163 (recipe-side follow-ups)

Route → #164 (template / docs / contract)

Route → #165 (parametrised guard test)

Route → #161 / #172 / #175 / #181 (shared package)

Verified clean across all thirteen

Overall judgement

CompleteDotTech commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions