Recipe 56: Adversarial harness evaluation #56

Recipe 56 of 60 · Level 5 (Expert) · Trust & security · Noul

Use case

Evaluate Jev screening at input, retrieved-context, proposed-action, and output boundaries using adversarial replay while measuring attack success, false positives, and task completion.

What to build

The folder recipes/56-adversarial-harness-evaluation/ with notebook.ipynb, README.md, and fixtures/, meeting the recipe contract. Start from python tools/new_recipe.py 56 and the template in recipes/_template/.

Build notes

• Screening at four boundaries: input, retrieved context, proposed action, and output.
• Replay an adversarial corpus and a benign task corpus through the harness.
• Report attack success, false positives, and task completion together.

Decision design

• Noul. Write each proposition as a statement that is true or false. One independent question per label or concern, sent together in one request. Noul has no confidence field, so each threshold is chosen from examples and evaluated.

Level 5 scope

Controlled optimization studies or closed-loop simulations requiring budgets, baselines, and held-out or trajectory evaluation.

Acceptance criteria

• recipes/56-adversarial-harness-evaluation/notebook.ipynb runs top to bottom offline, with no key and no network, and its outputs are committed
• Live mode is opt-in (JEV_COOKBOOK_LIVE=1 with TYPESAFE_API_KEY) and uses the same question definitions as the offline run
• Fixtures are synthetic, include the hard cases named in the build notes, carry gold labels, and pass the fixture validator
• Every response fixture records its provenance, and the notebook states which mode it ran in
• Evaluation: Attack success rate, false-positive rate, and task completion rate per boundary and combined.
• Budgets, seeds, and frozen configurations are recorded in the notebook output
• A baseline is run under the same conditions, and any held-out split is used exactly once after choices are frozen
• Results are reproducible from the committed seeds and fixtures
• No statement about Jev's quality, latency, or cost appears without recorded live inference; in a synthetic run, metrics are labelled as pipeline checks
• recipes/56-adversarial-harness-evaluation/README.md says what the recipe teaches, how to run it, and what was and was not measured live
• The pull request changes only recipes/56-adversarial-harness-evaluation/ and CI is green

Dependencies

• Blocked by Foundation: CI executes every notebook offline and validates fixtures #69 (CI executes every notebook offline and validates fixtures)
• Builds on Recipe 26: Prompt-injection screening #26 (Prompt-injection screening), Recipe 41: Approval scope review #41 (Approval scope review), Recipe 42: Outbound disclosure screening #42 (Outbound disclosure screening). Merge those first and reuse their patterns; do not copy their fixtures.
• Tracked in Epic: Level 5 · Expert recipes #79

Sources

Read these before writing the questions.

• S02 · TypeSafe AI: Primitives (Questions): Choice, Noul, Score, independent questions, and dependent follow-up calls.
• S03 · TypeSafe AI: Confidence: Distribution-derived confidence and task-specific thresholds informed by observed outcomes.
• S04 · TypeSafe AI: How to build with TypeSafe: Narrow semantic judgments integrated with workflows, deterministic logic, and side effects controlled by code.
• S07 · TypeSafe AI: Jev 1.13 jaggedness: Version-specific limitations affecting arithmetic, context selection, adversarial content, and generation.

Out of scope

• Changes to shared code, the catalog, or the root README. If the shared package is missing something, open a foundation issue and link it here.
• Real side effects of any kind. Actions are simulated.

Amendment (2026-10-20): multi-proposition Noul fixture coverage

Added under #283 before any builder is dispatched. This section adds to the acceptance criteria above; it is not part of "Out of scope". The same requirements are written for authors and builders in docs/recipe-template.md (the bullet beginning "Noul propositions are statements"), and tests/test_recipe_guard.py (noul_proposition_gold_coverage) enforces the first of them.

If this recipe asks more than one Noul proposition about the same state, its fixtures must satisfy all three requirements:

• Every proposition has at least one gold-true row and one gold-false row in each scored split, validation and test. Threshold selection uses only validation; the held-out test split stays untouched until choices are frozen.
• The recipe states which truth combinations the use case admits, and the fixtures include at least one row for each admissible combination. For two propositions, consider both true, both false, and each true alone; document any combination the use case excludes, and why.
• For every proposition, at least one row where that proposition's tag changes the composed Python outcome, with the other propositions' tags and the frozen policy settings held fixed. Identify the row.

These are separate obligations: positive and negative coverage alone does not establish truth-combination coverage, and neither establishes that each proposition can change the outcome.

• The fixture set meets the three multi-proposition Noul requirements above: gold-true and gold-false rows for every proposition in each scored split, every admissible truth combination, and an outcome-determining row for every proposition.