feat: grader file format with test-style assertions (result.shouldHave / result.assert)

[queso]

Problem

Writing a grader today means hand-rolling ~70 lines of executable: arg parsing, artifact reading, exit-code plumbing, and stderr diagnostics — for what is usually three lines of actual assertion. The command contract (run in the sandbox, exit 0/1, stderr = diagnostics) is the right base layer, but its ergonomics are hostile enough that the grader is the highest-activation-cost part of authoring an eval. It is also the snippet people see first in a README or blog post, and today it reads like plumbing, not like a test.

Real example — the core of a working grader (autocut NAR-006 eval), after the boilerplate:

const covered = segs.some(
  (s) => Number(s.source_in) <= arg && arg <= Number(s.source_out),
);
console.error(`grade: t=${arg} covered=${covered} | ranges: ${ranges}`);
process.exit(mode === "cover" ? (covered ? 0 : 1) : covered ? 1 : 0);

Proposal: a grader file format with test-style assertions

One file, named graders, scenario cases reference them by name. Something in the shape of:

// grade.eval.ts
import { grade } from 'promptdiff'

export default grade('02_debating/edit_plan.draft.json', {
  'product-named': ({ result }) => {
    // sugar for the common cases:
    result.shouldHave('cinema cutie')
    result.shouldNotHave('lorem')
    result.shouldMatch(/cutie line/i)
    // plain code for semantic cases, with labeled diagnostics for free:
    const kept = result.json().a_roll
    result.assert(
      kept.some(s => s.source_in <= 8.0 && 8.0 <= s.source_out),
      'no kept range covers t=8.0',
      { ranges: kept },
    )
  },
  'still-edits': ({ result }) => {
    const kept = keptSeconds(result.json())
    result.assert(kept / 64.672 <= 0.9, `kept ${kept}s — keep-everything degeneration`)
  },
})

Scenario side:

"grader": { "file": "./grade.eval.ts", "name": "product-named" }

The framework owns: artifact reading, JSON parsing, exit codes, collecting all failed assertions (not die-on-first), and printing what-actually-happened diagnostics on failure — the part authors reliably skip until they are debugging a bare exit 1.

Design constraints (learned from evals already in the field)

1. Do not let the sugar seduce authors into string matching. Substring assertions are the right primitive for maybe a third of real graders; the strongest existing evals are semantic (time-range coverage over an edit plan; regex rulers calibrated against labeled false positives). result.assert(expr, message, context) must be as front-and-center in docs and examples as shouldHave, or the DSL will mint brittle graders.
2. The command contract stays the base layer. Graders as arbitrary executables are why rulers exist in bash and range-math graders in bun. The DSL is a runner that compiles down to the same exit-code contract — sugar on top, never a replacement. Non-JS graders remain first-class.
3. Missing artifact becomes its own failure class. Today a missing draft exits nonzero for both positive (cover) and negative (uncover) assertions, and "grade a missing draft as no signal, never as a real failure" is manual discipline documented in eval READMEs. The framework should distinguish no-artifact from assertion-failed natively so negative assertions cannot silently pass — and reproduction stats cannot silently inflate — on an agent that produced nothing.

Scope notes

• Additive; no change to existing "grader": { "type": "command" } scenarios.
• feat: → ships as 1.1.0 through release-please + trusted publishing.

🤖 Generated with Claude Code

[queso]

Field evidence for design constraint 3 (missing-artifact as its own failure class), from a real measure run today: a 5-run reproduction probe reported 4/5 pass, but the one "fail" was grade: cannot read 02_debating/edit_plan.draft.json: ENOENT — the agent wrote no artifact at all. For a probe measuring how often a defect reproduces, that tally silently counts a no-signal as a reproduction; the honest number was 4/4 pass, 1 no-signal. The runner should surface no-artifact distinctly in the summary line, not just in grader stderr.

Related ergonomics papercut from the same session: compare on a baseline-only scenario dies with compare requires proposed skill paths — correct, but the error should point at measure, which is exactly the mode a reproduction probe wants.