docs: six runnable examples with captured output - #40
Conversation
Adds examples/ as a learning curve, each directory self-contained (agent, skills, scenario file) with its own README, cost label, and output from a real run: - 01-hello-compare: smallest baseline-vs-proposed compare - 02-fix-a-regression: target + regression pair on commit-message length - 03-measure-first: single-arm measure on an under-specified extraction - 04-judge-grader: calibrated LLM judge, rubric + fixtures + gate - 05-any-model: openai runner against local ollama - 06-token-economics: --max-turns and --raw-out for token-category analysis test/examples.test.ts loads every scenario.json through the real config loader, so schema drift breaks CI instead of the first user who copies an example. The root README gains a "See it work" section pointing at 01. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three inaccuracies the examples shipped with, plus one of the same kind found while checking: - examples/README.md and the root README claimed every example's output was captured from a real run. 05 was never executed (no ollama in the build environment) and its own README says so. Both now name 05 as the exception. - The root README's "See it work" block called itself real output while omitting the NOTE sampling-noise line and the two baseline failure lines that a real run of 01 prints. It now shows 01's full output. - 05's illustrative block claimed to be shaped exactly like formatCompareSummary's output but omitted the per-failing-run lines that a 1/3 baseline emits. Added. - 06's README did not say that its grader is a substring check, so a reader could assume the 2/2 pass rate measured the single-line format the prompt and skill.md ask for. Both passing runs put a sentence of reasoning ahead of the filename and still passed. Stated plainly, with the regex grader that would enforce the format. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
f89017f to
4f9af0e
Compare
There was a problem hiding this comment.
Nitpick review — approve
The change adds documentation and worked examples under examples/ with a small test that loads each scenario to catch schema drift in CI. The examples are additive and don't touch production code paths, and the accompanying test exercises every scenario's load path. No issues that warrant changes were identified.
0 inline comment(s).
Verdict was
approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.
There was a problem hiding this comment.
Nitpick review — comment
The change adds documentation and worked examples for the eval tool, with no production code touched. One arithmetic inconsistency survives in the README's captured-output block: the delta line reads +$0.0013 while the baseline and proposed costs ($0.0021 and $0.0033) imply +$0.0012, and the same block appears in examples/01-hello-compare/README.md. Worth correcting so the docs stay internally consistent; otherwise the change is sound.
1 inline comment(s).
| answers-with-summary-prefix (target) | ||
| baseline: 0/2 pass (0%) | $0.0021 | ||
| proposed: 2/2 pass (100%) | $0.0033 | ||
| delta: +100% pass | +$0.0013 |
There was a problem hiding this comment.
suggestion · Arithmetic error in documented sample output: +$0.0013 should be +$0.0012
The captured output block shows baseline $0.0021 and proposed $0.0033, so the delta line should read +$0.0012, not +$0.0013. The same block is duplicated in examples/01-hello-compare/README.md (line 24). Since the README explicitly claims the output is 'captured from a real run', an internally inconsistent cost figure undermines that claim.
| delta: +100% pass | +$0.0013 | |
| Correct the delta to +$0.0012 in both README.md and examples/01-hello-compare/README.md, or re-capture the block from an actual run so the numbers are self-consistent. |
What
Adds
examples/: six runnable, self-contained example directories ordered as a learning curve, each with its own agent, skills, scenario file, README, cost label, and captured output.01-hello-compare02-fix-a-regression03-measure-firstmeasureon an under-specified extraction prompt04-judge-gradercalibrate, then compare05-any-modelopenairunner against local ollama, no API key06-token-economics--max-turnsand--raw-outfor token-category analysisThe root README gains a "See it work" section pointing at
01.Why the test is in here
test/examples.test.tsloads everyexamples/*/scenario.jsonthrough the realloadCompareConfig, including its file-existence validation. Without it the examples are documentation that drifts from the schema and breaks for the first user who copies one. With it, drift breaks CI instead.Verification
bun run check: 120 tests pass, typecheck clean.audience-appropriate.mdhashes to therubricSha256in its committed calibration record, andassertJudgeCalibratedpasses on the real grader spec, so04does not refuse on a fresh clone.grader.ts:72,:84), the judge verdict format (judge.ts:282), the sampling-noise NOTE (compare.ts:157), the turn-cap failure (compare.ts:472), the calibration refusal (judge.ts:198), and the raw filename (compare.ts:529).02's^.{1,50}(\n|$)grader is correct becausegradeTextcompiles patterns with nomflag, so it anchors to the start of the output rather than to any body line.06's seed contains no occurrence of "backoff" or "exponential", so the task cannot be grepped andqueue.pyis unambiguous among the five files.Doc accuracy fixes (2d9d5d4)
Three inaccuracies the first commit shipped with, plus a fourth of the same
kind found while checking them:
examples/README.mdand the root README claimed every example's output wascaptured from a real run.
05was never executed (no ollama in the buildenvironment) and its own README says so. Both now name
05as theexception, and
05's table row is labeled illustrative.omitting the
NOTE: delta could be sampling noiseline and the twobaseline run N failedlines that a real run of01prints. It now shows01's full output.05's illustrative block claimed to be shaped exactly likeformatCompareSummary's output but omitted the per-failing-run lines a 1/3baseline emits (
compare.ts:161-168pushes one per failing rununconditionally). Added.
06's README did not state that its grader is a substring check, so areader could take the 2/2 pass rate as measuring the single-line format the
prompt and
skill.mdask for. Both committed raw results show a sentence ofreasoning ahead of the filename and still passed. Now stated, with the
{"type": "text", "regex": ["^queue\\.py\\s*$"]}grader that wouldenforce the format (
regex, notmatches—GraderSpechas nomatchesfield).
bun run checkstill green: 120 tests pass, typecheck clean. Markdown-onlychanges; no example was re-run, so no captured number moved.
🤖 Generated with Claude Code