Skip to content

docs: six runnable examples with captured output - #40

Merged
queso merged 2 commits into
mainfrom
docs/runnable-examples
Sep 18, 2026
Merged

queso merged 2 commits into
mainfrom
docs/runnable-examples

Conversation

@queso

@queso queso commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner

What

Adds examples/: six runnable, self-contained example directories ordered as a learning curve, each with its own agent, skills, scenario file, README, cost label, and captured output.

Example Shows Cost
01-hello-compare Smallest baseline-vs-proposed compare: one added instruction, one text grader, 0% to 100% ~$0.01
02-fix-a-regression Target + regression pair on commit-message subject length ~$0.10
03-measure-first Single-arm measure on an under-specified extraction prompt ~$0.01
04-judge-grader Calibrated LLM judge end to end: rubric, fixtures, calibrate, then compare ~$0.10
05-any-model openai runner against local ollama, no API key free
06-token-economics --max-turns and --raw-out for token-category analysis ~$0.19

The root README gains a "See it work" section pointing at 01.

Why the test is in here

test/examples.test.ts loads every examples/*/scenario.json through the real loadCompareConfig, including its file-existence validation. Without it the examples are documentation that drifts from the schema and breaks for the first user who copies one. With it, drift breaks CI instead.

Verification

  • bun run check: 120 tests pass, typecheck clean.
  • audience-appropriate.md hashes to the rubricSha256 in its committed calibration record, and assertJudgeCalibrated passes on the real grader spec, so 04 does not refuse on a fresh clone.
  • Every output string pasted into a README matches its source: the grader messages (grader.ts:72, :84), the judge verdict format (judge.ts:282), the sampling-noise NOTE (compare.ts:157), the turn-cap failure (compare.ts:472), the calibration refusal (judge.ts:198), and the raw filename (compare.ts:529).
  • 02's ^.{1,50}(\n|$) grader is correct because gradeText compiles patterns with no m flag, so it anchors to the start of the output rather than to any body line.
  • 06's seed contains no occurrence of "backoff" or "exponential", so the task cannot be grepped and queue.py is unambiguous among the five files.

Doc accuracy fixes (2d9d5d4)

Three inaccuracies the first commit shipped with, plus a fourth of the same
kind found while checking them:

  • examples/README.md and the root README claimed every example's output was
    captured from a real run. 05 was never executed (no ollama in the build
    environment) and its own README says so. Both now name 05 as the
    exception, and 05's table row is labeled illustrative.
  • The root README's "See it work" block called itself real output while
    omitting the NOTE: delta could be sampling noise line and the two
    baseline run N failed lines that a real run of 01 prints. It now shows
    01's full output.
  • 05's illustrative block claimed to be shaped exactly like
    formatCompareSummary's output but omitted the per-failing-run lines a 1/3
    baseline emits (compare.ts:161-168 pushes one per failing run
    unconditionally). Added.
  • 06's README did not state that its grader is a substring check, so a
    reader could take the 2/2 pass rate as measuring the single-line format the
    prompt and skill.md ask for. Both committed raw results show a sentence of
    reasoning ahead of the filename and still passed. Now stated, with the
    {"type": "text", "regex": ["^queue\\.py\\s*$"]} grader that would
    enforce the format (regex, not matches — GraderSpec has no matches
    field).

bun run check still green: 120 tests pass, typecheck clean. Markdown-only
changes; no example was re-run, so no captured number moved.

🤖 Generated with Claude Code

queso and others added 2 commits September 18, 2026 18:52
Adds examples/ as a learning curve, each directory self-contained (agent,
skills, scenario file) with its own README, cost label, and output from a
real run:

- 01-hello-compare: smallest baseline-vs-proposed compare
- 02-fix-a-regression: target + regression pair on commit-message length
- 03-measure-first: single-arm measure on an under-specified extraction
- 04-judge-grader: calibrated LLM judge, rubric + fixtures + gate
- 05-any-model: openai runner against local ollama
- 06-token-economics: --max-turns and --raw-out for token-category analysis

test/examples.test.ts loads every scenario.json through the real config
loader, so schema drift breaks CI instead of the first user who copies an
example. The root README gains a "See it work" section pointing at 01.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three inaccuracies the examples shipped with, plus one of the same kind
found while checking:

- examples/README.md and the root README claimed every example's output
  was captured from a real run. 05 was never executed (no ollama in the
  build environment) and its own README says so. Both now name 05 as the
  exception.
- The root README's "See it work" block called itself real output while
  omitting the NOTE sampling-noise line and the two baseline failure
  lines that a real run of 01 prints. It now shows 01's full output.
- 05's illustrative block claimed to be shaped exactly like
  formatCompareSummary's output but omitted the per-failing-run lines
  that a 1/3 baseline emits. Added.
- 06's README did not say that its grader is a substring check, so a
  reader could assume the 2/2 pass rate measured the single-line format
  the prompt and skill.md ask for. Both passing runs put a sentence of
  reasoning ahead of the filename and still passed. Stated plainly, with
  the regex grader that would enforce the format.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@queso
queso force-pushed the docs/runnable-examples branch from f89017f to 4f9af0e Compare September 18, 2026 18:52

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick review — approve

The change adds documentation and worked examples under examples/ with a small test that loads each scenario to catch schema drift in CI. The examples are additive and don't touch production code paths, and the accompanying test exercises every scenario's load path. No issues that warrant changes were identified.

0 inline comment(s).

Verdict was approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick review — comment

The change adds documentation and worked examples for the eval tool, with no production code touched. One arithmetic inconsistency survives in the README's captured-output block: the delta line reads +$0.0013 while the baseline and proposed costs ($0.0021 and $0.0033) imply +$0.0012, and the same block appears in examples/01-hello-compare/README.md. Worth correcting so the docs stay internally consistent; otherwise the change is sound.

1 inline comment(s).

Comment thread README.md
answers-with-summary-prefix (target)
baseline: 0/2 pass (0%) | $0.0021
proposed: 2/2 pass (100%) | $0.0033
delta: +100% pass | +$0.0013

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion · Arithmetic error in documented sample output: +$0.0013 should be +$0.0012

The captured output block shows baseline $0.0021 and proposed $0.0033, so the delta line should read +$0.0012, not +$0.0013. The same block is duplicated in examples/01-hello-compare/README.md (line 24). Since the README explicitly claims the output is 'captured from a real run', an internally inconsistent cost figure undermines that claim.

Suggested change
delta: +100% pass | +$0.0013
Correct the delta to +$0.0012 in both README.md and examples/01-hello-compare/README.md, or re-capture the block from an actual run so the numbers are self-consistent.

@queso
queso merged commit dddd104 into main Sep 18, 2026
2 checks passed
@queso queso mentioned this pull request Sep 18, 2026
@queso
queso deleted the docs/runnable-examples branch September 18, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant