feature-guide.md

Feature Guide

This page explains the main agent-eval primitives in plain English first, then shows when to use each one.

Overview

LLM agents can write code, drafts, research, plans, and actions. Evaluation measures whether the result met its requirements, whether another attempt is warranted, and whether a change improved outcomes.

agent-eval gives you reusable tools for that:

• Judges grade one output.
• Verifiers run several checks in order.
• Control loops let an agent keep working until it passes, gets blocked, or hits a budget.
• Feedback trajectories turn normal user approvals/rejections into training and eval data.
• Datasets and holdouts keep examples organized so you do not overfit.
• Optimizers and mutation loops try prompt, signature, or code variants and retain measured improvements.
• Traces and telemetry show what happened, step by step.

Which Primitive Should I Use?

Problem	Use	Why
“Did this single answer/draft pass?”	Judge or rubric	Fast quality signal for one artifact.
“Does generated code work?”	BuilderSession, MultiLayerVerifier	Build, test, and runtime checks catch failures judges miss.
“Should the agent keep trying?”	runAgentControlLoop	Budgeted observe -> validate -> decide -> act runtime.
“The agent should propose, verify, review, and revise.”	runProposeReview	Iterative propose/verify/review loop.
“Human feedback should become reusable eval data.”	FeedbackTrajectory	Captures approvals, rejections, edits, choices, metrics, and policy blocks.
“I want Vercel-style eval folders agents can add quickly.”	loadEvalFixtureScenarios, planEvalFixtureRun, runCampaign	Folder-per-eval DX with campaign scoring and cache-safe dry runs.
“Can this action run, or does it need approval?”	evaluateActionPolicy	Generic preflight for side effects, budgets, and required evidence.
“I need train/dev/test/holdout examples.”	Dataset plus feedback trajectory conversion	Stable splits and contamination control.
“Which optimization procedure wins?”	compareOptimizationMethods	Runs complete methods on shared train and selection cases, then compares them on separate final cases.
“Improve a multi-turn agent with candidates from my runtime.”	runImprovementLoop	Evaluates caller-generated candidates and applies a separate release rule.
“Improve prompts, then code if prompts plateau.”	gepaOptimizationMethod or externalTextOptimizationMethod for the prompt; agent-runtime's worktree path for the code	Text search stays here; executable code changes belong to the runtime.
“Find why a regression happened.”	bisector, traces, run records	Narrows changes and preserves evidence.
“Expose evals to another language.”	Wire protocol and Python client	HTTP/RPC boundary for non-TypeScript apps.

Integration Patterns

Recommended Product Shape

Use this shape when the product needs to keep pushing work forward instead of only answering once:

user intent
  -> product adapter observes typed state
  -> runAgentControlLoop decides the next driver action
  -> worker executes in the real product environment
  -> validators and judges grade the new state
  -> FeedbackTrajectory records user/environment/reviewer signal
  -> datasets and optimizers replay the same adapter

Keep the production state, validators, actions, budgets, and stop policies in the evaluation path. The adapter can supply a real user session, replay fixture, or sandbox. This tests the behavior that production executes. Transfer to future tasks still requires representative evaluation data and a measured comparison.

Agent Runtime Integration

Use when you have a coding, browser, computer-use, research, or documentation agent.

1. Represent the current task state.
2. Validate that state with objective checks first and judges second.
3. Use runAgentControlLoop to decide the next action.
4. Record user feedback as FeedbackTrajectory.
5. Convert trajectories into datasets and optimizer rows.

Result:

normal agent usage -> labeled examples -> replay/eval -> optimization

Implementation ownership:

• Put reusable loop mechanics, trace schemas, budget accounting, split assignment, and optimizer row conversion in agent-eval.
• Put product state readers, action executors, approval policy, credentials, workspace paths, and UI-specific storage in the downstream repo.
• Keep product execution adapters in the consuming repository.

Code Generator

Use when an agent writes or patches a repo.

1. Use BuilderSession or MultiLayerVerifier.
2. Always run static gates like typecheck/build/tests.
3. Add semantic judges only after build gates pass.
4. Store traces and run records for regression debugging.

Result:

generated code -> build/test/runtime gates -> score -> ship or revise

Prompt And Skill Optimization

Use gepaOptimizationMethod() for official GEPA recipes. Use skillOptOptimizationMethod() for Microsoft's official SkillOpt trainer. For DSPy programs, use official DSPy optimizers with Python's DspyJudgeMetric. Run complete methods through compareOptimizationMethods(). Use runImprovementLoop() only when your application or runtime owns candidate generation through SurfaceProposer.

1. Build train, selection, and final splits from the real product path.
2. Score complete trajectories rather than final text alone.
3. Give optimizers actionable evidence from train and selection failures.
4. Keep final cases out of candidate generation and method selection.
5. Record prompt hash, model, configuration, cost status, and source revision.

Result:

candidate variant -> repeated evaluations -> statistical comparison -> release decision

Do not optimize a toy harness if users run a different product loop. Build training examples from FeedbackTrajectory and controlRunToFeedbackTrajectory, then replay them through the same product adapter used at runtime. Train on train, tune on dev, report on test, and keep holdout untouched until a promotion decision.

Human Feedback Data

Use when operator or reviewer interaction should create labels.

Capture:

• approve/reject
• select A/B/C
• edit/rewrite
• rank/rate
• comment
• metric outcome
• policy block or budget block

Store as FeedbackTrajectory, then derive:

• preference memory for the next run
• dataset scenarios for regression
• optimizer rows for prompt/signature/code changes
• holdout examples to detect overfitting

Feature Map

Area	Key exports	Best for	Notes
Judging	llmJudge, semantic judges, anti-slop, wire rubrics	Content, voice, semantic quality	Pair with objective checks when possible.
Verification	MultiLayerVerifier, sandbox harness	Code and multi-step gates	Do not let semantic judges override failed builds.
Control	runAgentControlLoop, objectiveEval, subjectiveEval	Long-running agent tasks	Supports budgets, cost, stop policies, trace spans.
Propose/review	runProposeReview	Iterative artifact repair	Good for code, docs, plans, briefs.
Feedback data	FeedbackTrajectory, stores, converters	Human/environment labels	Domain adapters live in downstream repos.
Action policy	evaluateActionPolicy	Approval/budget preflight	Blocks or labels actions before act().
Datasets	Dataset, holdout tools, canaries	Train/dev/test/holdout corpora	Keeps optimization honest.
Eval fixtures	discoverEvalFixtures, loadEvalFixtureScenarios, planEvalFixtureRun	Coding-agent eval folders	Vercel-style input shape; still executes through runCampaign.
Optimization	compareOptimizationMethods, official optimizer adapters, runImprovementLoop	Prompt and skill comparison	Reserve separate final cases for method ranking.
Evolution	prompt/code mutators, sandbox pool, telemetry	Autoresearch and mutation loops	Use budgets and lineage; do not run unbounded.
Telemetry	TraceStore, OTLP, file sinks	Evidence and replay	Store traces as inspectable run evidence.
Trace analysis	AnalystRegistry, createDspyRlmTraceEngine, defineTraceAnalyst	Diagnosing failures across a batch of runs	See trace-analysis.md and the agent-eval analyst-benchmark CLI.
Reporting	summaries, pareto, cost tracker	Decision support	Useful for PRs, launch gates, research notes.

Guardrails

• Prefer deterministic checks before LLM judges.
• Keep holdout data out of optimization.
• Record model, prompt hash, config hash, commit, and cost for every serious run.
• Budget every loop: steps, wall time, and dollars.
• Treat external side effects as downstream policy. The runtime can stop loops, but your adapter decides what requires approval.
• Store user feedback with enough context to replay it later.
• Run harnesses and judges in the same sandbox when they need shared files, logs, screenshots, or browser state. Use separate sandboxes for parallel variants or destructive checks.

Highest-ROI Adoption Order

1. Wrap one real product workflow in runAgentControlLoop.
2. Add objective validators that can fail without an LLM.
3. Emit traces and convert completed runs into feedback trajectories.
4. Capture explicit user/reviewer feedback on attempts.
5. Replay those trajectories as train/dev/test/holdout scenarios.
6. Run prompt/signature optimization against that replay adapter.
7. Promote only when held-out trajectories and product telemetry both improve.

What Stays Out Of Core

Domain-specific adapters should usually stay in downstream repos until they prove reusable:

• browser site-specific actions
• repo-specific coding commands
• workspace-specific storage paths

Core should provide shapes, stores, runners, scoring, traces, and converters. Downstream integrations provide domain state, policy, tools, and storage.

Examples

• examples/same-sandbox-harness: one workdir for install/build/test plus evidence inspection.
• examples/eval-fixtures-quickstart: Vercel-style fixture folders with a dry-run plan and campaign cache reuse.
• examples/multi-shot-optimization: full-trajectory optimization with a holdout gate.
• examples/benchmarks: benchmark adapter contracts and reference wrappers.