ToolEval

An agent-agnostic harness that scores how well an LLM agent uses tools to complete multi-step tasks — separating outcome from process, measuring variance, and classifying failures, instead of reporting one pass/fail bit.

Built on the raw Anthropic SDK, no eval frameworks. See CONTEXT.md for the domain vocabulary and docs/adr/ for the three architectural decisions this project turns on.

---

The thesis

Most agent evals answer a single question: did it succeed? That one bit hides almost everything you need to trust an agent:

• An agent that succeeds by flailing through every tool isn't reliable.
• An agent that fails because a tool handed it garbage isn't broken.
• An agent that passes once might have just gotten lucky.

A single pass/fail score can't tell these apart. ToolEval refuses to collapse them. Every run is scored against four questions, not one:

Question	What it answers	How it's measured
Outcome	Did it achieve the goal?	Deterministic state comparison; LLM-as-judge only where state is too rigid
Process	Was the path reasonable?	Required calls present, forbidden calls absent, efficiency ratio
Variance	Consistent, or just lucky?	N runs per Task; pass rate + outcome/process mean & std
Failure cause	When it fails, why?	Mutually-exclusive, first-match failure taxonomy

These four are reported together as a Result vector. A single blended number is opt-in (--combine) and explicitly labelled lossy — leading with one number is the exact thing this project argues against.

---

What it looks like

The calc Suite and the test suite run fully offline from committed fixtures — no API key, no network:

pip install -e .
python run_eval.py --suite calc --runs 3        # trivial calc Suite (agnosticism proof)
python run_eval.py --suite job-search --runs 3  # full job-search Suite (the real target)
pytest                                           # fast, offline, deterministic

The scripted job-search Suite is offline too, with one exception: its graceful-degradation Task is graded by an LLM-as-judge (see §4) that calls Claude, so without ANTHROPIC_API_KEY that single Task is recorded as a run_error (flakiness-as-data — the eval still completes and the other four Tasks score normally). The four Step/Pipeline Tasks need no key in scripted mode.

A Result vector for one run, followed by the variance roll-up across runs:

                    Result vector — calc.arithmetic (run 0)
┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Outcome ┃ Process ┃ Variance       ┃ Failure        ┃ Do-nothing    ┃ Passed ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 1.00    │ 1.00    │ — not computed │ — not computed │ — not         │ ✓      │
└─────────┴─────────┴────────────────┴────────────────┴───────────────┴────────┘
                      Variance — calc.arithmetic (3 runs)
┏━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Runs ┃ Pass rate ┃ Out. mean ┃ Out. std  ┃ Proc mean ┃ Proc std  ┃ Var flag  ┃
┡━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ 3    │ 100%      │ 1.000     │ 0.000     │ 1.000     │ 0.000     │ —         │
└──────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘

Read it left to right: the agent achieved the goal (outcome 1.00) by the clean path (process 1.00), did so on every run (pass rate 100%, std 0.000), and never tripped a failure category. Variance and failure columns read — not computed on the single-run card because they're properties of the set of runs, summarised in the variance table beneath. Each run is also written to results/ as a standalone, human-readable JSON file (task + trajectory + score), never overwritten — the filename carries the task id, run index, and timestamp so history accumulates.

---

Architecture

Four ideas carry the design. The harness core contains zero job-search — or any-domain — logic; all domain knowledge lives in a Suite.

1. The Agent Contract

The harness reaches every agent through one interface and nothing else:

run(task, tool_handler) -> AgentResult

The harness never re-implements an agent's tool-use loop — doing so would score a copy of the agent, not the agent, and collapse the whole thesis (ADR 0001). Agents register named entry points in a registry; a Task names the entry point it targets. "Agent-agnostic" means agnostic behind the Contract — any agent that implements run(...) can be evaluated untouched.

2. Orchestrated decomposition (Steps + Pipeline)

The real Job Application Agent was re-architected from a monolithic tool-use loop into an orchestrator that composes focused Steps (extract → research → analyze → synthesize), each a mini-agent with a restricted tool subset. The full Pipeline is those Steps composed in order.

The payoff is honest multi-granularity evaluation: a single-step Task and the full-Pipeline Task exercise the same Step code, not a parallel test-only re-implementation. You can score job_search.extract in isolation and job_search.pipeline end-to-end, and both numbers describe shipped components.

3. Mock the world, not the agent's mind

The injected tool handler mocks only the world boundary — the flaky I/O tools (fetch_webpage, web_search, read_resume, save_output). The agent's reasoning always runs live, including any LLM call a Step makes internally. Swapping the mock handler for a live one (--live) needs no change to agent logic; the two modes are never mixed within a run.

Crucially, the mocks are not justified by reproducibility (ADR 0002). Cognition runs live, so runs aren't bit-reproducible — the variance machinery exists precisely to quantify that. Mocks earn their place three other ways:

1. Experimental control — holding the world fixed makes outcome variance attributable to the agent, not to a changing web. This is what lets the harness answer "consistent, or lucky?"
2. Authorable edge cases — adversarial world states (a paywalled posting, no company news, an empty resume, contradictory inputs) can't be summoned reliably from live tools. Fixtures are the only way to build them on demand.
3. A demo that doesn't rot — job-posting URLs die within months; frozen, committed fixtures keep run_eval.py working when a reviewer clones the repo long after the original postings are gone.

4. The vector-first Result model

Every run flows through plain dataclasses so all scoring is pure data transformation — no agent, no network:

• Task — id, description, target entry point, toolset, required_tool_calls, forbidden_tool_calls, minimal_necessary_calls, expected world state, tags.
• AgentResult — task id, ordered trajectory (every tool call: name, args, result, order), final response, final state (snapshotted before anything overwrites it), run index.
• TaskScore — outcome, process, opt-in combined, passed flag, failure category.
• EvalReport — the aggregate of one Eval run (a Suite run N times under one mind + one world): per Task, the run records, variance, and baseline check. Plain data — run_suite builds it; the CLI renders and persists it.

Process scoring is generic and explainable: required calls present; forbidden calls absent only scored where the entry point actually had access to the tool (otherwise N/A — never reward or penalise a violation that was impossible by construction); and an efficiency ratio = minimal_necessary_calls / actual_calls, capped at 1.0. No magic-number per-call penalties — a 9-call solution to a 4-call problem simply scores 0.44.

The harness ships do-nothing and random baselines (baselines.py) that validate the Task, not the agent: scoring a Task against a no-op answers "could a no-op pass this?", and a warning fires when it can — a broken Task. The baseline check runs once per Task on every Eval run (run_baselines in evaluation.py) and is reported as its own Baseline line beside the variance stats — a Task-design guardrail, never blended into an agent score. The failure taxonomy is a mutually-exclusive, first-match chain — run_error (a run that raised — live network/API flakiness recorded as data, never an aborted eval), no_tools (the do-nothing signature), called_forbidden, wrong_tool, missing_output, wrong_args, excessive_calls, good_process_bad_outcome, unknown — so a failed run tells you what to actually fix.

---

Layout

Path	Role
tooleval/core.py	Task, TrajectoryStep, AgentResult, TaskScore dataclasses
tooleval/contract.py	Agent Contract run(task, tool_handler) -> AgentResult + entry-point registry
tooleval/tool_handler.py	Mock tool handler (fixtures) + trajectory-recording wrapper
tooleval/runner.py	Invokes an entry point, times it, snapshots final state
tooleval/scoring.py	State-based outcome scorer + process scorer + opt-in combine
tooleval/baselines.py	Do-nothing + random baselines (warn on trivially-satisfied Tasks); Task-design guardrail, not in the default CLI loop
tooleval/variance.py	Per-Task variance stats across runs
tooleval/taxonomy.py	Mutually-exclusive failure-category classifier
tooleval/report.py	Rich Result-vector report (outcome-first)
tooleval/persistence.py	One JSON file per run, never overwritten
tooleval/suites/stub/	Minimal stub Suite (tracer-bullet integration proof)
tooleval/suites/calc/	Trivial calc Suite (agnosticism proof — the guide below)
tooleval/suites/job_search/	Full job-search Suite (the real agent target)
tests/	Seam tests: scorer, handler, persistence, report, end-to-end runner

---

CLI usage

python run_eval.py --suite SUITE [--runs N] [--agent MIND] [--combine] [--live]

Flag	Default	Effect
--suite	(required)	Name of the Suite to run, e.g. calc or job-search.
--runs	5 mock / 3 live	How many times to run each Task — this is how variance is measured. Defaults lower under --live (real tokens, real network); override explicitly anytime.
--agent	scripted	The mind axis. scripted runs the Suite's offline scripted twin; real imports the real agent in-process. Orthogonal to --live (the world axis). Only meaningful for a Suite that has a real backend (job-search).
--combine	off	Print a lossy blended score beneath each Result vector. Opt-in only, never the headline. Tune with --outcome-weight / --process-weight (default 0.7 / 0.3, normalised).
--live	off	Swap the Suite's mock handler for its real one. Proves the Contract against reality; never mixed with mock in one run. Fails clearly if a Suite has no live handler.

Two axes: the mind (--agent) and the world (--live)

The job-search Suite can run either its scripted twin (a canned stand-in, the default offline path) or the real JobPostingAgent (real LLM, real cognition). That mind axis is orthogonal to the existing world axis (--live):

--agent	--live	what runs
scripted (default)	mock	offline proof — no agent imported, ToolEval standalone (only the judge-graded graceful-degradation Task needs a key)
real	mock	Mode 1 — real LLM + mock tools
real	live	Mode 2 — real LLM + real tools (true end-to-end)
scripted	live	rejected — a canned mind calling real tools is meaningless

The real agent is a flat, non-packaged sibling repo, imported in-process from a path given by the JOBPOSTINGAGENT_PATH environment variable (it self-registers the real Pipeline and Step entry points; see ADR 0003). Exactly one backend registers per run, so the scripted twin and the real agent never collide on the shared job_search.* names. The agent has no dedicated graceful-degradation entry point, so in real mode that Task is aliased to the real extract Step (graceful degradation is extract fetching an unreadable page) — that's how the whole five-Task ladder runs against the real agent.

# Mode 1 — the real agent against canned tool fixtures
JOBPOSTINGAGENT_PATH=/path/to/JobPostingAgent \
    python run_eval.py --suite job-search --agent real --runs 1

# Mode 2 — the real agent end-to-end against the real web and disk
JOBPOSTINGAGENT_PATH=/path/to/JobPostingAgent \
    python run_eval.py --suite job-search --agent real --live

A missing/blank JOBPOSTINGAGENT_PATH, or a failed import, exits with a clear, actionable error before any task runs.

Mode 2 runs the whole altitude ladder against the real world, not just the Pipeline: each of the five Tasks swaps in a live-world variant (same Task id, so the same per-Task scorer applies) pointed at a real input instead of its mock fixture. The harness selects these live_tasks only under --live:

Task	Mode-2 real input	Fragile fixture / override
extract, pipeline	a real, public job posting	LIVE_JOB_URL_DEFAULT / JOBSEARCH_LIVE_JOB_URL
research	a real, well-known company (so web_search returns briefable results)	LIVE_COMPANY_DEFAULT (Anthropic) / JOBSEARCH_LIVE_COMPANY
analyze	the committed résumé on real disk (same text both worlds)	—
graceful-degradation	a real access-restricted URL the agent genuinely can't read	RESTRICTED_URL_DEFAULT (httpbin.org/status/403) / JOBSEARCH_RESTRICTED_URL

The job-posting and restricted URLs are deliberately fragile fixtures — postings expire, endpoints move — defined and documented in fixtures.py. When one rots the run does not crash: a failed fetch becomes a low-scoring run, a hard error becomes a run_error entry in the failure taxonomy — flakiness (DuckDuckGo throttling, a 404) is recorded as data, never an aborted eval. Each fragile fixture takes an env override so a stale default can be replaced for a single run without editing code:

JOBSEARCH_LIVE_JOB_URL=https://example.com/some-live-posting \
JOBPOSTINGAGENT_PATH=/path/to/JobPostingAgent \
    python run_eval.py --suite job-search --agent real --live

Running the full ladder in Mode 1 already exposes per-altitude differences — in a verified scripted-vs-real Mode-1 run, research scored process 0.75 and the Pipeline 0.86, against 1.00 elsewhere — which is the whole point of scoring each Step in isolation and end-to-end.

---

Add your own Suite

A Suite is the only place domain knowledge lives. The harness runner, scorer, variance stats, taxonomy, and persistence never change — you only add files under tooleval/suites/<name>/.

This guide is narrated from the real diff between the two shipped Suites. The calc Suite is the minimal reference — two tools, two Tasks, ~150 lines. The job_search Suite is the same shape scaled up to four Steps, a Pipeline, an adversarial Task, and an LLM judge. Everything job_search adds, it adds inside its own Suite directory — that's the agnosticism proof.

A Suite is five files:

tooleval/suites/calc/
├── fixtures.py   # committed mock world data + build_handler()
├── tasks.py      # Task definitions (one per entry point)
├── agent.py      # entry points honouring the Agent Contract
├── scorers.py    # custom outcome scorers (optional)
└── __init__.py   # registers the SuiteSpec

1 — Commit fixture data (fixtures.py)

Pair each world-boundary tool call (keyed by its args) with the result it should return, then hand the list to MockToolHandler.build:

from tooleval.tool_handler import MockToolHandler

EXPRESSION, CALCULATE_RESULT = "6 * 7", 42

FIXTURE_ENTRIES = [
    ("calculate", {"expression": EXPRESSION}, CALCULATE_RESULT),
]

def build_handler() -> MockToolHandler:
    return MockToolHandler.build(FIXTURE_ENTRIES)

What job_search adds: the same pattern, only bigger — a committed resume, posting HTML, search results, a paywalled posting fixture for the adversarial Task. The file is larger; the shape is identical.

2 — Define Tasks (tasks.py)

A Task names the entry point it targets and declares everything the scorer needs:

from tooleval.core import Task

ARITHMETIC_TASK = Task(
    id="calc.arithmetic",
    description=f"Compute the value of the expression '{EXPRESSION}'.",
    entry_point="calc.arithmetic",
    toolset=["calculate"],
    required_tool_calls=["calculate"],
    forbidden_tool_calls=["convert"],     # scored only because it's in the toolset
    minimal_necessary_calls=1,            # drives the efficiency ratio
    expected_state={"expression": EXPRESSION, "result": CALCULATE_RESULT},
    tags=["calc", "arithmetic"],
)

What job_search adds: one Task per Step plus a pipeline Task whose minimal_necessary_calls=5 spans the combined toolset, and a graceful-degradation Task with expected_state={} — there's no world state for "didn't hallucinate," so it leans on a judge (step 4).

3 — Write entry points (agent.py)

Register a function per entry point. Wrap the injected handler in RecordingHandler so every call lands in the trajectory, then return an AgentResult:

from tooleval.contract import ToolHandler, register_entry_point
from tooleval.core import AgentResult, Task
from tooleval.tool_handler import RecordingHandler

@register_entry_point("calc.arithmetic")
def arithmetic_eval(task: Task, tool_handler: ToolHandler) -> AgentResult:
    handler = RecordingHandler(tool_handler)
    result = handler("calculate", {"expression": EXPRESSION})
    return AgentResult(
        task_id=task.id,
        trajectory=handler.trajectory,
        final_response=f"{EXPRESSION} = {result}",
        final_state={"expression": EXPRESSION, "result": result},
    )

What job_search adds: its entry points invoke the real, re-architected Step code rather than scripted output, and the pipeline entry point composes the four Steps in order — the same Step code the single-step Tasks run (ADR 0001). The Contract is unchanged.

4 — Custom outcome scorers (scorers.py, optional)

Most Tasks use the harness's default state-based scorer. Register a custom one only where state comparison is too rigid — calc does it for floating-point tolerance:

def score_convert(task: Task, result: AgentResult) -> float:
    actual, expected = result.final_state.get("result"), task.expected_state.get("result")
    return 1.0 if abs(float(actual) - float(expected)) <= 1e-3 else 0.0

What job_search adds: a custom scorer for every Task, because a live LLM never reproduces a fixture's final state verbatim, so exact state-equality is meaningless against the real agent. Four are structural/invariant scorers that score the fraction of shape invariants met: one per Step (score_extract/score_research/score_analyze, each scoped to that Step's final state) and one for the whole package (score_pipeline — report saved, required sections present, job_details populated, briefing/analysis non-empty). The fifth, for the graceful-degradation Task, is an LLM-as-judge scorer — the one place state comparison can't express the goal ("absence of hallucination"); it applies a strict rubric and logs the judge's reasoning, so the score is auditable rather than a vibe. No other Task uses a judge.

5 — Register the Suite (__init__.py)

Bundle tasks, the mock handler, optional per-Task scorers, and an optional live handler into one SuiteSpec:

from tooleval.suites import SUITE_REGISTRY, SuiteSpec
from tooleval.suites.calc import agent  # noqa: F401 — registers entry points
from tooleval.suites.calc.fixtures import build_handler
from tooleval.suites.calc.scorers import score_convert
from tooleval.suites.calc.tasks import ARITHMETIC_TASK, CONVERT_TASK

SUITE_REGISTRY.register(SuiteSpec(
    name="calc",
    tasks=[ARITHMETIC_TASK, CONVERT_TASK],
    mock_handler_factory=build_handler,
    live_handler_factory=_build_live_handler,            # optional; --live needs it
    outcome_scorers={CONVERT_TASK.id: score_convert},    # default scorer otherwise
))

Finally, add one import to run_eval.py so the Suite registers on startup:

import tooleval.suites.calc  # noqa: F401

6 — Run it

python run_eval.py --suite calc --runs 5

That's the entire contract. The diff between calc and job_search is the whole "what scales up" story — more fixtures, real Step code, a Pipeline, an adversarial Task, one judge — and none of it touches the harness core.