Basecamp Bench

Basecamp Bench compares coding-agent harnesses on two realistic product tasks: a frontend prototype and a production-shaped API. Implementation agents receive only a concise product directive and the same seed/reference pack. Independent evaluator agents inspect each immutable submission, decide how to run and test it, and return evidence-backed dimension scores under a versioned contract.

The runner preserves raw attempts, computes scores and eligibility itself, and generates a self-contained HTML quality-versus-cost report. FE and BE results remain separate.

How it was built

Basecamp Bench was developed through three successive model-driven passes:

1. Grok 4.5 built the initial working prototype.
2. Claude Fable 5 cleaned up the implementation and improved its structure.
3. GPT-5.6 Sol professionalized the project, hardening the architecture, tests, documentation, safety controls, and release workflow.

Building the project with GPT-5.6 Sol consumed 682 million tokens and $449.33 in model usage. This development cost is separate from the benchmark-run costs reported below.

Results updated September 29, 2026

Basecamp Bench asks each agent to build two parts of a Basecamp 5 clone from a fixed specification: a single-file frontend SPA and a production-shaped backend API. An independent evaluator scores every submission across weighted quality dimensions. Fable 5.1 leads frontend at 8.183, ahead of GPT-6 Astra at 7.607. Astra leads backend at 9.305, ahead of Sonnet 5.5 at 9.011 and Fable 5.1 at 8.965. Astra costs about a quarter as much as Fable 5.1 across both tracks ($25.57 versus $92.96). Every model in the table except GPT-6 Sol scores higher on backend than frontend.

Model	Frontend	Backend	Total time	Total cost
Fable 5.1	8.183	8.965	3:04:06	$92.96
GPT-6 Astra	7.607	9.305	1:36:31	$25.57
Fable 5	7.578	8.392	2:06:40	$85.87
Sonnet 5.5	7.575	9.011	2:18:20	$52.30
Grok 4.6	7.104	7.926	57:58	$10.85
Sonnet 5	6.982	7.243	1:27:09	$36.23
Grok 4.7*	6.650	7.605	4:20:06	≥$10.46
Opus 5.5	6.625	8.683	1:59:02	$32.30
Grok 4.5	6.384	7.278	36:48	$9.30
GPT-6 Sol	6.212	6.155	1:33:42	$6.73
GPT-6.1 Sol	6.195	8.775	1:52:05	$4.92
GPT-5.6 Sol	5.765	7.310	59:48	$15.13
Muse Spark 1.3	5.745	6.711	42:24	$4.63
Gemini 3.8 Flash High	5.700	6.483	38:17	$6.07
GPT-5.5	5.670	7.084	44:14	$10.94
Gemini 3.7 Flash High	5.608	5.800	30:59	$7.54
GPT-6 Luna*	5.434	6.695	~5:17:32	≥$3.30

Total time and cost include implementation and evaluation across both tracks. GPT-6.1 Sol judged the model added on September 30, GPT-6 Sol judged the five models added on September 29, and GPT-5.6 Sol judged the rest. * GPT-6 Sol finished part of the Grok 4.7 and GPT-6 Luna work after their own attempts failed or stalled. Their costs are lower bounds because some failed attempts left no usage record.

Original baseline evidence is available in baseline/. Read the full report.

Getting Started

Basecamp Bench runs locally with Python 3.11 or newer and no runtime dependencies. Install and authenticate the agent CLIs you plan to compare: Codex, Claude Code, Grok, Pi, and/or Google Antigravity (agy). External OS isolation is optional for local runs and required for publication; the provided container is one way to supply it.

1. Clone this repository and enter its root.
2. Install the runner and copy the annotated local configuration.

python -m pip install -e .
cp bench.example.toml bench.toml

3. Adjust the selected models or executable paths, inspect the effective configuration, and start a run.

basecamp-bench show-config
basecamp-bench run --harness codex --track fe

Run

Local mode is intended for iteration and defaults to one repetition and one evaluator:

basecamp-bench run --harness codex --track fe

To benchmark GPT-6 Astra, enable the codex-astra block in bench.toml (shown in bench.example.toml) and run basecamp-bench run --harness codex-astra --track fe --track be. It uses gpt-6-astra at high reasoning effort through the Codex CLI; the configured independent evaluator remains unchanged.

Independent implementation attempts run concurrently. As soon as one submission is snapshotted, its evaluator calls run concurrently while other implementations continue. Live progress is written to stderr with attributable build, evaluate, aggregate, report, and final run events; use --quiet to suppress it. The completed run path remains the only stdout output. At most 32 paid agent processes run simultaneously by default; adjust the safety cap with --max-parallel-agents.

Harness and track flags may be repeated or comma-separated. Local jobs use isolated folders beneath runs/<run-id>/workspaces; full_access = true requires --allow-unsafe-host-execution locally or --confirmed-isolated-environment inside an external boundary.

Publication mode requires at least three implementation repetitions, two valid evaluator model IDs, exact or pinned pricing, a distributable reference pack, and confirmed isolation:

basecamp-bench run --mode publication --repetitions 3 \
  --confirmed-isolated-environment

Configure at least two enabled [[evaluators]] entries in bench.toml before running that command. Paid model calls can be substantial; inspect the complete fleet first with basecamp-bench show-config and enforce provider spending limits outside the runner.

Regenerate the report

Reports discover every matching attempt ledger (or legacy leaderboard JSON) beneath the supplied directories. Each ledger stores comparison identity, dimension profile, and raw attempts; statistics are derived at report time, so adding later model runs requires no hand-edited aggregates. Canonical machine artifacts are schema 2.0 JSON attempt ledgers (plus the HTML report). Normal runs do not write leaderboard CSV or Markdown files.

basecamp-bench report runs --output model-performance.html

Optional display-name overrides apply only at render time and never rewrite evidence:

basecamp-bench report runs --output model-performance.html \
  --rename model-id="Friendly Name"

The output is one deterministic, offline HTML decision surface: separate track/contract sections, a plain-language verdict, a cost-versus-quality chart with model labels, a comparison table (score, expected implementation cost per valid result, success rate, tokens, duration, evaluator overhead, total observed cost, and uncertainty when repetitions exist), dimension scores and weights, failures and mixed-eligibility footnotes, and a complete embedded JSON payload with provenance hashes, classifications, source IDs, and raw attempts. Tables remain usable without SVG; imported text is escaped. Methodology prose lives in docs/METHODOLOGY.md. Local reports combine matching benchmark evidence across runner revisions for exploratory comparison; publication reports keep those revisions separate.

Optional CSV and Markdown projections

When a spreadsheet or static Markdown table is needed, project one leaderboard JSON explicitly. Rows are derived only from the loaded attempts through the same aggregator the report uses; CSV/Markdown are never a second source of truth:

basecamp-bench export-tabular \
  runs/<run-id>/leaderboards/leaderboard_<track>_<version>_<sha>.json \
  --output-dir /tmp/tabular

The command writes two deterministic UTF-8 files named from the comparison identity (leaderboard_*.csv and leaderboard_*.md) and refuses to overwrite existing targets. Both schema 2.0 attempt ledgers and committed legacy schema 1.0 leaderboards are accepted as input.

Re-evaluate immutable submissions

Re-evaluation verifies an earlier run and its declared snapshot hashes, then creates a new run with fresh evaluator attempts. The prior run is never changed. Verified prior snapshots and normal implementations share one evaluator and attempt-finalization path; only the submission source differs.

basecamp-bench reevaluate runs/<run-id> --track fe

Current contracts and evaluator configuration apply to the new evaluation; lineage records the prior run manifest and snapshot hashes.

Verify and export

basecamp-bench verify-run runs/<run-id>
basecamp-bench export-run runs/<run-id> basecamp-bench-run.zip

Verification checks the strict manifest shape, identifiers, relative paths, and every declared artifact hash. Export includes only the manifest and declared public artifacts, scans them for likely credentials, rejects symlinks and path escapes, and writes a deterministic archive without overwriting an existing file. Workspaces and private logs are never exported. Local evaluation imports only run provenance builders; publication verification and ZIP export load at the verify/export (and reevaluation) boundary. Public APIs from basecamp_bench.manifest import verify_run, export_run remain stable.

Official baseline

The repository's baseline/ directory contains four verified, shareability-scanned reference runs and one combined self-contained HTML report. It preserves model snapshots, evaluator reports and results, raw attempts, leaderboards, and provenance manifests while excluding private logs, prompts, credentials, and execution workspaces.

for run in baseline/runs/*; do basecamp-bench verify-run "$run"; done
basecamp-bench report baseline/runs --output /tmp/basecamp-bench-report.html \
  --rename 'claude-fable-5=Fable 5' \
  --rename 'claude-sonnet-5=Sonnet 5' \
  --rename 'gpt-5.6-sol=GPT-5.6 Sol' \
  --commentary baseline/commentary.json
cmp baseline/report.html /tmp/basecamp-bench-report.html

New compatible model runs can be compared by regenerating a report from the baseline and additional run directories together. The committed baseline is an exploratory local run; its quality and cost points are auditable, while official publication eligibility and Pareto-frontier claims require the stricter repetition, evaluator, pricing, and isolation rules below.

Method

Each track lives under benchmarks/<track>/:

• prompt.md is passed byte-for-byte to implementation agents. It contains the task only—no benchmark, evaluator, rubric, output filename, or runner instructions.
• eval.md gives evaluator agents the full assessment context and evidence standard without prescribing the submission's language, runtime, filenames, commands, or architecture.
• contract.json is the canonical dimension, anchor, weight, and score policy.

Evaluators receive disposable copies of the original seed and immutable submission plus an exact JSON result schema. The runner rejects missing or unknown dimensions, non-finite/out-of-range scores, identity or hash mismatch, malformed evidence, failed evaluator processes, and any evidence mutation. It then takes the median evaluator score per dimension and computes the weighted overall score.

Future FE evaluations use a runner-owned static server over the immutable submission copy. The server binds only to loopback on an OS-assigned port, accepts GET/HEAD, checks HTTP readiness before launching the evaluator, and closes after evaluation, including failures. It serves index.html or a single root HTML entry point, consistent with the single-file frontend task.

The evaluator receives the exact HTTP URL and must use Chrome through computer use, capture desktop/mobile screenshots, and record click/type/reload outcomes in browser-evidence.json. The runner requires both a recorded Chrome load and complete browser evidence before accepting scores; source-only visual scoring cannot pass. Submission bugs are scored from observed browser behavior, while environmental browser failures leave the evaluation incomplete. Request audits and browser evidence are hashed run artifacts. This changes the FE methodology to contract 2026-10-15.1; historical runs retain their recorded versions.

Runs from September 30 onward use GPT-6.1 Sol at high effort as the evaluator. GPT-6 Sol judged the September 23 to 29 runs, and Sol 5.6 judged the earlier ones. bench.example.toml still defaults to Sol 5.6 and has commented GPT-6.1 Sol blocks for matching current runs. Comparative evaluator trials with Fable 5, Grok 4.5, and Sol 5.6 produced similarly fair judgments; Sol was the most detailed and thorough. Changing the evaluator is therefore a methodology and cost-routing choice that may affect the level of detail more than the general direction of the judgment.

Leaderboard aggregation is scoped by track, contract version, harness, and model. Failed attempts remain visible and contribute to success rate. The frontier's primary cost is median implementation cost per attempt divided by success rate; evaluation cost is reported separately.

See Methodology for claim boundaries and detailed rules.

Safety

Agent CLIs and generated applications are untrusted code. Directory separation, prompt instructions, environment allowlists, hash checks, log caps, and process group cleanup provide integrity and operational controls; some harnesses do not provide an OS security boundary. Use spend-limited credentials and keep personal files, ambient cloud credentials, Docker sockets, SSH agents, browser profiles, and production systems outside the run environment.

The isolated execution guide and reference containers/ recipe provide optional local hardening. Publication mode requires an explicit external-isolation confirmation.

Add models and harnesses

A new model on an existing CLI is configuration only: update the corresponding [harnesses.<id>] model and pricing override if necessary. A new harness needs an adapter implementing command construction, environment allowlisting, working directory, stdin, and output/usage parsing. Adapter tests must cover redaction, malformed output, permissions, timeouts, and vendor output drift.

The optional claude-haiku-55 configuration runs claude-haiku-5-5 through Claude Code at high effort. Enable its block in bench.example.toml, then run basecamp-bench run --harness claude-haiku-55 --track fe --track be. Claude Code can use its existing subscription login or an exported ANTHROPIC_API_KEY. Haiku 5.5 costs $0.10/$0.50 per million input/output tokens for prompts up to 100,000 tokens, and $0.50/$2.50 above that boundary; cache rates also depend on prompt length and cache lifetime. The runner uses Claude Code's reported session cost to preserve these request-level charges. If that cost is absent, Haiku's cost remains unknown rather than using aggregate token counts with the short-prompt rates. Rates are documented in the example config and Anthropic's pricing.

The muse adapter supports muse-wonder-1.0, muse-wonder-full-1.0, muse-spark-1.3, and muse-spark-1.2 through Muse Code 1.4.7 or later. Use the existing Muse subscription login or META_API_KEY. Enable the example blocks, then select one or all four:

basecamp-bench run --harness muse-wonder --harness muse-wonder-full --harness muse-spark-13 --harness muse-spark-12 --track fe --track be

Each job pins Standard mode with --contributor-mode off, preserves the native workspace sandbox, disables reminder agents and foreign personal context, and limits execution capacity to one. Prompts use --prompt-file; completed JSONL terminal events supply the final answer. Since 1.4.7's headless stream omits usage, the adapter exports the exact session offline into the job's log folder and sums its model-completion usage records. Missing usage, incomplete runs, model mismatches, and export failures fail the implementation before evaluation.

Built-in Meta Standard rates (verified October 14, 2026) are $4.00/$0.20/$20.00 per million fresh input, cached input, and output tokens for Wonder, and $1.25/$0.15/$4.25 for both Spark versions. Explicit pricing overrides take precedence. These are API token cost estimates when using a subscription, not an allocation of the monthly fee. Web-search charges are excluded. Wonder Full requires a Power Usage or Maximum Usage subscription and has no published token price: its cost stays unknown, and the run records missing pricing rather than inventing a zero cost. Remove any old free OpenCode pricing override for a Muse model before using the paid Muse harness; OpenCode's reported free cost remains separate from Muse's estimated token cost.

The optional pi adapter exposes the safe benchmark model ID glm-5.2 and routes it to OpenRouter's z-ai/glm-5.2; set OPENROUTER_API_KEY and use the commented example in bench.example.toml. Pi relies on the per-job workspace boundary in local mode; the optional container recipe adds OS isolation.

The optional opencode adapter exposes safe benchmark model IDs and routes them to OpenCode's native IDs: 0x-alpha maps to opencode/x-preview-f-free and requires the max effort, and muse-spark-1.3 maps to opencode/muse-spark-1.3-contributor-free at minimal through xhigh. The configured effort is passed through as OpenCode's --variant. It reads prompts from stdin, consumes OpenCode's JSON event stream for usage, and installs a per-job config that denies external-directory access in workspace-write mode. Use the commented example in bench.example.toml.

The optional agy adapter supports gemini-3.1-pro at low or high effort and gemini-3.5-flash, gemini-3.7-flash, and gemini-3.8-flash at low, medium, or high effort. It enables Antigravity's terminal sandbox and stages evaluator evidence as disposable workspace copies, preserving the immutable originals. See the commented bench.example.toml entry for setup.

Contract or evaluator-directive changes require a new contract version and changelog entry. Published contract versions are immutable.

Repository map

basecamp_bench/     runner, adapters, validation, aggregation, reporting, CLI
benchmarks/         FE/BE directives, evaluator rubrics, contracts, asset manifest
Repo/               identical seed and reference material supplied to agents
schemas/            public artifact JSON Schemas
docs/               methodology, security, rights, and community documentation
containers/         disposable non-root runner reference
tests/              unit and credential-free fake-harness integration tests

Rights and independence

Project code and original documentation are Apache-2.0. The vendored basecamp-sdk reference material is MIT-licensed at its recorded upstream commit. Personal-account screenshots are documented as Fair use in the hash-matched reference-pack manifest. See asset provenance, third-party notices, and trademarks.

This is an independent evaluation project and is not affiliated with or endorsed by Basecamp.