Extend prior outcome evals into product-wide task effectiveness and total-cost baselines #4219

Priority: P2 · Type: proposal

Problem

The product promises fewer tokens and tool calls, while exposing code discovery, traversal, edits, memory, session history, and additional surfaces. Decisions to expand or retain that surface need evidence that agents complete useful tasks more reliably or cheaply.

The repository already has evaluation and performance infrastructure. Completed #3654 qualifies learned-skill outcomes; completed #3655 evaluates optional repeated-query/Scout guidance. Reuse those capability-specific results and runners as prior work.

The adoption harness correctly says invocation alone is not usefulness. Its grader combines routing, tool budgets, and a bounded answer proxy. The efficiency scorecard measures useful runtime journeys such as initialization, queries, sync, restart, storage, and memory usage. Those measurements should support a real-task comparison rather than substitute for it.

Scope and audit basis

This proposal extends the inspected harnesses at 30c23baa1967d4bb04aee3f20deb7a19f302ba61 into a product-wide comparison of task effectiveness and total cost, including startup, indexing, and context overhead. It does not reopen the completed learned-skill or optional-guidance evaluation work, or imply there were no prior outcome measurements. Improving the common grader is tracked separately; this issue owns the combined experimental baseline and resulting product decisions.

Proposed change

Consolidate the prior capability-specific evaluation work and extend existing runners with a small, representative set of held-out coding tasks and independent success criteria. Compare the same model and task under native tools, the minimal TraceDecay core, and selected additional capabilities only where they plausibly help. Prompts must not require using the product. Change one capability or condition at a time rather than building a large Cartesian matrix.

Record actual task completion, regressions, elapsed time, tool attempts/failures, and observed input/output/cache token usage. Include schema/context overhead and distinguish cold indexing/setup cost from amortized repeated use. Preserve missing usage as unknown; do not invent zero costs. Use repeated runs within an explicit budget and report variability alongside effect size. Keep inspectable transcripts and task-result evidence.

Behavioral acceptance

• A reproducible comparison uses the same tasks, model settings, and explicit artifact identities across conditions.
• Successful native completion is credited equally; product invocation is reported separately.
• Task success is independently checked, including wrong answers and harmful edits.
• Reports include total observed cost and startup/indexing costs with clear attribution and uncertainty.
• Each tested additional capability receives an evidence-backed keep, simplify, defer, or remove decision.
• A slower or more expensive capability without a demonstrated benefit is not justified by a higher adoption score.

Audit provenance

Source evidence is pinned to 30c23baa. Relevant implementation was rechecked against master at 5daa37d1; the finding remains applicable. Verification performed and its limits are stated above.

Dependencies and related work

Use exact-run evidence from #4204, repaired outcome scoring from #4216, an owned isolated daemon lifecycle from #4205, and identified shipping artifacts from #4206. Extend the existing runners after those integrity fixes.