Add product-wide task effectiveness baselines

[Copilot]

Summary

• Add an opt-in comparison of native tools, TraceDecay core, and existing channel variants on independently checked tasks.
• Report matched task success, observed cost and timing variability, and artifact/setup identity without treating missing usage as zero.

Motivation

Adoption alone does not establish product value. Reproducible task and cost comparisons are needed to guide product-surface decisions while preserving prior qualified evaluations.

Changes

• Runner: Add a held-out task cohort, native-tools condition, per-run artifact/task/fixture identities, and setup timing. Require an explicit shipping binary for baseline runs.
• Grader: Report per-task success and invalid attempts, Wilson intervals, observed token/cache/USD/time ranges, and deltas against native runs. Optionally join the no-model efficiency scorecard only when binary SHA-256 matches.
• Evidence: Document the opt-in run and defer dashboard, Work/Workflow, and grammar changes where required empirical measurements are unavailable.

Example, with a shipping binary built first:

TRACEDECAY_AGENT_EVALS=1 EVAL_INCLUDE_BASELINE=1 \
  TRACEDECAY_BIN=/path/to/shipping/tracedecay \
  HOSTS=claude CLAUDE_MODELS=sonnet REPS=3 \
  CHANNELS="native bare no-hints no-skills full" \
  SCENARIOS="baseline_order_stock_failure baseline_stock_check_rename baseline_currency_decision" \
  evals/agent_adoption/run.sh

Test plan

• The affected Bazel targets pass
• bazel build //... passes when the change crosses package boundaries
• Tested manually (describe below if applicable)

Checklist

• CHANGELOG.md updated (under [Unreleased] if no version bump)
• No secrets, credentials, or .env files included
• Breaking changes documented (if any)

• Fixes Extend prior outcome evals into product-wide task effectiveness and total-cost baselines #4219

[changeset-bot]

⚠️ No Changeset found

Latest commit: 35f8574

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR