From afb31a7e7618f057fb386def72a4b036b8c90a56 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 19 Oct 2026 04:03:39 +0000 Subject: [PATCH 1/3] Initial plan From f7361ff169b7063c1eaafba8a4758fe71eaaa724 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 19 Oct 2026 04:36:47 +0000 Subject: [PATCH 2/3] feat(evals): add product task effectiveness comparisons Co-authored-by: ScriptedAlchemy <25274700+ScriptedAlchemy@users.noreply.github.com> --- evals/agent_adoption/README.md | 87 +++- evals/agent_adoption/grade.py | 375 +++++++++++++++++- evals/agent_adoption/run.sh | 115 +++++- .../scenarios/baseline_currency_decision.json | 31 ++ .../baseline_order_stock_failure.json | 30 ++ .../baseline_stock_check_rename.json | 52 +++ evals/agent_adoption/selftest.py | 166 ++++++++ 7 files changed, 829 insertions(+), 27 deletions(-) create mode 100644 evals/agent_adoption/scenarios/baseline_currency_decision.json create mode 100644 evals/agent_adoption/scenarios/baseline_order_stock_failure.json create mode 100644 evals/agent_adoption/scenarios/baseline_stock_check_rename.json diff --git a/evals/agent_adoption/README.md b/evals/agent_adoption/README.md index 6844d7cf17..8e95f940d2 100644 --- a/evals/agent_adoption/README.md +++ b/evals/agent_adoption/README.md @@ -77,22 +77,105 @@ paths, so do **not** commit run artifacts. |-----|---------|---------| | `TRACEDECAY_AGENT_EVALS` | unset | must be `1` to launch agents (else dry run) | | `HOSTS` | `claude` | `claude`, `codex`, or `claude codex` | -| `CHANNELS` | `full` | ablation conditions: any of `full no-hints no-skills bare cli-only` | +| `CHANNELS` | `full` | conditions: `native bare full no-hints no-skills cli-only` | | `SCENARIOS` | all active | space-separated scenario ids to run | | `EVAL_INCLUDE_DEFERRED` | `0` | also run `status:"deferred"` scenarios | +| `EVAL_INCLUDE_BASELINE` | `0` | include held-out product-effectiveness scenarios | | `CLAUDE_MODELS` | `opus sonnet` | space-separated `claude --model` matrix | | `CODEX_MODELS` | `gpt-5.6-sol` | space-separated `codex exec -m` matrix | | `CODEX_REASONING_EFFORT` | `medium` | Codex reasoning effort passed through `model_reasoning_effort` | | `SCENARIO_TIMEOUT` | `240` | per scenario wall-clock seconds | -| `REPS` | `1` | repetitions per scenario x host x model x condition cell; `>1` suffixes transcripts with `__r` and the grader pools the repetitions, so a cell of n=15 gives a usable exact-binomial interval instead of a single draw | +| `REPS` | `1` | repetitions per scenario x host x model x condition cell; repeated outcomes are reported with Wilson intervals and observed cost/latency ranges | | `PARALLEL` | `1` | concurrent live agent runs; scenarios are read-only against shared fixtures and each `claude` process spawns its own MCP server, so `4` is a safe default for high-rep cells | | `REPEATED_QUERY_HINTS` | `off` | `on` turns on the profile setting `user.repeated_query_hints.v1` for the eval's isolated profile before any agent launches, and suffixes transcripts with `__qhints`; see "Repeated-query hints on vs off" | | `REAL_CLAUDE_DIR` | `$CLAUDE_CONFIG_DIR` or `~/.claude` | where the Claude credential comes from: its `.claude_code_oauth_token` and `settings.json` `env`/`model` keys; see "Store isolation & auth" | | `TRACEDECAY_BIN` | candidate branch (live) | explicit tracedecay binary override | +| `EFFICIENCY_SCORECARD` | unset | optional no-model scorecard path, joined only when its binary digest matches | | `EVAL_OUT` | unset | dir to copy scoreboard + report into | | `LEARNED_SKILL_WORKFLOW` | unset | run only that workflow's held-out scenarios, once per variant, on `full` | | `LEARNED_SKILL_VARIANTS` | `none generated manual` | learned-skill variants to compare | +## Product-effectiveness baseline + +The opt-in `baseline_*` cohort adds three neutral, independently graded tasks: +code-path explanation, a behavior-preserving source edit (including a check +against a harmful boundary change), and recall of a seeded prior decision. +They are excluded from ordinary adoption runs. Run them with one host/model, +one artifact, identical prompts, and repeated trials: + +```sh +TRACEDECAY_AGENT_EVALS=1 EVAL_INCLUDE_BASELINE=1 HOSTS=claude \ + CLAUDE_MODELS=sonnet \ + SCENARIOS="baseline_order_stock_failure baseline_stock_check_rename baseline_currency_decision" \ + CHANNELS="native bare no-hints no-skills full" REPS=3 \ + evals/agent_adoption/run.sh +``` + +This is 45 live agent turns; the existing explicit `TRACEDECAY_AGENT_EVALS=1` +opt-in remains required. Native runs receive only Claude's standard file tools +and no TraceDecay MCP/plugin/CLI surface; the native condition is Claude-only. +`bare` exposes MCP tool descriptions without hooks or skills. `no-hints` and +`no-skills` are the existing channel ablations; `full` is the ordinary bundled +experience. Read per-task results rather than adoption averages: the same +successful answer or edit counts regardless of whether a product surface was +invoked. Channel ablations retain their documented steering differences, so +they are descriptive unless prompt steering is also controlled. + +The report includes per-task pass rates with Wilson intervals, observed +run-to-run cost/latency ranges, tool attempts/failures, and deltas against the +same task/model's native run. Token and USD values remain unknown when the host +does not report usage. One-time fixture initialization/indexing time is shown +separately and is not silently spread across tasks. For cold startup, indexing, +restart, incremental-sync, storage, and RSS measurements, run the existing +no-model scorecard against the same binary and join it when grading: + +```sh +TRACEDECAY_BIN="$(python3 - "$RUN_DIR/meta.json" <<'PY' +import json, sys +print(json.load(open(sys.argv[1]))["binary_path"]) +PY +)" +scripts/efficiency-scorecard.py --binary "$TRACEDECAY_BIN" --runs 3 \ + --output target/efficiency-scorecard +evals/agent_adoption/grade.py --run-dir "$RUN_DIR" \ + --efficiency-scorecard target/efficiency-scorecard/scorecard.json +``` + +The scorecard is rejected unless its binary SHA-256 equals the agent run's +recorded artifact digest. Its wall times and resource measurements remain a +separate cost dimension rather than being converted to model tokens or dollars. +No live-model task outcomes or usage values are checked into the repository. + +### Related product-surface dispositions + +The task cohort and runtime scorecard do not measure dashboard navigation +interactions, Work/Remote Brain resident-state footprint, or language-specific +quality and demand. Therefore they do not authorize removing those surfaces. + +| proposal | disposition from available evidence | +|---|---| +| #4220 dashboard navigation and Work projections | Defer removals. #3653 preserves distinct discovery semantics and shared Work query plumbing; this baseline has no dashboard interaction evidence. Keep deep links and all current projections. | +| #4223 Work/Workflow default footprint | Defer default or ownership changes. #3656 qualifies Work and Remote Brain jobs; this baseline does not measure their initialization, resident state, or operator steps. Preserve histories and mounted journeys. | +| #4229 additional grammar defaults | Defer profile changes. This baseline has no language-quality, demand, or matched core-versus-extras build measurement. Preserve advertised language support rather than inferring non-use. | + +These are explicit deferrals, not keep/remove verdicts. The related empirical +acceptance remains open until the corresponding task, resource, and quality +measurements exist. + +No paid-model runs were launched for this change, so there are no new observed +effectiveness or usage values and no capability keep/remove verdict. The +executable cohort can compare native tools with the MCP core and existing +channel variants. The prior learned-skill and optional repeated-query/Scout +evaluations (#3654/#3655) remain prior work; they were not rerun or replaced. +Hook/skill ablations retain the runner's existing steering differences and +must not be read as isolated causal effects until those are controlled. + +| capability | current disposition | +|---|---| +| MCP core versus native tools | Defer a product-value verdict until the held-out tasks are run; this checkout contains no model outcomes. | +| Hook guidance and skills | Defer individual keep/simplify/remove decisions; existing comparisons are executable, but this change has no run evidence and their steering conditions differ. | +| Learned skills and repeated-query/Scout guidance | Preserve the prior #3654/#3655 qualification; no new outcome result or default change is claimed here. | + ### Cost expectations One scenario x host x model is a single short headless agent turn. There are 44 diff --git a/evals/agent_adoption/grade.py b/evals/agent_adoption/grade.py index 0a1abb879a..171129ea5d 100755 --- a/evals/agent_adoption/grade.py +++ b/evals/agent_adoption/grade.py @@ -28,8 +28,10 @@ import argparse import fnmatch import json +import math import os import re +import statistics import sys from dataclasses import dataclass, field from typing import Any, Optional @@ -53,7 +55,7 @@ # Discovery-channel attribution # --------------------------------------------------------------------------- # # Ablation conditions (see run.sh CHANNELS). "full" = every discovery channel on. -CONDITIONS = ("full", "no-hints", "no-skills", "bare", "cli-only") +CONDITIONS = ("native", "bare", "full", "no-hints", "no-skills", "cli-only") KNOWN_HOSTS = ("claude", "codex") # Distinctive fragments of the hook-injected tool hints. These MIRROR the @@ -1016,7 +1018,7 @@ def score_scenario(scn: dict, tr: Transcript, seeded_facts: dict, run_meta: dict if "expected_skill" in scn or "allowed_skills" in scn: expected = scn.get("expected_skill") allowed = scn.get("allowed_skills", [expected] if expected else []) - measured = run_meta.get("channel_condition", "full") not in {"no-skills", "bare"} + measured = run_meta.get("channel_condition", "full") not in {"native", "no-skills", "bare"} unmeasured_reason = None if measured else "skills_unavailable_in_condition" selected = invoked_skills(tr) if measured else None details["skill_routing"] = { @@ -1071,6 +1073,7 @@ def score_scenario(scn: dict, tr: Transcript, seeded_facts: dict, run_meta: dict "repeated_query_hint_next_steps": repeated_query_hint_next_steps(tr), "final_answer_chars": len(tr.final_text), "cost": tr.cost, + "runner_elapsed_ms": run_meta.get("duration_ms"), "parse_note": tr.parse_note, "run_meta": run_meta, } @@ -1197,9 +1200,13 @@ def rate(key): return agg -def _prompt_tokens(cost: dict) -> Optional[int]: +def _prompt_tokens(cost: dict, host: Optional[str] = None) -> Optional[int]: + if host == "codex": + return cost.get("input_tokens") parts = [cost.get(k) for k in ("input_tokens", "cache_creation_input_tokens", "cache_read_input_tokens")] - return None if parts[0] is None else sum(p or 0 for p in parts) + if any(not isinstance(part, (int, float)) for part in parts): + return None + return sum(parts) def aggregate_cost_and_hints(results: list[dict], by_scenario: bool = False) -> dict: @@ -1231,7 +1238,7 @@ def aggregate_cost_and_hints(results: list[dict], by_scenario: bool = False) -> "mean_duration_s": _mean([None if u.get("duration_ms") is None else u["duration_ms"] / 1000 for u in costs]), "usd_per_pass": usd_per_pass(rs), - "mean_prompt_tokens": _mean([_prompt_tokens(u) for u in costs]), + "mean_prompt_tokens": _mean([_prompt_tokens(r["cost"], r["host"]) for r in rs]), "mean_output_tokens": _mean([u.get("output_tokens") for u in costs]), "hints_total": sum(next_steps.values()), "runs_with_hint": sum(1 for r in rs if r.get("repeated_query_hint_next_steps")), @@ -1240,6 +1247,243 @@ def aggregate_cost_and_hints(results: list[dict], by_scenario: bool = False) -> return out +def _sample_summary(values: list[Optional[float]]) -> dict: + known = sorted(v for v in values if isinstance(v, (int, float)) and math.isfinite(v)) + if not known: + return {"n": len(values), "observed": 0, "unknown": len(values), "mean": None, + "median": None, "range": None} + + def quantile(q: float) -> float: + position = (len(known) - 1) * q + lower = math.floor(position) + upper = math.ceil(position) + return known[lower] + (known[upper] - known[lower]) * (position - lower) + + return { + "n": len(values), + "observed": len(known), + "unknown": len(values) - len(known), + "mean": round(statistics.mean(known), 4), + "median": round(statistics.median(known), 4), + "q1": round(quantile(0.25), 4), + "q3": round(quantile(0.75), 4), + "range": [round(known[0], 4), round(known[-1], 4)], + } + + +def _wilson_interval(passed: int, total: int) -> Optional[list[float]]: + if total == 0: + return None + z = 1.96 + rate = passed / total + denominator = 1 + z * z / total + center = (rate + z * z / (2 * total)) / denominator + margin = z * math.sqrt(rate * (1 - rate) / total + z * z / (4 * total * total)) / denominator + return [round(max(0.0, center - margin), 4), round(min(1.0, center + margin), 4)] + + +def effectiveness_comparisons( + results: list[dict], invalid_runs: Optional[list[dict]] = None +) -> dict: + """Compare verified task outcomes and observed costs without blending adoption. + + Each condition remains grouped by task, host/model, and hint setting. A + condition delta is only reported against that exact cell's native-tools run. + """ + groups: dict[tuple[str, str, str, str], list[dict]] = {} + invalid_groups: dict[tuple[str, str, str, str], list[dict]] = {} + for result in results: + host_model = f"{result['host']}/{result['model']}" if result.get("model") else result["host"] + key = ( + result["id"], + host_model, + result.get("condition", "full"), + result.get("repeated_query_hints", "off"), + ) + groups.setdefault(key, []).append(result) + for result in invalid_runs or []: + model = result.get("model") + host_model = f"{result['host']}/{model}" if model else result["host"] + key = ( + result["id"], + host_model, + result.get("condition", "full"), + result.get("repeated_query_hints", "off"), + ) + invalid_groups.setdefault(key, []).append(result) + + cells: dict[str, dict] = {} + for scenario, host_model, condition, hints in sorted(set(groups) | set(invalid_groups)): + rows = groups.get((scenario, host_model, condition, hints), []) + invalid = invalid_groups.get((scenario, host_model, condition, hints), []) + graded = [r["correctness"]["status"] for r in rows + if r["correctness"]["status"] in ("pass", "fail")] + passed = graded.count("pass") + attempts = len(rows) + len(invalid) + unknown_outcomes = sum(r["correctness"]["status"] == "ungraded" for r in rows) + unknown_outcomes += sum(r.get("outcome_kind") == "ungraded" for r in invalid) + scored_attempts = attempts - unknown_outcomes + costs = [r["cost"] for r in rows] + tool_attempts = [r["tool_success"]["attempted"] for r in rows] + tool_failures = [r["tool_success"]["failed"] for r in rows] + cell = { + "scenario": scenario, + "host_model": host_model, + "condition": condition, + "repeated_query_hints": hints, + "correctness": { + "n": len(graded), + "passed": passed, + "failed": graded.count("fail"), + "pass_rate": round(passed / len(graded), 4) if graded else None, + "wilson_95": _wilson_interval(passed, len(graded)), + "ungraded": sum(r["correctness"]["status"] == "ungraded" for r in rows), + "unverified": sum(r["correctness"]["status"] == "unverified" for r in rows), + }, + "task_success": { + "attempts": attempts, + "scored_attempts": scored_attempts, + "passed": passed, + "unsuccessful_or_unverified": scored_attempts - passed, + "unknown_outcomes": unknown_outcomes, + "pass_rate": round(passed / scored_attempts, 4) if scored_attempts else None, + "wilson_95": _wilson_interval(passed, scored_attempts), + "invalid_runs": len(invalid), + }, + "observed": { + "input_tokens": _sample_summary([c.get("input_tokens") for c in costs] + [None] * len(invalid)), + "cache_read_tokens": _sample_summary( + [c.get("cache_read_input_tokens") for c in costs] + [None] * len(invalid) + ), + "cache_creation_tokens": _sample_summary( + [c.get("cache_creation_input_tokens") for c in costs] + [None] * len(invalid) + ), + "prompt_tokens": _sample_summary( + [_prompt_tokens(r["cost"], r["host"]) for r in rows] + [None] * len(invalid) + ), + "output_tokens": _sample_summary( + [c.get("output_tokens") for c in costs] + [None] * len(invalid) + ), + "total_tokens": _sample_summary( + [c.get("total_tokens") for c in costs] + [None] * len(invalid) + ), + "cost_usd": _sample_summary( + [c.get("total_cost_usd") for c in costs] + [None] * len(invalid) + ), + "elapsed_ms": _sample_summary( + [r.get("runner_elapsed_ms") for r in rows] + + [r.get("runner_elapsed_ms") for r in invalid] + ), + "tool_attempts": _sample_summary(tool_attempts + [None] * len(invalid)), + "tool_failures": _sample_summary(tool_failures + [None] * len(invalid)), + }, + } + cells[" / ".join((scenario, host_model, condition, f"hints-{hints}"))] = cell + + for cell in cells.values(): + if cell["condition"] == "native": + cell["comparison_to_native"] = {"status": "reference"} + continue + key = ( + cell["scenario"], + cell["host_model"], + "native", + cell["repeated_query_hints"], + ) + baseline = groups.get(key, []) + baseline_invalid = invalid_groups.get(key, []) + if key not in groups and key not in invalid_groups: + cell["comparison_to_native"] = { + "status": "missing_native_baseline", + "pass_rate_delta": None, + } + continue + baseline_passes = [r["correctness"]["status"] for r in baseline + if r["correctness"]["status"] in ("pass", "fail")] + native_attempts = len(baseline) + len(baseline_invalid) + native_unknown = sum(r["correctness"]["status"] == "ungraded" for r in baseline) + native_unknown += sum(r.get("outcome_kind") == "ungraded" for r in baseline_invalid) + native_scored_attempts = native_attempts - native_unknown + product_rate = cell["task_success"]["pass_rate"] + native_rate = ( + baseline_passes.count("pass") / native_scored_attempts if native_scored_attempts else None + ) + native_costs = [r["cost"] for r in baseline] + native_metrics = { + "prompt_tokens": _sample_summary( + [_prompt_tokens(r["cost"], r["host"]) for r in baseline] + )["median"], + "total_tokens": _sample_summary( + [c.get("total_tokens") for c in native_costs] + [None] * len(baseline_invalid) + )["median"], + "cost_usd": _sample_summary( + [c.get("total_cost_usd") for c in native_costs] + [None] * len(baseline_invalid) + )["median"], + "elapsed_ms": _sample_summary([r.get("runner_elapsed_ms") for r in baseline])["median"], + "tool_attempts": _sample_summary([r["tool_success"]["attempted"] for r in baseline])["median"], + } + if baseline_invalid: + native_metrics["prompt_tokens"] = _sample_summary( + [_prompt_tokens(r["cost"], r["host"]) for r in baseline] + [None] * len(baseline_invalid) + )["median"] + native_metrics["tool_attempts"] = _sample_summary( + [r["tool_success"]["attempted"] for r in baseline] + [None] * len(baseline_invalid) + )["median"] + if baseline_invalid: + native_metrics["elapsed_ms"] = _sample_summary( + [r.get("runner_elapsed_ms") for r in baseline] + + [r.get("runner_elapsed_ms") for r in baseline_invalid] + )["median"] + observed_deltas = {} + for metric, baseline_median in native_metrics.items(): + current_median = cell["observed"][metric]["median"] + observed_deltas[metric] = ( + round(current_median - baseline_median, 4) + if current_median is not None and baseline_median is not None else None + ) + cell["comparison_to_native"] = { + "status": "compared" if product_rate is not None and native_rate is not None else "ungraded", + "native_success_rate": round(native_rate, 4) if native_rate is not None else None, + "success_rate_delta": round(product_rate - native_rate, 4) + if product_rate is not None and native_rate is not None else None, + "observed_median_deltas": observed_deltas, + } + return cells + + +def load_efficiency_scorecard(path: str, run_meta: dict) -> dict: + with open(path) as stream: + scorecard = json.load(stream) + if not scorecard.get("schema", "").startswith("tracedecay.efficiency-scorecard/"): + raise ValueError("unsupported efficiency scorecard schema") + runs = scorecard.get("runs") or [] + if scorecard.get("verdict") != "ok" or not runs or any( + run.get("status") != "ok" for run in runs + ): + raise ValueError("efficiency scorecard contains a failed or incomplete run") + expected_sha = run_meta.get("binary_sha256") + actual_sha = (scorecard.get("binary") or {}).get("sha256") + if not expected_sha or expected_sha != actual_sha: + raise ValueError("efficiency scorecard binary SHA-256 does not match the agent eval") + median = scorecard.get("median") or {} + metric_names = ( + "daemon_start_empty.spawn_to_socket_seconds", + "cold_index.wall_seconds", + "cold_index.seal_to_activation_seconds", + "cold_index.daemon_build_elapsed_seconds", + "incremental_sync.wall_seconds", + ) + return { + "binary_sha256": actual_sha, + "binary_version": (scorecard.get("binary") or {}).get("version"), + "source_git_commit": (scorecard.get("source") or {}).get("git_commit"), + "fixture_sha256": (scorecard.get("fixture") or {}).get("tree_sha256"), + "runs": len(scorecard.get("runs", [])), + "median": {name: median.get(name) for name in metric_names}, + "attribution": "separate no-model runtime journey; not added to agent token or USD totals", + } + + def usd_per_pass(rs: list[dict]) -> Optional[float]: """Total spend divided by verified passes; None when spend is unknown or nothing passed.""" spend = [r["cost"].get("total_cost_usd") for r in rs] @@ -1333,8 +1577,19 @@ def render_report(scoreboard: dict) -> str: meta = scoreboard.get("meta", {}) lines.append(f"- run: `{meta.get('run_id','?')}`") lines.append(f"- git: `{meta.get('git_sha','?')}`") + lines.append(f"- binary: `{meta.get('binary_version') or 'unknown'}` " + f"(sha256 `{meta.get('binary_sha256') or 'unknown'}`)") + lines.append(f"- fixture sha256: `{meta.get('fixture_sha256') or 'unknown'}`") lines.append(f"- graded: {len(scoreboard['results'])} transcript(s)") lines.append(f"- invalid launches excluded: {len(meta.get('invalid_runs', []))}") + setup_ms = meta.get("fixture_setup_elapsed_ms") + lines.append(f"- one-time fixture initialization/indexing: " + f"{setup_ms} ms" if setup_ms is not None + else "- one-time fixture initialization/indexing: unknown") + scorecard = meta.get("efficiency_scorecard") + if scorecard: + lines.append(f"- no-model efficiency scorecard: binary SHA matched; " + f"{scorecard['runs']} run(s); fixture `{scorecard.get('fixture_sha256') or 'unknown'}`") lines.append("") lines.append("## Skill routing (separate from task outcomes)") lines.append("") @@ -1445,7 +1700,9 @@ def cell(name): lines.append("## Correctness, cost, latency, and repeated-query hints") lines.append("") - lines.append("Prompt tokens include cache reads and writes. `-` means the host did not report it.") + lines.append("Prompt tokens include cache reads and writes when the host reports them; schema/context " + "tokens are included in host-reported input usage, but hosts do not separately " + "attribute tool-schema overhead. `-` means unknown, not zero.") lines.append("") lines += render_cost_and_hints(scoreboard.get("cost_and_hints", {})) if scoreboard.get("repeated_query_scenarios"): @@ -1453,6 +1710,86 @@ def cell(name): lines.append("") lines += render_cost_and_hints(scoreboard["repeated_query_scenarios"]) + lines.append("## Matched task effectiveness and observed cost") + lines.append("") + lines.append("Task outcome is independently graded and native success counts equally. " + "Intervals are 95% Wilson intervals for the per-cell pass rate; cost ranges " + "describe observed run-to-run variability. Timeouts, failed launches, and " + "unverified outcomes count as unsuccessful attempts, not as successful task " + "completions. Prompt tokens include cache reads " + "and writes where reported; tool schema/context overhead is part of host-reported " + "input usage and cannot be separated by these hosts. Missing usage is unknown, " + "never zero. Cold startup/indexing is reported separately and is not converted " + "to dollars or silently amortized.") + lines.append("") + lines.append("| task / host / condition | successful / scored attempts | 95% interval | verified outcome pass rate | success Δ vs native | median cost Δ (prompt tok / USD / ms) | input tokens | cache read | cache create | output tokens | prompt total | total tokens | USD | elapsed ms | tool attempts / failures |") + lines.append("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|") + + def sample_cell(sample: dict, digits=0): + if not sample or sample["median"] is None: + return "-" + lo, hi = sample["range"] + return f"{sample['median']:.{digits}f} [{lo:.{digits}f}, {hi:.{digits}f}] (unknown {sample['unknown']})" + + for key, cell in sorted(scoreboard.get("effectiveness", {}).items()): + correctness = cell["correctness"] + task_success = cell["task_success"] + task_success_text = f"{task_success['passed']}/{task_success['scored_attempts']}" + if task_success["unknown_outcomes"]: + task_success_text += f" ({task_success['unknown_outcomes']} ungraded)" + if task_success["invalid_runs"]: + task_success_text += f" ({task_success['invalid_runs']} invalid)" + interval = task_success["wilson_95"] + interval_text = "-" if interval is None else f"[{interval[0]:.2f}, {interval[1]:.2f}]" + comparison = cell.get("comparison_to_native", {}) + delta = comparison.get("success_rate_delta") + delta_text = "-" if delta is None else f"{delta:+.2f}" + cost_deltas = comparison.get("observed_median_deltas", {}) + def delta_value(metric, digits): + value = cost_deltas.get(metric) + return "-" if value is None else f"{value:+.{digits}f}" + delta_cost_text = ( + f"{delta_value('prompt_tokens', 0)} / {delta_value('cost_usd', 4)} / " + f"{delta_value('elapsed_ms', 0)}" + ) + observed = cell["observed"] + attempts = observed["tool_attempts"]["mean"] + failures = observed["tool_failures"]["mean"] + tool_unknown = observed["tool_attempts"]["unknown"] + tool_text = "-" if attempts is None else f"{attempts:.1f} / {failures:.1f}" + if tool_unknown: + tool_text += f" ({tool_unknown} unknown)" + lines.append( + f"| {key} | {task_success_text} | {interval_text} | " + f"{fmt(correctness['pass_rate'])} | {delta_text} | " + f"{delta_cost_text} | " + f"{sample_cell(observed['input_tokens'])} | {sample_cell(observed['cache_read_tokens'])} | " + f"{sample_cell(observed['cache_creation_tokens'])} | {sample_cell(observed['output_tokens'])} | " + f"{sample_cell(observed['prompt_tokens'])} | {sample_cell(observed['total_tokens'])} | " + f"{sample_cell(observed['cost_usd'], 4)} | " + f"{sample_cell(observed['elapsed_ms'])} | {tool_text} |" + ) + lines.append("") + if scorecard: + lines.append("### Startup, indexing, and repeated-use resource baseline") + lines.append("") + lines.append("The efficiency scorecard is a separate no-model runtime journey. Its artifact " + "SHA-256 must match this agent evaluation. These wall times and resource " + "measurements are not model-token or monetary costs.") + lines.append("") + lines.append("| phase | median seconds |") + lines.append("|---|---:|") + for label, metric in ( + ("daemon start (empty profile)", "daemon_start_empty.spawn_to_socket_seconds"), + ("cold index (wall)", "cold_index.wall_seconds"), + ("cold index seal to activation", "cold_index.seal_to_activation_seconds"), + ("cold index daemon build", "cold_index.daemon_build_elapsed_seconds"), + ("incremental sync (wall)", "incremental_sync.wall_seconds"), + ): + value = scorecard["median"].get(metric) + lines.append(f"| {label} | {'-' if value is None else f'{value:.3f}'} |") + lines.append("") + if scoreboard.get("learned_skill"): lines += render_learned_skill(scoreboard["learned_skill"]) @@ -1546,6 +1883,10 @@ def main() -> int: here = os.path.dirname(os.path.abspath(__file__)) ap.add_argument("--run-dir") ap.add_argument("--scenarios", default=os.path.join(here, "scenarios")) + ap.add_argument( + "--efficiency-scorecard", + help="no-model efficiency-scorecard JSON from the exact same binary artifact", + ) ap.add_argument( "--lint-only", action="store_true", @@ -1627,6 +1968,15 @@ def main() -> int: if os.path.exists(meta_path): with open(meta_path) as f: run_meta_all = json.load(f) + efficiency_scorecard = None + if args.efficiency_scorecard: + try: + efficiency_scorecard = load_efficiency_scorecard( + args.efficiency_scorecard, run_meta_all + ) + except (OSError, ValueError) as error: + print(f"efficiency scorecard rejected: {error}", file=sys.stderr) + return 1 results = [] invalid_runs = [] @@ -1669,6 +2019,9 @@ def main() -> int: "host": host, "model": per_meta.get("model"), "condition": condition, + "repeated_query_hints": per_meta.get("repeated_query_hints", "off"), + "runner_elapsed_ms": per_meta.get("duration_ms"), + "outcome_kind": (scn.get("outcome") or {}).get("kind", "ungraded"), "exit_code": exit_code, "timed_out": bool(per_meta.get("timed_out")), "reason": "timed_out" if per_meta.get("timed_out") else "launch_failed", @@ -1684,6 +2037,9 @@ def main() -> int: "host": host, "model": per_meta.get("model"), "condition": condition, + "repeated_query_hints": per_meta.get("repeated_query_hints", "off"), + "runner_elapsed_ms": per_meta.get("duration_ms"), + "outcome_kind": (scn.get("outcome") or {}).get("kind", "ungraded"), "exit_code": exit_code, "timed_out": False, "reason": "unparseable_empty", @@ -1703,11 +2059,18 @@ def main() -> int: "meta": { "run_id": os.path.basename(os.path.normpath(run_dir)), "git_sha": run_meta_all.get("git_sha", "?"), + "binary_path": run_meta_all.get("binary_path"), + "binary_sha256": run_meta_all.get("binary_sha256"), + "binary_version": run_meta_all.get("binary_version"), + "fixture_sha256": run_meta_all.get("fixture_sha256"), + "fixture_setup_elapsed_ms": run_meta_all.get("fixture_setup_elapsed_ms"), + "efficiency_scorecard": efficiency_scorecard, "hosts": run_meta_all.get("hosts", {}), "invalid_runs": invalid_runs, }, "aggregate": aggregate(results), "cost_and_hints": aggregate_cost_and_hints(results), + "effectiveness": effectiveness_comparisons(results, invalid_runs), "repeated_query_scenarios": aggregate_cost_and_hints( [r for r in results if r.get("repeated_query_class")], by_scenario=True), "results": results, diff --git a/evals/agent_adoption/run.sh b/evals/agent_adoption/run.sh index c85245e7db..d3768dacd3 100755 --- a/evals/agent_adoption/run.sh +++ b/evals/agent_adoption/run.sh @@ -27,6 +27,7 @@ # HOSTS space-separated: "claude", "codex", or "claude codex" (default: claude) # SCENARIOS space-separated scenario ids to run (default: all active) # EVAL_INCLUDE_DEFERRED=1 also run scenarios with status="deferred" +# EVAL_INCLUDE_BASELINE=1 also run the held-out product-effectiveness cohort # CLAUDE_MODELS space-separated Claude matrix (default: opus sonnet) # CODEX_MODELS space-separated Codex matrix (default: gpt-5.6-sol) # CODEX_REASONING_EFFORT Codex reasoning effort (default: medium) @@ -76,9 +77,9 @@ REPEATED_QUERY_HINTS="${REPEATED_QUERY_HINTS:-off}" # Ablation matrix. Default is "full" only (all discovery channels on) to keep # cost bounded, each extra condition multiplies the number of live agent runs. -# Opt in with e.g. CHANNELS="full no-hints no-skills bare cli-only". See the README. +# Opt in with e.g. CHANNELS="native bare" or "full no-hints". See the README. CHANNELS="${CHANNELS:-full}" -KNOWN_CONDITIONS="full no-hints no-skills bare cli-only" +KNOWN_CONDITIONS="native bare full no-hints no-skills cli-only" for c in $CHANNELS; do case " $KNOWN_CONDITIONS " in *" $c "*) : ;; @@ -399,6 +400,7 @@ build_fixture() { fixture_main="$work/fixture-main" fixture_broken="$work/fixture-broken" +setup_started_ns="$(python3 -c 'import time; print(time.monotonic_ns())')" echo "indexing fixture-main (with seeded git history)..." build_fixture "$fixture_main" clean history if [[ -n "${LEARNED_SKILL_GENERATE:-}" ]]; then @@ -476,11 +478,53 @@ if [[ "$REPEATED_QUERY_HINTS" == on ]]; then fi fi -git_sha="$(git -C "$repo_root" rev-parse --short HEAD 2>/dev/null || echo unknown)" +setup_elapsed_ms=$(( ($(python3 -c 'import time; print(time.monotonic_ns())') - setup_started_ns) / 1000000 )) +git_sha="$(git -C "$repo_root" rev-parse HEAD 2>/dev/null || echo unknown)" +binary_path_json="$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$TD")" +binary_sha256="$(python3 - "$TD" <<'PY' +import hashlib, sys +digest = hashlib.sha256() +with open(sys.argv[1], "rb") as binary: + for chunk in iter(lambda: binary.read(1 << 20), b""): + digest.update(chunk) +print(digest.hexdigest()) +PY +)" +binary_version="$("$TD" --version 2>/dev/null | head -1 | tr -d '\n')" +fixture_digest() { + python3 - "$1" <<'PY' +import hashlib, os, sys +root = sys.argv[1] +digest = hashlib.sha256() +for base, dirs, files in os.walk(root): + dirs[:] = sorted(d for d in dirs if d != ".git") + for name in sorted(files): + path = os.path.join(base, name) + relative = os.path.relpath(path, root).replace(os.sep, "/") + digest.update(relative.encode("utf-8")) + digest.update(b"\0") + with open(path, "rb") as contents: + for chunk in iter(lambda: contents.read(1 << 20), b""): + digest.update(chunk) +print(digest.hexdigest()) +PY +} +fixture_main_sha256="$(fixture_digest "$fixture_main")" +fixture_broken_sha256="$(fixture_digest "$fixture_broken")" cat > "$run_dir/meta.json" < channels: +# native no MCP, plugin, hooks, skills, or TraceDecay CLI access +# bare MCP descriptions only; no hooks, skills, or steering # full hooks ON | skills ON | steering=plugin (candidate plugin, hermetic profile) # no-hints hooks OFF | skills ON | steering=fixed (isolates skill/description efficacy) # no-skills hooks ON | skills OFF | steering=fixed (isolates hint/description efficacy) @@ -515,6 +561,7 @@ fi provision_variant() { # $1 = condition; provisions $work/plugins/ (hermetic plugin copy). local cond="$1" d="$work/plugins/$1" + [[ "$cond" == "native" ]] && return 0 [[ -d "$d" ]] && return 0 mkdir -p "$work/plugins" cp -R "$plugin_src" "$d" @@ -539,6 +586,12 @@ CLAUDE_EXTRA=() claude_extra_for() { CLAUDE_EXTRA=() local cond="$1" fdir="$2" + if [[ "$cond" == "native" ]]; then + CLAUDE_EXTRA=(--setting-sources user,project,local + --strict-mcp-config --mcp-config "$empty_mcp_cfg" + --add-dir "$fdir") + return 0 + fi if [[ "$have_plugin" != "1" ]]; then echo "error: candidate plugin dir missing; cannot run '$cond' hermetically" >&2 return 2 @@ -560,9 +613,9 @@ claude_extra_for() { } # ---- select scenarios ------------------------------------------------------ # -python3 - "$EVAL_SCENARIOS_DIR" "${SCENARIOS:-}" "${EVAL_INCLUDE_DEFERRED:-0}" "$LEARNED_SKILL_WORKFLOW" > "$work/selected.tsv" <<'PY' +python3 - "$EVAL_SCENARIOS_DIR" "${SCENARIOS:-}" "${EVAL_INCLUDE_DEFERRED:-0}" "$LEARNED_SKILL_WORKFLOW" "${EVAL_INCLUDE_BASELINE:-0}" > "$work/selected.tsv" <<'PY' import json, os, sys -sdir, filt, incl_def, workflow = sys.argv[1], sys.argv[2].split(), sys.argv[3] == "1", sys.argv[4] +sdir, filt, incl_def, workflow, incl_baseline = sys.argv[1], sys.argv[2].split(), sys.argv[3] == "1", sys.argv[4], sys.argv[5] == "1" for fn in sorted(os.listdir(sdir)): if not fn.endswith(".json"): continue @@ -570,7 +623,9 @@ for fn in sorted(os.listdir(sdir)): if workflow: if (s.get("learned_skill") or {}).get("workflow") != workflow: continue - elif s.get("status") == "learned" or (s.get("status") == "deferred" and not incl_def): + elif (s.get("status") == "learned" + or (s.get("status") == "deferred" and not incl_def) + or (s.get("status") == "baseline" and not incl_baseline)): continue if filt and s["id"] not in filt: continue @@ -583,8 +638,11 @@ PY # ---- run each scenario x host x condition ---------------------------------- # run_claude() { # Uses the global CLAUDE_EXTRA array set by claude_extra_for. - local prompt="$1" fixture="$2" out="$3" err="$4" model="$5" home="$6" + local prompt="$1" fixture="$2" out="$3" err="$4" model="$5" home="$6" condition="$7" local allowed="Read,Glob,Grep,Task,Agent,Skill,mcp__tracedecay__*,mcp__plugin_tracedecay_graph__*,Bash(tracedecay tool *),Bash($TD tool *)" + if [[ "$condition" == "native" ]]; then + allowed="Read,Write,Edit,Glob,Grep,Task,Agent,Skill" + fi ( cd "$fixture" && HOME="$home" CLAUDE_CONFIG_DIR="$home/.claude" PATH="$EVAL_PATH" \ timeout "$SCENARIO_TIMEOUT" claude -p "$prompt" \ --output-format stream-json --verbose --include-hook-events \ @@ -635,35 +693,48 @@ PY # One live scenario x host x model x condition x repetition. Runs in a # background subshell when PARALLEL > 1, so it must not mutate shared state. run_one() { - local cond="$1" sid="$2" host="$3" fixture="$4" prompt="$5" model="$6" base="$7" variant="$8" effect="${9:-0}" - local fdir out err start end rc=0 timed_out=false home="$CLAUDE_EVAL_HOME" learned="" + local cond="$1" sid="$2" host="$3" fixture="$4" prompt="$5" model="$6" base="$7" variant="$8" effect="${9:-0}" rep="${10:-1}" + local fdir out err start end duration_ms scenario_sha256 fixture_sha model_settings_json="{}" rc=0 timed_out=false home="$CLAUDE_EVAL_HOME" learned="" if [[ -n "$variant" ]]; then home="${VARIANT_HOME[$variant]}" learned=",\"learned_skill_variant\":\"$variant\",\"learned_skill_id\":\"${VARIANT_SKILL[$variant]}\"" fi + [[ "$host" == "codex" ]] && model_settings_json="{\"reasoning_effort\":\"$CODEX_REASONING_EFFORT\"}" fdir="$(fixture_dir_for "$fixture")" + scenario_sha256="$(python3 - "$EVAL_SCENARIOS_DIR/$sid.json" <<'PY' +import hashlib, sys +digest = hashlib.sha256() +with open(sys.argv[1], "rb") as scenario: + for chunk in iter(lambda: scenario.read(1 << 20), b""): + digest.update(chunk) +print(digest.hexdigest()) +PY +)" + fixture_sha="$fixture_main_sha256" + [[ "$fixture" == "broken" ]] && fixture_sha="$fixture_broken_sha256" out="$run_dir/${base}.stdout.jsonl" err="$run_dir/${base}.stderr.log" echo "RUN $sid [$host/$model/$cond${variant:+/v-$variant}] ..." - start=$(date +%s) + start="$(python3 -c 'import time; print(time.monotonic_ns())')" if [[ "$host" == "claude" ]]; then claude_extra_for "$cond" "$fdir" - run_claude "$prompt" "$fdir" "$out" "$err" "$model" "$home" || rc=$? + run_claude "$prompt" "$fdir" "$out" "$err" "$model" "$home" "$cond" || rc=$? else run_codex "$prompt" "$fdir" "$out" "$err" "$model" || rc=$? fi - end=$(date +%s) + end="$(python3 -c 'import time; print(time.monotonic_ns())')" + duration_ms=$(( (end - start) / 1000000 )) [[ $rc -eq 124 ]] && timed_out=true [[ "$effect" == 1 ]] && snapshot_effects "$fdir" "$run_dir/${base}.effects.json" cat > "$run_dir/${base}.meta.json" <= PARALLEL )); do wait -n || true; done - run_one "$cond" "$sid" "$host" "$fixture" "$prompt" "$model" "$base" "$variant" 0 & + run_one "$cond" "$sid" "$host" "$fixture" "$prompt" "$model" "$base" "$variant" 0 "$rep" & else - run_one "$cond" "$sid" "$host" "$fixture" "$prompt" "$model" "$base" "$variant" 0 + run_one "$cond" "$sid" "$host" "$fixture" "$prompt" "$model" "$base" "$variant" 0 "$rep" fi done done @@ -715,7 +786,13 @@ wait # ---- grade ----------------------------------------------------------------- # if [[ "$live" == "1" ]]; then echo "grading..." - python3 "$here/grade.py" --run-dir "$run_dir" --scenarios "$EVAL_SCENARIOS_DIR" || true + grade_args=(--run-dir "$run_dir" --scenarios "$EVAL_SCENARIOS_DIR") + [[ -n "${EFFICIENCY_SCORECARD:-}" ]] && grade_args+=(--efficiency-scorecard "$EFFICIENCY_SCORECARD") + if [[ -n "${EFFICIENCY_SCORECARD:-}" ]]; then + python3 "$here/grade.py" "${grade_args[@]}" || exit 1 + else + python3 "$here/grade.py" "${grade_args[@]}" || true + fi if [[ -n "${EVAL_OUT:-}" ]]; then mkdir -p "$EVAL_OUT" cp "$run_dir/scoreboard.json" "$run_dir/report.md" "$EVAL_OUT/" 2>/dev/null || true diff --git a/evals/agent_adoption/scenarios/baseline_currency_decision.json b/evals/agent_adoption/scenarios/baseline_currency_decision.json new file mode 100644 index 0000000000..b0f5289a13 --- /dev/null +++ b/evals/agent_adoption/scenarios/baseline_currency_decision.json @@ -0,0 +1,31 @@ +{ + "id": "baseline_currency_decision", + "category": "product_effectiveness", + "hosts": [ + "claude" + ], + "fixture": "main", + "status": "baseline", + "prompt": "Was multi-currency support accepted or rejected, and what units are order totals stored in?", + "outcome": { + "kind": "answer", + "answer": { + "required": [ + { + "id": "identifies the rejection", + "pattern": "\\brejected\\b" + }, + { + "id": "identifies the 2026-05 review", + "pattern": "\\b2026\\-05\\b" + }, + { + "id": "states USD cents", + "pattern": "\\bUSD cents\\b" + } + ] + } + }, + "seeded_fact": "currency_fact_id", + "notes": "New held-out session-memory task with structured answer checks; this decision is seeded in memory but is not present in the fixture source." +} diff --git a/evals/agent_adoption/scenarios/baseline_order_stock_failure.json b/evals/agent_adoption/scenarios/baseline_order_stock_failure.json new file mode 100644 index 0000000000..4e3d59c05f --- /dev/null +++ b/evals/agent_adoption/scenarios/baseline_order_stock_failure.json @@ -0,0 +1,30 @@ +{ + "id": "baseline_order_stock_failure", + "category": "product_effectiveness", + "hosts": [ + "claude" + ], + "fixture": "main", + "status": "baseline", + "prompt": "If an order requests more units than a warehouse has, how does order placement fail, and which checks are involved?", + "outcome": { + "kind": "answer", + "answer": { + "required": [ + { + "id": "names reserve_stock", + "pattern": "\\breserve_stock\\b" + }, + { + "id": "names check_availability", + "pattern": "\\bcheck_availability\\b" + }, + { + "id": "explains failure", + "pattern": "\\b(?:Err|error|fails?|insufficient)\\b" + } + ] + } + }, + "notes": "New held-out answer task with an independently checked failure path; included only by explicit baseline-cohort opt-in." +} diff --git a/evals/agent_adoption/scenarios/baseline_stock_check_rename.json b/evals/agent_adoption/scenarios/baseline_stock_check_rename.json new file mode 100644 index 0000000000..6376cd990d --- /dev/null +++ b/evals/agent_adoption/scenarios/baseline_stock_check_rename.json @@ -0,0 +1,52 @@ +{ + "id": "baseline_stock_check_rename", + "category": "product_effectiveness", + "hosts": [ + "claude" + ], + "fixture": "main", + "status": "baseline", + "prompt": "Rename the warehouse availability check to has_sufficient_stock and update its callers. Keep the existing stock boundary behavior unchanged.", + "outcome": { + "kind": "effect", + "files": { + "src/inventory.rs": { + "required": [ + { + "id": "renamed function exists", + "pattern": "\\bfn has_sufficient_stock\\b" + }, + { + "id": "zero-default and inclusive stock boundary preserved", + "pattern": "unwrap_or\\(0\\)\\s*>=\\s*qty" + } + ], + "forbidden": [ + { + "id": "old function name remains", + "pattern": "\\bcheck_availability\\b" + }, + { + "id": "stock boundary changed to strict comparison", + "pattern": "unwrap_or\\(0\\)\\s*>\\s*qty" + } + ] + }, + "src/*.rs": { + "required": [ + { + "id": "new name is referenced", + "pattern": "\\bhas_sufficient_stock\\b" + } + ], + "forbidden": [ + { + "id": "old name remains in source", + "pattern": "\\bcheck_availability\\b" + } + ] + } + } + }, + "notes": "New held-out mutation task; post-run source snapshots verify the rename and reject a harmful boundary change." +} diff --git a/evals/agent_adoption/selftest.py b/evals/agent_adoption/selftest.py index 1def1e7adf..508971990c 100644 --- a/evals/agent_adoption/selftest.py +++ b/evals/agent_adoption/selftest.py @@ -125,6 +125,8 @@ def test_channels(): # unprompted: bare condition, adoption with nothing to credit. tr = transcript([claude_tool(TD, {}), claude_result("done")]) check("channel: unprompted under bare", grade.attribute_channel(tr, "bare") == grade.CH_UNPROMPTED) + check("channel: native run has no TraceDecay adoption", + grade.attribute_channel(transcript([claude_result("done")]), "native") == grade.CH_NONE) # CLI-only: the plugin has agents/skills but no MCP registration. A Bash # call to the TraceDecay CLI is still adoption and must retain tool identity. @@ -383,6 +385,14 @@ def test_end_to_end(): os.makedirs(rdir) with open(os.path.join(sdir, "explore_reserve_stock.json"), "w") as f: json.dump(scn, f) + with open(os.path.join(rdir, "meta.json"), "w") as f: + json.dump({ + "git_sha": "source-revision", + "binary_sha256": "artifact-digest", + "binary_version": "tracedecay test", + "fixture_sha256": "fixture-digest", + "fixture_setup_elapsed_ms": 1250, + }, f) # full condition, steering-driven, correct answer. answer = "The reserve_stock function decrements on-hand stock." @@ -430,8 +440,17 @@ def test_end_to_end(): agg = sb["aggregate"]["claude"] check("e2e: adoption_rate 1.0", agg["adoption_rate"] == 1.0) check("e2e: per-condition present", set(agg["conditions"]) == {"full", "bare"}) + check("e2e: no native baseline is reported as missing", + all(cell["comparison_to_native"]["status"] == "missing_native_baseline" + for cell in sb["effectiveness"].values())) + check("e2e: artifact and one-time setup identity preserved", + sb["meta"]["binary_sha256"] == "artifact-digest" + and sb["meta"]["fixture_setup_elapsed_ms"] == 1250) report = open(os.path.join(rdir, "report.md")).read() check("e2e: report has channel table", "Channel efficacy" in report) + check("e2e: report includes effectiveness and unknown setup attribution", + "Matched task effectiveness and observed cost" in report + and "1250 ms" in report) check("e2e: run meta hint setting reaches the scoreboard", by_cond["bare"].get("repeated_query_hints") == "on" and by_cond["full"].get("repeated_query_hints") == "off") @@ -669,6 +688,152 @@ def test_cost_and_corrections(): check("cost: scored result carries cost", scored["cost"] == tr.cost) +def test_product_effectiveness_comparisons(): + scenario = { + "id": "baseline_sample", + "category": "product_effectiveness", + "outcome": answer_outcome("correct"), + } + + def scored(condition, answer, usage, elapsed): + event = {"type": "result", "result": answer} + if usage is not None: + event["usage"] = usage + return grade.score_scenario( + scenario, + transcript([json.dumps(event)]), + {}, + { + "model": "sonnet", + "channel_condition": condition, + "duration_ms": elapsed, + "repeated_query_hints": "off", + }, + ) + + rows = [ + scored("native", "correct answer", { + "input_tokens": 100, "output_tokens": 20, + "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0, + }, 1000), + scored("native", "wrong answer", { + "input_tokens": 120, "output_tokens": 30, + "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0, + }, 1100), + scored("bare", "correct answer", None, 900), + scored("full", "correct answer", {"input_tokens": 80, "output_tokens": 10}, 800), + ] + for row in rows: + row["tool_success"] = {"attempted": 0, "failed": 0} + invalid = [{ + "id": "baseline_sample", + "host": "claude", + "model": "sonnet", + "condition": "native", + "repeated_query_hints": "off", + "runner_elapsed_ms": 1200, + "outcome_kind": "answer", + }] + cells = grade.effectiveness_comparisons(rows, invalid) + native = cells["baseline_sample / claude/sonnet / native / hints-off"] + bare = cells["baseline_sample / claude/sonnet / bare / hints-off"] + full = cells["baseline_sample / claude/sonnet / full / hints-off"] + check("effectiveness: task success is grouped independently of adoption", + native["correctness"]["pass_rate"] == 0.5 and native["correctness"]["wilson_95"] is not None) + check("effectiveness: condition effect is compared on the same task/model", + full["comparison_to_native"]["success_rate_delta"] == round(2 / 3, 4) + and bare["comparison_to_native"]["success_rate_delta"] == round(2 / 3, 4)) + check("effectiveness: invalid launches count as unsuccessful attempts", + native["task_success"]["attempts"] == 3 + and native["task_success"]["invalid_runs"] == 1 + and native["task_success"]["pass_rate"] == round(1 / 3, 4)) + check("effectiveness: missing usage remains unknown", + bare["observed"]["total_tokens"]["unknown"] == 1 + and bare["observed"]["total_tokens"]["mean"] is None) + check("effectiveness: repeated-run variability is reported", + native["observed"]["elapsed_ms"]["range"] == [1000, 1200] + and native["observed"]["prompt_tokens"]["range"] == [100, 120]) + check("effectiveness: cache usage remains separately observable", + native["observed"]["cache_read_tokens"]["unknown"] == 1 + and native["observed"]["cache_creation_tokens"]["unknown"] == 1 + and grade._prompt_tokens({"input_tokens": 100}, "claude") is None + and grade._prompt_tokens( + {"input_tokens": 1000, "cache_read_input_tokens": 800}, "codex" + ) == 1000) + ungraded_scenario = { + "id": "baseline_ungraded", + "category": "product_effectiveness", + "outcome": {"kind": "ungraded", "reason": "no independent checker"}, + } + ungraded = grade.score_scenario( + ungraded_scenario, + transcript([claude_result("an answer")]), + {}, + {"channel_condition": "native"}, + ) + ungraded["tool_success"] = {"attempted": 0, "failed": 0} + ungraded_cell = grade.effectiveness_comparisons([ungraded])[ + "baseline_ungraded / claude / native / hints-off" + ] + check("effectiveness: ungraded outcomes remain unknown, not failures", + ungraded_cell["task_success"]["pass_rate"] is None + and ungraded_cell["task_success"]["unknown_outcomes"] == 1) + + with tempfile.TemporaryDirectory() as work: + path = Path(work) / "scorecard.json" + path.write_text(json.dumps({ + "schema": "tracedecay.efficiency-scorecard/v1", + "binary": {"sha256": "abc", "version": "tracedecay 1"}, + "source": {"git_commit": "commit"}, + "fixture": {"tree_sha256": "fixture"}, + "verdict": "ok", + "runs": [{"status": "ok"}, {"status": "ok"}], + "median": {"cold_index.wall_seconds": 12.5}, + })) + attached = grade.load_efficiency_scorecard(str(path), {"binary_sha256": "abc"}) + check("effectiveness: cold-index baseline requires matching artifact identity", + attached["median"]["cold_index.wall_seconds"] == 12.5 + and attached["runs"] == 2) + try: + grade.load_efficiency_scorecard(str(path), {"binary_sha256": "other"}) + matched_only = False + except ValueError: + matched_only = True + check("effectiveness: scorecard from another artifact is rejected", matched_only) + value = json.loads(path.read_text()) + value["runs"][0]["status"] = "failed" + path.write_text(json.dumps(value)) + failed_scorecard_rejected = False + try: + grade.load_efficiency_scorecard(str(path), {"binary_sha256": "abc"}) + except ValueError: + failed_scorecard_rejected = True + check("effectiveness: incomplete efficiency scorecard is rejected", + failed_scorecard_rejected) + + baseline = {} + for name in ( + "baseline_order_stock_failure.json", + "baseline_stock_check_rename.json", + "baseline_currency_decision.json", + ): + scenario = json.loads((Path(HERE) / "scenarios" / name).read_text()) + baseline[scenario["id"]] = scenario + check("effectiveness: baseline tasks are neutral and independently graded", + not grade.lint_scenarios(baseline) and not grade.lint_outcomes(baseline)) + edit = baseline["baseline_stock_check_rename"] + good = { + "src/inventory.rs": "pub fn has_sufficient_stock(&self, sku: &str, qty: u32) -> bool { self.on_hand.get(sku).copied().unwrap_or(0) >= qty }", + "src/orders.rs": "has_sufficient_stock", + } + harmful = {**good, "src/inventory.rs": good["src/inventory.rs"].replace(">= qty", "> qty")} + edit_result = grade.score_scenario(edit, transcript([claude_result("done")]), {}, {}, good) + harmful_result = grade.score_scenario(edit, transcript([claude_result("done")]), {}, {}, harmful) + check("effectiveness: mutation outcome rejects harmful stock boundary edits", + edit_result["correctness"]["status"] == "pass" + and harmful_result["correctness"]["status"] == "fail") + + LEARNED_SKILL = "pricing-change-review" @@ -1023,6 +1188,7 @@ def main() -> int: test_repeated_query_hints_and_usage() test_skill_routing() test_cost_and_corrections() + test_product_effectiveness_comparisons() test_learned_skill_routing() test_learned_skill_lint() test_learned_skill_end_to_end() From 35f8574ec11fb2d3dcb3330c861b8d93726d539d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 19 Oct 2026 04:38:38 +0000 Subject: [PATCH 3/3] fix(evals): require shipping artifact for baseline Co-authored-by: ScriptedAlchemy <25274700+ScriptedAlchemy@users.noreply.github.com> --- evals/agent_adoption/README.md | 5 ++++- evals/agent_adoption/run.sh | 4 ++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/evals/agent_adoption/README.md b/evals/agent_adoption/README.md index 8e95f940d2..bab90e9aa9 100644 --- a/evals/agent_adoption/README.md +++ b/evals/agent_adoption/README.md @@ -89,7 +89,7 @@ paths, so do **not** commit run artifacts. | `PARALLEL` | `1` | concurrent live agent runs; scenarios are read-only against shared fixtures and each `claude` process spawns its own MCP server, so `4` is a safe default for high-rep cells | | `REPEATED_QUERY_HINTS` | `off` | `on` turns on the profile setting `user.repeated_query_hints.v1` for the eval's isolated profile before any agent launches, and suffixes transcripts with `__qhints`; see "Repeated-query hints on vs off" | | `REAL_CLAUDE_DIR` | `$CLAUDE_CONFIG_DIR` or `~/.claude` | where the Claude credential comes from: its `.claude_code_oauth_token` and `settings.json` `env`/`model` keys; see "Store isolation & auth" | -| `TRACEDECAY_BIN` | candidate branch (live) | explicit tracedecay binary override | +| `TRACEDECAY_BIN` | candidate branch (live) | explicit binary override; required for the product-effectiveness cohort | | `EFFICIENCY_SCORECARD` | unset | optional no-model scorecard path, joined only when its binary digest matches | | `EVAL_OUT` | unset | dir to copy scoreboard + report into | | `LEARNED_SKILL_WORKFLOW` | unset | run only that workflow's held-out scenarios, once per variant, on `full` | @@ -104,7 +104,10 @@ They are excluded from ordinary adoption runs. Run them with one host/model, one artifact, identical prompts, and repeated trials: ```sh +bazel build --config=release //crates/tracedecay-cli:tracedecay +TRACEDECAY_BIN="$(bazel info --config=release bazel-bin)/crates/tracedecay-cli/tracedecay" TRACEDECAY_AGENT_EVALS=1 EVAL_INCLUDE_BASELINE=1 HOSTS=claude \ + TRACEDECAY_BIN="$TRACEDECAY_BIN" \ CLAUDE_MODELS=sonnet \ SCENARIOS="baseline_order_stock_failure baseline_stock_check_rename baseline_currency_decision" \ CHANNELS="native bare no-hints no-skills full" REPS=3 \ diff --git a/evals/agent_adoption/run.sh b/evals/agent_adoption/run.sh index d3768dacd3..badee0820f 100755 --- a/evals/agent_adoption/run.sh +++ b/evals/agent_adoption/run.sh @@ -111,6 +111,10 @@ STEER_TEXT="This repository has indexed code-relationship evidence available. Ch live=0 if [[ "${TRACEDECAY_AGENT_EVALS:-}" == "1" ]]; then live=1; fi +if [[ "${EVAL_INCLUDE_BASELINE:-0}" == "1" && -z "${TRACEDECAY_BIN:-}" ]]; then + echo "error: product-effectiveness evals require an explicit shipping TRACEDECAY_BIN" >&2 + exit 2 +fi # Live runs default to the candidate branch binary. TRACEDECAY_BIN remains an # explicit override for release-binary comparisons; dry runs reuse a built