diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index 5eef1adc..3e95de17 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -551,3 +551,223 @@ divergences, so the deferred-work record is one place. Measurements in candidate — a real bug needing its own change, with a decision about whether legacy records can be distinguished from current ones at all. Caught in: timing-capture final review (gpt-5.6-sol). + +- [x] ~~No golden-corpus assertion of the four-bucket identity.~~ **DONE.** The + fixture clocks were unified (`_rebase_notifications` / `_rebase_lines` shift + codex's 2027 base and opencode's month-old base onto the replay's own clock, + keeping every derived duration exact) and `assert_timing_captured` now + asserts `Σ generation + ∪ tool + head + tail` against `duration_seconds`. + Mutation-verified: reintroducing the defect fails + `test_antigravity_golden[d_orphaned_tool]`, which previously passed. + 22 of 27 scenarios are checked. The remaining 5 are exempt via + `FICTIONAL_DURATIONS` for a reason rebasing cannot fix: they inject SDK + stamps in integer MILLISECONDS (17-900 ms of declared item time) while the + replay runs in well under one, so closing that last gap needs the agent's + own clock faked, not the fixtures' rebased. + +- [x] ~~No TypeScript counterpart to CE058.~~ **DONE.** + `evalboard/lib/__tests__/no-zero-coalesce.test.ts` is a vitest source scan + (there is no eslint in `evalboard/`) over `lib/runs.ts`, `lib/timing.ts` and + `_sections.tsx`. It is an ALLOWLIST rather than a ban, exactly because the + residual arithmetic uses `?? 0` correctly — each of its 14 entries carries a + one-line reason, and a new occurrence fails until its author justifies it or + keeps the value null. It is keyed on the codebase's own `…Ms` naming + convention rather than on every `?? 0`: a blanket scan matches 58 + occurrences, ~40 of them token buckets where zero is a fine answer, and an + allowlist that long is one nobody reads. Blind spots are declared in the + file. Two meta-tests keep it honest — a negative control (so the scan cannot + pass by matching nothing) and an assertion that every allowlist entry is + still present, so an entry cannot outlive its reason. + Caught in: turn head/tail timing final review. + +- [x] ~~**`timing.py::decompose_turn` raises an uncaught `TypeError` on a + naive/aware datetime mix**~~ **DONE.** `timing.py::_require_same_awareness` + now raises from five call sites (`decompose_turn`'s head and tail, + `busy_ms`'s window bounds and each span's two ends) with one message template + naming the field and which side is aware. Deliberately a GUARD and not a lint + rule: the invariant is still unviolated in-tree, and the exposure that + actually matters is a third-party agent registered through the + `coder_eval.plugins` SPI, which lives outside `src/coder_eval/agents/` and + which a rule scoped to that directory could never see — so the message + addresses that reader directly. An empty span list is checked not at all, + bounds included: nothing is compared, so there is no pair to be about. + Caught in: turn head/tail timing final review. + +- [x] ~~**`claude-code` does not subtract tool execution from its generation + windows.**~~ **FIXED** in `_ClaudeTurnState._subtract_tool_time_from_windows`, + which runs at finalization (it cannot run at flush time — a tool issued by an + earlier emission is still running when the next window closes). Re-measured + on the same task: 481 ms / 2.691% -> **1.4 ms / 0.006%** over four turns that + all carried overlapping tool calls. Original report kept below for the + reasoning. + + ORIGINAL: The other four + harnesses subtract the union (`timing.py::busy_ms`); claude-code is exempted + on the reasoning that it "marks the end of the previous SDK event and reads + again when the next message arrives, so a tool's execution falls between two + windows rather than inside one". But a tool's timer starts at the **emission** + carrying its `tool_use` block, and one assistant turn spans several emissions, + so a later emission's window runs concurrently with a tool already timing. + Measured live on a task with five parallel writes, five reads and two + concurrent `Bash` calls: the generation/tool overlap was **482 ms and 340 ms** + on two ~18-25 s turns, and the four-bucket residual came out at exactly + `-481 ms` / `-339 ms` — the overlap accounts for it to within 1.4 ms. The + other four harnesses overlapped by ~2.0-2.3 s on the same task and reconciled + to within 1.2 ms. Two claude-code turns with <1 ms of overlap reconciled to + within 0.1 ms, so the fault is precisely the missing subtraction. + Fix is to apply `busy_ms` in `on_assistant_message` as the other four do, but + it changes a PUBLISHED `generation_duration_ms` on the most-used harness, so + it needs its own golden regeneration and live pass. NOT introduced by the + head/tail work — generation-vs-tool timing predates it — but that work's + four-bucket identity is what made it visible. + Caught in: post-merge live verification of the head/tail buckets. + +### Deferred lint-rule widenings + +- [ ] **CE058 and CE059 still match `AssistantMessage` by a hardcoded constructor + NAME LIST** (`_MESSAGE_CONSTRUCTORS`), where CE060 derives the set from each + module's own `coder_eval.models` imports. The weakness is live, not + theoretical: `claude_code_agent.py` binds *only* + `AssistantMessage as AssistantMessageTelemetry` and never the bare name, so + the two shipped rules guard that file's two construction sites purely because + somebody wrote the current alias into a different file's frozenset — rename + the alias and both go silently blind there — and an arbitrary + `AssistantMessage as Msg` is missed outright by both. Adopting CE060's + alias-resolving `check()` pre-pass is about ten lines per rule, but it widens + two SHIPPED rules whose firing sets are load-bearing (CE058's constructor set + is a different, wider one: `CommandTelemetry`, `SlowestCommandInfo`, + `TurnRecord`), so it needs its own mutation check per rule and a re-measured + firing set over all of `src/`, not a drive-by edit. If a fourth same-scope + kwarg rule ever lands, extract `tests/lint/rules/_message_calls.py` at that + point rather than sooner. + Caught in: the CE060 / antigravity `message_id` run. + +- [ ] **Nothing pins that `message_id` is only ever a WITHIN-TURN identity.** Ids + repeat across retry attempts of one turn on every synthetic-id harness — + `Agent.discard_pending_turn` rolls the iteration counter back, so a crashed + partial and its retry both emit `-1-msg-0` (antigravity, codex, and + the out-of-tree delegate agent alike). Harmless today, and verified so: the + evalboard declares its grouping list INSIDE the per-turn loop + (`runs.ts:1822`, flushed at `:2217`) and only ever compares adjacent raws, and + no Python consumer reads the field at all. It stops being harmless the moment + anything joins on the id run-wide (a React key across turns, a cost join, a + dedup) — which is a natural thing to reach for once every harness populates + it. No cheap guard exists: the property to assert is "no consumer treats this + as run-unique", which is a negative over two languages, and asserting + within-turn uniqueness instead would pass today and catch nothing. Cheapest + real option is a comment on the model field; the durable one is a run-level + id if a consumer ever needs one. + Caught in: the CE060 / antigravity `message_id` final review. + +- [ ] **No evalboard test is fed by a Python golden snapshot.** The two halves of + a capture fix are pinned by two hand-written fixtures that never meet: the + golden (`tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json`) + pins what the reducer emits, and `evalboard/lib/__tests__/parseMessages.test.ts` + pins what the consumer does with a fixture an author typed from the same + understanding. Nothing feeds a real recorded shape through `parseMessages`, so + a reducer change that makes the TS fixture unrepresentative breaks no test on + either side. Deferred as architectural: it needs a loader, a scrub-aware + timestamp story (the goldens mask exactly the stamps the grouping reads), and + a convention for which snapshots the JS suite owns — well over 30 min, and + wider than any one capture fix. + Caught in: the CE060 / antigravity `message_id` final review. + +- [x] ~~**`AssistantMessage.message_id`'s field description names one harness of + five**~~ **DONE.** It said "Anthropic API message_id … when the Claude Code + CLI splits one API response", while five backends write the field and four + synthesize it — so `docs/agents/HARNESS_PARITY.md`'s row was the real SSOT + and the model, which this project's DRY principle designates as + authoritative, described claude-code only. Rewritten agent-agnostically: what + the id MEANS (the generation an emission belongs to), that all five write it + and four synthesize it, each scheme named, a pointer to the per-harness row, + and the fact that it is a WITHIN-TURN identity that repeats across retry + attempts. No mechanical guard was added and none is obvious — "a field + description must not name a single harness when the union has five writers" + needs a writer census per field, which is CE054-shaped but over a `str` + description rather than a key; the cheap version was exactly this, fixing the + sentence in the next change that touches the model. + Caught in: the CE060 / antigravity `message_id` final review. + +- [ ] **The golden corpus pins that a timing value EXISTS, never what it is.** + `tests/_fixtures/golden_streams/_scrub.py::SCRUB_KEYS` masks + `generation_duration_ms`, `started_at`, `completed_at` and both + `execution_*_at` to a placeholder, and the one assertion that does look at + magnitudes (`assert_timing_captured`'s four-bucket check) is an UPPER BOUND — + it catches a bucket claiming more time than the turn contains and says + nothing about one claiming less. So the committed suite cannot see a + per-harness generation number move at all, in either direction. Not + hypothetical: a whole phase of the timing plan was written on the premise + that changing those numbers would turn the golden master red, and it never + did. The two-sided check exists (`scripts/timing/decompose_run.py + --max-residual-pct`) but runs only against live `task.json` files, by hand. + Not cheap to guard: porting the two-sided residual into `_scrub.py` means + deciding a per-scenario tolerance for replays whose real wall clock is under + a millisecond while their SDK stamps declare hundreds — the same problem + `FICTIONAL_DURATIONS` already exempts six scenarios from, so the honest + version needs those scenarios to fake the agent's own clock too, not just + their item stamps. Interim cover is the per-reducer ms-exact identity test + added on pi and opencode + (`test_the_four_bucket_identity_closes_exactly_across_the_boundary`). + Caught in: the timing-architecture-standardization final review. + +## From the turn-timing P0–P3 run (2026-09-12) + +- [ ] **A golden scenario's justification comment can contradict its own + snapshot, and nothing notices.** Three did in this run: two orphan-tool + comments asserted bounds the committed JSON plainly carries (`pi_d`, + `opencode_d`), and `opencode_c`'s exemption claimed "the snapshot still + records the tiling" while `SCRUB_KEYS` masks both bounds and the duration. + Each was found by a human/model reading the JSON beside the prose — nothing + mechanically ties an exemption's stated reason to what its snapshot contains. + A rule would have to parse prose, so this is probably not guardable; the cheap + substitute is the review instruction that already exists ("read every new + snapshot before committing") plus the habit of quoting the actual JSON in the + comment. Caught in: turn-timing P0–P3, phases 2 and 5. + +- [ ] **A rationale comment asserting a now-false premise survives a ripple that + updated its siblings.** The "in-process SDK" claim was corrected in six files + and left standing in two (`test_event_collector.py`, + `message-timeline.test.tsx`), one of them directly beside a sibling that WAS + updated. Same shape as CE026/CE047 (doc-surface parity) but over a PHRASE + rather than a symbol, so a rule would be a phrase blocklist with an + ever-growing allowlist. Deferred on cost, not on value — a grep for the retired + phrase in the acceptance criteria is what actually caught these, and that is + cheap to write into a plan. + +- [ ] **`EventCollector` retains `_commands` and `_turn_starts` across a retry's + `AgentStartEvent`**, which resets only `_agent_end`. Pre-existing and NOT + introduced by the timing work. Blast radius is narrower than it first looks: + the persisted record, the reports and `max_turns` all read the AGENT's + collector, which is fresh per `communicate()`. Only `EarlyStopWatcher`'s + long-lived collector accumulates — where carrying a turn's whole engagement + across retry attempts is arguably what a live "did it engage the skill" + verdict wants, and `_check_round`'s docstring already reasons about crashed + attempts. Needs a decision on intent before any guard. Caught in: turn-timing + P0–P3 final review. + +- [ ] **claude-code has no `TurnClock`.** Its window bounds and span now share + one basis (raw `datetime.now()`), so they cannot disagree with each other — + but both carry the naive-local exposure `TurnClock` exists to remove: a DST + transition or NTP step inside a turn lands directly in a generation window, + and nightly runs are hours long. antigravity and pi already derive wall stamps + from monotonic; codex and opencode cannot (their spans are the CLI's epoch + stamps). claude-code is the one that could and does not. Caught in: turn-timing + P0–P3, phase 5. + +- [ ] **`pi_agent` publishes a `duration_ms` and a subtracted tool SPAN for an + UNRESOLVED orphan.** `_close_tool` guards on `execution_started_at is not + None` while its own comment claims it guards on "resolved", and the + `execution_completed_at` it stamps is only the instant the orphan sweep ran. + claude-code's `_finalize_commands` deliberately leaves the field `None` here, + for the reason CE058 exists. Captured in `pi_d_orphaned_tool.json`. Caught in: + turn-timing P0–P3, phase 2. + +- [ ] **`pi_agent` republishes a turn's content on a duplicate `turn_end`.** + `turn_text_parts` / `turn_tool_ids` are cleared only in `on_turn_start`, so a + second `turn_end` with no intervening start emits the previous turn's text as + its own assistant message and re-lists the same `tool_use_ids`. The TIMING + half of that same reset was deliberately fixed (`turn_started_at` moved into + `on_turn_end`, with a comment making exactly this argument); the content half + was not. Pi retries internally, so a replayed `turn_end` is a transport hiccup + rather than a hypothetical. Captured in `pi_f_duplicate_turn_end.json`. + Caught in: turn-timing P0–P3, phase 2. diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 58c450cd..ab8a1979 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -589,6 +589,21 @@ jobs: test "$FAILED" = "0" || { echo "smoke-pass had unexpected failures"; exit 1; } test "$ERRORED" = "0" || { echo "smoke-pass had errors"; exit 1; } + # The four wall-clock buckets (head + generation + UNION(tool) + tail) + # must account for each turn's own duration. This is the TWO-SIDED gate: + # the committed golden sensor only catches an OVERSHOOT, so a bucket that + # claims LESS time than it should — the defect class this area keeps + # producing — passes every test in the suite. It needs live task.json + # files, which the smoke-pass run above already leaves on disk. + # + # COVERS CLAUDE-CODE ONLY: experiments/default.yaml sets type: claude-code, + # so every turn here is that harness. The other four are covered by + # tests/test_timing_identity_contract.py, which is ms-exact but synthetic. + - name: Verify timing residual (claude-code only) + run: | + .venv/bin/python scripts/timing/decompose_run.py \ + $(find runs/ci-smoke-pass -name task.json) --max-residual-pct 5 + - name: Verify smoke-fail bucket run: | F=runs/ci-smoke-fail/experiment.json diff --git a/CLAUDE.md b/CLAUDE.md index 616945b8..cabde3f6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -236,7 +236,7 @@ make plugin-reference # the plugin's bundled criteria reference from the models Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction. -Recent additions, each traceable to a shipped defect: **CE058** (in `src/coder_eval/`, an unknown timing value may not become a numeric literal — `duration_ms is None` means *never timed* and `0.0` means *timed and instant*, so writing the literal publishes the second while meaning the first. One invariant, one id, five syntactic forms — a zero constructor keyword, `x or 0`, `x if x is not None else 0.0`, `if x.duration_ms is None: x.duration_ms = 0.0`, and a `model_copy(update={...})` dict (the shape the Antigravity DONE path writes through, which a keyword-only rule cannot see). Antigravity constructed EVERY message with `generation_duration_ms=0.0`, so the task page's Generation cell read `0ms` and its breakdown rendered `0%` for months with nothing failing; Codex published the SDK's `0.0` as a measured command duration, so `avg_command_time_ms` divided real milliseconds by a command count of which 70 of 211 in one nightly had never been timed. The fourth form is the one no existing rule shape covered and is where a live instance was hiding — `claude_code_agent._finalize_commands` set `0.0` on every command force-closed without a tool result, in the one harness a timing audit had called healthy. BLIND SPOT, stated in the rule's docstring: form 1 keys on the callee's spelling, so renaming the `AssistantMessageTelemetry` import alias silently disarms it there), **CE059** (in `src/coder_eval/agents/`, an `AssistantMessage` may not receive the same `ast.Name` for both `started_at` and `completed_at` — the Antigravity reducer read `datetime.now()` once and passed it as both bounds, so `started_at == completed_at` on 368 of 368 sampled messages. A separate id from CE058 because it is a separate invariant, a zero-length window whatever the duration field says, and one invariant per id is what makes a `# noqa` mean one thing. It does NOT fire when the same call passes `generation_duration_ms=None`: a call that says, in the field built to say it, that no window was measurable is not claiming one — that exemption is what keeps the rule pointed at the misleading case instead of accumulating four permanent suppressions on the rollout-rebuild and sub-agent-synthesis sites), **CE056** (no bare `CODER_EVAL_IN_CONTAINER` literal outside `models/container_paths.py` — the CE053 shape again: a rename-safety constant that shipped beside the literal it replaced, and the straggler was the single WRITER, so a rename would have disarmed four security/correctness gates at once with nothing failing; CE052 cannot catch it because that rule inspects `if` guards and the writer is not one), **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record or run-LOG filename literal outside `path_utils` — widened to `docker.log` / `grade.docker.log` / `task.log` / `grade.log` after the same shape recurred: `docker.log` was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, and because the consumer guards its copy with `is_file()`, a rename would have silently discarded the only record of why a grading container failed — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE057** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass). +Recent additions, each traceable to a shipped defect: **CE063** (no module in `src/coder_eval/agents/` may import `busy_ms` — tool execution comes out of a generation window in exactly ONE place, `streaming/collector.py::subtract_tool_time`. Five reducers used to do it themselves while the head and tail were already computed centrally at the same seam, and that asymmetry is where every timing defect on this branch lived — none of them in the arithmetic, all of them in the bookkeeping AROUND it: when to reset a per-step span list (clearing it at `step_start` wiped a span before the flush could subtract it, a 100% overstatement of that window), when to clear a spent start stamp (a second flush with no intervening start republished the previous span — 3000 ms of generation for a 2000 ms turn), when to advance the mark. A sixth harness reaching for `busy_ms` rebuilds that, and its tool time is then subtracted TWICE — by the reducer and again by the collector — under-reporting generation on one harness only, which takes a corpus comparison to notice. A separate id from CE061 rather than a rebody: CE061 asks where a window's ARITHMETIC came from and four reducers still call `close_window`, so its property is live and unsuperseded; this asks whether a reducer subtracts at all. It deliberately does NOT reuse CE061's `_imports_the_helper`, whose bare-module-import branch exists so `timing.close_window(...)` counts as reaching the helper — inverted into a ban that branch flags four of the five reducers. CE061 is now **exemption-free**: claude-code was its one permanent `# noqa` and, with the subtraction moved, calls the shrunken `close_window` like the other four), **CE060** (in `src/coder_eval/agents/`, every `AssistantMessage(...)` must pass `message_id` explicitly — an identity invariant, which is why it is its own id rather than a second arm of CE058/CE059, both of which are about timing. Antigravity omitted the kwarg, so the field defaulted to `None` on every message it ever recorded, and the evalboard — which groups assistant emissions by `message_id` and falls back to a `SAME_EMISSION_GAP_MS` wall-clock gap when either side lacks one — collapsed a whole turn's generations into ONE timeline row as soon as the harness's generation windows became contiguous (the gap is then exactly 0 ms, always). Nothing failed: the consumer SUMS the group, so the totals and the reconciliation invariant stayed right, and the golden snapshots had ratified the `null` on the day they were written — a snapshot is regenerated from whatever the code currently does, so it catches a later change and never an initial omission. The damage was not confined to the timeline, which is why "only granularity is lost" was the wrong way to describe it: a grouped emission is one API call to the evalboard's thinking-cost simulator, whose prompt-cache cascade is quadratic in that count, so a single-shot Antigravity run had every cascade coefficient pinned at zero; the `Messages` count and the 10 s slow-generation bar were per-turn too. Unlike its two siblings it **derives its constructor set from each module's own `coder_eval.models` imports** instead of hardcoding the spelling, which closes exactly the blind spot CE058's clause below concedes: `claude_code_agent.py` binds only `AssistantMessage as AssistantMessageTelemetry`, so a name list guards that file's two construction sites purely by coincidence, and an arbitrary `as Msg` is missed outright. Widening CE058/CE059 the same way is recorded in `.claude/harness-candidates.md`. BLIND SPOT, in the rule's docstring: the runtime `None` — the kwarg must be PRESENT, not statically non-`None`, because OpenCode's `messageID` and Pi's `responseId` legitimately evaluate to `None` when the CLI omits them, and passing a fallback expression *is* deciding), **CE058** (in `src/coder_eval/`, an unknown timing value may not become a numeric literal — `duration_ms is None` means *never timed* and `0.0` means *timed and instant*, so writing the literal publishes the second while meaning the first. One invariant, one id, five syntactic forms — a zero constructor keyword, `x or 0`, `x if x is not None else 0.0`, `if x.duration_ms is None: x.duration_ms = 0.0`, and a `model_copy(update={...})` dict (the shape the Antigravity DONE path writes through, which a keyword-only rule cannot see). Antigravity constructed EVERY message with `generation_duration_ms=0.0`, so the task page's Generation cell read `0ms` and its breakdown rendered `0%` for months with nothing failing; Codex published the SDK's `0.0` as a measured command duration, so `avg_command_time_ms` divided real milliseconds by a command count of which 70 of 211 in one nightly had never been timed. The fourth form is the one no existing rule shape covered and is where a live instance was hiding — `claude_code_agent._finalize_commands` set `0.0` on every command force-closed without a tool result, in the one harness a timing audit had called healthy. BLIND SPOT, stated in the rule's docstring: form 1 keys on the callee's spelling, so renaming the `AssistantMessageTelemetry` import alias silently disarms it there), **CE059** (in `src/coder_eval/agents/`, an `AssistantMessage` may not receive the same `ast.Name` for both `started_at` and `completed_at` — the Antigravity reducer read `datetime.now()` once and passed it as both bounds, so `started_at == completed_at` on 368 of 368 sampled messages. A separate id from CE058 because it is a separate invariant, a zero-length window whatever the duration field says, and one invariant per id is what makes a `# noqa` mean one thing. It does NOT fire when the same call passes `generation_duration_ms=None`: a call that says, in the field built to say it, that no window was measurable is not claiming one — that exemption is what keeps the rule pointed at the misleading case instead of accumulating four permanent suppressions on the rollout-rebuild and sub-agent-synthesis sites), **CE056** (no bare `CODER_EVAL_IN_CONTAINER` literal outside `models/container_paths.py` — the CE053 shape again: a rename-safety constant that shipped beside the literal it replaced, and the straggler was the single WRITER, so a rename would have disarmed four security/correctness gates at once with nothing failing; CE052 cannot catch it because that rule inspects `if` guards and the writer is not one), **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record or run-LOG filename literal outside `path_utils` — widened to `docker.log` / `grade.docker.log` / `task.log` / `grade.log` after the same shape recurred: `docker.log` was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, and because the consumer guards its copy with `is_file()`, a rename would have silently discarded the only record of why a grading container failed — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE057** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass). When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps..outputs.` / `needs..outputs.` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.) diff --git a/docs/agents/HARNESS_PARITY.md b/docs/agents/HARNESS_PARITY.md index 4b429032..84064a1c 100644 --- a/docs/agents/HARNESS_PARITY.md +++ b/docs/agents/HARNESS_PARITY.md @@ -24,34 +24,256 @@ wall clock its numbers account for. | Field | claude-code | codex | antigravity | opencode | pi | |---|---|---|---|---|---| -| `generation_duration_ms` source | harness clock: previous SDK event → this message | SDK item stamps, minus tool execution inside the window | harness clock: previous flush → this flush, minus tool execution inside the window | harness clock per CLI step, minus tool execution inside the step | harness clock per CLI turn, minus tool execution inside the turn | +| `generation_duration_ms` RAW window (the reducer's part) | harness clock: previous SDK event → this message | SDK item stamps | harness clock: previous flush → this flush | harness clock: previous `step_finish` → this one | harness clock: previous `turn_end` → this one | +| tool time subtracted from it | centrally | centrally | centrally | centrally | centrally | +| what the **first** window covers | the first `message_start`, so CLI boot + TTFT are OUTSIDE it | the first SDK item's own start, so CLI boot + TTFT are OUTSIDE it | the first MODEL-source `Step`, so dispatch + TTFT are OUTSIDE it | the first `step_start`, so CLI boot + TTFT are OUTSIDE it | the first `turn_start`, so CLI boot + TTFT are OUTSIDE it | +| `harness_startup_ms` (turn head) | ~3.6 s — CLI boot fused with TTFT | ~3.1 s — CLI boot fused with TTFT | ~4.7 s — dispatch fused with TTFT (its harness process is spawned once at startup, not per turn) | ~2.5 s — CLI boot fused with TTFT | ~0.23 s — CLI boot fused with TTFT | +| `harness_teardown_ms` (turn tail) | ~1.3 s | ~13 ms | ~7 ms | ~26 ms | ~19 ms | | tool `duration_ms` source | measured around the tool result | SDK `completed_at_ms − started_at_ms`; the item's own `duration_ms` only as a fallback | measured ACTIVE → DONE | measured around the tool event | measured around the tool event | | `execution_started_at` / `execution_completed_at` | derived from the measured duration | SDK stamps (both, or neither) | measured at ACTIVE / DONE | measured | measured | | `generation_completed_at` | set | `None` — see below | `None` | `None` | `None` | -| `Σ generation + Σ tool ≈ turn duration` | yes | yes | yes | yes | yes | +| `message_id` source | SDK `message_id`; `None` when the stream carries none; `subagent-` for a synthesized sub-agent terminal | synthetic `turn_id-msg-N`, shared across the sub-messages of one generation; `turn_id-subagent-N` for recovered sub-agent generations | synthetic `turn_id-msg-N`, one per generation | CLI `messageID`; `None` when absent | CLI `responseId`; `None` when absent | +| `Σ generation + ∪ tool + head + tail ≈ turn duration` | yes [^identity] | yes [^identity] | yes [^identity] | yes [^identity] | yes [^identity] | +| clock basis for recorded stamps | wall bounds, wall duration (raw `datetime.now()`) | SDK epoch ms — the subprocess's own clock, unreachable from the host | one `TurnClock` per turn | CLI epoch ms (`_epoch_ms_to_dt`), `datetime.now()` only as a fallback | one `TurnClock` per turn | +| window built by `timing.py::close_window` | yes | yes | yes | yes | yes | + +[^identity]: "yes" is load-bearing, and THREE sensors check it, each seeing +something the others cannot. + +`tests/test_timing_identity_contract.py` is the committed two-sided one: it +drives every built-in reducer off a scripted clock, through a real +`EventCollector`, and asserts the four buckets tile the turn to the +MILLISECOND. Magnitudes are only real where a scripted clock makes them real, +which is why it is not in the golden corpus. + +`tests/_fixtures/golden_streams/_scrub.py` replays recorded streams but asserts +only `overshoot <= …` — it catches a bucket claiming MORE time than the turn +contains and says nothing about one claiming less. It cannot be made two-sided +either: those replays run in ~0.3 ms of synthetic wall clock, where a relative +bound is vacuous. Nor can it see magnitudes at all — `SCRUB_KEYS` masks +`generation_duration_ms` and both bounds to a placeholder, so a snapshot records +that a window was measured, never what it measured. That is not a gap to close; +it is why the contract test exists. + +`scripts/timing/decompose_run.py --max-residual-pct N` is the two-sided check on +LIVE runs, gating each turn's `|residual|` as a share of its own wall clock. +`.github/workflows/pr-checks.yml` runs it over the `smoke-pass` bucket's real +`task.json` files, which covers claude-code only (`experiments/default.yaml` +sets that type); run it by hand for the others. **`generation_duration_ms` is model-generation time, not `completed_at − started_at`.** -Four of the five harnesses interleave tool execution into a single generation -window. Antigravity reports a `Step` for the tool and only a later +All five harnesses can have tool execution inside a generation window, and it is +subtracted out of every one of them — **once, centrally**, by +`streaming/collector.py::subtract_tool_time`. No reducer does it itself; each +publishes the raw window (see the two sections below). Every harness has the +problem: Antigravity reports a `Step` for the tool and only a later `usage_metadata` `Step` cuts the message; Codex's message window is seeded from -the first item's start and extended to the last item's completion; OpenCode -opens its window at `step_start` and closes it at `step_finish`, and Pi at -`turn_start` / `turn_end`, with every tool call running inside. In all four the +the first item's start and extended to the last item's completion; OpenCode, Pi +and claude-code tile, each window opening where the previous one closed and +running to the next, with every tool call in between running inside. In each the span between the recorded bounds legitimately CONTAINS tool time that the model -did not spend generating, so all four subtract it — the **union** of the closed tool intervals -clipped to the window (`agents/_timing.py::busy_ms`), never the sum, because -tool calls overlap: Antigravity resolves several from one `Step` and backgrounds -anything over ten seconds, and Codex spawns collab agents concurrently. Summing -them over-subtracts by exactly the overlap and, with enough concurrency, drives -the result to a clamped zero. +did not spend generating. What comes out is the **union** of the resolved +main-thread tool intervals clipped to the window +(`coder_eval/timing.py::busy_ms`), never the sum, because tool calls overlap: +Antigravity resolves several from one `Step` and backgrounds anything over ten +seconds, and Codex spawns collab agents concurrently. Summing them +over-subtracts by exactly the overlap and, with enough concurrency, drives the +result to a clamped zero. The consequence worth knowing: on an emission that carries *only* a tool call, the whole measured window was that tool running, so the recorded generation time is legitimately `0.0`. That is a measurement, not a placeholder — `None` -is what "never measured" looks like. Only `claude-code` does not need the -subtraction: it marks the end of the previous SDK event and reads again when -the next message arrives, so a tool's execution falls between two windows -rather than inside one. +is what "never measured" looks like. + +**One helper opens all five windows, and the subtraction is not in it.** Every +reducer calls `coder_eval/timing.py::close_window`, which is now only the +window's own geometry: tile from the mark, keep a stamp that went backwards +from inverting the span, clamp at zero. It had been copy-pasted four times, and +Pi shipped a variant that measured from its own turn start — so every +inter-turn gap fell into no bucket, and nothing failed, because the identity +above is asserted on one side only. **CE061** requires any module in `agents/` +publishing a measured `generation_duration_ms` to import the helper, and is now +**exemption-free**: claude-code was its one permanent `# noqa` and no longer +needs it. + +**Tool execution comes out of the windows ONCE, at the collector.** +`streaming/collector.py::subtract_tool_time` takes the union of the main-thread +tool intervals, clipped to each window, out of the raw spans the reducers +publish. Before, that happened five times in five places — four inside +`close_window` as the reducer flushed, claude-code once at finalization — while +the head and the tail were already computed centrally at the same seam. That +asymmetry was the complexity, and every timing defect on this branch lived in +the per-reducer bookkeeping around the subtraction rather than in the +subtraction: when to reset a span list (clearing it at `step_start` wiped a span +before the flush could subtract it — a 100% overstatement of that window), when +to clear a spent start stamp (a second flush with no intervening start +republished the previous span — 3000 ms of generation for a 2000 ms turn), when +to advance the mark. Those three lists, their reset rules, and the bounding of +still-open calls are all deleted. **CE063** stops a sixth harness rebuilding +them: no module in `agents/` may import `busy_ms`. + +Two consequences worth stating, because both are behaviour changes: + +- **A call still open when a window closes is no longer subtracted at that + boundary.** The reducer used to bound it at the window's end and take that + slice. The collector sees every span at once, so the call is subtracted from + the windows its REAL interval overlaps, once it resolves — no approximation. + A call that never resolves has no `execution_completed_at`, contributes + nothing, and says so. +- **Codex's two sub-messages are one group.** They share a pair of bounds and + divide the window by output-token share; the collector groups on the bounds + (not on `message_id`, which OpenCode and Pi can legitimately leave `None`), + subtracts the overlap once, and re-apportions so the parts still sum. + +**Three clock bases remain, and the row above says which.** Antigravity and Pi +derive every recorded wall stamp from one `TurnClock` per turn, so a turn's +bounds and the tool spans subtracted from them cannot disagree. Antigravity +needed it: its span was monotonic while its tool intervals were wall, which is +the only reason its window could go negative, and the clamp that caught it was +indistinguishable from a real instant generation. Pi needed it for a different +reason — its stamps were naive-local, so a DST transition or an NTP step inside +a turn lands directly in a generation window. + +Codex and OpenCode are **not** converted and the hazard is narrowed rather than +removed. Their tool spans are the CLI's own epoch-millisecond stamps +(`codex_agent.py::_ms_to_dt`, `opencode_agent.py::_epoch_ms_to_dt`), which +cannot be re-derived host-side; converting only the window bounds would put two +bases inside one `busy_ms` subtraction, relocating the defect instead of +removing it. Both therefore keep the naive-local exposure. + +claude-code is the third case and the newest. Its window duration used to be a +monotonic delta while its bounds were wall stamps — the split `TurnClock` +exists to remove — and central subtraction made that untenable, because it +clips WALL tool spans against those WALL bounds. It now measures the span from +the bounds, so the two agree; but the bounds are still raw `datetime.now()`, +so it keeps the same naive-local exposure as codex and opencode, for a +different reason: no epoch-stamp constraint, it simply has not been converted. +That conversion is the remaining improvement here and is not done. + +Deadlines on every harness stay on raw `time.monotonic()` and must — a deadline +may not move when the wall clock steps. + +**HISTORY — why claude-code needed a special case at all.** It was once exempt +from subtracting entirely, on the premise that because it marks the end of the +previous SDK event and reads again when the next message arrives, a tool's +execution falls *between* two windows rather than inside one. Measured, that +premise does not hold: a tool's timer starts at the **emission** carrying its +`tool_use` block, and one assistant turn spans several emissions, so a later +emission's window runs concurrently with a tool already timing. On a task +issuing five parallel writes, five reads and two concurrent `Bash` calls the +overlap was 482 ms and 340 ms on two ~18-25 s turns, and the four-bucket +residual came out at exactly `-481 ms` and `-339 ms`. (That run is pinned at +`tests/_fixtures/timing_runs/claude-code.json`, which still reconciles at +-481 ms — it is a RECORD of the defect, not of current behaviour; see the README +there.) + +It could not subtract while flushing, because a tool issued by an earlier +emission is still running when the next window closes and its interval does not +exist yet — so it subtracted once at finalization instead, in a method of its +own. Central subtraction dissolves the special case: the collector is *already* +the place where every span is known, so claude-code needs no separate pass and +no exemption. + +Its window is also now measured on ONE clock. The duration used to be a +monotonic delta while the bounds were wall stamps, which is exactly the split +`TurnClock` exists to eliminate — and it became load-bearing with central +subtraction, which clips WALL tool spans against those WALL bounds. A +monotonic-measured duration would have had the two disagreeing inside one +subtraction, which is the defect that let Antigravity's window go negative. +`turn_start_time` stays monotonic and is untouched: `duration_seconds` and the +turn deadline read it, and a deadline must not move when the wall clock steps. +Adopting a full `TurnClock` here (deriving the wall stamps from monotonic, as +antigravity and pi do) is the remaining improvement and is not done. + +**The head and tail are measured, not normalized.** Generation and tool are +only two of the four buckets. The turn's **head** (turn start → first +generation window) and **tail** (last window → turn end) are booked as +`TurnRecord.harness_startup_ms` / `harness_teardown_ms`, computed once at the +`EventCollector` seam by `coder_eval/timing.py::decompose_turn`. The tool term +is the **union** of the command intervals, for the same reason the subtraction +above is — Pi resolved a `Write` and a `Bash` overlapping by 18.4 ms in one +measured turn, and summing their durations books that overlap twice. The head +and tail exclude tool execution by that same rule and that same helper, which +is what keeps the four buckets disjoint: a tool is not confined to a +generation window (Antigravity force-closes an orphan at finalization, inside +the tail, and backgrounds anything over ten seconds), so a span that escapes +one would otherwise be counted both as tool and as head or tail. With all +four buckets and the union, six live turns per harness reconcile to within +1.7 ms of `duration_seconds` (worst case 0.014% of wall clock; the residual is +clock skew, since head and tail are measured between wall-clock event stamps +while `duration_seconds` is the agent's own monotonic span, and its sign flips +between harnesses). `scripts/timing/decompose_run.py` reproduces the table. The head and tail +figures in the table above are means of six live `tasks/hello_date` turns per +harness and move with CLI cache warmth, so read their ORDER OF MAGNITUDE, not +the digits. + +**The head means one thing on all five.** It is the wall clock from the turn +starting until the harness first observed **model output**, and that instant is +also where the harness opens its first generation window — which is what keeps +the head and the generation disjoint so the four-bucket identity still closes. +The per-harness first-output signal: + +| harness | first observed model output | +|---|---| +| claude-code | the first `message_start` stream event | +| codex | the first SDK item's own start | +| antigravity | the first MODEL-source `Step` (a SYSTEM/USER Step does not seed) | +| opencode | the first `step_start` | +| pi | the first `turn_start` | + +What the head CONTAINS still differs, and that part is deliberately **not** +decomposed. **All five spawn a process** — the distinction is WHEN. claude-code, +codex, opencode and pi spawn theirs per turn, so their head fuses that boot with +provider resolution, dispatch and TTFT, and the stream carries no marker between +them (measured on OpenCode: the process spawns in ~3 ms and its first +`step_start` lands at ~3.9 s). Antigravity spawns its bundled `localharness` +binary ONCE, in `start()`, and holds it across every `communicate()` — so there +is no boot inside the turn for its head to contain, and its head is dispatch plus +TTFT. That is a real property of the harness rather than a measurement artifact, +which is as far as unification can honestly go. + +So the fields are named for the **interval they measure**, never for what they +contain. Do not rename them `cli_boot_ms` or `ttft_ms` — that would claim a +split nobody performed. A measured `0.0` head is an answer; `None` is what +"never measured" looks like (a turn that produced no assistant message). + +**The table's head figures are SINGLE-TURN.** They are means of six live +`tasks/hello_date` turns. A simulation (dialog) task runs each turn as its own +`communicate()`, so on the per-turn-spawn harnesses turns 2..N book a full +process boot *plus* session-transcript replay into `harness_startup_ms`, and +will read well above these numbers. That is correct under the definition and is +an improvement — the same time was previously hidden inside the first +generation — but do not read a dialog run's larger head as a regression against +this table. + +**HISTORY — why claude-code and antigravity used to report `0.0`.** Both +stamped their first window's mark when the turn state was built, *before* +`AgentStartEvent` was emitted, so the head was a small negative that +`decompose_turn` clamped. The `0.0` was therefore a clamped inversion published +as "measured, and instant" — the exact confusion CE058 exists to prevent +everywhere else — and everything those harnesses spent before their first model +output was booked as the first generation instead: **~3.6 s per turn on +claude-code and ~4.7 s on antigravity**, inflating every generation figure, the +Generation split percentages and the 10 s slow-generation bar on the two +most-used harnesses. + +The re-seed was rejected once, on the premise that claude-code runs the model +in-process so "the interval from turn entry to the first message is msg0's +generation". That premise was simply wrong: `claude-agent-sdk` spawns the +`claude` CLI as a subprocess (`anyio.open_process`) and `_pump_messages` calls +`query()` once per `communicate()` — a fresh CLI per turn, the same shape as +codex, opencode and pi. Nor was antigravity ever the in-process counterexample +it was described as: it spawns `localharness` too, just once at `start()` +rather than per turn. + +Both re-seeds are **once per turn**. `message_start` and `Step` each arrive +many times; re-seeding on every one would stop the windows tiling and drop the +gap before the next emission — a tool result landing, then the next request +going out — into no bucket at all, which is the defect Pi shipped with. Neither +flag needs a reset: both harnesses build a fresh turn state per +`communicate()`, so it is per-attempt by construction. A turn that streams no +`message_start` / no `Step` never re-seeds, keeps the turn-entry mark and +clamps to `0.0` exactly as before. **Why Codex leaves `generation_completed_at` as `None`.** It means "when the model finished emitting the `tool_use` block". Codex's stream does not carry @@ -69,6 +291,94 @@ generic tool items now carry a duration where they previously carried none, so `avg_command_time_ms` and `total_command_time_ms` for a Codex run describe every tool call rather than shell commands alone. +**`message_id` is what splits the timeline.** The evalboard groups assistant +emissions by `message_id`, and falls back to a wall-clock gap threshold +(`SAME_EMISSION_GAP_MS`, 100 ms, in `evalboard/lib/runs.ts`) when either side +lacks one. Antigravity's `Step` stream carries no message id, so the harness +synthesizes one — and it must, because this harness's generation windows are +*contiguous* by construction: each opens exactly where the previous one closed, +so the gap between two of them is always 0 ms and the fallback would fold a +whole turn's generations into a single row. CE060 makes the kwarg mandatory in +`src/coder_eval/agents/` for that reason. + +The collapse is a *display* defect, not an accounting one — the consumer SUMS a +group's token buckets and durations, so every total, percentage and cost is +identical either way, as is the reconciliation residual. But it is not +cosmetic, and three displayed figures do move when a turn stops collapsing: +the thinking-cost simulator's per-call cache cascade (`calls` in +`evalboard/lib/thinkingSim.ts` is the number of grouped emissions, and the +cascade is quadratic in it — on a single-shot run it was pinned at one call, +so every coefficient was zero), the `Messages` count and timeline heading, and +the "slow generation" count, whose 10 s bar was being applied to a whole turn's +summed generation time. All three move toward the figure they were always +meant to report, so the fix corrects them rather than breaking them — but a +trend compared across this change is not comparing like with like. + +The two synthetic schemes read differently on purpose: Codex deliberately REPEATS one +id across the sub-messages of a single generation — that is exactly the "the +CLI split one API response" signal the field exists to carry — while +Antigravity's are all distinct, because it emits one message per generation +with every block inside it. Runs recorded before a harness captured the field +still carry `null` and still depend on the gap fallback, which is why it stays +— and so does a current OpenCode or Pi message whose payload omitted the id, +which is the case CE060 cannot see (it requires the kwarg to be present, not +non-`None` at runtime). OpenCode tiles its windows contiguously too, so it is +the other harness where a missing id can still collapse a turn. + +### Time to first token is not measured + +Nothing records it **as its own field** today — there is no `ttft` or +`first_token` symbol anywhere in `src/`, `evalboard/`, `docs/` or `tests/`. + +But most of its value for the TURN is already delivered: `harness_startup_ms` +now measures the wall clock up to the harness's first observed model output on +every harness, which is a time-to-first-output latency for the first generation. +Two things a separate `first_delta_latency_ms` would still add — and the design +below is about both, so do not read this paragraph as retiring it: + +1. **Per-generation latency**, not just the first. The design measures from + EVERY window's mark, so it reports a first-delta latency for each emission; + the head covers only the interval before the first one. +2. **The boot/prefill split** inside the head on the per-turn-spawn harnesses — + which is the part that genuinely cannot be derived, because no stream carries + a marker between them. + +This section is the design, so the next person to want it does not re-derive it. +Nothing below is implemented. + +**The mark is the measure-from point, and every reducer already keeps one.** +Each one records the moment its current generation window opened — which is +exactly what a latency is measured from. Read the current attribute off `src/` +rather than trusting a table here; the last note that transcribed those names +went stale in precisely that way. + +**The first-delta signal already exists in every reducer.** claude-code has raw +`content_block_delta` (already delivered — `include_partial_messages=True`), +codex `item/agentMessage/delta`, antigravity `step.content_delta`, OpenCode the +text part event, Pi `text_delta`. + +Four rules, each of which changes what gets built: + +- **Name it `first_delta_latency_ms`, never `ttft_ms`.** Four harnesses' windows + tile, so the mark is the *previous step's close* and the interval fuses + queueing and tool time. That is queue latency, not prefill latency. Only + claude-code's `message_start` sits near "the request went out". This is the + same rule the head and tail already follow: a field is named for the interval + it MEASURES, never for what it contains. +- **It is never a fifth bucket.** It is a sub-interval of head + first window. + Adding it to the four-bucket identity breaks the disjointness the whole + design rests on. Report it beside the identity, never inside it. +- **Take the first delta of ANY kind**, not the first visible-text delta. The + codex, OpenCode and Pi handlers ignore thinking deltas, so a reasoning-heavy + turn would report its first token late by the entire thinking phase. +- **Never write `0.0` for "not measured"** (CE058). Use `None` when no delta + arrived. + +The verification hook is `tests/_fixtures/golden_streams/_scrub.py`'s +`assert_timing_captured`, where a floor belongs; the five +`tests/_fixtures/golden_streams/*_fixtures.py` modules already carry the deltas +needed to drive it. + ### Known divergences - **Delegate (`delegate-sdk`, out of tree)** records `duration_ms` but no @@ -82,7 +392,19 @@ tool call rather than shell commands alone. records `execution_completed_at` while leaving `duration_ms` as `None` (audit P2-1). -Both are deliberately deferred; see `c/time-bugs-audit.md` for the measurements. +- **`TurnStartEvent` is emitted at inconsistent points.** Antigravity and Codex + fire it at turn entry, before the pump; claude-code, OpenCode and Pi fire it + when a generation begins. Nothing in the timing accounting reads it — the + head and tail are measured from the first and last `AssistantMessage` + instead, which is uniform across all five — so this is recorded rather than + fixed. It is NOT a `max_turns` hazard: `EventCollector.visible_turn_count` is + `len(self._commands)`, derived from `ToolEndEvent`, and `_turn_starts` feeds + only `assistant_turn_count` on the no-`AgentEndEvent` fallback path. The real + cost of normalizing it is that the event drives the live renderers, so moving + it changes the turn boundaries users watch during a run. + +All three are deliberately deferred; see `c/time-bugs-audit.md` for the +measurements. ## `max_turns` counts visible turns on Codex and Antigravity diff --git a/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx b/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx index afa01102..55e6bc33 100644 --- a/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx +++ b/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx @@ -678,6 +678,146 @@ describe("MessageTimelineSection — Unaccounted cell", () => { }); }); +describe("MessageTimelineSection — Startup and Teardown cells", () => { + function cell(label: string): HTMLElement { + const parent = screen.getByText(label).parentElement as HTMLElement; + return parent.children[1] as HTMLElement; + } + + // Same 4s generation + 1s tool exec fixture the Unaccounted block uses, so + // the two blocks' numbers are directly comparable. + function renderStrip(props: { + taskDurationSeconds?: number | null; + harnessStartupMs?: number | null; + harnessTeardownMs?: number | null; + }) { + const m = makeMessage({ + generationMs: 4000, + textMs: 4000, + toolUses: [ + { + toolName: "Bash", + toolUseId: "tu_1", + summary: "ls", + argText: "ls", + description: null, + genMs: null, + durationMs: 1000, + isError: false, + resultPreview: null, + outputTokens: null, + resultTokens: null, + execStartMs: null, + execEndMs: null, + }, + ], + }); + return render(); + } + + test("both buckets render their measured value", () => { + renderStrip({ + taskDurationSeconds: 10, + harnessStartupMs: 3000, + harnessTeardownMs: 1500, + }); + expect(cell("Startup").textContent).toBe("3.0s"); + expect(cell("Teardown").textContent).toBe("1.5s"); + }); + + test("a measured zero renders as 0ms, not as an em-dash", () => { + // A head of 0 stays representable: a turn can reach its first model + // output with nothing measurable in front of it, and a clamped + // inversion is still a measurement because both ends were observed. + // "—" would report that as a missing one. + renderStrip({ + taskDurationSeconds: 10, + harnessStartupMs: 0, + harnessTeardownMs: 834.7, + }); + expect(cell("Startup").textContent).toBe("0ms"); + expect(cell("Teardown").textContent).toBe("835ms"); + }); + + test("Unaccounted shrinks by exactly startup + teardown", () => { + // 10s − 4s gen − 1s tool = 5s before; minus 3s + 1.5s = 500ms after. + renderStrip({ + taskDurationSeconds: 10, + harnessStartupMs: 3000, + harnessTeardownMs: 1500, + }); + expect(cell("Unaccounted").textContent).toBe("500ms (5%)"); + }); + + test("a corrected residual still above 25% stays red", () => { + // The other direction: naming the buckets must not disable the tint, + // only move the number it reads. 20s − 4s gen − 1s tool − 3s − 1s + // = 11s, still 55% unexplained. + renderStrip({ + taskDurationSeconds: 20, + harnessStartupMs: 3000, + harnessTeardownMs: 1000, + }); + expect(cell("Unaccounted").textContent).toBe("11.0s (55%)"); + expect(cell("Unaccounted").className).toContain("text-red-700"); + }); + + test("a residual that was red goes grey once the buckets are named", () => { + // The 25% threshold applies to the CORRECTED residual: 50% before, + // 5% after, so the red tint must follow the correction. + renderStrip({ + taskDurationSeconds: 10, + harnessStartupMs: 3000, + harnessTeardownMs: 1500, + }); + expect(cell("Unaccounted").className).not.toContain("text-red-700"); + }); + + test("an older run with neither field renders — and today's residual", () => { + const { container } = renderStrip({ taskDurationSeconds: 10 }); + expect(cell("Startup").textContent).toBe("—"); + expect(cell("Teardown").textContent).toBe("—"); + // Byte-identical to the pre-existing Unaccounted expectation. + expect(cell("Unaccounted").textContent).toBe("5.0s (50%)"); + expect(cell("Unaccounted").className).toContain("text-red-700"); + expect(container.textContent).not.toContain("NaN"); + }); + + test("only the present bucket is subtracted", () => { + renderStrip({ taskDurationSeconds: 10, harnessStartupMs: 3000 }); + expect(cell("Startup").textContent).toBe("3.0s"); + expect(cell("Teardown").textContent).toBe("—"); + expect(cell("Unaccounted").textContent).toBe("2.0s (20%)"); + }); + + test("the residual still goes negative and stays amber", () => { + // Naming the buckets does not clamp the overlap signal. + renderStrip({ + taskDurationSeconds: 5, + harnessStartupMs: 1000, + harnessTeardownMs: 500, + }); + expect(cell("Unaccounted").textContent).toBe("-1.5s (-30%)"); + expect(cell("Unaccounted").className).toContain("text-amber-700"); + }); + + test("each bucket says what it measures and that it is not decomposed", () => { + renderStrip({ + taskDurationSeconds: 10, + harnessStartupMs: 3000, + harnessTeardownMs: 1500, + }); + expect(screen.getByText("Startup").parentElement).toHaveAttribute( + "title", + expect.stringContaining("time-to-first-token"), + ); + expect(screen.getByText("Teardown").parentElement).toHaveAttribute( + "title", + expect.stringContaining("teardown"), + ); + }); +}); + // A mixed-kind emission's per-kind split is apportioned by content size, so // the page must say so and must not let the unattributable part distort the // thinking share. diff --git a/evalboard/app/runs/[id]/[...task]/_sections.tsx b/evalboard/app/runs/[id]/[...task]/_sections.tsx index eecdf1ad..671ca434 100644 --- a/evalboard/app/runs/[id]/[...task]/_sections.tsx +++ b/evalboard/app/runs/[id]/[...task]/_sections.tsx @@ -318,6 +318,8 @@ export function MessageTimelineSection({ subAgentUsageByToolId = {}, impactByIndex, taskDurationSeconds, + harnessStartupMs, + harnessTeardownMs, }: { messages: MessageEvent[]; // Per-Agent-call sub-agent token breakdown (input/output/cache-create/ @@ -332,6 +334,13 @@ export function MessageTimelineSection({ // tool execution do NOT account for. Null/absent on a run predating // duration capture — the cell then renders "—" rather than a fake residual. taskDurationSeconds?: number | null; + // The turn-level head and tail, summed over the task's turns: wall clock + // before the first generation window opened and after the last one closed. + // Turn-scoped, so they cannot be derived from the per-message stream the + // other stats come from. Null/absent on a run predating the capture, and + // the cells then read "—" while Unaccounted keeps exactly its old meaning. + harnessStartupMs?: number | null; + harnessTeardownMs?: number | null; }) { // Token columns can be shown as counts or as their estimated USD value. const [unit, setUnit] = useState("tokens"); @@ -398,11 +407,23 @@ export function MessageTimelineSection({ const attributableGenMs = totalGenMs - mixedMs; const thinkingShare = attributableGenMs > 0 ? thinkingMs / attributableGenMs : 0; - // Wall clock the agent stream does not explain. Negative means generation - // and tool execution overlapped, which is a real signal — never clamped. + // Wall clock the agent stream does not explain, AFTER every named bucket. + // Startup and teardown are subtracted because they are measured intervals, + // not residual — leaving them in reported a harness's CLI boot as + // unexplained time. `?? 0` subtracts only what was actually measured, so an + // older run with neither field keeps exactly its previous number. + // Negative means generation and tool execution overlapped, which is a real + // signal — never clamped. const taskMs = taskDurationSeconds != null ? taskDurationSeconds * 1000 : null; - const unaccountedMs = taskMs != null ? taskMs - totalGenMs - toolExecMs : null; + const unaccountedMs = + taskMs != null + ? taskMs - + totalGenMs - + toolExecMs - + (harnessStartupMs ?? 0) - + (harnessTeardownMs ?? 0) + : null; const unaccountedShare = taskMs != null && taskMs > 0 && unaccountedMs != null ? unaccountedMs / taskMs @@ -419,19 +440,28 @@ export function MessageTimelineSection({

MIXED = multiple block types · red = slow (gen ≥10s, tool ≥5s)

- {/* TWO LEVELS, two rows. The top row's Generation, Tool exec and - Unaccounted sum to the task's wall clock; the bottom row splits - Generation alone and sums to IT. Rendering the split as a - sub-cell of one top-row cell put both sums on one line, where - nothing said which total each part belonged to. */} + {/* TWO LEVELS, two rows. The top row's five time cells — Startup, + Generation, Tool exec, Teardown, Unaccounted — sum to the task's + wall clock; the bottom row splits Generation alone and sums to + IT. Rendering the split as a sub-cell of one top-row cell put + both sums on one line, where nothing said which total each part + belonged to. The time cells are ordered as the turn runs. */}
-
+
Messages
{messageCount}
+
+
+ Startup +
+
+ {fmtMs(harnessStartupMs ?? null)} +
+
Generation @@ -448,7 +478,15 @@ export function MessageTimelineSection({ {fmtMs(toolExecMs)}
-
+
+
+ Teardown +
+
+ {fmtMs(harnessTeardownMs ?? null)} +
+
+
Unaccounted
@@ -771,6 +809,8 @@ export function CostExplorerSection({ tokens, recordedCostUsd, taskDurationSeconds, + harnessStartupMs, + harnessTeardownMs, }: { messages: MessageEvent[]; subAgentUsageByToolId?: Record; @@ -778,6 +818,9 @@ export function CostExplorerSection({ recordedCostUsd: number | null; // Forwarded verbatim to the timeline's Unaccounted cell. taskDurationSeconds?: number | null; + // Forwarded verbatim to the timeline's Startup/Teardown cells. + harnessStartupMs?: number | null; + harnessTeardownMs?: number | null; }) { const [scale, setScale] = useState(1); const [toolScale, setToolScale] = useState(1); @@ -816,6 +859,8 @@ export function CostExplorerSection({ subAgentUsageByToolId={subAgentUsageByToolId} impactByIndex={impactByIndex} taskDurationSeconds={taskDurationSeconds} + harnessStartupMs={harnessStartupMs} + harnessTeardownMs={harnessTeardownMs} /> {model && tokens.total > 0 && (
diff --git a/evalboard/app/runs/[id]/[...task]/page.tsx b/evalboard/app/runs/[id]/[...task]/page.tsx index 12e8b334..84efa461 100644 --- a/evalboard/app/runs/[id]/[...task]/page.tsx +++ b/evalboard/app/runs/[id]/[...task]/page.tsx @@ -365,6 +365,8 @@ export default async function TaskPage({ tokens={task.tokens} recordedCostUsd={task.totalCostUsd} taskDurationSeconds={task.durationSeconds} + harnessStartupMs={task.harnessStartupMs} + harnessTeardownMs={task.harnessTeardownMs} /> )} diff --git a/evalboard/lib/__tests__/harnessOverhead.test.ts b/evalboard/lib/__tests__/harnessOverhead.test.ts new file mode 100644 index 00000000..e1ae7139 --- /dev/null +++ b/evalboard/lib/__tests__/harnessOverhead.test.ts @@ -0,0 +1,80 @@ +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, test, vi } from "vitest"; + +// End-to-end: the two turn-level timing buckets survive the trip from +// task.json's `iterations` onto TaskDetail. `sumHarnessOverhead` is unit-tested +// in runs.test.ts; what only a read off disk can catch is a misspelled raw key, +// since every TurnEntry field is optional and a typo would just parse as +// absent. Mirrors providerCalls.test.ts's env-stub + fresh-import pattern. +const RUN = "2026-01-01_00-00-00"; +const TASK = "demo-task"; +let tmp: string; + +async function write(rel: string, body: string): Promise { + const abs = path.join(tmp, rel); + await fs.mkdir(path.dirname(abs), { recursive: true }); + await fs.writeFile(abs, body); +} + +async function loadRuns() { + vi.resetModules(); + vi.stubEnv("EVALBOARD_LOCAL_RUNS_DIR", tmp); + return import("../runs"); +} + +async function writeTask(iterations: unknown[]): Promise { + await write( + `${RUN}/run.json`, + JSON.stringify({ + run_id: RUN, + task_results: [{ task_id: TASK, status: "success" }], + }), + ); + await write( + `${RUN}/default/${TASK}/00/task.json`, + JSON.stringify({ final_status: "success", iterations }), + ); +} + +beforeEach(async () => { + tmp = await fs.mkdtemp(path.join(os.tmpdir(), "evalboard-overhead-")); +}); + +afterEach(async () => { + vi.unstubAllEnvs(); + await fs.rm(tmp, { recursive: true, force: true }); +}); + +describe("readTaskDetail: harness startup/teardown", () => { + test("sums both buckets across the task's turns", async () => { + await writeTask([ + { harness_startup_ms: 3047.9, harness_teardown_ms: 33.1 }, + { harness_startup_ms: 120.5, harness_teardown_ms: 4.2 }, + ]); + const { readTaskDetail } = await loadRuns(); + const detail = await readTaskDetail(RUN, TASK); + expect(detail?.harnessStartupMs).toBeCloseTo(3168.4, 3); + expect(detail?.harnessTeardownMs).toBeCloseTo(37.3, 3); + }); + + test("an older run without the fields reports null, not zero", async () => { + await writeTask([{ model_used: "claude-haiku-4-5" }]); + const { readTaskDetail } = await loadRuns(); + const detail = await readTaskDetail(RUN, TASK); + expect(detail?.harnessStartupMs).toBeNull(); + expect(detail?.harnessTeardownMs).toBeNull(); + }); + + test("a measured zero head is preserved as 0", async () => { + // A head of 0.0 stays representable: a turn can reach its first model + // output with nothing measurable in front of it. It must not read as + // "never measured", which is what null means. + await writeTask([{ harness_startup_ms: 0.0, harness_teardown_ms: 834.7 }]); + const { readTaskDetail } = await loadRuns(); + const detail = await readTaskDetail(RUN, TASK); + expect(detail?.harnessStartupMs).toBe(0); + expect(detail?.harnessTeardownMs).toBeCloseTo(834.7, 3); + }); +}); diff --git a/evalboard/lib/__tests__/no-zero-coalesce.test.ts b/evalboard/lib/__tests__/no-zero-coalesce.test.ts new file mode 100644 index 00000000..444ef86f --- /dev/null +++ b/evalboard/lib/__tests__/no-zero-coalesce.test.ts @@ -0,0 +1,182 @@ +import { readFileSync } from "node:fs"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, test } from "vitest"; + +// The TypeScript counterpart to CE058. +// +// CE058 stops an unmeasured TIMING value becoming a numeric literal in `src/`: +// `durationMs == null` means *never timed* and `0` means *timed and instant*, +// so writing the literal publishes the second while meaning the first. The +// evalboard carries the same contract — `sumMeasured` (lib/runs.ts) implements +// it correctly, returning null when nothing was measured — but nothing stopped +// the next author writing `?? 0` where an unmeasured value must stay null. +// +// There is no eslint in evalboard/ (package.json has only typecheck / test / +// build), so this is a vitest source scan rather than a lint plugin. +// +// WHY AN ALLOWLIST RATHER THAN A BAN. The residual arithmetic in _sections.tsx +// uses `?? 0` *correctly*: it subtracts only what was measured, which is the +// whole point. A blanket ban fires on right code. So every occurrence must be +// listed with a reason, and a NEW one fails until its author either justifies +// it here or uses null. +// +// WHY IT KEYS ON TIMING NAMES, and this is a deliberate narrowing. A scan for +// every `?? 0` in these files matches 58 occurrences, roughly half of them +// token or cache buckets (`inputTokens ?? 0`, `prev?.input ?? 0`) where zero is +// a perfectly good answer — tokens are COUNTED, not measured, so there is no +// None-vs-0 ambiguity to protect. An allowlist that long, much of it saying +// "token bucket, fine", is one nobody reads and everybody appends to. CE058's +// contract is about TIMING, so this matches the codebase's own naming +// conventions for a measured interval: an identifier ending in `Ms`, `Seconds`, +// or `duration`/`Duration`. +// +// DECLARED BLIND SPOTS, stated the way the Python rules state theirs: +// * only the three files below are covered; anything else gains no protection; +// * a timing field named by NONE of those conventions is invisible — the +// convention IS the rule, and it is a convention rather than a type; +// * `?? 0.0`, `|| 0.0` and an `if (x == null) x = 0` assignment are not matched; +// * comment stripping cuts from the FIRST `//` on a line, so a coalesce that +// follows a URL or a `//` inside a string literal is invisible. Cheap to +// hit only on purpose, and the alternative is parsing TypeScript. + +const here = dirname(fileURLToPath(import.meta.url)); +const root = resolve(here, "../.."); + +const COVERED = [ + "lib/runs.ts", + "lib/timing.ts", + "app/runs/[id]/[...task]/_sections.tsx", +]; + +// Keyed on the trimmed source LINE, not a line number: numbers move on every +// edit and the test would fail for unrelated reasons. +const ALLOWED = new Map([ + [ + "? executed.reduce((a, t) => a + (t.duration ?? 0), 0)", + "Guarded: the enclosing branch only runs when `allHaveDuration` is true, so every term was measured.", + ], + [ + "const totalGenMs = mainThread.reduce((s, m) => s + (m.generationMs ?? 0), 0);", + "Summing a breakdown: an unmeasured emission contributes nothing to the total, which is what a sum of the measured means.", + ], + [ + "const thinkingMs = mainThread.reduce((s, m) => s + (m.thinkingMs ?? 0), 0);", + "Summing a breakdown: an unmeasured emission contributes nothing, which is what a sum of the measured means.", + ], + [ + "const textMs = mainThread.reduce((s, m) => s + (m.textMs ?? 0), 0);", + "Summing a breakdown: an emission with no text time contributes nothing to the text total.", + ], + [ + "const toolGenMs = mainThread.reduce((s, m) => s + (m.toolGenMs ?? 0), 0);", + "Summing a breakdown: an emission with no tool-gen time contributes nothing to that total.", + ], + [ + "const mixedMs = mainThread.reduce((s, m) => s + (m.mixedGenMs ?? 0), 0);", + "Summing a breakdown: an emission with no mixed-block time contributes nothing to that total.", + ], + [ + "(m) => (m.generationMs ?? 0) >= SLOW_GEN_MS,", + "A threshold comparison: an unmeasured generation is not a slow one, and 0 is the right answer to the question asked.", + ], + [ + "s + m.toolUses.filter((t) => (t.durationMs ?? 0) >= SLOW_TOOL_MS).length,", + "A threshold comparison: an untimed call is not a slow call, so 0 answers the question asked.", + ], + [ + "(harnessStartupMs ?? 0) -", + "The residual. Subtracting only what was measured is the whole point; an unmeasured head leaves its time IN the residual rather than silently claiming it.", + ], + [ + "(harnessTeardownMs ?? 0)", + "The residual's tail half: subtracting only what was measured leaves unmeasured time IN the residual.", + ], + [ + "const slowExec = (execMs ?? 0) >= SLOW_TOOL_MS;", + "A threshold comparison: an untimed execution is not a slow one, so 0 answers the question asked.", + ], + [ + "const slowGen = (m.generationMs ?? 0) >= SLOW_GEN_MS;", + "A threshold comparison: an unmeasured generation is not a slow one.", + ], + [ + "const slowTool = m.toolUses.some((t) => (t.durationMs ?? 0) >= SLOW_TOOL_MS);", + "A threshold comparison: an untimed call is not a slow call.", + ], + [ + "const execMs = m.toolUses.reduce((a, t) => a + (t.durationMs ?? 0), 0);", + "Summing the measured calls of one message; an untimed call adds nothing to that sum.", + ], +]); + +// `x ?? 0` / `x || 0` where the coalesced identifier names a measured interval. +// Not anchored to the line start, so several on one line are all found. +const COALESCE = /\b[\w$]*(?:Ms|Seconds|[Dd]uration)\s*(?:\?\?|\|\|)\s*0(?![.\d\w])/g; + +export function findCoalesces(source: string): string[] { + const hits: string[] = []; + for (const raw of source.split("\n")) { + // Strip line comments so prose about `?? 0` is not a violation. + const line = raw.replace(/\/\/.*$/, ""); + if (COALESCE.test(line)) hits.push(raw.trim()); + COALESCE.lastIndex = 0; + } + return hits; +} + +describe("no unguarded zero-coalesce on a timing value", () => { + for (const relative of COVERED) { + test(relative, () => { + const source = readFileSync(resolve(root, relative), "utf8"); + const unlisted = findCoalesces(source).filter((line) => !ALLOWED.has(line)); + expect( + unlisted, + `${relative} coalesces an unmeasured timing value to 0 without a reason. ` + + `\`x ?? 0\` on a \`…Ms\` field publishes "measured, and instant" where null means ` + + `"never measured" — the contract CE058 enforces on the Python side. If the zero is ` + + `correct here (a sum, or a threshold comparison), add the line to ALLOWED with a ` + + `one-line reason; otherwise keep it null.`, + ).toEqual([]); + }); + } + + test("the scanner actually matches something (a scan that finds nothing proves nothing)", () => { + // The failure mode that makes a source scanner worthless: a regex that + // silently stops matching, after which every file "passes". + const found = COVERED.flatMap((relative) => + findCoalesces(readFileSync(resolve(root, relative), "utf8")), + ); + expect(found.length).toBeGreaterThan(10); + }); + + test("every allowlist entry is still present in a covered file", () => { + // An entry nobody needs is an entry that outlived its reason. + const all = new Set( + COVERED.flatMap((relative) => + findCoalesces(readFileSync(resolve(root, relative), "utf8")), + ), + ); + expect([...ALLOWED.keys()].filter((line) => !all.has(line))).toEqual([]); + }); + + test("every allowlist entry carries a non-empty reason", () => { + expect([...ALLOWED.entries()].filter(([, why]) => why.trim().length < 20)).toEqual([]); + }); + + test("NEGATIVE CONTROL: an un-allowlisted occurrence is reported", () => { + // Without this the suite could pass by matching nothing at all. + const hits = findCoalesces("const x = someTimingMs ?? 0;\n"); + expect(hits).toEqual(["const x = someTimingMs ?? 0;"]); + expect(ALLOWED.has(hits[0])).toBe(false); + }); + + test("NEGATIVE CONTROL: prose and non-timing fields are not matched", () => { + expect(findCoalesces("// the residual uses `?? 0` deliberately\n")).toEqual([]); + expect(findCoalesces("const n = inputTokens ?? 0;\n")).toEqual([]); + expect(findCoalesces("const n = count ?? 0;\n")).toEqual([]); + // The widened conventions, so a regression to `Ms`-only is caught here. + expect(findCoalesces("const s = taskSeconds ?? 0;\n")).toEqual(["const s = taskSeconds ?? 0;"]); + expect(findCoalesces("const d = t.duration ?? 0;\n")).toEqual(["const d = t.duration ?? 0;"]); + }); +}); diff --git a/evalboard/lib/__tests__/parseMessages.test.ts b/evalboard/lib/__tests__/parseMessages.test.ts index 04580ba1..3bffe27b 100644 --- a/evalboard/lib/__tests__/parseMessages.test.ts +++ b/evalboard/lib/__tests__/parseMessages.test.ts @@ -228,7 +228,7 @@ describe("parseMessages — message_id collapsing", () => { expect(e.generationMs).toBe(5500); }); - test("splits when message_ids differ even with tight gap", () => { + test("splits differing message_ids across contiguous windows (0ms gap — the Antigravity shape)", () => { const turns: TurnEntry[] = [ { messages: [ @@ -242,9 +242,11 @@ describe("parseMessages — message_id collapsing", () => { }, { role: "assistant", - started_at: "2026-01-01T00:00:01.010Z", // 10ms gap + // Opens exactly where the previous one closed, the + // way Antigravity tiles its generation windows. + started_at: "2026-01-01T00:00:01.000Z", completed_at: "2026-01-01T00:00:02.000Z", - generation_duration_ms: 990, + generation_duration_ms: 1000, message_id: "msg_b", content_blocks: [{ block_type: "text", text: "hi" }], }, diff --git a/evalboard/lib/__tests__/runs.test.ts b/evalboard/lib/__tests__/runs.test.ts index 0ac7b635..2a0cced0 100644 --- a/evalboard/lib/__tests__/runs.test.ts +++ b/evalboard/lib/__tests__/runs.test.ts @@ -24,6 +24,7 @@ import { parseCriterionResults, type RawTaskResult, sortArtifacts, + sumHarnessOverhead, toTaskRow, visibleTurnsFromRaw, walkArtifacts, @@ -246,6 +247,62 @@ describe("aggregateSubAgentUsage", () => { }); }); +describe("sumHarnessOverhead", () => { + test("sums both buckets across iterations", () => { + expect( + sumHarnessOverhead([ + { harness_startup_ms: 3000, harness_teardown_ms: 800 }, + { harness_startup_ms: 120, harness_teardown_ms: 40 }, + ]), + ).toEqual({ startupMs: 3120, teardownMs: 840 }); + }); + + test("a measured zero is a measurement and still sums", () => { + // A harness that reached its first model output with nothing + // measurable in front of it legitimately reports 0.0 — a clamped + // inversion where both ends were still observed. That is a number, + // not a gap, and the assertion holds however the head is produced. + expect( + sumHarnessOverhead([{ harness_startup_ms: 0, harness_teardown_ms: 3.5 }]), + ).toEqual({ startupMs: 0, teardownMs: 3.5 }); + }); + + test("is null when EVERY iteration is null — never 0", () => { + // 0 would claim the harness started instantly; null says nobody looked. + expect( + sumHarnessOverhead([ + { harness_startup_ms: null, harness_teardown_ms: null }, + {}, + ]), + ).toEqual({ startupMs: null, teardownMs: null }); + }); + + test("sums the measured iterations and ignores the unmeasured ones", () => { + expect( + sumHarnessOverhead([ + { harness_startup_ms: 500 }, + { harness_teardown_ms: 90 }, + ]), + ).toEqual({ startupMs: 500, teardownMs: 90 }); + }); + + test("is null on an empty turn list", () => { + expect(sumHarnessOverhead([])).toEqual({ + startupMs: null, + teardownMs: null, + }); + }); + + test("a non-finite value is dropped rather than poisoning the sum", () => { + expect( + sumHarnessOverhead([ + { harness_startup_ms: NaN, harness_teardown_ms: 10 }, + { harness_startup_ms: 25 }, + ]), + ).toEqual({ startupMs: 25, teardownMs: 10 }); + }); +}); + describe("isExcludedArtifact", () => { test("hides build artifacts, local state, and secrets", () => { for (const rel of [ diff --git a/evalboard/lib/__tests__/timing-union-parity.test.ts b/evalboard/lib/__tests__/timing-union-parity.test.ts index 734c33f6..58bcfd39 100644 --- a/evalboard/lib/__tests__/timing-union-parity.test.ts +++ b/evalboard/lib/__tests__/timing-union-parity.test.ts @@ -5,7 +5,7 @@ import { describe, expect, test } from "vitest"; import { busyMs, toolExecutionMs } from "../timing"; import type { MessageEvent, MessageToolUse } from "../runs"; -// Parity guard: `busyMs` here and `coder_eval.agents._timing.busy_ms` in the +// Parity guard: `busyMs` here and `coder_eval.timing.busy_ms` in the // Python harness answer the same question about the same task.json — how much // wall clock the tools occupied — one to subtract it from a generation window, // the other to subtract it from the task's duration. A divergence makes the @@ -25,7 +25,19 @@ interface UnionCase { expected_ms: number; } -const corpus: { cases: UnionCase[] } = JSON.parse(readFileSync(fixture, "utf8")); +// `union_cases` carries no window: the extent is the spans' own bounds. It is +// the half that pins `toolExecutionMs`, which derives that extent with its own +// min/max instead of being handed one — the Python twin is +// `coder_eval.timing.union_ms`. +interface ExtentCase { + name: string; + spans: [number, number][]; + expected_ms: number; +} + +const corpus: { cases: UnionCase[]; union_cases: ExtentCase[] } = JSON.parse( + readFileSync(fixture, "utf8"), +); describe("busyMs matches the shared union corpus", () => { test("the corpus is non-empty (a silently emptied file must not pass)", () => { @@ -138,3 +150,30 @@ describe("toolExecutionMs", () => { expect(toolExecutionMs([message([toolUse({})])])).toBe(0); }); }); + +describe("toolExecutionMs matches the shared extent corpus", () => { + test("the extent corpus is non-empty (a silently emptied file must not pass)", () => { + expect(corpus.union_cases.length).toBeGreaterThan(5); + }); + + // Each span becomes one bounded tool call on one message, so + // `toolExecutionMs` has to derive the extent itself — the one part of the + // union rule the windowed `cases` above cannot reach. Python replays the + // same array through `coder_eval.timing.union_ms`. + for (const c of corpus.union_cases) { + test(c.name, () => { + const ms = toolExecutionMs([ + message( + c.spans.map(([s, e], i) => + toolUse({ + toolUseId: `t${i}`, + execStartMs: s, + execEndMs: e, + }), + ), + ), + ]); + expect(ms).toBeCloseTo(c.expected_ms, 6); + }); + } +}); diff --git a/evalboard/lib/runs.ts b/evalboard/lib/runs.ts index 3cfaf894..82431659 100644 --- a/evalboard/lib/runs.ts +++ b/evalboard/lib/runs.ts @@ -364,6 +364,12 @@ export interface TaskDetail extends TaskResultSummary { // the Agent row). The cost simulator consumes the values via Object.values(). // Empty for runs/turns with no spawned sub-agents. subAgentUsageByToolId: Record; + // The task's harness head and tail, summed over its turns. `null` when no + // turn measured that end — never 0, which would claim the harness started + // or finished instantly. Subtracted from the timeline's Unaccounted cell so + // that residual is what is left after every named bucket. + harnessStartupMs: number | null; + harnessTeardownMs: number | null; // Per-call ACTUAL cost + cache audit rows, grouped by turn iteration. Only // turns whose `provider_call_costs` list is non-empty appear (LiteLLM/ // open-weight backend; empty on Claude/Bedrock). Rendered as a standalone @@ -394,6 +400,35 @@ export interface SubAgentTotals { cacheRead: number; } +// Sum one optional per-turn measurement across a task's turns. `null` — never +// 0 — when no turn carried the value, because 0 means "measured, and instant" +// while null means nobody measured (the `TurnRecord` fields' own contract, and +// what CE058 guards on the Python side). Non-finite values are dropped rather +// than poisoning the total with NaN. +function sumMeasured(values: (number | null | undefined)[]): number | null { + let total: number | null = null; + for (const v of values) { + if (typeof v !== "number" || !Number.isFinite(v)) continue; + total = (total ?? 0) + v; + } + return total; +} + +// The task's harness head and tail, summed over its turns. The per-turn values +// are measured by `coder_eval/timing.py::decompose_turn`; the summation is +// evalboard-only, and the arithmetic that consumes it — the Unaccounted +// residual in `_sections.tsx` — is the deliberate second implementation that +// helper's docstring names (as `pricing.ts` mirrors `pricing.py`). +export function sumHarnessOverhead(turns: TurnEntry[]): { + startupMs: number | null; + teardownMs: number | null; +} { + return { + startupMs: sumMeasured(turns.map((t) => t.harness_startup_ms)), + teardownMs: sumMeasured(turns.map((t) => t.harness_teardown_ms)), + }; +} + // Group the parsed assistant messages by `parentToolUseId` into a per-sub-agent // token breakdown. A sub-agent's generations all carry the spawning Agent call's // tool_use_id; main-thread messages (parentToolUseId null/undefined) are skipped. @@ -1531,6 +1566,12 @@ export interface TurnEntry { // reconciliation row, which carries no model of its own. model_used?: string | null; token_usage?: TokenUsageEntry | null; + // The turn's head and tail: wall ms before the first generation window + // opened and after the last one closed. Absent on runs predating the + // capture, and null on a turn that produced no assistant message — in both + // cases nobody measured, which is a different fact from a measured 0. + harness_startup_ms?: number | null; + harness_teardown_ms?: number | null; // Per-call actual cost + cache audit rows (LiteLLM/open-weight backend); // empty/absent on Claude/Bedrock. Surfaced as a standalone per-call table. provider_call_costs?: ProviderCallEntryRaw[]; @@ -2522,6 +2563,8 @@ export async function readTaskDetail( const tokens = selectTokenTotals(messages, task?.iterations ?? []); const subAgentUsageByToolId = aggregateSubAgentUsage(messages); + const { startupMs: harnessStartupMs, teardownMs: harnessTeardownMs } = + sumHarnessOverhead(task?.iterations ?? []); const taskDescription = task?.task_config?.resolved?.initial_prompt ?? @@ -2562,6 +2605,8 @@ export async function readTaskDetail( messages, tokens, subAgentUsageByToolId, + harnessStartupMs, + harnessTeardownMs, providerCalls, }; } diff --git a/evalboard/lib/timing.ts b/evalboard/lib/timing.ts index 59819599..37174600 100644 --- a/evalboard/lib/timing.ts +++ b/evalboard/lib/timing.ts @@ -136,10 +136,11 @@ export function epochMs(value: string | null | undefined): number | null { // Milliseconds inside [lo, hi] where at least ONE span was running: the UNION, // not the sum. // -// The TypeScript twin of `coder_eval.agents._timing.busy_ms`, deliberately the -// same algorithm — the agents subtract tool time from a generation window with -// it, and this file subtracts tool time from a task's wall clock, so the two -// must agree about what "tool execution took N ms" means. Held in step by +// The TypeScript twin of `coder_eval.timing.busy_ms`, deliberately the +// same algorithm — the harness subtracts tool time from its generation windows +// with it (once, in streaming/collector.py::subtract_tool_time), and this file +// subtracts tool time from a task's wall clock, so the two must agree about +// what "tool execution took N ms" means. Held in step by // tests/_fixtures/timing_union_cases.json, which both suites replay. export function busyMs( spans: [number, number][], diff --git a/pyproject.toml b/pyproject.toml index b1e08974..395e3ea5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -277,6 +277,9 @@ external = [ "CE057", "CE058", "CE059", + "CE060", + "CE061", + "CE063", ] # custom architectural lint rules (tests/lint/) [tool.ruff.lint.pylint] diff --git a/scripts/timing/decompose_run.py b/scripts/timing/decompose_run.py new file mode 100644 index 00000000..6c3bd68c --- /dev/null +++ b/scripts/timing/decompose_run.py @@ -0,0 +1,312 @@ +#!/usr/bin/env python3 +"""Decompose recorded turns into the four wall-clock buckets, per harness. + +Reads `task.json` files, groups their turns by `agent_type`, and prints the +mean generation / tool / startup / teardown against the mean turn duration, +plus the residual as a percentage of wall clock. A healthy harness reconciles +to well under 1%. The tool bucket is the UNION of the command intervals, never +their sum — tool calls overlap, and summing them books the overlap twice. + + uv run python scripts/timing/decompose_run.py runs//default/*/00/task.json + +Pass `--max-residual-pct` to turn the report into a GATE: a non-zero exit when +any single turn's |residual| exceeds that share of its own wall clock. The gate +is deliberately TWO-SIDED, because the only other sensor for this identity is +not. `tests/_fixtures/golden_streams/_scrub.py` asserts `overshoot <= ...`, +which catches a bucket that claims MORE time than the turn contains and says +nothing at all about a bucket that claims less — so an unmeasured bucket, the +exact defect this file exists to find, passes every test in the suite. Gating +on `abs(share)` covers both signs. + +Not wired into `make`: it needs live runs, not fixtures. NOTE `scripts/` is +outside the Makefile's LINT_PATHS, so this file is neither formatted nor +ruff-checked — keep it small and dependency-free (stdlib plus the one shared +`union_ms` import, so the union rule has a single definition). +""" + +from __future__ import annotations + +import argparse +import json +import statistics +import sys +from collections import defaultdict +from datetime import datetime +from pathlib import Path + +from coder_eval.timing import union_ms + + +def _parse(stamp: object) -> datetime | None: + if not isinstance(stamp, str): + return None + try: + return datetime.fromisoformat(stamp) + except ValueError: + return None + + +def _sub_agent_tool_ids(turn: dict) -> set: + """Tool ids owned by a SUB-AGENT generation, which the main thread excludes. + + Derived the only way it can be: a child generation carries + `parent_tool_use_id`, and its `tool_use_ids` are the calls it made. + + This MUST match `EventCollector._main_thread_tool_spans`, which applies the + same filter when it computes the head, the tail and the generation + subtraction. The two used to disagree — the collector passed every command + while filtering its generations — and they agreed only by luck, because a + child nests inside the parent Agent call whose interval the union already + covers. Codex's recovered child tools carry the CHILD's clock, so the + nesting is not guaranteed, and a gate computing a different tool total than + the harness reports a residual that is an artifact of the disagreement + rather than a bucket error. This is the only two-sided live sensor for the + identity, so that is the worst place for the two to drift. + """ + ids = set() + for message in turn.get("messages") or []: + if message.get("role") == "assistant" and message.get("parent_tool_use_id") is not None: + ids.update(message.get("tool_use_ids") or []) + return ids + + +def _tool_ms(turn: dict) -> float: + """Wall ms this turn's MAIN-THREAD tools occupied — the UNION, not the sum. + + The same rule `coder_eval.timing.union_ms` applies when the collector + subtracts tool time out of a generation window, and it has to be the same + rule here or the identity does not close: Pi resolved a `Write` and a `Bash` + that overlapped by 18.4 ms in one measured turn, and summing their durations + booked that overlap twice, which is precisely the 18.3 ms residual that + found this. A command with no recorded bounds cannot be placed on the + timeline at all, so it contributes nothing rather than being summed in + blind — see docs/agents/HARNESS_PARITY.md's Delegate divergence. Sub-agent + tools are excluded for the same reason their generations are; see + `_sub_agent_tool_ids`. + """ + excluded = _sub_agent_tool_ids(turn) + spans = [] + for command in turn.get("commands") or []: + if command.get("tool_id") in excluded: + continue + start = _parse(command.get("execution_started_at")) + end = _parse(command.get("execution_completed_at")) + if start is not None and end is not None and end >= start: + spans.append((start, end)) + return union_ms(spans) + + +def _turn_buckets(turn: dict) -> tuple[float, float, float, float, float] | None: + """(wall_ms, generation_ms, tool_ms, startup_ms, teardown_ms) for one turn. + + None when the turn was never timed at all — a crash partial with no + generation. A bucket the harness could not measure counts as 0 toward the + sums while the turn still contributes its wall clock, so an unmeasured + bucket shows up as residual rather than silently vanishing. + """ + duration_seconds = turn.get("duration_seconds") + if not isinstance(duration_seconds, (int, float)): + return None + # MAIN THREAD ONLY. A sub-agent's generations bubble into the same stream + # tagged with the spawning Agent call's tool_use_id, and that call's own + # interval already spans the sub-agent's entire run. Counting both books the + # sub-agent twice — the evalboard's timeline strip filters on exactly this + # field for exactly this reason (a 120 s Agent call containing 90 s of + # sub-agent generation drove its residual to -57%). + messages = turn.get("messages") or [] + generation_ms = sum( + m.get("generation_duration_ms") or 0.0 + for m in messages + if m.get("role") == "assistant" + and m.get("parent_tool_use_id") is None + and isinstance(m.get("generation_duration_ms"), (int, float)) + ) + startup_ms = turn.get("harness_startup_ms") + teardown_ms = turn.get("harness_teardown_ms") + return ( + duration_seconds * 1000.0, + generation_ms, + _tool_ms(turn), + startup_ms if isinstance(startup_ms, (int, float)) else 0.0, + teardown_ms if isinstance(teardown_ms, (int, float)) else 0.0, + ) + + +def _residual_ms(buckets: tuple[float, float, float, float, float]) -> float: + wall, gen, tool, up, down = buckets + return wall - gen - tool - up - down + + +def _never_measured(turn: dict) -> bool: + """True when the collector recorded NO generation window for this turn. + + Both head and tail `None` is how `EventCollector` says a turn had nothing + measurable — `_scrub.py` asserts exactly that pairing. There is nothing to + reconcile against, so a 100% residual here is an artifact of the absence, + not a bucket the harness failed to fill. One of the two set and the other + `None` is the opposite case and is NOT skipped: that IS a real unmeasured + bucket, and `_turn_buckets` counts it as 0.0 so it surfaces as residual. + """ + return turn.get("harness_startup_ms") is None and turn.get("harness_teardown_ms") is None + + +def main(argv: list[str]) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("task_json", nargs="+", type=Path, help="task.json files to decompose") + parser.add_argument( + "--max-residual-pct", + type=float, + default=None, + help="fail (exit 1) if any gateable turn's |residual| exceeds this share of its own wall clock", + ) + parser.add_argument( + "--min-turn-ms", + type=float, + default=1000.0, + help="turns shorter than this are excluded from the share columns and the gate (default 1000)", + ) + parser.add_argument( + "--include-crashed", + action="store_true", + help="keep crashed and never-measured turns instead of skipping them", + ) + args = parser.parse_args(argv) + + # (path, turn_index, buckets) rather than bare buckets: a breach that cannot + # name the file it came from is a gate nobody can act on, and a gate nobody + # can act on gets muted. + by_harness: dict[str, list[tuple[Path, int, tuple[float, float, float, float, float]]]] = defaultdict(list) + skipped_crashed = 0 + skipped_no_window = 0 + # A turn `_turn_buckets` cannot place on the timeline at all (no numeric + # `duration_seconds`). Counted rather than silently dropped, for the same + # reason as the two above: an exclusion nobody can see understates how much + # of the corpus the gate actually looked at. + skipped_untimed = 0 + for path in args.task_json: + try: + record = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + print(f"skipping {path}: {exc}", file=sys.stderr) + continue + harness = record.get("agent_type") or "unknown" + for index, turn in enumerate(record.get("iterations") or []): + # Filter on the TURN, never on the record's `final_status`. The + # orchestrator preserves a crashed partial TurnRecord across a + # retry, so a SUCCESS record can hold a crashed turn; and a + # `coder-eval execute` corpus finalizes EVERY row as NOT_GRADED, + # which says nothing about timing — a status allowlist would skip + # all of it and then report a clean gate over nothing measured. + # + # The two conditions are counted INDEPENDENTLY and a turn matching + # both is counted under each. Short-circuiting on the first would + # leave the second's tally reading 0 on the only corpus that + # contains it — the measured case: all three excluded turns are + # crashed and two of them are also never-measured — which reads as + # a filter that never fires rather than one with no evidence. + crashed = turn.get("crashed") is True + no_window = _never_measured(turn) + if not args.include_crashed and (crashed or no_window): + skipped_crashed += int(crashed) + skipped_no_window += int(no_window) + continue + buckets = _turn_buckets(turn) + if buckets is None: + skipped_untimed += 1 + continue + by_harness[harness].append((path, index, buckets)) + + if not by_harness: + print("no timed turns found", file=sys.stderr) + return 1 + + header = ( + f"{'harness':<14} {'n':>3} {'wall':>10} {'generation':>11} {'tool':>9} " + f"{'startup':>9} {'teardown':>9} {'residual':>10} {'%':>7} {'worst turn':>11} " + f"{'gated':>6} {'med|%|':>7} {'worst|%|':>9}" + ) + print(header) + print("-" * len(header)) + worst_share = 0.0 + worst_turn = 0.0 + skipped_short = 0 + breaches: list[tuple[str, float, float, float, Path, int]] = [] + gateable_total = 0 + for harness in sorted(by_harness): + rows = by_harness[harness] + turns = [buckets for _, _, buckets in rows] + n = len(turns) + wall, gen, tool, up, down = (sum(col) / n for col in zip(*turns, strict=True)) + residual = wall - gen - tool - up - down + share = (residual / wall * 100.0) if wall else 0.0 + # The MEAN residual can hide an outlier by cancellation — the sign + # flips between harnesses because head/tail are measured between event + # stamps while duration_seconds is the agent's own monotonic span. So + # report the worst single turn beside it; that is the real bound. + per_turn = max(abs(_residual_ms(buckets)) for buckets in turns) + worst_share = max(worst_share, abs(share)) + worst_turn = max(worst_turn, per_turn) + + # Per-turn |residual| as a share of that turn's OWN wall clock. A 30 ms + # turn with a 5 ms residual is not a 17% defect, so short turns are out + # of the share columns and out of the gate — but they stay in the means + # above, where their absolute contribution is honest and tiny. + # `wall_ms <= 0` is guarded here and not in `_turn_buckets`, which + # returns a real tuple for a literal 0 duration. + shares: list[tuple[float, Path, int, tuple[float, float, float, float, float]]] = [] + for path, index, buckets in rows: + turn_wall = buckets[0] + if turn_wall <= 0 or turn_wall < args.min_turn_ms: + skipped_short += 1 + continue + shares.append((abs(_residual_ms(buckets)) / turn_wall * 100.0, path, index, buckets)) + gateable_total += len(shares) + if shares: + median_share = statistics.median(s for s, _, _, _ in shares) + worst_row = max(shares, key=lambda row: row[0]) + med_col = f"{median_share:>6.3f}%" + worst_col = f"{worst_row[0]:>8.3f}%" + else: + med_col = f"{'—':>7}" + worst_col = f"{'—':>9}" + if args.max_residual_pct is not None: + for turn_share, path, index, buckets in shares: + if turn_share > args.max_residual_pct: + breaches.append((harness, turn_share, _residual_ms(buckets), buckets[0], path, index)) + + print( + f"{harness:<14} {n:>3} {wall:>9.1f}ms {gen:>10.1f}ms {tool:>8.1f}ms " + f"{up:>8.1f}ms {down:>8.1f}ms {residual:>9.3f}ms {share:>6.2f}% {per_turn:>9.3f}ms " + f"{len(shares):>6} {med_col} {worst_col}" + ) + print(f"\nworst mean |residual| = {worst_share:.2f}% of wall clock") + print(f"worst single-turn |residual| = {worst_turn:.3f}ms") + print( + f"skipped: {skipped_crashed} crashed, {skipped_no_window} no-window " + f"(a turn can be both), {skipped_untimed} untimed, " + f"{skipped_short} short (< {args.min_turn_ms:.0f}ms)" + ) + + if not gateable_total: + # A gate that passes because it measured nothing is the exact failure + # this script exists to remove, so it only passes when none was asked for. + print("no gateable turns", file=sys.stderr) + return 1 if args.max_residual_pct is not None else 0 + + if args.max_residual_pct is None: + return 0 + if not breaches: + print(f"\ngate OK: every one of {gateable_total} gateable turns is within {args.max_residual_pct}%") + return 0 + print(f"\ngate FAILED: {len(breaches)} turn(s) over {args.max_residual_pct}% of wall clock", file=sys.stderr) + for harness, turn_share, residual_ms, turn_wall, path, index in sorted(breaches, key=lambda b: -b[1]): + print( + f" {harness:<14} {turn_share:>8.3f}% residual {residual_ms:>10.3f}ms " + f"wall {turn_wall:>10.1f}ms {path} turn {index}", + file=sys.stderr, + ) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/src/coder_eval/agents/_timing.py b/src/coder_eval/agents/_timing.py deleted file mode 100644 index 6d568e56..00000000 --- a/src/coder_eval/agents/_timing.py +++ /dev/null @@ -1,42 +0,0 @@ -"""Shared timing helpers for agent implementations. - -Two harnesses interleave tool execution into a single generation window — -Antigravity (the Step for the tool arrives and only a later ``usage_metadata`` -Step cuts the message) and Codex (``_flush_message``'s window is extended to -the last item's ``completed_at_ms``). Both must therefore subtract the tool -time from the window before publishing ``generation_duration_ms``, and both -must subtract the same thing: the UNION of the closed intervals, clipped to -the window. -""" - -from datetime import datetime - - -def busy_ms(spans: list[tuple[datetime, datetime]], lo: datetime, hi: datetime) -> float: - """Wall milliseconds inside ``[lo, hi]`` where at least ONE span was running. - - The union, not the sum. Tool intervals overlap in practice — Antigravity - resolves several calls from one ``Step`` and backgrounds anything over ten - seconds; Codex spawns collab agents that run concurrently — so adding - their durations over-counts the busy time by exactly the overlap. - Subtracting such a sum from a generation window understates generation - and, with enough concurrency, drives it negative: four concurrent 400 ms - calls inside a 1000 ms window sum to 1600 ms, clamping the result to the - ``0.0`` that "unknown timing says unknown" exists to eliminate. - - Clipping to ``[lo, hi]`` is the other half: a tool that opened before this - window only spent part of its life inside it, and only that part is not - generation time here. - """ - clipped = sorted((max(s, lo), min(e, hi)) for s, e in spans if min(e, hi) > max(s, lo)) - if not clipped: - return 0.0 - total = 0.0 - open_start, open_end = clipped[0] - for start, end in clipped[1:]: - if start > open_end: # disjoint — bank the run and start a new one - total += (open_end - open_start).total_seconds() * 1000.0 - open_start, open_end = start, end - else: # overlapping or adjacent — extend the run - open_end = max(open_end, end) - return total + (open_end - open_start).total_seconds() * 1000.0 diff --git a/src/coder_eval/agents/antigravity_agent.py b/src/coder_eval/agents/antigravity_agent.py index 6c3269ec..5d016f4a 100644 --- a/src/coder_eval/agents/antigravity_agent.py +++ b/src/coder_eval/agents/antigravity_agent.py @@ -31,7 +31,6 @@ from coder_eval.agent import Agent, AgentState from coder_eval.agents._logging import PrefixedAdapter -from coder_eval.agents._timing import busy_ms from coder_eval.agents.registry import AgentRegistry from coder_eval.agents.watchdog import ThreadedWatchdog from coder_eval.config import settings @@ -68,6 +67,7 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.timing import TurnClock, close_window from coder_eval.utils import expand_env_vars @@ -554,8 +554,15 @@ async def communicate( assert self.config.type is not None, "AntigravityAgent requires AgentConfig.type before communicate()" self._begin_turn() + # Raw monotonic, and deliberately not the turn clock: this seeds the + # poll deadline below and `duration_seconds`, neither of which may move + # when the wall clock steps. `TurnClock` is for the RECORDED stamps. turn_start_time = time.monotonic() - turn_start_wall = datetime.now() + # ONE clock per turn. This is the (monotonic, wall) pair the reducer + # already captured here and then failed to use for its later stamps — + # which is why its window span was monotonic while its tool intervals + # were wall, and why the two could disagree. + clock = TurnClock() task_id = str(self.config.type) model = self._effective_model() collector = EventCollector() @@ -572,7 +579,7 @@ async def communicate( iteration=self._iteration, model=model, turn_start_time=turn_start_time, - turn_start_wall=turn_start_wall, + clock=clock, max_turns=max_turns, ) @@ -806,7 +813,7 @@ def __init__( iteration: int, model: str, turn_start_time: float, - turn_start_wall: datetime, + clock: TurnClock, max_turns: int | None = None, ) -> None: self._agent = agent @@ -818,6 +825,11 @@ def __init__( self.iteration = iteration self.model = model self.turn_start_time = turn_start_time + # Every wall stamp below derives from this, so the tool spans and the + # window bounds they are subtracted from share one basis. Injected, not + # read from a module global, so a test supplies a fake instead of + # monkeypatching `datetime` out from under the reducer. + self.clock = clock self.max_turns = max_turns self.timeout_hit = False @@ -853,15 +865,10 @@ def __init__( # come from the SAME instant, captured by communicate(), so the # recorded bounds and the measured duration describe one span. # Advanced only by a flush that actually emitted a message. - self._gen_mark_monotonic: float = turn_start_time - self._gen_mark_wall: datetime = turn_start_wall - # Execution intervals of tools that CLOSED since the mark. This harness - # interleaves tool calls into one generation — the Step for the tool - # arrives and only a later usage_metadata Step cuts the message — so a - # window legitimately contains tool time that is not model time. Kept - # as intervals, not a running total, because they overlap (see - # busy_ms). - self._tool_spans_since_mark: list[tuple[datetime, datetime]] = [] + self._gen_mark_wall: datetime = clock.now() + # Re-seeded ONCE, at the first observed Step. See + # `_seed_first_generation_window`. + self._first_output_seen: bool = False @property def ended_cleanly(self) -> bool: @@ -883,11 +890,65 @@ def max_turns_reached(self) -> bool: """ return self.max_turns is not None and self.collector.visible_turn_count >= self.max_turns + def _seed_first_generation_window(self, source: Any) -> None: + """Move the first window's mark to the first observed MODEL output. + + ``harness_startup_ms`` is defined as the wall clock from the turn + starting until the harness first observed model output, and that instant + is also where the first generation window opens — which is what keeps + the head and the generation disjoint so the four-bucket identity still + closes. + + Without this ``_gen_mark_wall`` is stamped when the turn state is built, + BEFORE ``AgentStartEvent`` is emitted, so the head is a small negative + that ``decompose_turn`` clamps to ``0.0`` — a clamped inversion + published as "measured, and instant", which is the exact confusion CE058 + exists to prevent everywhere else. Everything before the first ``Step`` + — dispatch and time to first token — was booked as the first + generation instead: ~4.7 s per turn on this harness, measured against a + later-window median of 3.3 s. + + What differs from claude-code is not in-process versus subprocess — + this harness spawns a ``localharness`` binary too. It is spawned ONCE, + in ``start()``, and held across every ``communicate()``, so there is no + boot inside a turn for the head to contain: it is dispatch plus time to + first token. claude-code spawns a fresh CLI per turn and so fuses that + boot in. The head means the same thing on both; only its COMPOSITION + differs, which is a real property of the harness rather than a + measurement artifact. + + GATED ON ``source``, because the field is defined as model output and + the SDK streams Steps that are not. ``StepSource`` carries ``SYSTEM`` + and ``USER`` besides ``MODEL``, and ``StepType`` carries + ``SYSTEM_MESSAGE`` / ``COMPACTION`` / ``FINISH``; the SDK's event + processor queues every ``step_update`` verbatim, so a turn can + legitimately open with one. Seeding on such a Step would put the mark + BEFORE the model spoke and hand the remainder back to msg0's + generation, which is the defect this method exists to remove. The same + gate guards text streaming a few lines below, for the same reason. + + ONCE PER TURN, and that is the whole contract. ``process_step`` runs for + every Step in the turn; re-seeding on each would stop the windows tiling + and drop the gap before the next emission into no bucket at all, which + is the defect pi shipped with. The flag needs no reset: a fresh turn + state (and a fresh ``TurnClock``) is built per ``communicate()``, so it + is per-attempt by construction. + + A turn that streams no MODEL Step at all never latches, keeps the + turn-entry mark and clamps to ``0.0`` exactly as before — the same + fail-safe degradation as an unrecognized source. + """ + if self._first_output_seen or _enum_value(source) != _SOURCE_MODEL: + return + self._first_output_seen = True + self._gen_mark_wall = self.clock.now() + def process_step(self, step: Any) -> None: """Route one streamed ``Step`` to events + transcript reconstruction.""" stype = _enum_value(step.type) sstatus = _enum_value(step.status) ssource = _enum_value(step.source) + self._seed_first_generation_window(ssource) starget = _enum_value(step.target) done = sstatus in (_STATUS_DONE, _STATUS_ERROR) @@ -939,7 +1000,7 @@ def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call self._next_seq += 1 tool_name = _ANTIGRAVITY_TO_CLAUDE_TOOL_MAP.get(raw_name, str(raw_name)) self._tool_input_keys[cid] = set(call.args) - now = datetime.now() + now = self.clock.now() tel = CommandTelemetry( tool_name=tool_name, tool_id=cid, @@ -965,7 +1026,7 @@ def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call or step.content or None ) - completed = datetime.now() + completed = self.clock.now() started = start_tel.execution_started_at or completed tool_ms = max((completed - started).total_seconds() * 1000.0, 0.0) end_tel = start_tel.model_copy( @@ -978,12 +1039,6 @@ def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call "duration_ms": tool_ms, } ) - # This tool closed inside the open generation window, so its time is - # not model time. The INTERVAL is recorded, not the duration: tool - # calls overlap here, and only their union may be subtracted (see - # busy_ms). Only the DONE path records one — a tool force-closed at - # finalize has duration_ms None and was never timed. - self._tool_spans_since_mark.append((started, completed)) self.commands.append(end_tel) self.emit.on_event( ToolEndEvent( @@ -1024,61 +1079,38 @@ def _flush_generation(self, gen: TokenUsage, reasoning_tokens: int) -> None: """ if not self._blocks and gen.is_empty(): return - now_monotonic = time.monotonic() - now_wall = datetime.now() - # Model-generation time = the whole window MINUS the tool execution - # that happened inside it. - # + now_wall = self.clock.now() # Do NOT "simplify" this to resetting the mark when a tool ends. That # loses real model time: measured on run 2026-09-09_04-18-50, task # skill-rpa-uia-google-search, a harness-local Read closed 8 ms after # it opened while 6.4 s of model time separated the two flushes around # it — a reset would have reported 8 ms and dropped the 6.4 s. - # Subtracting closed tool time handles that case AND its opposite (a - # 43 s Bash, where the model time really is the flush-to-DONE - # remainder). + # Publishing the RAW window and letting the collector subtract the tool + # union handles that case AND its opposite (a 43 s Bash, where the + # model time really is the flush-to-DONE remainder). # - # A tool that is still OPEN at flush time counts too, bounded at - # `now_wall`. Subtracting only CLOSED intervals published the portion - # of a straddling call that ran before the boundary as generation, - # while the call's own duration_ms counted it again — the one - # double-count that this harness's contiguous windows have no slack to - # absorb. Measured on tasks/hello_date: a Bash opening 1.7 ms before - # the flush drove Sum(generation) + Sum(command) 0.26 ms PAST the turn - # wall, on a turn whose whole headroom was 1.4 ms. The four sibling - # runs passed by 1.2-8.7 ms out of ~12 s, so this was a coin flip, not - # a rounding artifact. + # This harness interleaves a tool INTO a window rather than tiling + # around it, so the window legitimately contains time that is not model + # time. `EventCollector.subtract_tool_time` clips the union to these + # bounds and takes it out. Measured here before any of that existed: a + # Bash opening 1.7 ms before the flush drove Sum(generation) + + # Sum(command) 0.26 ms PAST the turn wall, on a turn whose whole + # headroom was 1.4 ms. # - # No double subtraction: when the call later closes, the DONE path - # appends its full interval to the NEXT window's list, where busy_ms - # clips it to the post-flush remainder. - span_ms = (now_monotonic - self._gen_mark_monotonic) * 1000.0 - still_open = [ - (tel.execution_started_at, now_wall) - for cid, tel in self._open_tools.items() - if cid not in self._closed_tools and tel.execution_started_at is not None - ] - tool_ms = busy_ms(self._tool_spans_since_mark + still_open, self._gen_mark_wall, now_wall) - generation_ms = span_ms - tool_ms - if generation_ms < 0: - # busy_ms clips to this window and unions overlaps, so it cannot - # exceed the window's own wall span. Reaching here means the two - # clocks disagree (the span is monotonic, the tool intervals are - # wall), i.e. jitter — worth a line in the task log, because the - # clamped 0.0 below is otherwise indistinguishable from a real - # instant generation. Numbers only: no agent output is logged. - self._agent._log.debug( - "Generation window went negative (span=%.1fms tool=%.1fms); clamping to 0.", - span_ms, - tool_ms, - ) + # The span used to be read off `time.monotonic()` while these intervals + # were wall, and subtracting one from the other is the only reason this + # window could go negative — a clamp that was indistinguishable from a + # real instant generation. Both bounds now derive from `self.clock`, so + # the disagreement is unrepresentable and the branch that hid it is + # gone. + _, generation_ms = close_window(mark=self._gen_mark_wall, now=now_wall) for i, block in enumerate(self._blocks): block.sequence = i self.messages.append( AssistantMessage( started_at=self._gen_mark_wall, completed_at=now_wall, - generation_duration_ms=max(0.0, generation_ms), + generation_duration_ms=generation_ms, content_blocks=list(self._blocks), tool_use_ids=[b.tool_use_id for b in self._blocks if b.block_type == "tool_use" and b.tool_use_id], input_tokens=gen.uncached_input_tokens, @@ -1087,6 +1119,10 @@ def _flush_generation(self, gen: TokenUsage, reasoning_tokens: int) -> None: cache_read_tokens=gen.cache_read_input_tokens, reasoning_tokens=reasoning_tokens, model=self.model, + # The Step stream carries no message id, and the evalboard's + # SAME_EMISSION_GAP_MS fallback cannot split this harness's + # contiguous windows — see docs/agents/HARNESS_PARITY.md. + message_id=f"{self.turn_id}-msg-{self._assistant_turns}", ) ) self._assistant_turns += 1 @@ -1094,9 +1130,7 @@ def _flush_generation(self, gen: TokenUsage, reasoning_tokens: int) -> None: # Advance the mark ONLY after a message was actually appended. The # early return above means a no-op flush leaves the window open, so a # later real generation still measures from where it began. - self._gen_mark_monotonic = now_monotonic self._gen_mark_wall = now_wall - self._tool_spans_since_mark = [] def _agent_output(self) -> str: if self._output_parts: @@ -1138,7 +1172,7 @@ def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reaso for cid, tel in self._open_tools.items(): if cid in self._closed_tools: continue - orphan = tel.model_copy(update={"result_status": "unknown", "execution_completed_at": datetime.now()}) + orphan = tel.model_copy(update={"result_status": "unknown", "execution_completed_at": self.clock.now()}) self.emit.on_event( ToolEndEvent( task_id=self.task_id, diff --git a/src/coder_eval/agents/claude_code_agent.py b/src/coder_eval/agents/claude_code_agent.py index 7175cb7f..114f054d 100644 --- a/src/coder_eval/agents/claude_code_agent.py +++ b/src/coder_eval/agents/claude_code_agent.py @@ -76,6 +76,7 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.timing import close_window from coder_eval.utils import dump_dataclass, process_plugins @@ -243,8 +244,19 @@ def __init__( self.sequence_number = 0 self.last_assistant_message_index: int | None = None - self.last_event_monotonic: float = turn_start_time + # ONE clock basis for the window. The duration used to be a MONOTONIC + # delta while these bounds were wall, which is the split + # `timing.TurnClock` exists to eliminate: the central subtraction clips + # WALL tool spans to these WALL bounds, so a monotonic-measured + # duration would have the two disagreeing inside one subtraction — + # exactly the defect that let antigravity's window go negative. + # `turn_start_time` stays monotonic and is untouched: `duration_seconds` + # and the turn deadline read it, and a deadline must not move when the + # wall clock steps. self.last_event_wall: datetime = datetime.now() + # Re-seeded ONCE, at the first observed model output. See + # `_seed_first_generation_window`. + self.first_output_seen: bool = False # SDK ResultMessage capture. self.sdk_result_usage: dict[str, Any] | None = None @@ -311,10 +323,8 @@ def dispatch(self, message: Message) -> None: def on_assistant_message(self, message: Message) -> None: """Capture ToolUseBlocks + build the AssistantMessage telemetry record.""" - message_arrival_monotonic = time.monotonic() message_arrival_wall = datetime.now() generation_started_wall = self.last_event_wall - generation_duration_ms = (message_arrival_monotonic - self.last_event_monotonic) * 1000 current_turn_index = len(self.sdk_messages) self.assistant_turn_count += 1 @@ -424,10 +434,17 @@ def on_assistant_message(self, message: Message) -> None: out_tok = int(msg_usage.get("output_tokens", 0) or 0) self.pending_delta_output_tokens = None + # The RAW window. Tool execution comes out of it once, centrally, in + # `EventCollector.build_turn_record` — so this harness now asks the same + # helper as the other four and CE061 no longer needs its one permanent + # exception. The mark is the only harness-shaped decision left, and it + # stays here: `started` is the mark, since this stream carries no + # per-emission item start to pull the window open to. + started, raw_generation_ms = close_window(mark=generation_started_wall, now=message_arrival_wall) assistant_telemetry = AssistantMessageTelemetry( - started_at=generation_started_wall, + started_at=started, completed_at=message_arrival_wall, - generation_duration_ms=max(0.0, generation_duration_ms), + generation_duration_ms=raw_generation_ms, content_blocks=turn_content_blocks, tool_use_ids=turn_tool_use_ids, input_tokens=in_tok, @@ -450,7 +467,6 @@ def on_assistant_message(self, message: Message) -> None: self.emission_proxies_by_id.setdefault(message_id, []).append(emission_content_chars) self.last_assistant_message_index = len(self.sdk_messages) - 1 - self.last_event_monotonic = message_arrival_monotonic self.last_event_wall = message_arrival_wall def on_task_notification(self, message: Message) -> None: @@ -489,12 +505,64 @@ def on_result_message(self, message: Message) -> None: last_msg.cache_read_tokens = int(self.sdk_result_usage.get("cache_read_input_tokens", 0) or 0) last_msg.reasoning_tokens = int(self.sdk_result_usage.get("reasoning_tokens", 0) or 0) + def _seed_first_generation_window(self) -> None: + """Move the first window's mark to the first observed model output. + + ``harness_startup_ms`` is defined as the wall clock from the turn + starting until the harness first observed model output, and that instant + is also where the first generation window opens — which is what keeps + the head and the generation disjoint so the four-bucket identity still + closes. + + Without this the mark is stamped in ``__init__``, BEFORE + ``AgentStartEvent`` is emitted, so the head is a small negative that + ``decompose_turn`` clamps to ``0.0`` — a clamped inversion published as + "measured, and instant", which is the exact confusion CE058 exists to + prevent everywhere else. Everything the CLI spent booting, resolving a + provider and reaching its first token was booked as msg0's generation + instead: ~3.6 s per turn on this harness, inflating every generation + figure, the Generation split and the 10 s slow-generation bar. + + The old rejection rested on this harness running the model in-process. + It does not: ``claude-agent-sdk`` spawns the ``claude`` CLI over + ``anyio.open_process`` and ``_pump_messages`` calls ``query()`` once + per ``communicate()`` — a fresh CLI per turn, the same shape as codex, + opencode and pi. + + ONCE PER TURN, and that is the whole contract. ``message_start`` arrives + for every API call in the turn; re-seeding on each would stop the + windows tiling and drop the gap before the next emission — a tool result + landing, then the next request going out — into no bucket at all, which + is the defect pi shipped with. The flag needs no reset: a fresh + ``_ClaudeTurnState`` is built per ``communicate()``, so it is + per-attempt by construction. If a future harness reuses a turn state, + the reset belongs there and not here. + + A turn with no ``message_start`` — a mocked ``query()``, a crash before + the first event — never calls this, keeps the turn-entry mark and clamps + to ``0.0`` exactly as before. That is the correct degradation rather + than a gap. + + One route to it is OPERATOR-REACHABLE and worth knowing: this harness + sets ``include_partial_messages=True`` BEFORE spreading + ``**self.config.sdk_options``, so + ``-D agent.sdk_options.include_partial_messages=false`` turns the raw + stream off, and with it this re-seed — the head silently returns to the + clamped ``0.0`` it used to publish. Nothing warns; the degradation is + safe but the number changes meaning. + """ + if self.first_output_seen: + return + self.first_output_seen = True + self.last_event_wall = datetime.now() + def on_stream_event(self, message: Message) -> None: """Recover cumulative output_tokens from raw ``message_start`` / ``message_delta`` stream events (handles both sub-cases internally).""" evt: dict[str, Any] = getattr(message, "event", None) or {} evt_type = evt.get("type") if evt_type == "message_start": + self._seed_first_generation_window() mid = (evt.get("message") or {}).get("id") self.current_stream_message_id = mid if isinstance(mid, str) else None elif evt_type == "message_delta": @@ -517,7 +585,6 @@ def on_user_message(self, message: Message) -> None: """Process tool results (and a sub-agent's terminal generation) from a tool-result UserMessage. The sub-agent message is appended BEFORE the tool-result loop — its position in ``sdk_messages`` is observable.""" - self.last_event_monotonic = time.monotonic() self.last_event_wall = datetime.now() sub_msg = self._agent._synthesize_subagent_terminal_message(message, self.sdk_model_used) diff --git a/src/coder_eval/agents/codex_agent.py b/src/coder_eval/agents/codex_agent.py index b8ffea86..cb438f68 100644 --- a/src/coder_eval/agents/codex_agent.py +++ b/src/coder_eval/agents/codex_agent.py @@ -17,7 +17,6 @@ from coder_eval.agent import Agent, AgentState from coder_eval.agents._logging import PrefixedAdapter, log_raw_sdk_event -from coder_eval.agents._timing import busy_ms from coder_eval.agents.registry import AgentRegistry from coder_eval.agents.watchdog import ThreadedWatchdog from coder_eval.config import settings @@ -54,6 +53,7 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.timing import close_window from coder_eval.utils import expand_env_vars @@ -473,47 +473,22 @@ def _flush_message(self, last: Any) -> None: # attributed to nothing. Across that turn only 15.8% of the 17 s wall # clock was accounted for. Tiling matches Antigravity and claude-code, # and is what lets Sum(generation) + Sum(tool) reconcile to the turn. - # - # min() is defensive: a stamp that goes backwards must never push the - # window start PAST the first item and invert the span. - window_start_ms = self.gen_mark_ms if self.gen_mark_ms is not None else self.open_start_ms - if window_start_ms is not None and self.open_start_ms is not None: - window_start_ms = min(window_start_ms, self.open_start_ms) + mark_ms = self.gen_mark_ms if self.gen_mark_ms is not None else self.open_start_ms window_end_ms = self.open_end_ms if self.open_end_ms is not None else self.open_start_ms - started = _ms_to_dt(window_start_ms) + mark = _ms_to_dt(mark_ms) completed = _ms_to_dt(window_end_ms) - # The window is extended to the LAST item's completion, so any + # The RAW window. It is extended to the LAST item's completion, so a # generation containing a tool call already CONTAINS that tool's - # execution. Publishing the raw span as generation time double-counts - # it against the tool's own duration_ms: a tool-only emission reported - # 250ms of "generation" for a 250ms `echo hi`, and the task page's - # Generation + Tool exec then exceeded the wall clock they must - # reconcile to. - # - # Same treatment, and the same shared helper, as Antigravity: subtract - # the UNION of the tool intervals clipped to this window. A sum would - # over-subtract wherever they overlap, which Codex produces natively - # via concurrent collab agents. - # - # Closed intervals, plus any call still OPEN at this flush bounded at - # the window end. Excluding the open ones publishes the part of a - # straddling call that ran inside this window as generation while the - # call's own duration_ms counts it again — harmless while the windows - # were too narrow to overlap a tool, and a live double-count now that - # they tile. Antigravity hit exactly that and broke the invariant by - # 0.26 ms; the fix travels with the tiling that makes it reachable. - tool_spans = [ - (c.execution_started_at, c.execution_completed_at) - for c in self.commands - if c.execution_started_at is not None and c.execution_completed_at is not None - ] - tool_spans += [ - (t.execution_started_at, completed) - for t in self.open_tools.values() - if t.execution_started_at is not None and t.execution_started_at < completed - ] - span_ms = max((completed - started).total_seconds() * 1000.0, 0.0) - gen_ms = max(0.0, span_ms - busy_ms(tool_spans, started, completed)) + # execution — but taking it back out is no longer this reducer's job. + # `EventCollector.subtract_tool_time` does it for all five, which is + # also what makes the sub-message split below safe: the two specs share + # these bounds, so the collector groups them and subtracts the overlap + # ONCE rather than once per part. + started, gen_ms = close_window( + mark=mark, + now=completed, + item_start=_ms_to_dt(self.open_start_ms) if self.open_start_ms is not None else None, + ) message_id = f"{self.turn_id}-msg-{self.gen_index}" # Output split: reasoning portion to the thinking row, the remainder to diff --git a/src/coder_eval/agents/opencode_agent.py b/src/coder_eval/agents/opencode_agent.py index 41904253..c7bd6d56 100644 --- a/src/coder_eval/agents/opencode_agent.py +++ b/src/coder_eval/agents/opencode_agent.py @@ -43,7 +43,6 @@ from typing import Any, ClassVar, Literal, NoReturn from coder_eval.agent import Agent -from coder_eval.agents._timing import busy_ms from coder_eval.errors import AgentCrashError, TurnTimeoutError from coder_eval.isolation.docker_runner import STDOUT_LINE_LIMIT_BYTES from coder_eval.models import ( @@ -77,6 +76,7 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.timing import close_window from ._skills import _plugin_skill_dirs from .registry import AgentRegistry @@ -325,14 +325,6 @@ def __init__(self, *, task_id: str, iteration: int, user_input: str, model: str self.gen_mark: datetime | None = None self.step_text_parts: list[str] = [] self.step_tool_ids: list[str] = [] - # Execution intervals of tools that CLOSED inside the open - # generation window. Every tool call runs INSIDE the window, so - # publishing the raw span as generation time counts the same - # milliseconds twice — once here and once as the tool's own - # duration_ms. Intervals, not a running total: they overlap - # whenever the harness runs tools concurrently, and only their - # union may be subtracted (agents/_timing.py::busy_ms). - self.step_tool_spans: list[tuple[datetime, datetime]] = [] # callID -> (telemetry, started_at) for tools awaiting a result. self.open_tools: dict[str, CommandTelemetry] = {} @@ -370,7 +362,13 @@ def on_step_start(self, part: dict[str, Any]) -> None: self.step_started_at = datetime.now() self.step_text_parts = [] self.step_tool_ids = [] - self.step_tool_spans = [] + # There is no per-step span list to reset here any more, and that whole + # class of defect is gone with it: `EventCollector.subtract_tool_time` + # sees every span at once and clips each to the window it overlaps, so + # a call closing in the gap before this `step_start` needs nobody to + # remember it. The reset rule that used to live here was wrong once + # (clearing at `step_start` wiped the span before `step_finish` could + # subtract it — a 100% overstatement of that window). self.emit( TurnStartEvent( task_id=self.task_id, @@ -488,10 +486,6 @@ def _close_tool( telemetry.execution_completed_at = completed if telemetry.execution_started_at is not None: telemetry.duration_ms = (completed - telemetry.execution_started_at).total_seconds() * 1000 - # This tool ran inside the open generation window, so its time is - # not model time. Only a RESOLVED tool contributes: one force-closed - # without a result was never timed. - self.step_tool_spans.append((telemetry.execution_started_at, completed)) telemetry.result_status = _RESULT_STATUS[status] # Stored untruncated by design (sub-agent returns must survive whole). telemetry.result_summary = summary @@ -701,9 +695,6 @@ def on_step_finish(self, part: dict[str, Any]) -> None: completed = datetime.now() step_start = self.step_started_at or completed - # Tile from the previous step's finish; min() keeps a clock that went - # backwards from inverting the span. - started = min(self.gen_mark, step_start) if self.gen_mark is not None else step_start blocks: list[ContentBlock] = [] step_text = "".join(self.step_text_parts) if step_text: @@ -711,24 +702,19 @@ def on_step_finish(self, part: dict[str, Any]) -> None: for i, tool_id in enumerate(self.step_tool_ids, start=len(blocks)): blocks.append(ContentBlock(block_type="tool_use", sequence=i, tool_use_id=tool_id)) - # A call still OPEN at this boundary counts too, bounded at `completed`. - # Subtracting only CLOSED intervals publishes the part of a straddling - # call that ran inside this window as generation, while the call's own - # duration_ms counts it again — a live double-count now that the windows - # tile contiguously from `gen_mark`. No double subtraction: when the call - # later closes, `_finish_tool` appends its full interval to the NEXT - # window's list, where busy_ms clips it to the post-boundary remainder. - spans = self.step_tool_spans + [ - (t.execution_started_at, completed) for t in self.open_tools.values() if t.execution_started_at is not None - ] + # Tile from the previous step's finish. The RAW window only — + # `EventCollector.subtract_tool_time` takes the tool union back out of + # it, once, for every harness. + started, generation_ms = close_window( + mark=self.gen_mark if self.gen_mark is not None else step_start, + now=completed, + item_start=step_start, + ) self.messages.append( AssistantMessage( started_at=started, completed_at=completed, - generation_duration_ms=max( - 0.0, - (completed - started).total_seconds() * 1000 - busy_ms(spans, started, completed), - ), + generation_duration_ms=generation_ms, content_blocks=blocks, tool_use_ids=list(self.step_tool_ids), input_tokens=step_in, @@ -744,8 +730,18 @@ def on_step_finish(self, part: dict[str, Any]) -> None: # A message was appended, so the next window starts where this one # ended. Only `step_finish` advances the mark: a step that never # finished published nothing, so tiling past it would attribute its - # time to whichever step finishes next. + # time to whichever step finishes next. There is no span list to clear + # alongside it any more — see `on_step_start`. self.gen_mark = completed + # And so is this step's own start stamp, because it has now been SPENT. + # It is passed to `close_window` as `item_start`, whose `min()` pulls + # the window open to cover it; left in place, a second `step_finish` + # with no intervening `step_start` would reopen the next window back at + # the previous step's start and publish that whole span a second time. + # The `min()` still defends a genuinely OPEN step against a backwards + # clock, which is what it is for — this reducer's stamps are raw + # `datetime.now()` and are not on a `TurnClock`. + self.step_started_at = None self.emit( TurnEndEvent( task_id=self.task_id, diff --git a/src/coder_eval/agents/pi_agent.py b/src/coder_eval/agents/pi_agent.py index f7ae7d6d..f53af2ca 100644 --- a/src/coder_eval/agents/pi_agent.py +++ b/src/coder_eval/agents/pi_agent.py @@ -77,7 +77,6 @@ from coder_eval.agent import Agent from coder_eval.agents._skills import _plugin_skill_dirs # shared plugin->skills resolver -from coder_eval.agents._timing import busy_ms from coder_eval.errors import AgentCrashError, TurnTimeoutError from coder_eval.isolation.docker_runner import STDOUT_LINE_LIMIT_BYTES from coder_eval.models import ( @@ -110,6 +109,7 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.timing import TurnClock, close_window from .registry import AgentRegistry @@ -262,12 +262,27 @@ class _PiTurnState: to force-close orphans when a turn dies mid-flight. """ - def __init__(self, *, task_id: str, iteration: int, user_input: str, model: str | None) -> None: + def __init__( + self, + *, + task_id: str, + iteration: int, + user_input: str, + model: str | None, + clock: TurnClock | None = None, + ) -> None: self.task_id = task_id self.iteration = iteration self.user_input = user_input self.model = model + # ONE clock per turn, and every wall stamp below derives from it, so + # the tool spans and the window bounds they are subtracted from cannot + # end up on different bases. Injectable so a test can supply a fake + # rather than monkeypatching this module's `datetime` global — which a + # derived stamp would silently escape, leaving the test passing against + # the real clock instead of failing. + self.clock = clock or TurnClock() self.started_at = time.monotonic() self.thread_id: str | None = None @@ -288,14 +303,16 @@ def __init__(self, *, task_id: str, iteration: int, user_input: str, model: str self.turn_started_at: datetime | None = None self.turn_text_parts: list[str] = [] self.turn_tool_ids: list[str] = [] - # Execution intervals of tools that CLOSED inside the open - # generation window. Every tool call runs INSIDE the window, so - # publishing the raw span as generation time counts the same - # milliseconds twice — once here and once as the tool's own - # duration_ms. Intervals, not a running total: they overlap - # whenever the harness runs tools concurrently, and only their - # union may be subtracted (agents/_timing.py::busy_ms). - self.turn_tool_spans: list[tuple[datetime, datetime]] = [] + # Where the NEXT generation window starts: the previous turn's end. + # Pi was the only harness measuring from its own `turn_start`, so the + # wall clock between one `turn_end` and the next `turn_start` — the + # model time that PRODUCED that turn — fell into no bucket at all. + # + # None until the first turn finishes, and deliberately so: the first + # window keeps its own `turn_start`, because everything before it is + # CLI process spawn, not model time. Same shape as OpenCode's + # `gen_mark` and Codex's `gen_mark_ms`. + self.gen_mark: datetime | None = None # toolCallId -> telemetry for tools awaiting a result. self.open_tools: dict[str, CommandTelemetry] = {} @@ -353,10 +370,13 @@ def on_turn_start(self) -> None: self.turn_count += 1 self.turn_open = True self.turn_id = f"turn_{self.turn_count}" - self.turn_started_at = datetime.now() + self.turn_started_at = self.clock.now() self.turn_text_parts = [] self.turn_tool_ids = [] - self.turn_tool_spans = [] + # No per-turn span list to reset here any more — see the identical note + # in `opencode_agent.on_step_start`. The collector subtracts from final + # bounds with every span known, so nothing has to remember a call that + # closed in the gap before this `turn_start`. self.emit( TurnStartEvent( task_id=self.task_id, @@ -387,7 +407,7 @@ def on_tool_execution_start(self, obj: dict[str, Any]) -> None: tool_name = _TOOL_NAME_MAP.get(raw_tool.lower(), raw_tool) args = obj.get("args") params = args if isinstance(args, dict) else {} - started = datetime.now() + started = self.clock.now() telemetry = CommandTelemetry( tool_name=tool_name, tool_id=call_id, @@ -434,17 +454,13 @@ def _close_tool( tool_name="unknown", tool_id=call_id, assistant_turn_index=self.turn_count, - timestamp=datetime.now(), + timestamp=self.clock.now(), sequence_number=self.sequence, ) - completed = datetime.now() + completed = self.clock.now() telemetry.execution_completed_at = completed if telemetry.execution_started_at is not None: telemetry.duration_ms = (completed - telemetry.execution_started_at).total_seconds() * 1000 - # This tool ran inside the open generation window, so its time is - # not model time. Only a RESOLVED tool contributes: one force-closed - # without a result was never timed. - self.turn_tool_spans.append((telemetry.execution_started_at, completed)) telemetry.result_status = _RESULT_STATUS[status] # Stored untruncated by design (sub-agent returns must survive whole). telemetry.result_summary = summary @@ -573,8 +589,7 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: else: self.error_message = None - started = self.turn_started_at or datetime.now() - completed = datetime.now() + completed = self.clock.now() blocks: list[ContentBlock] = [] turn_text = "".join(self.turn_text_parts) if turn_text: @@ -582,23 +597,20 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: for i, tool_id in enumerate(self.turn_tool_ids, start=len(blocks)): blocks.append(ContentBlock(block_type="tool_use", sequence=i, tool_use_id=tool_id)) - # A call still OPEN at this boundary counts too, bounded at `completed`. - # Subtracting only CLOSED intervals publishes the part of a straddling - # call that ran inside this window as generation, while the call's own - # duration_ms counts it again. No double subtraction: when the call later - # closes, `_finish_tool` appends its full interval to the NEXT turn's - # list, where busy_ms clips it to the post-boundary remainder. - spans = self.turn_tool_spans + [ - (t.execution_started_at, completed) for t in self.open_tools.values() if t.execution_started_at is not None - ] + # Tile from the previous turn's end. The RAW window only — + # `EventCollector.subtract_tool_time` takes the tool union back out of + # it, once, for every harness. + turn_start = self.turn_started_at if self.turn_started_at is not None else completed + started, generation_ms = close_window( + mark=self.gen_mark if self.gen_mark is not None else turn_start, + now=completed, + item_start=turn_start, + ) self.messages.append( AssistantMessage( started_at=started, completed_at=completed, - generation_duration_ms=max( - 0.0, - (completed - started).total_seconds() * 1000 - busy_ms(spans, started, completed), - ), + generation_duration_ms=generation_ms, content_blocks=blocks, tool_use_ids=list(self.turn_tool_ids), input_tokens=step_in, @@ -611,6 +623,20 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: message_id=str(message.get("responseId") or "") or None, ) ) + # A message was appended, so the next window starts where this one + # ended. Only a finished turn advances the mark: one that never + # finished published nothing, so tiling past it would attribute its + # time to whichever turn finishes next. There is no span list to clear + # alongside it any more — see `on_turn_start`. + self.gen_mark = completed + # And so is this turn's own start stamp, because it has now been SPENT. + # It is passed to `close_window` as `item_start`, whose `min()` pulls + # the window open to cover it; left in place, a second `turn_end` with + # no intervening `turn_start` — a duplicate or replayed line, which + # this reducer promises to survive — would reopen the next window back + # at the previous turn's start and publish that whole span a second + # time. Reproduced: 3000 ms of generation for a 2000 ms turn. + self.turn_started_at = None self.emit( TurnEndEvent( task_id=self.task_id, @@ -991,6 +1017,10 @@ def emit(event: StreamEvent) -> None: ) ) + # Deadlines stay on `time.monotonic()` and are deliberately NOT routed + # through the turn clock: a deadline must not move when the wall clock + # steps. `TurnClock` exists to give the RECORDED stamps one basis; this + # is the one place a raw monotonic reading is the right answer. deadline = None if timeout is None else time.monotonic() + timeout stopped_early = False stderr_drain: asyncio.Future[bytes] | None = None diff --git a/src/coder_eval/models/results.py b/src/coder_eval/models/results.py index a12bddd2..9b8f152e 100644 --- a/src/coder_eval/models/results.py +++ b/src/coder_eval/models/results.py @@ -325,6 +325,34 @@ class TurnRecord(BaseModel): ) timestamp: datetime = Field(default_factory=datetime.now, description="When this turn occurred") duration_seconds: float = Field(default=0.0, description="How long this turn took") + harness_startup_ms: float | None = Field( + default=None, + description=( + "Wall milliseconds from the agent turn starting until the harness first observed " + "MODEL OUTPUT — one definition on all five, and the same instant at which the harness " + "opens its first generation window, which is what keeps the two buckets disjoint. " + "Measured between AGENT EVENT stamps, not from timestamp/duration_seconds above, " + "which are orchestrator-level and a slightly different clock, so a consumer " + "recomputing this from those will get a near-but-not-equal number. Only its " + "COMPOSITION differs per harness, and that difference is a real property rather than " + "a measurement artifact: a harness that spawns its process PER TURN (claude-code, " + "codex, opencode, pi) fuses that boot, provider resolution, dispatch and TTFT here, " + "while one that spawns it once at startup and holds it across turns (antigravity) has " + "no boot inside the turn to fuse in. It is deliberately not decomposed further " + "— no stream carries a marker between those parts. " + "See docs/agents/HARNESS_PARITY.md. " + "None when the turn produced no assistant message — never 0.0, which would mean " + "'measured, and instant'." + ), + ) + harness_teardown_ms: float | None = Field( + default=None, + description=( + "Wall milliseconds between the last generation window closing and the agent turn " + "ending: SDK/CLI finalization, result assembly and process teardown. Same clock " + "caveat as harness_startup_ms. None when the turn produced no assistant message." + ), + ) token_usage: TokenUsage | None = Field( default=None, description="Token usage for this turn (if available from agent SDK)" ) diff --git a/src/coder_eval/models/telemetry.py b/src/coder_eval/models/telemetry.py index 15a18430..7079b2da 100644 --- a/src/coder_eval/models/telemetry.py +++ b/src/coder_eval/models/telemetry.py @@ -227,11 +227,25 @@ class AssistantMessage(BaseModel): description=( "Model-generation time for this emission, in milliseconds. None when the harness " "surfaced the message with no measurable window (a rollout rebuild, or a sub-agent " - "generation delivered as a tool result). Equals completed_at - started_at only when " - "no tool execution closed inside the window; a harness whose stream interleaves tool " - "calls into one generation (Antigravity) subtracts those. The property this field exists " - "to make true — once every harness records a real window — is: " - "sum(generation_duration_ms) + sum(command duration_ms) ~= turn duration_seconds. " + "generation delivered as a tool result). " + "WRITTEN BY THE COLLECTOR, not by the agent: a reducer publishes the RAW window it " + "measured, and streaming/collector.py::subtract_tool_time takes the UNION of the " + "main-thread tool intervals back out of it, once, for every harness. So this equals " + "completed_at - started_at only when no tool execution overlapped the window, and a " + "reader of an agent's own AssistantMessage(...) call is NOT looking at the published " + "value. Messages sharing one pair of bounds (Codex splits a window into thinking and " + "action sub-messages) are one group: the overlap comes out once and is re-apportioned " + "across them, so the parts still sum to the group's total. " + "The property this field exists " + "to make true — once every harness records a real window — is the FOUR-bucket identity: " + "sum(generation_duration_ms) + UNION(command execution intervals) " + "+ TurnRecord.harness_startup_ms + TurnRecord.harness_teardown_ms ~= turn duration_seconds. " + "The tool term is the union and not the sum for the same reason the subtraction above " + "uses one (timing.py::busy_ms): concurrent tool calls otherwise book their overlap twice. " + "The last two are the turn's " + "head and tail, which no message can carry because they are the wall clock OUTSIDE every " + "generation window; without them the identity holds only on a harness with nothing to " + "boot or dispatch before its first model output. " "Per-harness status is in docs/agents/HARNESS_PARITY.md; do not assume it holds " "for a harness that table does not yet claim it for." ), @@ -276,9 +290,20 @@ class AssistantMessage(BaseModel): message_id: str | None = Field( default=None, description=( - "Anthropic API message_id. Multiple AssistantMessage records can share this id " - "when the Claude Code CLI splits one API response into per-block-kind events. " - "Downstream tooling can group by this id to recover one logical generation." + "Identity of the generation this emission belongs to. Several AssistantMessage " + "records can share one id, and downstream tooling groups by it to recover a single " + "logical generation — the evalboard's timeline renders one row per group. " + "ALL FIVE backends write it, by three different schemes. Passed THROUGH from the " + "harness: claude-code (a real Anthropic API message_id, which repeats because the CLI " + "splits one API response into per-block-kind events — the case this field was named " + "for), and opencode and pi (the CLI's own id, so they can legitimately leave this None " + "when the payload omits it). SYNTHESIZED: codex, which deliberately repeats one id " + "across the sub-messages of a generation, and antigravity, which mints a distinct one " + "per generation because its Step stream carries none. claude-code also synthesizes in " + "ONE place — the sub-agent terminal message, which is delivered as a tool result and " + "never streamed. See docs/agents/HARNESS_PARITY.md for the per-harness row — and note " + "the id is a WITHIN-TURN identity only: it repeats across retry attempts of one turn " + "on every synthesizing harness." ), ) diff --git a/src/coder_eval/reports_html.py b/src/coder_eval/reports_html.py index 7f8fcade..c0c1c8eb 100644 --- a/src/coder_eval/reports_html.py +++ b/src/coder_eval/reports_html.py @@ -20,7 +20,7 @@ from coder_eval.models import FinalStatus, eval_result_total_cost, sum_costs from .reports import early_stop_gate_note -from .reports_stats import format_score, is_env_table_key +from .reports_stats import format_score, is_env_table_key, turn_time_buckets if TYPE_CHECKING: @@ -932,8 +932,20 @@ def _render_token_usage(result: EvaluationResult) -> str: """ +def _format_signed_ms(ms: float | None) -> str: + """Like `_format_ms`, but keeps a NEGATIVE residual visible and signed. + + A negative Unaccounted is real and means generation and tool execution + overlapped, so it is rendered rather than clamped — the evalboard does the + same. Clamping would turn a measurable inconsistency into a clean zero. + """ + if ms is None: + return "—" + return f"-{_format_ms(-ms)}" if ms < 0 else _format_ms(ms) + + def _render_generation_metrics(result: EvaluationResult) -> str: - """Render Generation Metrics — latency, turns.""" + """Render Generation Metrics — latency, turns, and the four wall-clock buckets.""" from .reports import count_partials_by_outcome, group_consecutive_by_iteration turns = result.iterations or [] @@ -951,6 +963,22 @@ def _render_generation_metrics(result: EvaluationResult) -> str: f'
Crashed Partials
' f'
{_esc(breakdown)}
' ) + # The four wall-clock buckets. The arithmetic is in reports_stats; this + # only formats it. An unmeasured bucket renders as an em dash, never 0ms — + # a run predating the head/tail capture measured nothing, and a zero would + # claim it measured instantly (CE058). + buckets = turn_time_buckets(result) + startup = _format_ms(buckets.startup_ms) + generation = _format_ms(buckets.generation_ms) + tool_exec = _format_ms(buckets.tool_ms) + teardown = _format_ms(buckets.teardown_ms) + unaccounted = _format_signed_ms(buckets.unaccounted_ms) + unaccounted_title = _esc( + "the task's wall clock minus the four buckets. Measured against the whole task, so it " + + "legitimately includes sandbox setup and grading — it is LARGER than the per-turn residual " + + "scripts/timing/decompose_run.py reports, and the two are not comparable. Negative means " + + "generation and tool execution overlapped." + ) return f"""

Generation Metrics

@@ -961,6 +989,15 @@ def _render_generation_metrics(result: EvaluationResult) -> str:
Avg Turn Latency
{avg_latency}
{crashed_stat}
+
+
Startup
{startup}
+
Generation
{generation}
+
Tool exec
{tool_exec}
+
Teardown
{teardown}
+
+
Unaccounted (incl. setup + grading)
{unaccounted}
+
+
""" diff --git a/src/coder_eval/reports_stats.py b/src/coder_eval/reports_stats.py index ba9f9de3..5bba846f 100644 --- a/src/coder_eval/reports_stats.py +++ b/src/coder_eval/reports_stats.py @@ -11,10 +11,20 @@ import math import random import statistics as _stats +from collections.abc import Iterable from pathlib import Path from typing import NamedTuple -from coder_eval.models import EvaluationResult, ExperimentResult, ExperimentVariant, TaskExperimentSummary +from coder_eval.models import ( + AssistantMessage, + EvaluationResult, + ExperimentResult, + ExperimentVariant, + TaskExperimentSummary, + TurnRecord, +) +from coder_eval.streaming.collector import main_thread_tool_spans +from coder_eval.timing import union_ms from .path_utils import TASK_JSON_FILENAME @@ -336,6 +346,109 @@ def format_score(score: float | None) -> str: return UNGRADED_SCORE_TEXT if score is None else f"{score:.3f}" +class TurnTimeBuckets(NamedTuple): + """The four wall-clock buckets of a whole run, plus what they leave over. + + Each is ``None`` when NOTHING in the run measured it — a run recorded before + the head and tail were captured has no startup at all, a run that recorded + no bounded tool span has no tool total, and a run with no duration has no + residual. Rendering any of those as ``0ms`` claims a measurement nobody + took (CE058, and the reason the evalboard's ``sumMeasured`` returns + ``null``). A MEASURED zero stays ``0.0`` and renders as ``0ms``. + + DISPLAY AND ARITHMETIC DIFFER HERE, on purpose. An unmeasured bucket renders + as a dash and counts as ``0.0`` toward ``unaccounted``, so the missing time + surfaces as residual rather than vanishing. That is the rule + ``scripts/timing/decompose_run.py::_turn_buckets`` already applies, and + keeping the two the same is what lets a reader compare them. + """ + + startup_ms: float | None + generation_ms: float | None + tool_ms: float | None + teardown_ms: float | None + unaccounted_ms: float | None + + +def turn_time_buckets(result: EvaluationResult) -> TurnTimeBuckets: + """Sum the four timing buckets across a run's turns, and the residual. + + The arithmetic lives HERE rather than in the renderer because this module is + the designated home for shared report statistics: the evalboard, the + markdown report and the HTML report must not each grow their own version. + ``reports_html`` formats what this returns and decides nothing. + + ``unaccounted`` is measured against ``EvaluationResult.duration_seconds`` — + the TASK's wall clock, which is what the card's existing Total Latency uses + and what the evalboard's own Unaccounted cell uses. It therefore legitimately + contains sandbox setup and grading, and is LARGER than the per-turn residual + ``decompose_run.py`` reports. The two are not comparable and the label says + so. + """ + turns = result.iterations or [] + startup = _sum_measured(t.harness_startup_ms for t in turns) + teardown = _sum_measured(t.harness_teardown_ms for t in turns) + # MAIN THREAD ONLY, the same filter the collector and the evalboard apply: + # a sub-agent's generations bubble into the same stream, and the spawning + # Agent call's own interval already spans them. + generation = _sum_measured( + m.generation_duration_ms + for t in turns + for m in t.messages + if isinstance(m, AssistantMessage) and m.parent_tool_use_id is None + ) + # `None` only when NO turn recorded a bounded tool span. A turn that ran + # tools and timed none is indistinguishable from a turn that ran none, so + # the presence of a SPAN — not the presence of a turn — is what decides + # measured-versus-not. `_sum_measured` over a list of plain floats could + # never return None, which made this read `0ms` ("measured and instant") + # for a run nobody timed. + per_turn = [_turn_tool_union_ms(t) for t in turns] + tool = _sum_measured(per_turn) if any(ms is not None for ms in per_turn) else None + + # `duration_seconds` is a non-optional float defaulting to 0.0, so there is + # no None arm to write — but a 0.0 duration is a run that was never timed, + # and subtracting real buckets from it renders a fabricated negative + # residual. The evalboard keeps that null for the same reason; so do we. + unaccounted = ( + result.duration_seconds * 1000.0 - (startup or 0.0) - (generation or 0.0) - (tool or 0.0) - (teardown or 0.0) + if result.duration_seconds > 0.0 + else None + ) + return TurnTimeBuckets(startup, generation, tool, teardown, unaccounted) + + +def _sum_measured(values: Iterable[float | None]) -> float | None: + """Sum what was measured, or ``None`` when nothing was. + + The Python twin of the evalboard's ``sumMeasured``: a run with no measured + value anywhere returns ``None`` (never measured), while a run that measured + a genuine zero returns ``0.0``. + """ + total: float | None = None + for value in values: + if isinstance(value, (int, float)) and math.isfinite(value): + total = (total or 0.0) + value + return total + + +def _turn_tool_union_ms(turn: TurnRecord) -> float | None: + """One turn's tool execution — the UNION of its main-thread command spans. + + ``None`` when the turn recorded no bounded span at all, which is different + from a turn whose tools took no time. Never the sum: concurrent calls + occupy the wall clock once, and summing them books the overlap twice. + + The span SELECTION is ``streaming.collector.main_thread_tool_spans``, not a + copy of it. That rule (which commands count, and the sub-agent exclusion) + is what the collector measures the generation subtraction and the head and + tail against, so a second typed implementation here is how two surfaces + come to publish two different tool totals for one run. + """ + spans = main_thread_tool_spans(turn.messages, turn.commands) + return union_ms(spans) if spans else None + + def collect_variant_series(result: ExperimentResult) -> dict[str, VariantSeries]: """Per-variant (scores, durations, tokens, assistant-turns) series, keyed by variant id. diff --git a/src/coder_eval/streaming/collector.py b/src/coder_eval/streaming/collector.py index ebac3ee9..9fa5bfeb 100644 --- a/src/coder_eval/streaming/collector.py +++ b/src/coder_eval/streaming/collector.py @@ -22,6 +22,9 @@ from __future__ import annotations +from collections.abc import Iterable +from datetime import datetime + from coder_eval.models import ( AssistantMessage, CommandTelemetry, @@ -37,6 +40,134 @@ ToolEndEvent, TurnStartEvent, ) +from coder_eval.timing import busy_ms, decompose_turn + + +def main_thread_tool_spans( + messages: Iterable[TranscriptMessage], commands: Iterable[CommandTelemetry] +) -> list[tuple[datetime, datetime]]: + """Bounded execution intervals of the MAIN THREAD's tool calls. + + The span set the generation subtraction, the head and the tail are all + measured against, so they cannot disagree about which calls exist. Shared + with ``reports_stats.turn_time_buckets``, which answers the same question + about a finished ``TurnRecord`` — a second typed copy of this rule is how + two report surfaces come to publish two different tool totals for one run. + (``scripts/timing/decompose_run.py`` keeps its own, over raw ``task.json`` + dicts rather than models; that is the sanctioned third reader, and + ``tests/test_timing_close_window.py::TestTheThreeToolUnionsAgree`` pins all + three together.) + + Sub-agent tools are excluded, and that used to be the gap: ``_overhead_ms`` + filtered its GENERATIONS to the main thread and then passed EVERY command, + so its claim to keep all four buckets measuring one thread was true only by + luck. It held because a child nests inside the parent Agent call, whose own + interval the union already covers — but Codex's recovered child tools carry + the CHILD's clock, so nothing made it true by construction. The evalboard's + twin (``toolExecutionMs``) does filter, so the two agreed by accident. + + A sub-agent's tool ids are reachable only through the messages that own + them: a child generation carries ``parent_tool_use_id``, and its + ``tool_use_ids`` are the calls it made. + + An inverted pair (``end`` before ``start``) is dropped here rather than + passed on. ``busy_ms`` would discard it anyway, but ``timing.union_ms`` + documents that it does NOT filter them because its callers do — so this is + the caller keeping that true. + """ + sub_agent_tool_ids = { + tool_id + for m in messages + if isinstance(m, AssistantMessage) and m.parent_tool_use_id is not None + for tool_id in m.tool_use_ids + } + return [ + (c.execution_started_at, c.execution_completed_at) + for c in commands + if c.execution_started_at is not None + and c.execution_completed_at is not None + and c.execution_completed_at >= c.execution_started_at + and c.tool_id not in sub_agent_tool_ids + ] + + +def subtract_tool_time( + messages: list[TranscriptMessage], + spans: list[tuple[datetime, datetime]], +) -> list[TranscriptMessage]: + """Take tool execution back out of the generation windows it overlapped. + + THE one place this happens. Five reducers used to do it themselves — four + through ``close_window`` as they flushed, claude-code once at finalization — + while the head and tail were already computed centrally, right here. That + asymmetry was the complexity, and every timing defect this branch fixed + lived in the per-reducer bookkeeping around the subtraction rather than in + the subtraction itself: when to reset a span list, when to clear a start + stamp, when to advance a mark. A reducer now publishes the RAW window and + keeps only the genuinely harness-shaped decision, which is where its window + opens. + + NON-MUTATING, and the reason is aliasing rather than repeated calls. Every + agent builds its terminal event as ``AgentEndEvent(messages=list(...))`` — + that copies the LIST, not the message objects — so writing in place would + reach back into the agent's own live state from the collector, which is + exactly the layering "the collector is the sole capture seam" exists to + prevent. ``model_copy`` keeps it one-directional. It is also unconditionally + safe for any caller that builds a record twice: ``EarlyStopWatcher`` holds + one collector across a turn's tool-call rounds and calls + ``build_turn_record`` on every one. + + GROUPED BY IDENTICAL BOUNDS, not by ``message_id``. Codex splits one window + across two sub-messages (thinking and action) that share ``started_at`` and + ``completed_at`` and divide the window by output-token share; subtracting + the group's overlap from each part separately would subtract it twice and + stop the parts summing to the window. Bounds identity covers that, and it + also covers OpenCode and Pi, which can legitimately carry + ``message_id is None`` — so keying on the id would silently collapse every + id-less message of a turn into one group. + + MAIN THREAD ONLY. A sub-agent generation (``parent_tool_use_id`` set) is + skipped: its own tools are not in this span set, and the Agent call that + spawned it already covers its whole run. + + A ``generation_duration_ms`` of ``None`` means no window was ever measured + (codex's rollout rebuild, claude's synthesized sub-agent terminal), so there + is nothing to subtract from and it passes through untouched — never + coerced to ``0.0`` (CE058). Every non-``AssistantMessage`` entry — a + simulation ``UserMessage``, the appended ``ReconciliationMessage`` — passes + through by identity. + + A window entirely covered by tool execution reaches ``0.0``, and that is a + measurement rather than an absence. + """ + # (index, raw window ms) per group. The raw value is captured HERE, where + # the message is already narrowed to AssistantMessage, so the apportioning + # loop below needs no second narrowing. + groups: dict[tuple[datetime, datetime], list[tuple[int, float]]] = {} + for index, message in enumerate(messages): + if not isinstance(message, AssistantMessage): + continue + raw = message.generation_duration_ms + if raw is None or message.parent_tool_use_id is not None: + continue + groups.setdefault((message.started_at, message.completed_at), []).append((index, raw)) + + out = list(messages) + for (started, completed), members in groups.items(): + raw_total = sum(raw for _, raw in members) + # Nothing to apportion, and dividing by it is a ZeroDivisionError. A + # group already at zero stays at zero. + if raw_total <= 0: + continue + net = max(raw_total - busy_ms(spans, started, completed), 0.0) + assigned = 0.0 + for n, (index, raw) in enumerate(members): + # The last member takes the remainder so the parts reconstruct the + # group's net exactly, rather than drifting by the rounding. + share = net - assigned if n == len(members) - 1 else round(net * (raw / raw_total), 6) + out[index] = out[index].model_copy(update={"generation_duration_ms": share}) + assigned += share + return out class EventCollector: @@ -54,6 +185,8 @@ def __init__(self) -> None: self._user_input: str = "" self._model: str | None = None self._turn_starts: int = 0 + # Stamped by AgentStartEvent; the head is measured from it. + self._agent_start_at: datetime | None = None # tool_id -> finalized telemetry (last ToolEnd wins, mirroring last-result-wins). self._commands: dict[str, CommandTelemetry] = {} self._agent_end: AgentEndEvent | None = None @@ -71,6 +204,13 @@ def on_event(self, event: StreamEvent) -> None: if isinstance(event, AgentStartEvent): self._iteration = event.iteration self._user_input = event.prompt + self._agent_start_at = event.timestamp + # A new turn has begun, so the previous turn's terminal event is no + # longer this turn's. Every agent builds a fresh collector per + # communicate(), but EarlyStopWatcher keeps ONE across retries: left + # stale, it would pair this attempt's start with the last attempt's + # end and publish the clamped inversion as a measured 0.0. + self._agent_end = None if event.model: self._model = event.model elif isinstance(event, TurnStartEvent): @@ -103,6 +243,65 @@ def visible_turn_count(self) -> int: def _ordered_commands(self) -> list[CommandTelemetry]: return sorted(self._commands.values(), key=lambda c: c.sequence_number) + def _overhead_ms( + self, messages: list[TranscriptMessage], tool_spans: list[tuple[datetime, datetime]] | None = None + ) -> tuple[float | None, float | None]: + """The turn's head and tail — the wall clock the generations do not cover. + + Measured against ``AssistantMessage`` entries only: a simulation turn + interleaves ``UserMessage`` entries, and a reconciled turn ends with a + ``ReconciliationMessage`` that carries no timestamps at all, so indexing + the raw list would measure the wrong thing or raise. + + Two further restrictions, both of which are the difference between a + measurement and an invention: + + A message whose ``generation_duration_ms`` is ``None`` is SKIPPED. That + field is the codebase's own marker for "no window was measurable here", + and every producer of one stamps ``started_at == completed_at == + datetime.now()`` at *append* time as an admitted placeholder — Codex's + rollout rebuild (``_messages_from_items``), both Codex sub-agent + recovery builders, and Claude's ``_synthesize_subagent_terminal_message``. + Reading those stamps as window bounds turns a placeholder into a + measurement: a Codex turn rebuilt from its rollout stamps every message + at turn END, which would book the entire turn as harness startup. It is + the same exemption CE059 makes for exactly the same reason. + + ``min`` / ``max`` rather than the first and last list entries, because + the list is not ordered by time — Codex appends recovered sub-agent + messages after the parent's last flush. Positional access made the + result depend on append order, which nothing enforces. + + MAIN THREAD ONLY, the third restriction and the same rule its two + sibling call sites already apply (``codex_agent._token_usage_from_messages`` + and ``scripts/timing/decompose_run.py``). A sub-agent's generations + carry the spawning Agent call's ``parent_tool_use_id``, and the identity + these two values complete sums generation over the main thread ONLY — + the parent tool call's own interval already spans the sub-agent's whole + run. Bracketing the span with a sub-agent message therefore shrinks the + head or the tail by time no other bucket claims, and Codex's recovered + child messages carry the CHILD's clock, so the bracket can move either + way. Excluding them keeps all four buckets measuring one thread. + """ + generations = [ + m + for m in messages + if isinstance(m, AssistantMessage) and m.generation_duration_ms is not None and m.parent_tool_use_id is None + ] + if not generations: + return None, None + return decompose_turn( + min(m.started_at for m in generations), + max(m.completed_at for m in generations), + self._agent_start_at, + self._agent_end.timestamp if self._agent_end is not None else None, + tool_spans if tool_spans is not None else self._main_thread_tool_spans(messages), + ) + + def _main_thread_tool_spans(self, messages: list[TranscriptMessage]) -> list[tuple[datetime, datetime]]: + """This turn's main-thread tool spans, from the reduced ToolEnd stream.""" + return main_thread_tool_spans(messages, self._commands.values()) + @staticmethod def _reconciled_messages(messages: list[TranscriptMessage], usage: TokenUsage) -> list[TranscriptMessage]: """Append a ``ReconciliationMessage`` so the transcript's token buckets @@ -190,9 +389,23 @@ def build_turn_record(self) -> TurnRecord: # to the total — making the stream self-reconciling for any downstream # consumer (e.g. the evalboard) without a competing aggregate. messages: list[TranscriptMessage] = list(end.messages) + # Tool execution comes out of the generation windows HERE, once, for + # every harness — the reducers publish raw windows. + # + # The span set is computed ONCE and handed to both consumers. That is + # the invariant worth protecting, and it is the one that is easy to + # break: the subtraction and the head/tail must agree about which calls + # exist, or the buckets stop being disjoint. (The ORDER of the two is + # not load-bearing — `_overhead_ms` reads only the bounds, the + # main-thread flag and whether the duration is `None`, none of which + # `subtract_tool_time` changes. Do not add a comment claiming it is.) + tool_spans = self._main_thread_tool_spans(messages) + messages = subtract_tool_time(messages, tool_spans) if token_usage is not None: messages = self._reconciled_messages(messages, token_usage) + startup_ms, teardown_ms = self._overhead_ms(messages, tool_spans) + return TurnRecord( iteration=end.iteration or self._iteration, user_input=end.user_input or self._user_input, @@ -208,4 +421,6 @@ def build_turn_record(self) -> TurnRecord: result_summary=end.result_summary, crashed=end.crashed, crash_reason=end.crash_reason, + harness_startup_ms=startup_ms, + harness_teardown_ms=teardown_ms, ) diff --git a/src/coder_eval/timing.py b/src/coder_eval/timing.py new file mode 100644 index 00000000..e764b555 --- /dev/null +++ b/src/coder_eval/timing.py @@ -0,0 +1,313 @@ +"""Wall-clock arithmetic for a turn, defined once and shared. + +A cycle-free leaf (the ``models/cli_match.py`` rationale): it sits outside +``agents/`` because ``EventCollector`` consumes it, and importing anything +under ``agents/`` pulls in every agent, which imports ``streaming/``. + +NO harness subtracts tool execution from its own generation windows. Each +publishes the RAW window it measured, and ``streaming/collector.py::subtract_tool_time`` +takes the UNION of the tool intervals back out of them once, for all five, at +the single capture seam — the same place the head and the tail are already +computed. A reducer's only remaining timing decision is where its window +opens, which is the one genuinely harness-shaped part: two interleave a tool +into a single window outright (Antigravity, whose Step for the tool arrives and +only a later ``usage_metadata`` Step cuts the message, and Codex, whose +``_flush_message`` window extends to the last item's ``completed_at_ms``) while +the other three tile the turn contiguously, so a call open at a boundary runs +inside two windows. Central subtraction handles both without either reducer +knowing which it is. + +There is a TypeScript twin, ``evalboard/lib/timing.ts::busyMs``, which +subtracts tool time from a task's WALL CLOCK to produce the Unaccounted +residual. It answers the same question about the same ``task.json``, so the +two must agree — neither owns the numbers: ``tests/_fixtures/timing_union_cases.json`` +does, and both suites replay it. +""" + +import time +from datetime import datetime, timedelta + + +class TurnClock: + """One (wall, monotonic) pair per turn; every later stamp derives from it. + + A turn's bounds and its durations have to share a basis or they can + disagree, and the disagreement lands in a field measured in milliseconds. + Two concrete failures this removes: + + * Antigravity computed its window span on the MONOTONIC clock while + unioning WALL-clock tool intervals and subtracting one from the other. + That is the only reason its window could go negative at all, and the + clamp that hid it was indistinguishable from a real instant generation. + * Pi stamped with naive-LOCAL ``datetime.now()``. A DST transition or an + NTP step inside a turn lands directly in a generation window — an + hour-long jump in a millisecond field. Nightly runs start at 04:18 and + run for hours, so it is reachable rather than theoretical. A + monotonic-derived stamp cannot express it. + + It is an EXTRACTION, not an invention: antigravity already captured this + exact pair at the top of ``communicate`` and simply did not use it for + later stamps. + + Stamps stay NAIVE LOCAL, matching what the rest of the telemetry and the + persisted ``execution_started_at`` already are, so no consumer changes. + + Within a turn the derived stamp is monotonic-accurate and may drift from + real wall time; each turn re-anchors. That is intended — do not "fix" it by + re-reading the wall clock, which is the property being removed. + + ONE PER TURN, never module-level and never reused across turns: a long run + would accumulate drift between the pair and real wall time. The turn-state + constructors take it as an argument so the lifetime is visible in the + signature, and so tests can inject a fake instead of monkeypatching a + module global out from under the reducer. + + NOT for deadlines. Those stay on ``time.monotonic()`` directly: a deadline + must not move when the wall clock steps. + + Codex and OpenCode deliberately do NOT use it. Their tool spans are the + CLI's own epoch-millisecond stamps, unreachable from the host, so + converting only the window bounds would put two bases inside one + ``busy_ms`` subtraction — relocating the defect instead of removing it. + + claude-code does not use it either, but for no good reason: it has no + epoch-stamp constraint, it simply has not been converted. Its window bounds + and its span now share one basis (raw ``datetime.now()``), so the two cannot + disagree with each other — but both carry the naive-local exposure this + class removes. Converting it is the remaining work; see + docs/agents/HARNESS_PARITY.md. + """ + + def __init__(self) -> None: + self._wall0 = datetime.now() + self._mono0 = time.monotonic() + + def now(self) -> datetime: + return self._wall0 + timedelta(seconds=time.monotonic() - self._mono0) + + +def _require_same_awareness(a: datetime, b: datetime, *, field: str) -> None: + """Raise if one stamp is timezone-aware and the other is naive. + + Subtracting the two raises ``TypeError: can't subtract offset-naive and + offset-aware datetimes`` deep inside the arithmetic below, which surfaces + out of ``EventCollector.build_turn_record`` and kills the turn with a + message naming neither the field nor the harness. This turns that into a + statement of which pair disagreed and which side is aware. + + Unreachable from this repo today, and that is the point: every stamp in + ``agents/`` and ``streaming/`` is a naive ``datetime.now()`` (verified by + grep — zero ``timezone.utc`` / ``astimezone`` / ``tzinfo`` hits), so this + guards the SEAM rather than a live defect. The exposure it is actually for + is a third-party agent registered through the ``coder_eval.plugins`` SPI, + which lives outside ``src/coder_eval/agents/`` and which no lint rule + scoped to that directory could ever see. That is why this is a runtime + guard and not a rule. + + Only the MIX raises. An agent that is internally consistent in UTC is not + this function's problem, and neither is one that is consistently naive. + """ + if (a.tzinfo is None) == (b.tzinfo is None): + return + aware, naive = ("first", "second") if a.tzinfo is not None else ("second", "first") + raise TypeError( + f"{field}: one stamp is timezone-aware and the other is naive (the {aware} is aware, " + + f"the {naive} is not), so the interval between them cannot be measured. Every stamp " + + "this harness records is a naive local `datetime.now()`; if you are writing an agent " + + "outside this repo (the `coder_eval.plugins` SPI), make its stamps naive local too " + + "rather than normalizing here, so its tool spans and its window bounds keep one basis." + ) + + +def busy_ms(spans: list[tuple[datetime, datetime]], lo: datetime, hi: datetime) -> float: + """Wall milliseconds inside ``[lo, hi]`` where at least ONE span was running. + + The union, not the sum. Tool intervals overlap in practice — Antigravity + resolves several calls from one ``Step`` and backgrounds anything over ten + seconds; Codex spawns collab agents that run concurrently — so adding + their durations over-counts the busy time by exactly the overlap. + Subtracting such a sum from a generation window understates generation + and, with enough concurrency, drives it negative: four concurrent 400 ms + calls inside a 1000 ms window sum to 1600 ms, clamping the result to the + ``0.0`` that "unknown timing says unknown" exists to eliminate. + + Clipping to ``[lo, hi]`` is the other half: a tool that opened before this + window only spent part of its life inside it, and only that part is not + generation time here. + + Every stamp reaching this function is a naive ``datetime.now()`` today — + that is true of all of ``agents/`` and ``streaming/`` — so a mixed pair + means an agent has started recording aware stamps, and + ``_require_same_awareness`` names which pair rather than letting a bare + ``TypeError`` escape from the arithmetic. The spans are checked as well as + the bounds, not instead of them: the clipping below compares each span + against BOTH ``lo`` and ``hi``, so a guard on the bounds alone would leave + this function uncovered by it. + + An EMPTY span list is checked NOT AT ALL, bounds included. The comprehension + never runs, nothing is compared and nothing is subtracted, so there is no + pair for the guard to be about — and raising there would reject a call that + has always returned ``0.0``. + """ + if not spans: + return 0.0 + _require_same_awareness(lo, hi, field="busy_ms window") + for span_start, span_end in spans: + _require_same_awareness(lo, span_start, field="busy_ms window vs a tool span's start") + _require_same_awareness(hi, span_end, field="busy_ms window vs a tool span's end") + clipped = sorted((max(s, lo), min(e, hi)) for s, e in spans if min(e, hi) > max(s, lo)) + if not clipped: + return 0.0 + total = 0.0 + open_start, open_end = clipped[0] + for start, end in clipped[1:]: + if start > open_end: # disjoint — bank the run and start a new one + total += (open_end - open_start).total_seconds() * 1000.0 + open_start, open_end = start, end + else: # overlapping or adjacent — extend the run + open_end = max(open_end, end) + return total + (open_end - open_start).total_seconds() * 1000.0 + + +def union_ms(spans: list[tuple[datetime, datetime]]) -> float: + """Wall milliseconds at least ONE span was running, over their full extent. + + ``busy_ms`` with the window set to the spans' own bounds. It exists because + two callers had copy-pasted that same ``min``/``max``/``busy_ms`` tail — + ``tests/_fixtures/golden_streams/_scrub.py`` (the golden sensor) and + ``scripts/timing/decompose_run.py`` (the live residual gate) — and they + answer the same question about the same recorded commands, so a divergence + would let one pass while the other failed. Each keeps its OWN stamp parsing + and span building, because their input shapes genuinely differ; only this + tail is shared. + + It does NOT filter ``end < start``. EVERY caller drops those while building + its span list — ``streaming.collector.main_thread_tool_spans`` (shared by + the collector and the report layer), ``_scrub.py`` and + ``decompose_run.py`` — so guarding again here would be a second rule about + the same input in a second place. That reasoning holds only while it stays + true of every caller: a new one that skips the check gets whatever + ``busy_ms`` does with an inverted pair, which is to discard it, but + silently rather than by this function's stated contract. + """ + if not spans: + return 0.0 + return busy_ms(spans, min(s for s, _ in spans), max(e for _, e in spans)) + + +def close_window(*, mark: datetime, now: datetime, item_start: datetime | None = None) -> tuple[datetime, float]: + """Open one generation window at ``mark`` and close it at ``now``: its ``(started, span_ms)``. + + The shape all five reducers share. What it returns is the RAW window — + tool execution is taken back out of it once, centrally, in + ``streaming/collector.py::subtract_tool_time``, which is the only place + that arithmetic lives. It used to happen here too, per flush, and in + claude-code at finalization; the per-reducer bookkeeping that required + (a span list, its reset rule, the set of still-open calls) is where every + timing defect on this branch actually lived. + + ``mark`` is where the window opens: the previous flush's close, which is + what makes the windows TILE the turn contiguously instead of leaving the + model time that PRODUCED an item attributed to nothing. It is keyword-only + and has NO default so that no reducer can open a window without stating + what it tiles from — which is the defect pi shipped with, measuring from + its own turn start so that every inter-turn gap fell into no bucket at all. + Note what the signature does and does not buy: it constrains the call + SHAPE, not the VALUE. A reducer can still pass the wrong mark; what it + cannot do is fail to have one. + + ``item_start`` is this emission's own first stamp, when the harness has + one. The ``min()`` against ``mark`` is the tiling defense and nothing else: + a stamp that went backwards must never push the window start PAST the first + item and invert the span. claude-code passes none — its stream carries no + per-emission item start — so its window opens exactly at the mark. + + The result is clamped at ``0.0``: an inverted window (``now`` before + ``mark``, two clocks disagreeing) is a measured zero, not a negative + generation. + + It deliberately does NOT return ``completed``. The window always ends at + ``now``, which the caller passed in, so handing it back would be an + argument returned unchanged — redundancy dressed as symmetry. Call sites + write ``completed_at=now`` directly. + """ + started = min(mark, item_start) if item_start is not None else mark + return started, max(0.0, (now - started).total_seconds() * 1000.0) + + +def decompose_turn( + first_started_at: datetime | None, + last_completed_at: datetime | None, + agent_started_at: datetime | None, + agent_ended_at: datetime | None, + tool_spans: list[tuple[datetime, datetime]] | None = None, +) -> tuple[float | None, float | None]: + """Wall ms before the first generation window opens, and after the last closes. + + The turn's two unexplained ends. Between them the windows tile (each + harness's generation mark runs to the next) and tool execution is already + subtracted inside them, so head + generation + UNION(tool) + tail is the + whole turn — the union and not the sum, because concurrent tool calls + otherwise book their overlap twice (``busy_ms`` above, and measured: one + live Pi turn overlapped a ``Write`` and a ``Bash`` by 18.4 ms). + + ``tool_spans`` is what keeps those four buckets DISJOINT, and omitting it + is a double-count rather than a lost refinement. A tool is not confined to + a generation window: Antigravity force-closes an orphan at finalization + (``antigravity_agent.py``), which stamps its completion inside the tail, + and it backgrounds anything over ten seconds, which can straddle either + end. Such a span is subtracted out of the windows AND counted in the tool + bucket, so leaving it in the head or tail books it twice — measured on the + committed ``antigravity_d_orphaned_tool`` fixture as a residual of -86% of + wall clock. So the head and tail exclude tool time by the same rule and + the same helper the windows use. + + ``EventCollector`` is the SOLE caller, and deliberately so: this is the one + place the two values are computed, after which they are persisted on + ``TurnRecord`` and every later consumer READS them rather than recomputing. + The golden-stream sensor asserts on the dumped record, and + ``scripts/timing/decompose_run.py`` reads the stored fields — neither can + call this, because ``task.json`` carries no ``AgentStartEvent`` stamp to + recompute a head from. + + The head means ONE thing on all five: wall clock from the turn starting + until the harness first observed model output. Every reducer opens its first + generation window at that same instant, which is what keeps the two buckets + disjoint. What the head CONTAINS still differs and is deliberately NOT + split: a harness that spawns its process PER TURN fuses that boot, provider + resolution, dispatch and TTFT — measured on OpenCode, the process spawns in + 3 ms and the first event lands at 3921 ms — while one that spawns it once at + startup and holds it across turns has no boot inside the turn to fuse in. No + stream carries a marker between those parts. Naming these for the + interval they MEASURE rather than for what they contain is the whole point; + see docs/agents/HARNESS_PARITY.md for the per-harness composition. + + Every stamp reaching this function is a naive ``datetime.now()`` — that is + true of all of ``agents/`` and ``streaming/`` today — so a mixed pair means + an agent has started recording aware stamps, and ``_require_same_awareness`` + says so rather than letting a bare ``TypeError`` escape and kill the turn. + + ``None`` means never measured — a turn that produced no generation, or a + snapshot taken before the terminal event. Never 0.0, which would claim a + measurement was taken and came back instant (CE058). A measured inversion + (the two clocks disagreeing) IS a real zero and clamps, because both ends + were observed. + + NOTE the four-bucket identity has a second implementation in TypeScript — + the evalboard's Unaccounted cell (``_sections.tsx``) subtracts the same + buckets from the same wall clock, as ``pricing.ts`` mirrors ``pricing.py``. + It does not recompute a head or a tail (it reads the stored fields), so a + change HERE needs a TS change only when it alters what the buckets mean; + adding a fifth bucket means touching that cell and ``sumHarnessOverhead``. + """ + spans = tool_spans or [] + head = tail = None + if first_started_at is not None and agent_started_at is not None: + _require_same_awareness(agent_started_at, first_started_at, field="harness_startup_ms") + elapsed = (first_started_at - agent_started_at).total_seconds() * 1000.0 + head = max(elapsed - busy_ms(spans, agent_started_at, first_started_at), 0.0) + if last_completed_at is not None and agent_ended_at is not None: + _require_same_awareness(last_completed_at, agent_ended_at, field="harness_teardown_ms") + elapsed = (agent_ended_at - last_completed_at).total_seconds() * 1000.0 + tail = max(elapsed - busy_ms(spans, last_completed_at, agent_ended_at), 0.0) + return head, tail diff --git a/tests/_fixtures/golden_streams/_scrub.py b/tests/_fixtures/golden_streams/_scrub.py index c84cc77d..8afba06a 100644 --- a/tests/_fixtures/golden_streams/_scrub.py +++ b/tests/_fixtures/golden_streams/_scrub.py @@ -2,8 +2,11 @@ from __future__ import annotations +from datetime import datetime from typing import Any +from coder_eval.timing import union_ms + SCRUB_PLACEHOLDER = "" @@ -24,6 +27,11 @@ "duration_ms", "duration_seconds", "generation_duration_ms", + # Measured wall intervals like the two above, so they vary run to run; + # masking keeps None-vs-set (the meaningful distinction) visible while + # the value itself stays out of the snapshot. + "harness_startup_ms", + "harness_teardown_ms", # Cost is a rate-card-dependent float (and is backfilled from the rate # card on timeout/kill), so it is masked too — keeping the snapshot # rate-card-independent. The integer TOKEN buckets stay EXACT; those are @@ -99,7 +107,61 @@ def assert_reconciliation(record: dict[str, Any]) -> None: assert cr_sum == usage["cache_read_input_tokens"], "cache_read bucket does not reconcile" -def assert_timing_captured(record: dict[str, Any], *, expect_generation_window: bool) -> None: +# The four buckets are disjoint by construction, so their sum cannot exceed the +# turn's own wall clock. Flag only an overshoot past BOTH bounds: the relative +# one is what catches the defect (an orphaned tool double-booked into the tail +# read +55% of wall on ``antigravity_d_orphaned_tool``), and the absolute floor +# keeps a replay whose whole turn is 40 microseconds from failing on scheduler +# jitter. Healthy fixtures overshoot by at most 0.003 ms / 2%. +_IDENTITY_FLOOR_MS = 0.1 +_IDENTITY_SHARE = 0.20 + + +def _sub_agent_tool_ids(record: dict[str, Any]) -> set[str]: + """Tool ids owned by a SUB-AGENT generation. + + Must match `EventCollector._main_thread_tool_spans` and + `scripts/timing/decompose_run.py::_sub_agent_tool_ids`: all three recompute + the tool union for the same identity, so a filter applied by one and not + the others reports a residual that is an artifact of the disagreement. + """ + ids: set[str] = set() + for message in record.get("messages") or []: + if message.get("role") == "assistant" and message.get("parent_tool_use_id") is not None: + ids.update(message.get("tool_use_ids") or []) + return ids + + +def _tool_union_ms(record: dict[str, Any]) -> float: + """Wall ms this turn's MAIN-THREAD tools occupied — the union, never the sum. + + Sub-agent tools are excluded for the same reason their generations are: the + spawning Agent call's own interval already spans the child's whole run. + """ + excluded = _sub_agent_tool_ids(record) + spans: list[tuple[datetime, datetime]] = [] + for command in record.get("commands") or []: + if command.get("tool_id") in excluded: + continue + start = _parse_stamp(command.get("execution_started_at")) + end = _parse_stamp(command.get("execution_completed_at")) + if start is not None and end is not None and end >= start: + spans.append((start, end)) + return union_ms(spans) + + +def _parse_stamp(value: Any) -> datetime | None: + if not isinstance(value, str): + return None + try: + return datetime.fromisoformat(value) + except ValueError: + return None + + +def assert_timing_captured( + record: dict[str, Any], *, expect_generation_window: bool, check_identity: bool = True +) -> None: """Assert a TurnRecord dump actually recorded the timing it could measure. Run on the UNSCRUBBED dump. ``scrub()`` masks values but preserves ``None`` @@ -130,6 +192,50 @@ def assert_timing_captured(record: dict[str, Any], *, expect_generation_window: bounds are the same ``ast.Name``; when they are two different names holding the same value it cannot, and this is the check that does. + **Unconditional, and keyed on the messages rather than on the flag.** A + turn's head and tail (``harness_startup_ms`` / ``harness_teardown_ms``) are + set exactly when the turn produced an assistant message with a MEASURABLE + window, because that is what the collector measures them against — so both + are non-``None`` when one exists and both are ``None`` when none does. + + Both halves of that key are load-bearing. The flag is the wrong one: + ``codex_e_orphan_tool`` streams a generation whose window subtracts to + zero, so it clears the flag while still having a head and a tail to report. + And "any assistant message" is too weak: ``codex_g_items_rebuild`` rebuilds + its transcript from the rollout after the turn ended, with + ``generation_duration_ms=None`` and placeholder ``now()`` bounds, so there + is nothing there to measure an end against and the honest answer is + ``None`` for both. + + PRESENCE is all the fixtures can support, and it is the thing worth + asserting: the replays run in ~0.3 ms of synthetic wall clock, so their + head and tail are microseconds and any bound or ordering check would be + noise. A ``>= 0`` check would be worse than noise — ``decompose_turn`` + clamps with ``max(..., 0.0)``, so it would restate the implementation and + could never fail. + + **The four-bucket identity**, when ``check_identity``. Generation plus the + UNION of the tool intervals plus the head plus the tail cannot exceed the + turn's ``duration_seconds``, because the four are disjoint: the windows are + tool-subtracted and so are the head and tail. This is the one assertion + that catches a DOUBLE-COUNT rather than an absence — it is how an orphaned + tool force-closed inside the tail, booked both as tool and as teardown, was + found reconciling at -86% of wall clock while all 72 golden tests passed. + + The check is ONE-SIDED on purpose and stays that way. A symmetric bound + would be a sensor in name only here: the replays run in ~0.3 ms of + synthetic wall clock, so ``abs(residual) <= max(0.1 ms, 20% x wall)`` + passes essentially any magnitude. The two-sided, millisecond-exact check + lives in ``tests/test_timing_identity_contract.py``, where a scripted clock + makes the magnitudes real, and the live two-sided gate is + ``scripts/timing/decompose_run.py --max-residual-pct``. + + ``check_identity`` is off for the scenarios that inject their own SDK + timestamps (see ``FICTIONAL_DURATIONS``): those declare integer-millisecond + item durations of 17-900 ms while the replay itself takes ~0.3 ms of real + wall clock, so no rebasing can make the two commensurable — the SDK's + stamps are milliseconds and the replay is faster than one. + Why a scenario-level floor rather than a per-entry rule: no per-entry form works against the real snapshots. ``claude_d_subagent_terminal`` holds two content-bearing assistant messages of which exactly one is legitimately @@ -149,9 +255,56 @@ def assert_timing_captured(record: dict[str, Any], *, expect_generation_window: "returned was timed, so the record must say when and for how long" ) + assistant = [m for m in record.get("messages") or [] if m.get("role") == "assistant"] + measurable = [m for m in assistant if m.get("generation_duration_ms") is not None] + for field in ("harness_startup_ms", "harness_teardown_ms"): + value = record.get(field) + if measurable: + assert value is not None, ( + f"{field} is None on a turn carrying {len(measurable)} measurable generation " + "window(s): the collector measures the head and tail against the earliest and " + "latest of those, so a turn that generated has both — None says never measured" + ) + else: + assert value is None, ( + f"{field} is {value!r} on a turn with no measurable generation window " + f"({len(assistant)} assistant message(s), none reporting a duration): there is " + "nothing to measure an end against, and a number here claims a measurement " + "nobody could have taken" + ) + + if check_identity: + wall_ms = (record.get("duration_seconds") or 0.0) * 1000.0 + # Main thread only: a sub-agent's generations bubble into the same + # stream, and the spawning Agent call's own interval already spans them. + generation_ms = sum( + m.get("generation_duration_ms") or 0.0 + for m in record.get("messages") or [] + if m.get("role") == "assistant" and m.get("parent_tool_use_id") is None + ) + tool_ms = _tool_union_ms(record) + bucket_sum = ( + generation_ms + + tool_ms + + (record.get("harness_startup_ms") or 0.0) + + (record.get("harness_teardown_ms") or 0.0) + ) + overshoot = bucket_sum - wall_ms + assert overshoot <= max(_IDENTITY_FLOOR_MS, _IDENTITY_SHARE * wall_ms), ( + f"the four buckets sum to {bucket_sum:.4f} ms against a {wall_ms:.4f} ms turn " + f"(over by {overshoot:.4f} ms): generation={generation_ms:.4f}, tool_union={tool_ms:.4f}, " + f"startup={record.get('harness_startup_ms')!r}, teardown={record.get('harness_teardown_ms')!r}. " + "They are meant to be DISJOINT, so a sum this far over the turn means something is " + "booked twice — most likely a tool that ran outside every generation window and was " + "left in the head or tail as well as in the tool union, or a generation window that " + "kept tool time it should have subtracted (see docs/agents/HARNESS_PARITY.md — the " + "subtraction happens once, in streaming/collector.py::subtract_tool_time, so a " + "double-count is a span the collector saw twice or a reducer publishing a window it " + "already narrowed)" + ) + if not expect_generation_window: return - assistant = [m for m in record.get("messages") or [] if m.get("role") == "assistant"] windows = [(m.get("generation_duration_ms"), m.get("started_at"), m.get("completed_at")) for m in assistant] assert any( duration is not None and duration > 0 and started is not None and completed is not None and completed > started diff --git a/tests/_fixtures/golden_streams/codex_fixtures.py b/tests/_fixtures/golden_streams/codex_fixtures.py index 30008af7..e0a157fd 100644 --- a/tests/_fixtures/golden_streams/codex_fixtures.py +++ b/tests/_fixtures/golden_streams/codex_fixtures.py @@ -13,6 +13,7 @@ import os from dataclasses import dataclass +from datetime import datetime from pathlib import Path from types import SimpleNamespace from typing import Any @@ -26,12 +27,23 @@ CODEX_MODEL = "gpt-5-codex" +# How far after the replay's start the rebased timeline begins. Small, but +# non-zero so the first generation window opens AFTER the AgentStartEvent and +# the head is a measured interval instead of a clamped inversion. +_REPLAY_LEAD_MS = 2 + # --- Notification factories (mirror test_codex_agent) ----------------------- # Fixed epoch milliseconds, so every derived duration is deterministic and the -# golden snapshots pin a real value rather than a scrubbed clock read. +# golden snapshots pin a real value rather than a scrubbed clock read. It is a +# BASE, not a wall-clock claim: ``_rebase_notifications`` shifts the whole +# timeline onto the replay's own clock before the scenario runs, so the SDK +# stamps and the agent's own event stamps are commensurable. Left absolute, +# a codex replay recorded a ``harness_startup_ms`` of ~126 DAYS — the agent +# events are stamped ``now()`` while these sat in 2027 — which is a number no +# presence-only assertion can catch. _T0_MS = 1_800_000_000_000 @@ -209,9 +221,21 @@ def _build_catalogue() -> list[CodexScenario]: CodexScenario( name="c_reasoning_placeholder", notifications=[ - _item("item/completed", _reasoning(text="")), + # Real bounds, and they are load-bearing rather than decorative: + # with none, `_flush_message` takes `_ms_to_dt(None)` for BOTH + # ends, which is two adjacent `datetime.now()` reads. Those + # collide at microsecond resolution often enough that this + # scenario failed `assert_timing_captured`'s + # `completed_at > started_at` roughly one run in twenty under + # parallel load, naming a different scenario each time. + _item("item/completed", _reasoning(text=""), started_at_ms=_T0_MS, completed_at_ms=_T0_MS + 40), _delta("final answer"), - _item("item/completed", _agent_message("final answer")), + _item( + "item/completed", + _agent_message("final answer"), + started_at_ms=_T0_MS + 40, + completed_at_ms=_T0_MS + 300, + ), _token_usage(inp=100, out=50, cached=8, reasoning=20), _turn_completed(), ], @@ -282,7 +306,13 @@ def _build_catalogue() -> list[CodexScenario]: name="h_no_turn_completed_crash", notifications=[ _delta("partial"), - _item("item/completed", _agent_message("partial")), + # Bounded for the same reason as (c) above. + _item( + "item/completed", + _agent_message("partial"), + started_at_ms=_T0_MS, + completed_at_ms=_T0_MS + 200, + ), _token_usage(inp=100, out=40, cached=8), ], expects=AgentCrashError, @@ -314,6 +344,40 @@ def turn(self, _user_input: str) -> _FakeTurnHandle: return _FakeTurnHandle(self._notifications) +def _rebase_notifications(notifications: list[Any]) -> list[Any]: + """Shift every SDK item stamp from ``_T0_MS`` onto the replay's own clock. + + The scenario catalogue is built once at import with an absolute base, which + keeps every DERIVED duration deterministic (a 250 ms command stays 250 ms). + But the agent stamps its own lifecycle events with ``datetime.now()``, so + left absolute the two clocks are months apart and the recorded head and + tail are nonsense. Rebasing keeps the deltas and fixes the era. + + The offset puts the first item a beat AFTER the replay starts, so the head + is a small positive interval rather than an inversion clamped to 0.0. + """ + offset = int(datetime.now().timestamp() * 1000) - _T0_MS + _REPLAY_LEAD_MS + rebased: list[Any] = [] + for note in notifications: + payload = getattr(note, "payload", None) + started = getattr(payload, "started_at_ms", None) + completed = getattr(payload, "completed_at_ms", None) + if payload is None or (started is None and completed is None): + rebased.append(note) + continue + rebased.append( + SimpleNamespace( + method=note.method, + payload=SimpleNamespace( + item=payload.item, + started_at_ms=None if started is None else started + offset, + completed_at_ms=None if completed is None else completed + offset, + ), + ) + ) + return rebased + + async def run_codex_scenario(scenario: CodexScenario, working_dir: str) -> dict[str, Any]: """Run ``scenario`` with fakes and return the TurnRecord/pending_turn dump.""" import pytest @@ -322,7 +386,7 @@ async def run_codex_scenario(scenario: CodexScenario, working_dir: str) -> dict[ agent = CodexAgent(config) agent.working_directory = Path(working_dir) agent.codex_client = SimpleNamespace(close=lambda: None) - agent.thread = _FakeThread(scenario.notifications) + agent.thread = _FakeThread(_rebase_notifications(scenario.notifications)) # Point CODEX_HOME at a sessions-less dir so sub-agent rollout recovery # short-circuits instead of polling the real ~/.codex. diff --git a/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json b/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json index e388dae4..915390ff 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ @@ -25,7 +27,7 @@ ], "generation_duration_ms": "", "input_tokens": 100, - "message_id": null, + "message_id": "antigravity-1-msg-0", "model": "gemini-3.5-flash", "output_tokens": 20, "parent_tool_use_id": null, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json b/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json index 062c9556..bda60126 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ @@ -54,7 +56,7 @@ ], "generation_duration_ms": "", "input_tokens": 120, - "message_id": null, + "message_id": "antigravity-1-msg-0", "model": "gemini-3.5-flash", "output_tokens": 15, "parent_tool_use_id": null, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json b/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json index c767a652..56138f54 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ @@ -63,7 +65,7 @@ ], "generation_duration_ms": "", "input_tokens": 200, - "message_id": null, + "message_id": "antigravity-1-msg-0", "model": "gemini-3.5-flash", "output_tokens": 50, "parent_tool_use_id": null, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json index 2cc1c723..02f13064 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ @@ -45,7 +47,7 @@ ], "generation_duration_ms": "", "input_tokens": 90, - "message_id": null, + "message_id": "antigravity-1-msg-0", "model": "gemini-3.5-flash", "output_tokens": 10, "parent_tool_use_id": null, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json b/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json index ee0f36cb..1b69cc78 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ @@ -25,7 +27,7 @@ ], "generation_duration_ms": "", "input_tokens": 100, - "message_id": null, + "message_id": "antigravity-1-msg-0", "model": "gemini-3.5-flash", "output_tokens": 15, "parent_tool_use_id": null, @@ -52,7 +54,7 @@ ], "generation_duration_ms": "", "input_tokens": 110, - "message_id": null, + "message_id": "antigravity-1-msg-1", "model": "gemini-3.5-flash", "output_tokens": 18, "parent_tool_use_id": null, @@ -79,7 +81,7 @@ ], "generation_duration_ms": "", "input_tokens": 120, - "message_id": null, + "message_id": "antigravity-1-msg-2", "model": "gemini-3.5-flash", "output_tokens": 14, "parent_tool_use_id": null, diff --git a/tests/_fixtures/golden_streams/expected/claude_a_single_text_turn.json b/tests/_fixtures/golden_streams/expected/claude_a_single_text_turn.json index febe95e0..7dd57bd4 100644 --- a/tests/_fixtures/golden_streams/expected/claude_a_single_text_turn.json +++ b/tests/_fixtures/golden_streams/expected/claude_a_single_text_turn.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/claude_b_tool_use_result.json b/tests/_fixtures/golden_streams/expected/claude_b_tool_use_result.json index 3ef5f669..d94df272 100644 --- a/tests/_fixtures/golden_streams/expected/claude_b_tool_use_result.json +++ b/tests/_fixtures/golden_streams/expected/claude_b_tool_use_result.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/claude_c_multi_emission_delta.json b/tests/_fixtures/golden_streams/expected/claude_c_multi_emission_delta.json index 00d6f778..0644c58c 100644 --- a/tests/_fixtures/golden_streams/expected/claude_c_multi_emission_delta.json +++ b/tests/_fixtures/golden_streams/expected/claude_c_multi_emission_delta.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/claude_d_subagent_terminal.json b/tests/_fixtures/golden_streams/expected/claude_d_subagent_terminal.json index 943e0ef7..37c18d97 100644 --- a/tests/_fixtures/golden_streams/expected/claude_d_subagent_terminal.json +++ b/tests/_fixtures/golden_streams/expected/claude_d_subagent_terminal.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/claude_e_model_usage_and_backfill.json b/tests/_fixtures/golden_streams/expected/claude_e_model_usage_and_backfill.json index 782fa203..88ee7a64 100644 --- a/tests/_fixtures/golden_streams/expected/claude_e_model_usage_and_backfill.json +++ b/tests/_fixtures/golden_streams/expected/claude_e_model_usage_and_backfill.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json index cae8354c..9195adde 100644 --- a/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json +++ b/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json @@ -26,6 +26,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/claude_g_crash_format_placeholder.json b/tests/_fixtures/golden_streams/expected/claude_g_crash_format_placeholder.json index d615eae2..4e60e81d 100644 --- a/tests/_fixtures/golden_streams/expected/claude_g_crash_format_placeholder.json +++ b/tests/_fixtures/golden_streams/expected/claude_g_crash_format_placeholder.json @@ -5,6 +5,8 @@ "crash_reason": "Communication with agent failed: crash after poison\nStderr output:\nNo stderr captured", "crashed": true, "duration_seconds": "", + "harness_startup_ms": null, + "harness_teardown_ms": null, "iteration": 1, "max_turns_exhausted": false, "messages": [], diff --git a/tests/_fixtures/golden_streams/expected/claude_h1_timeout_process_error.json b/tests/_fixtures/golden_streams/expected/claude_h1_timeout_process_error.json index 2e021896..c87fb5be 100644 --- a/tests/_fixtures/golden_streams/expected/claude_h1_timeout_process_error.json +++ b/tests/_fixtures/golden_streams/expected/claude_h1_timeout_process_error.json @@ -5,6 +5,8 @@ "crash_reason": "Agent turn timed out after 30s", "crashed": true, "duration_seconds": "", + "harness_startup_ms": null, + "harness_teardown_ms": null, "iteration": 1, "max_turns_exhausted": false, "messages": [], diff --git a/tests/_fixtures/golden_streams/expected/claude_h2_process_error_crash.json b/tests/_fixtures/golden_streams/expected/claude_h2_process_error_crash.json index dcdd6042..bf0c6715 100644 --- a/tests/_fixtures/golden_streams/expected/claude_h2_process_error_crash.json +++ b/tests/_fixtures/golden_streams/expected/claude_h2_process_error_crash.json @@ -5,6 +5,8 @@ "crash_reason": "CLI process failed (exit code 1): bad config", "crashed": true, "duration_seconds": "", + "harness_startup_ms": null, + "harness_teardown_ms": null, "iteration": 1, "max_turns_exhausted": false, "messages": [], diff --git a/tests/_fixtures/golden_streams/expected/claude_i_in_loop_deadline_break.json b/tests/_fixtures/golden_streams/expected/claude_i_in_loop_deadline_break.json index da8254bd..e0a00c6b 100644 --- a/tests/_fixtures/golden_streams/expected/claude_i_in_loop_deadline_break.json +++ b/tests/_fixtures/golden_streams/expected/claude_i_in_loop_deadline_break.json @@ -5,6 +5,8 @@ "crash_reason": "Agent turn timed out after 100s", "crashed": true, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json b/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json index 1ef1685a..b3b0e23e 100644 --- a/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json +++ b/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json b/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json index 4f1ce309..df652657 100644 --- a/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json +++ b/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json b/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json index 4db03065..93b83762 100644 --- a/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json +++ b/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json b/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json index 55959efe..32092379 100644 --- a/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json +++ b/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json b/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json index 438c798e..81459ad0 100644 --- a/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json +++ b/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json b/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json index 74d457e3..b1f83b09 100644 --- a/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json +++ b/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json @@ -46,6 +46,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json b/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json index aed8db92..75285012 100644 --- a/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json +++ b/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": null, + "harness_teardown_ms": null, "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json b/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json index d58244c0..4828acb6 100644 --- a/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json +++ b/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json @@ -5,6 +5,8 @@ "crash_reason": "Codex turn failed: Turn did not complete (no turn/completed notification received)", "crashed": true, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json b/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json index 3d65a879..bc1b0dea 100644 --- a/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json +++ b/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json @@ -5,6 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json b/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json index 571de997..8abfd499 100644 --- a/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json +++ b/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json @@ -25,6 +25,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/opencode_c_multi_step_tiling.json b/tests/_fixtures/golden_streams/expected/opencode_c_multi_step_tiling.json new file mode 100644 index 00000000..ee573afc --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/opencode_c_multi_step_tiling.json @@ -0,0 +1,108 @@ +{ + "agent_output": "Listed it.", + "assistant_turn_count": 2, + "commands": [ + { + "assistant_turn_index": 1, + "duration_ms": "", + "error_message": null, + "execution_completed_at": "", + "execution_started_at": "", + "generation_completed_at": null, + "parameters": { + "command": "ls" + }, + "result_data": null, + "result_status": "success", + "result_summary": "main.py", + "result_tokens": 2, + "sequence_number": 1, + "timestamp": "", + "tool_id": "call_1", + "tool_name": "Bash" + } + ], + "crash_reason": null, + "crashed": false, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "max_turns_exhausted": false, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "tool_use", + "is_error": false, + "sequence": 0, + "signature": null, + "text": null, + "thinking": null, + "tool_use_id": "call_1" + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": "msg_1", + "model": "deepseek/deepseek-v4-pro", + "output_tokens": 20, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "tool-calls", + "tool_use_ids": [ + "call_1" + ] + }, + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "Listed it.", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 50, + "message_id": "msg_2", + "model": "deepseek/deepseek-v4-pro", + "output_tokens": 30, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [] + } + ], + "model_used": "deepseek/deepseek-v4-pro", + "num_turns": 2, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": "stop", + "subtype": "completed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 150, + "output_tokens": 50, + "total_cost_usd": "", + "uncached_input_tokens": 150 + }, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/opencode_d_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/opencode_d_orphaned_tool.json new file mode 100644 index 00000000..3ebb5691 --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/opencode_d_orphaned_tool.json @@ -0,0 +1,90 @@ +{ + "agent_output": "Waiting.", + "assistant_turn_count": 1, + "commands": [ + { + "assistant_turn_index": 1, + "duration_ms": null, + "error_message": "no result observed", + "execution_completed_at": "", + "execution_started_at": null, + "generation_completed_at": null, + "parameters": { + "command": "sleep 600" + }, + "result_data": null, + "result_status": "unknown", + "result_summary": null, + "result_tokens": 0, + "sequence_number": 1, + "timestamp": "", + "tool_id": "call_1", + "tool_name": "Bash" + } + ], + "crash_reason": null, + "crashed": false, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "max_turns_exhausted": false, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "Waiting.", + "thinking": null, + "tool_use_id": null + }, + { + "block_type": "tool_use", + "is_error": false, + "sequence": 1, + "signature": null, + "text": null, + "thinking": null, + "tool_use_id": "call_1" + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": "msg_1", + "model": "deepseek/deepseek-v4-pro", + "output_tokens": 20, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [ + "call_1" + ] + } + ], + "model_used": "deepseek/deepseek-v4-pro", + "num_turns": 1, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": "stop", + "subtype": "completed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 100, + "output_tokens": 20, + "total_cost_usd": "", + "uncached_input_tokens": 100 + }, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/opencode_e_error_after_generation.json b/tests/_fixtures/golden_streams/expected/opencode_e_error_after_generation.json new file mode 100644 index 00000000..a757b0c8 --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/opencode_e_error_after_generation.json @@ -0,0 +1,59 @@ +{ + "agent_output": "Starting.", + "assistant_turn_count": 1, + "commands": [], + "crash_reason": "OpenCode error: 401 from the provider", + "crashed": true, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "max_turns_exhausted": false, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "Starting.", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": "msg_1", + "model": "deepseek/deepseek-v4-pro", + "output_tokens": 20, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [] + } + ], + "model_used": "deepseek/deepseek-v4-pro", + "num_turns": 1, + "result_summary": { + "is_error": true, + "result": "OpenCode error: 401 from the provider", + "stop_reason": "stop", + "subtype": "crashed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 100, + "output_tokens": 20, + "total_cost_usd": "", + "uncached_input_tokens": 100 + }, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json b/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json index 49083eac..f83268c0 100644 --- a/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json +++ b/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json @@ -1,10 +1,12 @@ { - "agent_output": "", + "agent_output": "All done.", "assistant_turn_count": 1, "commands": [], "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ @@ -12,7 +14,17 @@ "cache_creation_tokens": 0, "cache_read_tokens": 64, "completed_at": "", - "content_blocks": [], + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "All done.", + "thinking": null, + "tool_use_id": null + } + ], "generation_duration_ms": "", "input_tokens": 100, "message_id": null, diff --git a/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json b/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json index a1c1aa85..f256747c 100644 --- a/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json +++ b/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json @@ -45,6 +45,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", "iteration": 1, "max_turns_exhausted": false, "messages": [ diff --git a/tests/_fixtures/golden_streams/expected/pi_c_multi_turn_tiling.json b/tests/_fixtures/golden_streams/expected/pi_c_multi_turn_tiling.json new file mode 100644 index 00000000..2a06910a --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/pi_c_multi_turn_tiling.json @@ -0,0 +1,108 @@ +{ + "agent_output": "Listed it.", + "assistant_turn_count": 2, + "commands": [ + { + "assistant_turn_index": 1, + "duration_ms": "", + "error_message": null, + "execution_completed_at": "", + "execution_started_at": "", + "generation_completed_at": null, + "parameters": { + "command": "ls" + }, + "result_data": null, + "result_status": "success", + "result_summary": "main.py", + "result_tokens": 2, + "sequence_number": 1, + "timestamp": "", + "tool_id": "call_1", + "tool_name": "Bash" + } + ], + "crash_reason": null, + "crashed": false, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "max_turns_exhausted": false, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "tool_use", + "is_error": false, + "sequence": 0, + "signature": null, + "text": null, + "thinking": null, + "tool_use_id": "call_1" + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": null, + "model": "openrouter/moonshotai/kimi-k3", + "output_tokens": 20, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [ + "call_1" + ] + }, + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "Listed it.", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 50, + "message_id": null, + "model": "openrouter/moonshotai/kimi-k3", + "output_tokens": 30, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [] + } + ], + "model_used": "openrouter/moonshotai/kimi-k3", + "num_turns": 2, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": "stop", + "subtype": "completed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 150, + "output_tokens": 50, + "total_cost_usd": "", + "uncached_input_tokens": 150 + }, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/pi_d_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/pi_d_orphaned_tool.json new file mode 100644 index 00000000..e2efaaad --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/pi_d_orphaned_tool.json @@ -0,0 +1,90 @@ +{ + "agent_output": "Waiting.", + "assistant_turn_count": 1, + "commands": [ + { + "assistant_turn_index": 1, + "duration_ms": "", + "error_message": "no result observed", + "execution_completed_at": "", + "execution_started_at": "", + "generation_completed_at": null, + "parameters": { + "command": "sleep 600" + }, + "result_data": null, + "result_status": "unknown", + "result_summary": null, + "result_tokens": 0, + "sequence_number": 1, + "timestamp": "", + "tool_id": "call_1", + "tool_name": "Bash" + } + ], + "crash_reason": null, + "crashed": false, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "max_turns_exhausted": false, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "Waiting.", + "thinking": null, + "tool_use_id": null + }, + { + "block_type": "tool_use", + "is_error": false, + "sequence": 1, + "signature": null, + "text": null, + "thinking": null, + "tool_use_id": "call_1" + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": null, + "model": "openrouter/moonshotai/kimi-k3", + "output_tokens": 20, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [ + "call_1" + ] + } + ], + "model_used": "openrouter/moonshotai/kimi-k3", + "num_turns": 1, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": "stop", + "subtype": "completed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 100, + "output_tokens": 20, + "total_cost_usd": "", + "uncached_input_tokens": 100 + }, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/pi_e_error_after_generation.json b/tests/_fixtures/golden_streams/expected/pi_e_error_after_generation.json new file mode 100644 index 00000000..33a4d72e --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/pi_e_error_after_generation.json @@ -0,0 +1,76 @@ +{ + "agent_output": "Starting.", + "assistant_turn_count": 2, + "commands": [], + "crash_reason": "Pi error: provider returned 529 after 5 retries", + "crashed": true, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "max_turns_exhausted": false, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "Starting.", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": null, + "model": "openrouter/moonshotai/kimi-k3", + "output_tokens": 20, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [] + }, + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [], + "generation_duration_ms": "", + "input_tokens": 0, + "message_id": null, + "model": "openrouter/moonshotai/kimi-k3", + "output_tokens": 0, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "error", + "tool_use_ids": [] + } + ], + "model_used": "openrouter/moonshotai/kimi-k3", + "num_turns": 2, + "result_summary": { + "is_error": true, + "result": "Pi error: provider returned 529 after 5 retries", + "stop_reason": "error", + "subtype": "crashed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 100, + "output_tokens": 20, + "total_cost_usd": "", + "uncached_input_tokens": 100 + }, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/pi_f_duplicate_turn_end.json b/tests/_fixtures/golden_streams/expected/pi_f_duplicate_turn_end.json new file mode 100644 index 00000000..66c7e39d --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/pi_f_duplicate_turn_end.json @@ -0,0 +1,86 @@ +{ + "agent_output": "First.", + "assistant_turn_count": 1, + "commands": [], + "crash_reason": null, + "crashed": false, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "max_turns_exhausted": false, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "First.", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": null, + "model": "openrouter/moonshotai/kimi-k3", + "output_tokens": 20, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [] + }, + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "First.", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 10, + "message_id": null, + "model": "openrouter/moonshotai/kimi-k3", + "output_tokens": 5, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [] + } + ], + "model_used": "openrouter/moonshotai/kimi-k3", + "num_turns": 1, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": "stop", + "subtype": "completed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 110, + "output_tokens": 25, + "total_cost_usd": "", + "uncached_input_tokens": 110 + }, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/opencode_fixtures.py b/tests/_fixtures/golden_streams/opencode_fixtures.py index f88c6a32..5f138147 100644 --- a/tests/_fixtures/golden_streams/opencode_fixtures.py +++ b/tests/_fixtures/golden_streams/opencode_fixtures.py @@ -25,23 +25,61 @@ import json import os from dataclasses import dataclass +from datetime import datetime from typing import Any from unittest.mock import patch from coder_eval.agents.opencode_agent import OpenCodeAgent +from coder_eval.errors import AgentCrashError from coder_eval.models import OpenCodeAgentConfig SESSION = "ses_test123" +# Base epoch milliseconds for the recorded stream. A BASE, not a wall-clock +# claim: `_rebase_lines` shifts the whole timeline onto the replay's own clock +# before the scenario runs, so these stamps and the agent's own `datetime.now()` +# event stamps are commensurable. Left absolute they sit a month away from the +# replay, which puts the recorded tool interval outside every measured window. +_T0_MS = 1_786_663_016_802 + +# How far after the replay's start the rebased timeline begins — small, but +# non-zero so the first window opens after the AgentStartEvent. +_REPLAY_LEAD_MS = 2 + + def _evt(event_type: str, part: dict[str, Any]) -> str: """One CLI event line: payload under ``part``, sessionID on the envelope.""" return json.dumps( - {"type": event_type, "timestamp": 1786663016802, "sessionID": SESSION, "part": {"sessionID": SESSION, **part}} + {"type": event_type, "timestamp": _T0_MS, "sessionID": SESSION, "part": {"sessionID": SESSION, **part}} ) +def _rebase_lines(lines: list[str]) -> list[str]: + """Shift every recorded stamp from ``_T0_MS`` onto the replay's own clock. + + Keeps every DERIVED duration exact (a 17 ms tool stays 17 ms) and fixes + only the era, so the head and tail the collector records against the + agent's `datetime.now()` stamps are meaningful rather than a month wide. + """ + offset = int(datetime.now().timestamp() * 1000) - _T0_MS + _REPLAY_LEAD_MS + + def shift(node: Any) -> Any: + if isinstance(node, dict): + return {k: (v + offset if k in _STAMP_KEYS and isinstance(v, int) else shift(v)) for k, v in node.items()} + if isinstance(node, list): + return [shift(v) for v in node] + return node + + return [json.dumps(shift(json.loads(line))) for line in lines] + + +# Millisecond-epoch keys anywhere in an event payload: the envelope's own +# stamp, and a tool's `state.time` bounds. +_STAMP_KEYS = frozenset({"timestamp", "start", "end"}) + + def _tokens(inp: int, out: int, *, write: int = 0, read: int = 0, reasoning: int = 0) -> dict[str, Any]: """Token payload in the NESTED convention (total = input+output+reasoning, cache counted inside `input`); see TestTokenShapeIsObservable for the flat one.""" @@ -164,18 +202,23 @@ def _agent() -> OpenCodeAgent: class OpenCodeScenario: """One recorded CLI event stream. - No ``expects`` knob: every scenario here replays cleanly. The crash and - timeout paths live in the agent's own test module, which asserts on the - exception rather than on a snapshot. + ``expects`` names the exception a scenario is supposed to raise, and the + runner then snapshots ``pending_turn`` instead of the returned record — + the same knob ``ClaudeScenario`` carries, for the same reason: the partial + a crash preserves is a real capture path, and one nobody was comparing + against a snapshot on this harness. """ name: str lines: list[str] + expects: type[BaseException] | None = None async def run_opencode_scenario(scenario: OpenCodeScenario, working_dir: str) -> dict[str, Any]: """Replay one scenario and return the resulting record as a plain dump.""" - proc = _FakeProcess(scenario.lines) + import pytest + + proc = _FakeProcess(_rebase_lines(scenario.lines)) async def fake_exec(*_argv: str, **_kwargs: Any) -> _FakeProcess: proc.stderr = proc # type: ignore[assignment] @@ -188,7 +231,13 @@ async def fake_exec(*_argv: str, **_kwargs: Any) -> _FakeProcess: patch.object(os, "killpg", lambda _pgid, _sig: None, create=True), ): await agent.start(working_dir) - record = await agent.communicate("do it") + if scenario.expects is not None: + with pytest.raises(scenario.expects): + await agent.communicate("do it") + record = agent.pending_turn + assert record is not None, f"{scenario.name}: pending_turn was not set on the failure path" + else: + record = await agent.communicate("do it") return record.model_dump(mode="json") @@ -221,6 +270,105 @@ def _build_catalogue() -> list[OpenCodeScenario]: OpenCodeScenario(name="b_tool_call_resolved", lines=list(HAPPY_STREAM)), ) + # (c) two generations with a tool resolving between them. The TILING case: + # the second window opens at the first `step_finish`, not at its own + # `step_start`, so the wall clock between the two steps — the model time + # that produced the second one — lands inside a window rather than in no + # bucket at all. That is the defect this harness shipped with, and it had + # a unit test but no golden. + scenarios.append( + OpenCodeScenario( + name="c_multi_step_tiling", + lines=[ + _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}), + _evt( + "tool_use", + { + "id": "prt_2", + "messageID": "msg_1", + "tool": "bash", + "callID": "call_1", + "state": { + "status": "completed", + "input": {"command": "ls"}, + "output": "main.py", + "time": {"start": _T0_MS, "end": _T0_MS + 5}, + }, + }, + ), + _evt( + "step_finish", + {"id": "prt_3", "messageID": "msg_1", "reason": "tool-calls", "tokens": _tokens(100, 20)}, + ), + _evt("step_start", {"id": "prt_4", "messageID": "msg_2"}), + _evt("text", {"id": "prt_5", "messageID": "msg_2", "text": "Listed it."}), + _evt( + "step_finish", + {"id": "prt_6", "messageID": "msg_2", "reason": "stop", "tokens": _tokens(50, 30)}, + ), + ], + ) + ) + + # (d) a tool the CLI opens and never resolves — force-closed as `unresolved` + # by the orphan sweep at finalization. It carries NO `state.time`, which is + # the honest shape for a call that never returned: with no + # `execution_started_at` there is no `duration_ms` and no span. + # + # READ THE SNAPSHOT: the sweep still stamps `execution_completed_at`, which + # it does on every close path, so the record holds an end with no + # beginning. Compare `pi_d_orphaned_tool`, where the start IS stamped and a + # manufactured duration follows from it. + scenarios.append( + OpenCodeScenario( + name="d_orphaned_tool", + lines=[ + _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}), + _evt( + "tool_use", + { + "id": "prt_2", + "messageID": "msg_1", + "tool": "bash", + "callID": "call_1", + "state": {"status": "pending", "input": {"command": "sleep 600"}}, + }, + ), + _evt("text", {"id": "prt_3", "messageID": "msg_1", "text": "Waiting."}), + _evt( + "step_finish", + {"id": "prt_4", "messageID": "msg_1", "reason": "stop", "tokens": _tokens(100, 20)}, + ), + ], + ) + ) + + # (e) the CLI's own structured error AFTER a complete generation. `_settle_turn` + # crashes on it, and the partial `pending_turn` must still carry that + # generation and its head/tail — a crash does not un-measure what was + # measured before it. + scenarios.append( + OpenCodeScenario( + name="e_error_after_generation", + lines=[ + _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}), + _evt("text", {"id": "prt_2", "messageID": "msg_1", "text": "Starting."}), + _evt( + "step_finish", + {"id": "prt_3", "messageID": "msg_1", "reason": "stop", "tokens": _tokens(100, 20)}, + ), + json.dumps( + { + "type": "error", + "sessionID": SESSION, + "error": {"name": "ProviderAuthError", "data": {"message": "401 from the provider"}}, + } + ), + ], + expects=AgentCrashError, + ) + ) + return scenarios diff --git a/tests/_fixtures/golden_streams/pi_fixtures.py b/tests/_fixtures/golden_streams/pi_fixtures.py index 26297241..3dfb5499 100644 --- a/tests/_fixtures/golden_streams/pi_fixtures.py +++ b/tests/_fixtures/golden_streams/pi_fixtures.py @@ -24,6 +24,7 @@ from unittest.mock import patch from coder_eval.agents.pi_agent import PiAgent +from coder_eval.errors import AgentCrashError from coder_eval.models import PiAgentConfig @@ -63,10 +64,45 @@ def _turn_end(*, inp: int, out: int, cache_read: int = 0, cache_write: int = 0, ) +def _text(delta: str) -> str: + """One streamed assistant text delta. + + The CLI's real shape, which is `message_update` carrying an + `assistantMessageEvent` of type `text_delta` — NOT a bare `{"type": "text"}` + line. `_handle_line` dispatches on the outer `type`, so a bare `text` line + is unrecognized vocabulary: it reaches no handler, appends no text, and + leaves `agent_output` empty while the scenario still passes. + `a_single_text_turn` was written that way and asserted nothing about the + text capture its own name claims. + """ + return json.dumps({"type": "message_update", "assistantMessageEvent": {"type": "text_delta", "delta": delta}}) + + def _tool_start(call_id: str, name: str, args: dict[str, Any]) -> str: return json.dumps({"type": "tool_execution_start", "toolCallId": call_id, "toolName": name, "args": args}) +def _turn_end_error(message: str) -> str: + """A `turn_end` whose `stopReason` is the provider error pi could not retry away. + + `pi -p` exits 0 after exhausting its internal retries, so this is the only + signal that the turn died — `_settle_turn` crashes on it precisely so the + row does not book as a clean failure and silently depress the pass rate. + """ + return json.dumps( + { + "type": "turn_end", + "message": { + "role": "assistant", + "usage": {"input": 0, "output": 0, "cost": {"total": 0.0}}, + "stopReason": "error", + "errorMessage": message, + }, + "toolResults": [], + } + ) + + def _tool_end(call_id: str, name: str, text: str, *, is_error: bool = False) -> str: return json.dumps( { @@ -155,17 +191,22 @@ def _agent() -> PiAgent: class PiScenario: """One recorded CLI event stream. - No ``expects`` knob: every scenario here replays cleanly. The crash and - timeout paths live in the agent's own test module, which asserts on the - exception rather than on a snapshot. + ``expects`` names the exception a scenario is supposed to raise, and the + runner then snapshots ``pending_turn`` instead of the returned record — + the same knob ``ClaudeScenario`` carries, for the same reason: the partial + a crash preserves is a real capture path, and one nobody was comparing + against a snapshot on this harness. """ name: str lines: list[str] + expects: type[BaseException] | None = None async def run_pi_scenario(scenario: PiScenario, working_dir: str) -> dict[str, Any]: """Replay one scenario and return the resulting record as a plain dump.""" + import pytest + proc = _FakeProcess(scenario.lines) async def fake_exec(*_argv: str, **_kwargs: Any) -> _FakeProcess: @@ -179,7 +220,13 @@ async def fake_exec(*_argv: str, **_kwargs: Any) -> _FakeProcess: patch.object(os, "killpg", lambda _pgid, _sig: None, create=True), ): await agent.start(working_dir) - record = await agent.communicate("do it") + if scenario.expects is not None: + with pytest.raises(scenario.expects): + await agent.communicate("do it") + record = agent.pending_turn + assert record is not None, f"{scenario.name}: pending_turn was not set on the failure path" + else: + record = await agent.communicate("do it") return record.model_dump(mode="json") @@ -192,7 +239,7 @@ def _build_catalogue() -> list[PiScenario]: name="a_single_text_turn", lines=[ _turn_start(), - json.dumps({"type": "text", "text": "All done."}), + _text("All done."), _turn_end(inp=100, out=20, cache_read=64, cost=0.001), ], ) @@ -201,6 +248,98 @@ def _build_catalogue() -> list[PiScenario]: # (b) the captured live stream: three turns with resolved tool calls. scenarios.append(PiScenario(name="b_tool_call_resolved", lines=list(HAPPY_STREAM))) + # (c) two generations with a tool resolving between them. The TILING case: + # the second window opens at the first `turn_end`, not at its own + # `turn_start`, so the wall clock between the two turns — the model time + # that produced the second one — lands inside a window rather than in no + # bucket at all. Pi was the harness that shipped that defect, and it had a + # unit test but no golden. Every stamp here comes from the reducer's own + # TurnClock, so this scenario stays inside the identity check. + scenarios.append( + PiScenario( + name="c_multi_turn_tiling", + lines=[ + _turn_start(), + _tool_start("call_1", "bash", {"command": "ls"}), + _tool_end("call_1", "bash", "main.py"), + _turn_end(inp=100, out=20, cost=0.001), + _turn_start(), + _text("Listed it."), + _turn_end(inp=50, out=30, cost=0.002), + ], + ) + ) + + # (d) a tool the CLI opens and never resolves — force-closed as `unresolved` + # by the orphan sweep at finalization. + # + # READ THE SNAPSHOT: it carries a `duration_ms` and BOTH execution bounds, + # and that span is subtracted from the generation window. Its + # `execution_completed_at` is the instant the sweep ran, not a completion + # anybody observed, so the duration is manufactured — and `_close_tool`'s + # own comment ("Only a RESOLVED tool contributes: one force-closed without + # a result was never timed") describes a guard it does not have: the test + # is `execution_started_at is not None`, which an orphan passes. + # claude-code's `_finalize_commands` deliberately leaves `duration_ms` + # None in exactly this case, and says why. Captured rather than fixed: + # this scenario is what makes it visible. + scenarios.append( + PiScenario( + name="d_orphaned_tool", + lines=[ + _turn_start(), + _tool_start("call_1", "bash", {"command": "sleep 600"}), + _text("Waiting."), + _turn_end(inp=100, out=20, cost=0.001), + ], + ) + ) + + # (e) the provider error pi's internal retries could not clear, AFTER a + # complete generation. The CLI still exits 0, so `_settle_turn` crashes on + # `stopReason=error` alone — and the partial `pending_turn` must still carry + # that generation and its head/tail. A crash does not un-measure what was + # measured before it. + scenarios.append( + PiScenario( + name="e_error_after_generation", + lines=[ + _turn_start(), + _text("Starting."), + _turn_end(inp=100, out=20, cost=0.001), + _turn_start(), + _turn_end_error("provider returned 529 after 5 retries"), + ], + expects=AgentCrashError, + ) + ) + + # (f) a duplicate `turn_end` with no `turn_start` between — a transport + # hiccup this reducer explicitly promises to survive, since pi retries + # internally. A spent `turn_started_at` left in place reopens the next + # window at the PREVIOUS turn's start and republishes that whole span: + # reproduced as 3000 ms of generation for a 2000 ms turn. It had a unit test + # and no golden. + # + # READ THE SNAPSHOT: it records that the TIMING half of that reset is fixed + # and the CONTENT half is not. `turn_text_parts` / `turn_tool_ids` are + # cleared in `on_turn_start` only, so the second `turn_end` publishes the + # first turn's text a second time, as its own assistant message. The + # argument `on_turn_end`'s comment makes for moving `turn_started_at` out of + # `on_turn_start` applies to those two lists unchanged. Captured here rather + # than fixed: this scenario is what makes it visible at all. + scenarios.append( + PiScenario( + name="f_duplicate_turn_end", + lines=[ + _turn_start(), + _text("First."), + _turn_end(inp=100, out=20, cost=0.001), + _turn_end(inp=10, out=5, cost=0.0001), + ], + ) + ) + return scenarios diff --git a/tests/_fixtures/timing_runs/README.md b/tests/_fixtures/timing_runs/README.md new file mode 100644 index 00000000..838b0c21 --- /dev/null +++ b/tests/_fixtures/timing_runs/README.md @@ -0,0 +1,53 @@ +# Pinned per-harness timing corpus + +One scrubbed, representative `task.json` per harness, from a real +`tasks/timing-parallel-tools` run. Prompts, outputs, tokens and cost are +stripped; only the wall-clock fields `scripts/timing/decompose_run.py` reads +survive. + +``` +uv run python scripts/timing/decompose_run.py tests/_fixtures/timing_runs/*.json --min-turn-ms 0 +``` + +## What this is NOT + +**It cannot show a before/after for a code change, and it was originally asked +to.** `decompose_run.py` READS STORED FIELDS — `harness_startup_ms`, +`generation_duration_ms`, the command bounds — out of a recorded record. It +recomputes nothing from `src/`. Run over a fixed corpus it prints the identical +table before and after any change to the harness, so a green result would be +meaningless rather than reassuring. + +Two rows here make that concrete: + +- **claude-code reconciles at −481 ms (−2.69%).** That is the pre-subtraction + defect `_subtract_tool_time_from_windows` was written to fix, frozen in a + record written before the fix landed. The live code has not had that defect + for some time. +- **claude-code and antigravity both book a `0.0` head.** That is the clamped + inversion the "one meaning for `harness_startup_ms`" change removed. Neither + harness produces it any more. + +Re-recording the corpus after a change would fix both, and would also destroy +the only thing the corpus is good for. + +## What it IS + +A reproducible statement of what real runs of each harness look like — the +shape of the buckets, the per-harness spread in the head, and a fixed input for +`decompose_run.py` itself. It is what the timing audit's P0 was read off: the +two harnesses reporting a `0.0` head had a FIRST generation window 2.4–3.8× their +own later median, and that excess was the startup they were not booking. + +## The instruments that DO move with the code + +- `tests/test_timing_identity_contract.py` — drives each of the five reducers + off a scripted clock and asserts `head + Σgeneration + UNION(tool) + tail` + equals the turn span to the millisecond. This is the sensor for the + arithmetic. +- `tests/test_agent_golden_master.py` — replays recorded streams end to end. + Note it masks every timing VALUE (`_scrub.py::SCRUB_KEYS`), so it sees shape + and not magnitude. +- `scripts/timing/decompose_run.py --max-residual-pct` over a **fresh** run, + which `.github/workflows/pr-checks.yml` does against the smoke-pass bucket. + A live run is the only way this script can say anything about current code. diff --git a/tests/_fixtures/timing_runs/antigravity.json b/tests/_fixtures/timing_runs/antigravity.json new file mode 100644 index 00000000..a662f2f9 --- /dev/null +++ b/tests/_fixtures/timing_runs/antigravity.json @@ -0,0 +1,134 @@ +{ + "_comment": "Scrubbed representative run for the timing residual report. Prompts, outputs, tokens and cost are removed; only the wall-clock fields decompose_run.py reads are kept. NOT a before/after instrument \u2014 see the header in this directory.", + "agent_type": "antigravity", + "iterations": [ + { + "duration_seconds": 11.191662207944319, + "iteration": 1, + "harness_startup_ms": 0.0, + "harness_teardown_ms": 0.116, + "crashed": false, + "messages": [ + { + "completed_at": "2026-09-11T21:25:56.137300", + "started_at": "2026-09-11T21:25:51.666942", + "generation_duration_ms": 4454.441, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T21:26:00.773915", + "started_at": "2026-09-11T21:25:56.137300", + "generation_duration_ms": 4574.724999999999, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T21:26:02.858513", + "started_at": "2026-09-11T21:26:00.773915", + "generation_duration_ms": 0.3619999999996253, + "role": "assistant", + "parent_tool_use_id": null + } + ], + "commands": [ + { + "tool_name": "Edit", + "execution_completed_at": "2026-09-11T21:25:55.126590", + "execution_started_at": "2026-09-11T21:25:55.122510", + "tool_id": "b7747e76d7fb722e11ea267731099383:2", + "result_status": "success", + "duration_ms": 4.08 + }, + { + "tool_name": "Edit", + "execution_completed_at": "2026-09-11T21:25:55.455388", + "execution_started_at": "2026-09-11T21:25:55.452684", + "tool_id": "b7747e76d7fb722e11ea267731099383:3", + "result_status": "success", + "duration_ms": 2.7039999999999997 + }, + { + "tool_name": "Edit", + "execution_completed_at": "2026-09-11T21:25:55.798964", + "execution_started_at": "2026-09-11T21:25:55.796438", + "tool_id": "b7747e76d7fb722e11ea267731099383:4", + "result_status": "success", + "duration_ms": 2.5260000000000002 + }, + { + "tool_name": "Edit", + "execution_completed_at": "2026-09-11T21:25:56.101913", + "execution_started_at": "2026-09-11T21:25:56.099180", + "tool_id": "b7747e76d7fb722e11ea267731099383:5", + "result_status": "success", + "duration_ms": 2.733 + }, + { + "tool_name": "Edit", + "execution_completed_at": "2026-09-11T21:25:56.140758", + "execution_started_at": "2026-09-11T21:25:56.133426", + "tool_id": "b7747e76d7fb722e11ea267731099383:6", + "result_status": "success", + "duration_ms": 7.332 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T21:25:59.328201", + "execution_started_at": "2026-09-11T21:25:59.323643", + "tool_id": "b7747e76d7fb722e11ea267731099383:8", + "result_status": "success", + "duration_ms": 4.558000000000001 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T21:25:59.550767", + "execution_started_at": "2026-09-11T21:25:59.549154", + "tool_id": "b7747e76d7fb722e11ea267731099383:9", + "result_status": "success", + "duration_ms": 1.613 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T21:25:59.801832", + "execution_started_at": "2026-09-11T21:25:59.800660", + "tool_id": "b7747e76d7fb722e11ea267731099383:10", + "result_status": "success", + "duration_ms": 1.1720000000000002 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T21:26:00.087008", + "execution_started_at": "2026-09-11T21:26:00.086176", + "tool_id": "b7747e76d7fb722e11ea267731099383:11", + "result_status": "success", + "duration_ms": 0.832 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T21:26:00.395861", + "execution_started_at": "2026-09-11T21:26:00.394123", + "tool_id": "b7747e76d7fb722e11ea267731099383:12", + "result_status": "success", + "duration_ms": 1.738 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T21:26:02.768580", + "execution_started_at": "2026-09-11T21:26:00.725396", + "tool_id": "b7747e76d7fb722e11ea267731099383:13", + "result_status": "success", + "duration_ms": 2043.1840000000002 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T21:26:02.858151", + "execution_started_at": "2026-09-11T21:26:00.773556", + "tool_id": "b7747e76d7fb722e11ea267731099383:14", + "result_status": "success", + "duration_ms": 2084.5950000000003 + } + ] + } + ] +} diff --git a/tests/_fixtures/timing_runs/claude-code.json b/tests/_fixtures/timing_runs/claude-code.json new file mode 100644 index 00000000..1eff89cc --- /dev/null +++ b/tests/_fixtures/timing_runs/claude-code.json @@ -0,0 +1,225 @@ +{ + "_comment": "Scrubbed representative run for the timing residual report. Prompts, outputs, tokens and cost are removed; only the wall-clock fields decompose_run.py reads are kept. NOT a before/after instrument \u2014 see the header in this directory.", + "agent_type": "claude-code", + "iterations": [ + { + "duration_seconds": 17.885937000159174, + "iteration": 1, + "harness_startup_ms": 0.0, + "harness_teardown_ms": 857.9449999999999, + "crashed": false, + "messages": [ + { + "completed_at": "2026-09-11T07:13:59.899445", + "started_at": "2026-09-11T07:13:56.052088", + "generation_duration_ms": 3847.4723750259727, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:00.221777", + "started_at": "2026-09-11T07:13:59.899445", + "generation_duration_ms": 322.3441250156611, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:01.010903", + "started_at": "2026-09-11T07:14:00.221777", + "generation_duration_ms": 789.1439578961581, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:01.672766", + "started_at": "2026-09-11T07:14:01.032139", + "generation_duration_ms": 640.6468339264393, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:02.386421", + "started_at": "2026-09-11T07:14:01.676561", + "generation_duration_ms": 709.8735831677914, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:03.064864", + "started_at": "2026-09-11T07:14:02.403329", + "generation_duration_ms": 661.5535838063806, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:03.747205", + "started_at": "2026-09-11T07:14:03.070861", + "generation_duration_ms": 676.3597088865936, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:04.328511", + "started_at": "2026-09-11T07:14:03.768013", + "generation_duration_ms": 560.5133329518139, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:04.917330", + "started_at": "2026-09-11T07:14:04.338140", + "generation_duration_ms": 579.204916022718, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:05.485979", + "started_at": "2026-09-11T07:14:04.920393", + "generation_duration_ms": 565.5976661946625, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:06.071761", + "started_at": "2026-09-11T07:14:05.493927", + "generation_duration_ms": 577.8485001064837, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:06.671228", + "started_at": "2026-09-11T07:14:06.078218", + "generation_duration_ms": 593.0241669993848, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:07.253322", + "started_at": "2026-09-11T07:14:06.678189", + "generation_duration_ms": 575.1475831493735, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:07.734996", + "started_at": "2026-09-11T07:14:07.253322", + "generation_duration_ms": 481.6872498486191, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:12.089632", + "started_at": "2026-09-11T07:14:09.587005", + "generation_duration_ms": 2502.6892910245806, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:13.079613", + "started_at": "2026-09-11T07:14:12.089632", + "generation_duration_ms": 990.0077090132982, + "role": "assistant", + "parent_tool_use_id": null + } + ], + "commands": [ + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:14:01.032594", + "execution_started_at": "2026-09-11T07:14:01.011427", + "tool_id": "toolu_01BqWiy1hXCaepichHH9dnXC", + "result_status": "success", + "duration_ms": 21.16704103536904 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:14:01.676585", + "execution_started_at": "2026-09-11T07:14:01.672806", + "tool_id": "toolu_01YN7oVP3MEuUGhNgtoYXDhV", + "result_status": "success", + "duration_ms": 3.7791249342262745 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:14:02.403402", + "execution_started_at": "2026-09-11T07:14:02.386537", + "tool_id": "toolu_01JNtoHGHMzD4QamxrePBR9T", + "result_status": "success", + "duration_ms": 16.864625038579106 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:14:03.070898", + "execution_started_at": "2026-09-11T07:14:03.064935", + "tool_id": "toolu_01DaWJKb1Hm6bxcpiKjVzeYd", + "result_status": "success", + "duration_ms": 5.963291972875595 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:14:03.768071", + "execution_started_at": "2026-09-11T07:14:03.747296", + "tool_id": "toolu_014MD2sCpV9RhaF9TjAQccFT", + "result_status": "success", + "duration_ms": 20.77529113739729 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:14:04.338200", + "execution_started_at": "2026-09-11T07:14:04.328605", + "tool_id": "toolu_01M2q1dfDu3Vsxte9dSv6AhA", + "result_status": "success", + "duration_ms": 9.594874922186136 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:14:04.920425", + "execution_started_at": "2026-09-11T07:14:04.917406", + "tool_id": "toolu_01FueR7TLLvWTEWXEbfGErzE", + "result_status": "success", + "duration_ms": 3.019041148945689 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:14:05.493992", + "execution_started_at": "2026-09-11T07:14:05.486085", + "tool_id": "toolu_017wr4qR5X22aegAGbtx9PVE", + "result_status": "success", + "duration_ms": 7.906709099188447 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:14:06.078272", + "execution_started_at": "2026-09-11T07:14:06.071847", + "tool_id": "toolu_01HDYj2UeY1Nagd1ecjbs81V", + "result_status": "success", + "duration_ms": 6.425125058740377 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:14:06.678240", + "execution_started_at": "2026-09-11T07:14:06.671314", + "tool_id": "toolu_01UbZp7DRorw4cUbo5PB2Bia", + "result_status": "success", + "duration_ms": 6.9256669376045465 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:09.527272", + "execution_started_at": "2026-09-11T07:14:07.253341", + "tool_id": "toolu_0194Py8X1atsEPAA5WAWVjyT", + "result_status": "success", + "duration_ms": 2273.930750088766 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:09.587030", + "execution_started_at": "2026-09-11T07:14:07.734993", + "tool_id": "toolu_018Nbqg6YSdYk6fb9c4gWAGy", + "result_status": "success", + "duration_ms": 1852.037207921967 + } + ] + } + ] +} diff --git a/tests/_fixtures/timing_runs/codex.json b/tests/_fixtures/timing_runs/codex.json new file mode 100644 index 00000000..0042aaf6 --- /dev/null +++ b/tests/_fixtures/timing_runs/codex.json @@ -0,0 +1,109 @@ +{ + "_comment": "Scrubbed representative run for the timing residual report. Prompts, outputs, tokens and cost are removed; only the wall-clock fields decompose_run.py reads are kept. NOT a before/after instrument \u2014 see the header in this directory.", + "agent_type": "codex", + "iterations": [ + { + "duration_seconds": 19.704716542037204, + "iteration": 1, + "harness_startup_ms": 4209.975, + "harness_teardown_ms": 6.189, + "crashed": false, + "messages": [ + { + "completed_at": "2026-09-11T07:14:25.044000", + "started_at": "2026-09-11T07:14:22.104000", + "generation_duration_ms": 1593.815668, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:25.044000", + "started_at": "2026-09-11T07:14:22.104000", + "generation_duration_ms": 1337.184332, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:31.755000", + "started_at": "2026-09-11T07:14:25.044000", + "generation_duration_ms": 6436.0, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:14:37.592000", + "started_at": "2026-09-11T07:14:31.755000", + "generation_duration_ms": 4040.0, + "role": "assistant", + "parent_tool_use_id": null + } + ], + "commands": [ + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:14:25.044000", + "execution_started_at": "2026-09-11T07:14:25.035000", + "tool_id": "call_MFDN1SquQKjnPMAC7cdID6es", + "result_status": "success", + "duration_ms": 9.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:31.452000", + "execution_started_at": "2026-09-11T07:14:31.452000", + "tool_id": "call_XoyG0xCaUZkVh4gs4HscEQY6", + "result_status": "success", + "duration_ms": 0.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:31.474000", + "execution_started_at": "2026-09-11T07:14:31.474000", + "tool_id": "call_3xwdnJ9958Qws766mtNMUmum", + "result_status": "success", + "duration_ms": 0.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:31.480000", + "execution_started_at": "2026-09-11T07:14:31.480000", + "tool_id": "call_kQU4wxYIqLLd7MxdhVBXt1fz", + "result_status": "success", + "duration_ms": 0.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:33.552000", + "execution_started_at": "2026-09-11T07:14:31.480000", + "tool_id": "call_EHI6niBRHKpW9GSxYgbSGPQn", + "result_status": "success", + "duration_ms": 2072.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:31.483000", + "execution_started_at": "2026-09-11T07:14:31.483000", + "tool_id": "call_5UNrspGHKnOkwpcR5PedDt5B", + "result_status": "success", + "duration_ms": 0.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:14:31.487000", + "execution_started_at": "2026-09-11T07:14:31.487000", + "tool_id": "call_xOqzas8ctiA0ri1wIsfiJobK", + "result_status": "success", + "duration_ms": 0.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": null, + "execution_started_at": null, + "tool_id": "call_zAeVW6TDp7vxHy3HilRTAa3i", + "result_status": null, + "duration_ms": null + } + ] + } + ] +} diff --git a/tests/_fixtures/timing_runs/opencode.json b/tests/_fixtures/timing_runs/opencode.json new file mode 100644 index 00000000..749e62e3 --- /dev/null +++ b/tests/_fixtures/timing_runs/opencode.json @@ -0,0 +1,171 @@ +{ + "_comment": "Scrubbed representative run for the timing residual report. Prompts, outputs, tokens and cost are removed; only the wall-clock fields decompose_run.py reads are kept. NOT a before/after instrument \u2014 see the header in this directory.", + "agent_type": "opencode", + "iterations": [ + { + "duration_seconds": 22.314462833106518, + "iteration": 1, + "harness_startup_ms": 3018.525, + "harness_teardown_ms": 26.273, + "crashed": false, + "messages": [ + { + "completed_at": "2026-09-11T20:13:55.133594", + "started_at": "2026-09-11T20:13:53.488937", + "generation_duration_ms": 1640.657, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T20:13:59.771406", + "started_at": "2026-09-11T20:13:55.133594", + "generation_duration_ms": 4597.812, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T20:14:03.986747", + "started_at": "2026-09-11T20:13:59.771406", + "generation_duration_ms": 4171.340999999999, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T20:14:07.755801", + "started_at": "2026-09-11T20:14:03.986747", + "generation_duration_ms": 1705.054, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T20:14:10.259492", + "started_at": "2026-09-11T20:14:07.755801", + "generation_duration_ms": 2501.691, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T20:14:12.758747", + "started_at": "2026-09-11T20:14:10.259492", + "generation_duration_ms": 2499.2549999999997, + "role": "assistant", + "parent_tool_use_id": null + } + ], + "commands": [ + { + "tool_name": "TodoWrite", + "execution_completed_at": "2026-09-11T20:13:55.119000", + "execution_started_at": "2026-09-11T20:13:55.115000", + "tool_id": "toolu_01WfZGLMB5HLRS9E15jLYJq8", + "result_status": "success", + "duration_ms": 4.0 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T20:13:56.979000", + "execution_started_at": "2026-09-11T20:13:56.968000", + "tool_id": "toolu_01WLeJLQCihZjNtCgwCi1xpL", + "result_status": "success", + "duration_ms": 11.0 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T20:13:57.684000", + "execution_started_at": "2026-09-11T20:13:57.678000", + "tool_id": "toolu_019tjDMgudrqyntWUAh8Aupy", + "result_status": "success", + "duration_ms": 6.0 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T20:13:58.385000", + "execution_started_at": "2026-09-11T20:13:58.376000", + "tool_id": "toolu_013J2dP6wxbm4q1acY5PXr6s", + "result_status": "success", + "duration_ms": 9.0 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T20:13:59.080000", + "execution_started_at": "2026-09-11T20:13:59.075000", + "tool_id": "toolu_01DZAPiHZrcfEiDeUDVdur6H", + "result_status": "success", + "duration_ms": 5.0 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T20:13:59.760000", + "execution_started_at": "2026-09-11T20:13:59.751000", + "tool_id": "toolu_01S2Kd2KUueP11tuBZ3hX53j", + "result_status": "success", + "duration_ms": 9.0 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T20:14:01.518000", + "execution_started_at": "2026-09-11T20:14:01.506000", + "tool_id": "toolu_01RbNdCyWmL3XcWUZ1BZBMVs", + "result_status": "success", + "duration_ms": 12.0 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T20:14:02.089000", + "execution_started_at": "2026-09-11T20:14:02.079000", + "tool_id": "toolu_01SDGHLgG9HR9aCieXd81Wz3", + "result_status": "success", + "duration_ms": 10.0 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T20:14:02.660000", + "execution_started_at": "2026-09-11T20:14:02.654000", + "tool_id": "toolu_01XER7fGNDxwMaaucfPhKS7C", + "result_status": "success", + "duration_ms": 6.0 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T20:14:03.239000", + "execution_started_at": "2026-09-11T20:14:03.232000", + "tool_id": "toolu_01TFRCQq7bYkiBwHvSpc3Cxr", + "result_status": "success", + "duration_ms": 7.0 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T20:14:03.974000", + "execution_started_at": "2026-09-11T20:14:03.965000", + "tool_id": "toolu_01MHcPMKPGMihm8MkmhAWBtu", + "result_status": "success", + "duration_ms": 9.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T20:14:06.181000", + "execution_started_at": "2026-09-11T20:14:06.125000", + "tool_id": "toolu_014B6sD12tnYcpCAHbC7YJbS", + "result_status": "success", + "duration_ms": 56.0 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T20:14:07.745000", + "execution_started_at": "2026-09-11T20:14:05.681000", + "tool_id": "toolu_01KCeAUpoHtjG2ssT85e4o9c", + "result_status": "success", + "duration_ms": 2064.0 + }, + { + "tool_name": "TodoWrite", + "execution_completed_at": "2026-09-11T20:14:10.249000", + "execution_started_at": "2026-09-11T20:14:10.247000", + "tool_id": "toolu_01GMeN29huU3tN1BHpusugFh", + "result_status": "success", + "duration_ms": 2.0 + } + ] + } + ] +} diff --git a/tests/_fixtures/timing_runs/pi.json b/tests/_fixtures/timing_runs/pi.json new file mode 100644 index 00000000..e6a2a592 --- /dev/null +++ b/tests/_fixtures/timing_runs/pi.json @@ -0,0 +1,141 @@ +{ + "_comment": "Scrubbed representative run for the timing residual report. Prompts, outputs, tokens and cost are removed; only the wall-clock fields decompose_run.py reads are kept. NOT a before/after instrument \u2014 see the header in this directory.", + "agent_type": "pi", + "iterations": [ + { + "duration_seconds": 14.706855208845809, + "iteration": 1, + "harness_startup_ms": 344.118, + "harness_teardown_ms": 22.406, + "crashed": false, + "messages": [ + { + "completed_at": "2026-09-11T07:42:08.410088", + "started_at": "2026-09-11T07:42:03.676385", + "generation_duration_ms": 4716.634, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:42:10.831543", + "started_at": "2026-09-11T07:42:08.410734", + "generation_duration_ms": 2409.315, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:42:14.599514", + "started_at": "2026-09-11T07:42:10.831791", + "generation_duration_ms": 1718.625, + "role": "assistant", + "parent_tool_use_id": null + }, + { + "completed_at": "2026-09-11T07:42:18.016455", + "started_at": "2026-09-11T07:42:14.600077", + "generation_duration_ms": 3416.3779999999997, + "role": "assistant", + "parent_tool_use_id": null + } + ], + "commands": [ + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:42:08.406411", + "execution_started_at": "2026-09-11T07:42:08.389754", + "tool_id": "toolu_016h4AfvKbh7JeGEBNGX9XXf", + "result_status": "success", + "duration_ms": 16.657 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:42:08.405764", + "execution_started_at": "2026-09-11T07:42:08.398691", + "tool_id": "toolu_01PBQpT5a3TeTqKhkyNP8x2t", + "result_status": "success", + "duration_ms": 7.073 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:42:08.406823", + "execution_started_at": "2026-09-11T07:42:08.399067", + "tool_id": "toolu_01G3VRUbZC8WUetG97MvmRML", + "result_status": "success", + "duration_ms": 7.756 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:42:08.406580", + "execution_started_at": "2026-09-11T07:42:08.399201", + "tool_id": "toolu_01C9J9jQya5aZZQvvARyi2Cn", + "result_status": "success", + "duration_ms": 7.3790000000000004 + }, + { + "tool_name": "Write", + "execution_completed_at": "2026-09-11T07:42:08.406723", + "execution_started_at": "2026-09-11T07:42:08.399309", + "tool_id": "toolu_01DitFddEYKRKVBhSzeeGqZB", + "result_status": "success", + "duration_ms": 7.414 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:42:10.824862", + "execution_started_at": "2026-09-11T07:42:10.813486", + "tool_id": "toolu_0171JeX8QfzobMv3YZxqafuM", + "result_status": "success", + "duration_ms": 11.376000000000001 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:42:10.824535", + "execution_started_at": "2026-09-11T07:42:10.814175", + "tool_id": "toolu_01PoWyWCXneLVWEJoBbRTiYQ", + "result_status": "success", + "duration_ms": 10.36 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:42:10.822806", + "execution_started_at": "2026-09-11T07:42:10.814507", + "tool_id": "toolu_01NVqJypM1gJY6To9KfihgJL", + "result_status": "success", + "duration_ms": 8.299000000000001 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:42:10.823747", + "execution_started_at": "2026-09-11T07:42:10.815282", + "tool_id": "toolu_01MgTWDsatAnY18osCK4qTTr", + "result_status": "success", + "duration_ms": 8.465 + }, + { + "tool_name": "Read", + "execution_completed_at": "2026-09-11T07:42:10.824980", + "execution_started_at": "2026-09-11T07:42:10.815562", + "tool_id": "toolu_01ACtUYSTWuGSkr34qH2Lvqd", + "result_status": "success", + "duration_ms": 9.418 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:42:14.598034", + "execution_started_at": "2026-09-11T07:42:12.548936", + "tool_id": "toolu_01SGnPjtpNCtQnCrxwKLvqQ4", + "result_status": "success", + "duration_ms": 2049.098 + }, + { + "tool_name": "Bash", + "execution_completed_at": "2026-09-11T07:42:12.626587", + "execution_started_at": "2026-09-11T07:42:12.549868", + "tool_id": "toolu_016kzccEcHbwwo1P9USYsXKM", + "result_status": "success", + "duration_ms": 76.719 + } + ] + } + ] +} diff --git a/tests/_fixtures/timing_union_cases.json b/tests/_fixtures/timing_union_cases.json index 1154740a..e4ac69c8 100644 --- a/tests/_fixtures/timing_union_cases.json +++ b/tests/_fixtures/timing_union_cases.json @@ -2,14 +2,20 @@ "_comment": [ "Shared replay corpus for the tool-execution UNION, in milliseconds relative", "to an arbitrary base instant. Two implementations must agree on it:", - " * Python — coder_eval.agents._timing.busy_ms, which subtracts tool time", + " * Python — coder_eval.timing.busy_ms, which subtracts tool time", " from an agent's generation window.", - " * TypeScript — evalboard/lib/runs.ts::busyMs, which subtracts tool time", + " * TypeScript — evalboard/lib/timing.ts::busyMs, which subtracts tool time", " from a task's wall clock to produce the Unaccounted residual.", "They answer the same question about the same task.json, so a divergence", "means the evalboard and the harness disagree about how long the tools ran.", "tests/test_timing_union_parity.py and evalboard/lib/__tests__/timing-union-parity.test.ts", - "both replay this file; neither owns the numbers." + "both replay this file; neither owns the numbers.", + "", + "`union_cases` is the same question with no window given: the extent is the", + "spans' own min/max. Python replays it through coder_eval.timing.union_ms;", + "TypeScript through evalboard/lib/timing.ts::toolExecutionMs, which derives", + "that extent itself rather than being handed one — which is exactly why it", + "needs its own cases instead of being assumed to agree." ], "cases": [ { @@ -96,5 +102,58 @@ "spans": [[100, 500], [150, 550], [200, 600], [250, 650]], "expected_ms": 550 } + ], + + "union_cases": [ + { + "name": "no spans at all", + "spans": [], + "expected_ms": 0 + }, + { + "name": "one span is its own length", + "spans": [[200, 700]], + "expected_ms": 500 + }, + { + "name": "two disjoint spans add up", + "spans": [[100, 200], [400, 900]], + "expected_ms": 600 + }, + { + "name": "overlapping spans are counted once, not summed", + "spans": [[100, 600], [200, 700]], + "expected_ms": 600 + }, + { + "name": "a span fully contained in another adds nothing", + "spans": [[100, 900], [300, 400]], + "expected_ms": 800 + }, + { + "name": "adjacent spans merge without a gap", + "spans": [[100, 400], [400, 700]], + "expected_ms": 600 + }, + { + "name": "unsorted input gives the same answer as sorted", + "spans": [[600, 800], [100, 300], [200, 250]], + "expected_ms": 400 + }, + { + "name": "four concurrent calls are the union, never the sum", + "spans": [[100, 500], [150, 550], [200, 600], [250, 650]], + "expected_ms": 550 + }, + { + "name": "a zero-length span contributes nothing", + "spans": [[500, 500], [600, 900]], + "expected_ms": 300 + }, + { + "name": "the extent is the spans' own bounds, not a window", + "spans": [[10000, 10250]], + "expected_ms": 250 + } ] } diff --git a/tests/lint/rules/_model_ctor.py b/tests/lint/rules/_model_ctor.py new file mode 100644 index 00000000..b5074cd6 --- /dev/null +++ b/tests/lint/rules/_model_ctor.py @@ -0,0 +1,89 @@ +"""Resolve `coder_eval.models` constructor calls inside one module's AST. + +CE060 and CE061 ask the same first question — *is this call building an +`AssistantMessage`?* — and answering it takes more than matching a name: a +module may bind the class under any alias, reach it through a relative import, +or never bind it at all and spell it `models.AssistantMessage(...)`. CE060 +worked that out once; duplicating it into CE061 would mean a model rename or a +new import spelling needs two fixes in two rules, and the second one is the one +that gets missed. So it lives here and both rules consume it. + +The class name is taken from the model itself rather than written as a string, +the way CE056 imports `IN_CONTAINER_ENV` and CE057 derives its target set from +`SIDECAR_MODULES`: renaming the model moves both rules with it. + +BLIND SPOT, inherited by every consumer: a re-export through an intermediate +module (`from .sibling import AssistantMessage`) is invisible, because +resolving it means following imports across files and no rule in this package +does that. +""" + +import ast +import re + +from coder_eval.models import AssistantMessage + + +# Reducers live here; nothing outside it builds a generation window. +AGENTS_ROOT = re.compile(r"(?:^|[/\\])src[/\\]coder_eval[/\\]agents[/\\]") + +_MODELS_MODULE = "coder_eval.models" +_MODELS_TAIL = _MODELS_MODULE.rpartition(".")[2] + +# Taken from the model, never spelled here: a rename then moves the rules too. +ASSISTANT_MESSAGE = AssistantMessage.__name__ + + +def reaches_models_module(node: ast.ImportFrom) -> bool: + """True if this `from ... import` reaches `coder_eval.models`. + + A relative import inside `agents/` (`from ..models import ...`) carries only + the tail in `node.module`, so testing the absolute path alone would leave a + rule silently blind for a whole file — and `agents/` does use relative + imports. + """ + module = node.module or "" + if module.startswith(_MODELS_MODULE): + return True + return bool(node.level) and (module == _MODELS_TAIL or module.startswith(f"{_MODELS_TAIL}.")) + + +def local_bindings(tree: ast.AST, class_name: str) -> set[str]: + """Every local name this module binds `coder_eval.models.` to. + + Built per file: caching it across files would leak one module's alias into + another's matching. + """ + names: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and reaches_models_module(node): + names.update(a.asname or a.name for a in node.names if a.name == class_name) + return names + + +def constructor_name(func: ast.expr, names: set[str], class_name: str) -> str | None: + """The spelling this call used to name the model, or None if it did not. + + A bare name has to be bound in this module to be ours; the attribute + spelling is matched on the attribute alone, since the module binding it + arrives through (`import coder_eval.models as models`, `from coder_eval + import models`) is what a walk over one file's CLASS bindings cannot see. + """ + if isinstance(func, ast.Name) and func.id in names: + return func.id + if isinstance(func, ast.Attribute) and func.attr == class_name: + return func.attr + return None + + +def keywords_of(node: ast.Call) -> dict[str, ast.expr]: + """The call's named arguments. A `**`-expansion contributes nothing. + + That is deliberate rather than an oversight: such a call has not declared + the field AT THE SITE, which is what these rules are about. + """ + return {kw.arg: kw.value for kw in node.keywords if kw.arg is not None} + + +def is_none(node: ast.expr | None) -> bool: + return isinstance(node, ast.Constant) and node.value is None diff --git a/tests/lint/rules/ce058_no_timing_literal.py b/tests/lint/rules/ce058_no_timing_literal.py index 45bae047..2abe303f 100644 --- a/tests/lint/rules/ce058_no_timing_literal.py +++ b/tests/lint/rules/ce058_no_timing_literal.py @@ -15,6 +15,16 @@ milliseconds by a command count of which 70 of 211 in one nightly had never been timed at all. +A third field family joined the first two: ``TurnRecord.harness_startup_ms`` +and ``harness_teardown_ms``, the turn's head and tail buckets. They are the +same invariant one level up — a turn whose stream carried no assistant message +was never timed at either end, and a ``0.0`` there would claim the harness +started instantly, which is exactly the reading that sends a real gap into the +evalboard's ``Unaccounted`` cell while a named bucket says it was measured at +zero. A measured ``0.0`` remains a legitimate answer — a window subtracted +down to nothing by the tool execution inside it, or a clamped inversion where +both ends really were observed — so the two values must stay distinguishable. + Five syntactic forms, one invariant, one id — the shapes the codebase actually produced: @@ -47,16 +57,20 @@ # Trailing-segment match, so `cmd.duration_ms` and `generation_duration_ms` -# fire while `duration_ms_limit` does not. +# fire while `duration_ms_limit` does not. The `_startup_ms` / `_teardown_ms` +# arms need a leading segment for the same reason the `_duration_ms` arm does: +# the shipped fields are `harness_*`, and a bare `startup_ms` is more likely a +# budget than a measurement. _TIMING_NAME = re.compile( - r"^(duration_ms|generation_duration_ms|total_command_time_ms|avg_command_time_ms|[a-z_]*_duration_ms)$" + r"^(duration_ms|generation_duration_ms|total_command_time_ms|avg_command_time_ms" + r"|[a-z_]*_duration_ms|[a-z_]*_(?:startup|teardown)_ms)$" ) # The constructors that carry a timing field. Keying on the callee name is what # makes the alias hazard above real; it is also the only thing an AST rule can # see without type inference. _TIMING_CONSTRUCTORS = frozenset( - {"AssistantMessage", "AssistantMessageTelemetry", "CommandTelemetry", "SlowestCommandInfo"} + {"AssistantMessage", "AssistantMessageTelemetry", "CommandTelemetry", "SlowestCommandInfo", "TurnRecord"} ) _SRC_ROOT = re.compile(r"(?:^|[/\\])src[/\\]coder_eval[/\\]") diff --git a/tests/lint/rules/ce060_message_id_declared.py b/tests/lint/rules/ce060_message_id_declared.py new file mode 100644 index 00000000..fbdb4820 --- /dev/null +++ b/tests/lint/rules/ce060_message_id_declared.py @@ -0,0 +1,112 @@ +"""CE060: an assistant message must declare its identity. + +``AssistantMessage.message_id`` is what lets a consumer tell two generations +apart. Antigravity simply omitted the kwarg, so the field defaulted to ``None`` +on every message it ever recorded, and the evalboard — which groups assistant +emissions by ``message_id`` and falls back to a wall-clock ``SAME_EMISSION_GAP_MS`` +threshold when either side lacks one — folded a whole turn's generations into a +single timeline row once the harness's windows became contiguous. Nothing +failed: the consumer sums a group, so every total came out right, and the +golden snapshots had ratified the ``null`` the day they were written. It was +not confined to the timeline either — a grouped emission is one API call to the +thinking-cost simulator, so its whole cache cascade was computed from one call +per turn. The mechanism and the blast radius live in +``docs/agents/HARNESS_PARITY.md`` § Timing capture; neither is restated here. + +Separate id from CE058 and CE059 deliberately: those two are about *timing* +(an unknown duration published as a literal, a window built from one clock +read), this one is about *identity*. One invariant per id is what makes a +``# noqa`` mean one thing. + +WHY IT RESOLVES ALIASES where CE058 and CE059 hardcode constructor names: +CE058's own docstring already concedes that spelling-based matching dies on a +rename, and the weakness is live — ``claude_code_agent.py`` binds *only* +``AssistantMessage as AssistantMessageTelemetry`` and never the bare name, so a +name list guards that file's two construction sites purely because somebody +wrote the current alias into a different rule. CE060 instead derives its +constructor set from each module's own ``coder_eval.models`` imports, which +removes the gap rather than documenting it and catches an arbitrary +``AssistantMessage as Msg`` besides. Widening the other two rules the same way +is recorded in ``.claude/harness-candidates.md``; it is a change to two shipped +rules and needs its own mutation checks. + +What it removes is the *local binding* spelling, not every rename: the class's +own name still has to be known, so it is taken from the model itself +(``AssistantMessage.__name__``) rather than written here as a string, the way +CE056 imports ``IN_CONTAINER_ENV`` and CE057 derives its target set from +``SIDECAR_MODULES``. Renaming the model therefore moves this rule with it. + +That resolution lives in ``_model_ctor.py`` and is shared with CE061, which +needs the identical answer to a different question. Keeping two copies would +mean a new import spelling needs two fixes in two rules. + +BLIND SPOT 1: the runtime ``None``. The rule requires the kwarg to be +*present*, not non-``None`` when it runs. ``opencode_agent.py`` passes +``str(part.get("messageID") or "") or None`` and ``pi_agent.py`` the same shape +for ``responseId``, so either records ``None`` whenever the id is missing from +the payload (`pi_a_single_text_turn.json` is a snapshot of that shape, though +its null comes from a fixture that emits no ``responseId`` rather than from a +live CLI omission). No AST rule can see it, and demanding a statically +non-``None`` value would be wrong: passing a fallback expression *is* deciding +what the id is. The sensor for that case is the golden corpus, and only +partially — a snapshot is written from whatever the code currently does, so it +catches a later change, never an initial omission. + +BLIND SPOT 2: a binding the resolver cannot follow. It reads one module's own +imports, so it sees the direct forms — absolute or relative ``from ... import +AssistantMessage``, under any alias — and the attribute spelling +``.AssistantMessage(...)``. What remains invisible is a re-export +through an intermediate module (``from .sibling import AssistantMessage``); see +``_model_ctor.py``. + +A ``**``-expanded call fires: such a call has not declared the field at the +site. There is no carve-out because no site in ``src/coder_eval/agents/`` uses +``**`` expansion for these constructors; if one is ever added, pass +``message_id=`` explicitly beside it. +""" + +import ast + +from tests.lint.rules._model_ctor import ( + AGENTS_ROOT, + ASSISTANT_MESSAGE, + constructor_name, + is_none, + keywords_of, + local_bindings, +) +from tests.lint.rules.base import BaseRule +from tests.lint.violation import Violation + + +class MessageIdDeclared(BaseRule): + id = "CE060" + + def __init__(self, filepath: str) -> None: + super().__init__(filepath) + self._in_scope = bool(AGENTS_ROOT.search(filepath)) + self._names: set[str] = set() + + def check(self, tree: ast.AST) -> list[Violation]: + self._names = local_bindings(tree, ASSISTANT_MESSAGE) + return super().check(tree) + + def visit_Call(self, node: ast.Call) -> None: + if self._in_scope: + name = constructor_name(node.func, self._names, ASSISTANT_MESSAGE) + if name is not None: + kwargs = keywords_of(node) + if "message_id" not in kwargs or is_none(kwargs["message_id"]): + self.violation( + node, + f"{name}(...) leaves 'message_id' undeclared — absent, or an explicit None — " + "so every message it builds shares one empty identity. Pass the field " + "with a real value: the CLI's own " + "id where the stream carries one, else synthesize it the way Codex does " + "(f'{turn_id}-msg-{gen_index}'). Antigravity shipped without it: the " + "evalboard then falls back to its SAME_EMISSION_GAP_MS wall-clock gap to " + "group emissions, and a harness whose generation windows are contiguous " + "has every one of a turn's generations collapse into one row. See " + "docs/agents/HARNESS_PARITY.md.", + ) + self.generic_visit(node) diff --git a/tests/lint/rules/ce061_window_via_close_window.py b/tests/lint/rules/ce061_window_via_close_window.py new file mode 100644 index 00000000..eacd99bb --- /dev/null +++ b/tests/lint/rules/ce061_window_via_close_window.py @@ -0,0 +1,141 @@ +"""CE061: a generation window must come from the shared helper. + +Pi shipped measuring its window from its own ``turn_start`` while four sibling +reducers tiled from a mark, so the wall clock between one turn's end and the +next turn's start — the model time that PRODUCED that turn — fell into no +bucket at all. Nothing failed. ``docs/agents/HARNESS_PARITY.md`` asserted the +four-bucket identity, and the only sensor for it +(``tests/_fixtures/golden_streams/_scrub.py``) checks ONE side: it catches a +bucket claiming more time than the turn contains and says nothing about one +claiming less. Pi's own tests passed because they were written against Pi's +own arithmetic. + +That is the shape this rule guards against: not a reducer that computes the +window wrongly, but a reducer that computes it AT ALL instead of asking +``coder_eval.timing.close_window``. A new harness whose author reimplements the +arithmetic inline arrives with a green test suite by construction. + +Separate id from CE058, CE059 and CE060 deliberately. Those three are about the +VALUES a message carries — an unknown duration published as a literal, a window +built from one clock read, a missing identity. This one is about PROVENANCE: +where the arithmetic came from. One invariant per id is what makes a ``# noqa`` +mean one thing. + +NOTE what this rule no longer covers, and deliberately: the tool SUBTRACTION is +not part of a window's geometry any more, so "did this reducer subtract +correctly" is not a question here. CE063 owns it — no module in ``agents/`` may +import ``busy_ms`` at all. + +BLIND SPOT, and it is the whole weakness of the chosen shape: this proves the +module IMPORTS the helper, never that any particular call used it. The value +passed to ``generation_duration_ms=`` is always a local (``generation_ms``, +``gen_parts[idx]``), so no AST rule can trace it back to a call. The sensors for +the arithmetic itself are ``tests/test_timing_close_window.py`` and the +per-reducer window tests; this rule adds only the cheap structural half that +neither can reach — a sixth harness rolling its own. + +It costs NO suppression. It used to cost exactly one: ``claude_code_agent.py`` +computed its window from a monotonic delta and subtracted tool time once at +finalization, because a call issued by an earlier emission is still running when +the next window closes — and forcing that into ``close_window`` would have meant +a mode flag on a helper whose whole value is having one shape. Moving the +subtraction to ``EventCollector.subtract_tool_time`` dissolved the exception: +the collector is already the place where every span is known, so claude-code +needs no separate pass and calls the same shrunken helper as the other four. +``tests/test_custom_lint.py::TestCE061WindowViaCloseWindow::test_the_rule_is_now_exemption_free`` +pins the suppression set EMPTY, so a new exemption has to be argued for. + +EXEMPT, because both are honest claims that no window was measured: an explicit +``generation_duration_ms=None`` (codex's rollout rebuild, claude-code's +sub-agent synthesis) and the kwarg absent altogether, which defaults to +``None``. Not matched: ``**``-expansion and ``model_copy(update={...})`` — CE058 +already covers the ``model_copy`` dict shape for timing literals. + +Alias resolution, and its blind spot, live in ``_model_ctor.py``, shared with +CE060. The helper's own name is taken from the function object rather than +written here as a string, so renaming it moves this rule too. +""" + +import ast + +from coder_eval.timing import close_window +from tests.lint.rules._model_ctor import ( + AGENTS_ROOT, + ASSISTANT_MESSAGE, + constructor_name, + is_none, + keywords_of, + local_bindings, +) +from tests.lint.rules.base import BaseRule +from tests.lint.violation import Violation + + +_TIMING_MODULE = "coder_eval.timing" +_TIMING_TAIL = _TIMING_MODULE.rpartition(".")[2] + +# Taken from the function, never spelled here: a rename then moves the rule too. +_HELPER = close_window.__name__ + + +def _imports_the_helper(tree: ast.AST) -> bool: + """True if this module can reach `close_window` under any spelling. + + Both the `from`-import (under any alias) and the module import that makes + `timing.close_window(...)` possible count — a rule that recognized only the + first would tell an author to change a working call site. + """ + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom): + module = node.module or "" + reaches = module.startswith(_TIMING_MODULE) or ( + bool(node.level) and (module == _TIMING_TAIL or module.startswith(f"{_TIMING_TAIL}.")) + ) + if reaches and any(a.name == _HELPER for a in node.names): + return True + # `from coder_eval import timing` / `from .. import timing`. The + # package is checked too: `from anywhere import timing` is not this + # module, and accepting it would let an unrelated name disarm the + # rule for a whole file. + package = module == _TIMING_MODULE.rpartition(".")[0] or (bool(node.level) and not module) + if package and any(a.name == _TIMING_TAIL for a in node.names): + return True + elif isinstance(node, ast.Import): + if any(a.name == _TIMING_MODULE for a in node.names): + return True + return False + + +class WindowViaCloseWindow(BaseRule): + id = "CE061" + + def __init__(self, filepath: str) -> None: + super().__init__(filepath) + self._in_scope = bool(AGENTS_ROOT.search(filepath)) + self._names: set[str] = set() + self._has_helper = False + + def check(self, tree: ast.AST) -> list[Violation]: + self._names = local_bindings(tree, ASSISTANT_MESSAGE) + self._has_helper = _imports_the_helper(tree) + return super().check(tree) + + def visit_Call(self, node: ast.Call) -> None: + if self._in_scope and not self._has_helper: + name = constructor_name(node.func, self._names, ASSISTANT_MESSAGE) + duration = keywords_of(node).get("generation_duration_ms") + if name is not None and duration is not None and not is_none(duration): + self.violation( + node, + f"{name}(...) publishes a measured 'generation_duration_ms' but this module " + f"never imports {_TIMING_MODULE}.{_HELPER} — so it is computing a generation " + "window of its own. Every window is the same geometry: tile from the mark, and " + "keep a backwards item stamp from inverting the span. Publish that RAW span; do " + "NOT subtract tool time here — EventCollector.subtract_tool_time does it once, " + "for every harness, and doing it in the reducer too takes it out twice (CE063 " + "guards that half). Pi got the mark wrong by measuring from its own turn start, " + "and nothing caught it because the golden identity check is one-sided; " + "tests/test_timing_identity_contract.py is the two-sided one. " + f"Call {_HELPER} instead.", + ) + self.generic_visit(node) diff --git a/tests/lint/rules/ce063_no_busy_ms_in_agents.py b/tests/lint/rules/ce063_no_busy_ms_in_agents.py new file mode 100644 index 00000000..6a6d901a --- /dev/null +++ b/tests/lint/rules/ce063_no_busy_ms_in_agents.py @@ -0,0 +1,105 @@ +"""CE063: a reducer may not compute its own tool subtraction. + +Tool execution comes out of a generation window in exactly ONE place: +``coder_eval.streaming.collector.subtract_tool_time``. Before that, five +reducers each did it themselves — four through ``close_window`` as they +flushed, claude-code once at finalization — while the head and the tail were +already computed centrally at the collector seam. That asymmetry is where every +timing defect on this branch actually lived, and none of them was in the +arithmetic: they were in the bookkeeping AROUND it. When to reset a per-step +span list (clearing it at ``step_start`` wiped a span before the flush could +subtract it — a 100% overstatement of that window). When to clear a spent start +stamp (a second flush with no intervening start republished the previous span — +3000 ms of generation for a 2000 ms turn). When to advance the mark. + +A sixth harness whose author reaches for ``busy_ms`` is rebuilding exactly that +bookkeeping, and its tool time would then be subtracted TWICE: once by the +reducer and once by the collector, which subtracts from every window it is +handed. The result is a silently under-reported generation figure on one +harness only — the shape that takes a corpus comparison to notice. + +Separate id from CE061 deliberately, and CE061 is NOT rebodied into this. +CE061 asks where a window's ARITHMETIC came from, and four reducers still call +``close_window``, so its property is still live and still worth guarding — it +is not superseded. This one asks a different question: whether a reducer +subtracts tool time at all. One invariant per id is what makes a ``# noqa`` +mean one thing. (Phase 5 did make CE061 exemption-free: claude-code now calls +the shrunken ``close_window`` like the other four, so its one permanent +suppression is gone.) + +WHY NOT ``_imports_the_helper``, which CE061 uses. That function deliberately +returns True for a bare module import (``from coder_eval import timing``), so +that ``timing.close_window(...)`` counts as reaching the helper — its own +comment says a rule that missed it "would tell an author to change a working +call site." Inverted into a BAN that branch flags any reducer importing the +module and calling ``timing.close_window(...)``, which after Phase 5 is four of +them. So this rule keys on the ``busy_ms`` NAME binding plus an +``ast.Attribute`` match for the ``timing.busy_ms`` spelling, and leaves the +module import alone. + +The name is taken from the function object rather than written here as a +string, the way CE061 takes ``close_window``: renaming it moves this rule too. + +BLIND SPOT: a reducer that re-implements the union inline, without importing +anything, is invisible — as is one reaching ``busy_ms`` through a re-export. +The sensor for the arithmetic itself is +``tests/test_timing_identity_contract.py``, which drives every harness off a +scripted clock and asserts the four buckets tile the turn to the millisecond; +this rule adds only the cheap structural half that a static check can reach. +""" + +import ast + +from coder_eval.timing import busy_ms +from tests.lint.rules._model_ctor import AGENTS_ROOT +from tests.lint.rules.base import BaseRule + + +_TIMING_MODULE = "coder_eval.timing" +_TIMING_TAIL = _TIMING_MODULE.rpartition(".")[2] + +# Taken from the function, never spelled here: a rename then moves the rule too. +_BANNED = busy_ms.__name__ + +_MESSAGE = ( + f"imports '{_BANNED}', but a reducer does not subtract tool time any more — " + "coder_eval.streaming.collector.subtract_tool_time does it once, for every harness, " + "at the single capture seam. Publish the RAW window (close_window gives you its bounds " + "and span) and let the collector clip the tool union out of it. Subtracting here too " + "takes it out twice and silently under-reports generation on this harness alone." +) + + +class NoBusyMsInAgents(BaseRule): + id = "CE063" + + def __init__(self, filepath: str) -> None: + super().__init__(filepath) + self._in_scope = bool(AGENTS_ROOT.search(filepath)) + + def visit_ImportFrom(self, node: ast.ImportFrom) -> None: + """`from coder_eval.timing import busy_ms`, under any alias. + + Relative forms (`from ..timing import busy_ms`) count too: the module + is the same one whatever the path to it looks like. + """ + if not self._in_scope: + return + module = node.module or "" + reaches = module.startswith(_TIMING_MODULE) or ( + bool(node.level) and (module == _TIMING_TAIL or module.startswith(f"{_TIMING_TAIL}.")) + ) + if reaches and any(alias.name == _BANNED for alias in node.names): + self.violation(node, _MESSAGE) + + def visit_Attribute(self, node: ast.Attribute) -> None: + """The `timing.busy_ms` spelling. + + Defensive: no reducer uses it today (all five import plain names), but + a name-binding check alone would let it through, and it is one arm. + """ + if not self._in_scope: + return + if node.attr == _BANNED and isinstance(node.value, ast.Name) and node.value.id == _TIMING_TAIL: + self.violation(node, _MESSAGE) + self.generic_visit(node) diff --git a/tests/lint/runner.py b/tests/lint/runner.py index 8800c424..2014c7bd 100644 --- a/tests/lint/runner.py +++ b/tests/lint/runner.py @@ -37,6 +37,9 @@ from tests.lint.rules.ce057_sidecar_shim_stdlib_only import SidecarShimStdlibOnly from tests.lint.rules.ce058_no_timing_literal import NoTimingLiteral from tests.lint.rules.ce059_generation_window_is_two_reads import GenerationWindowIsTwoReads +from tests.lint.rules.ce060_message_id_declared import MessageIdDeclared +from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow +from tests.lint.rules.ce063_no_busy_ms_in_agents import NoBusyMsInAgents from tests.lint.rules.no_agent_timing_access import NoAgentTimingAccess from tests.lint.rules.no_blocking_io_in_async import NoBlockingIoInAsync from tests.lint.rules.no_cli_imports_in_core import NoCliImportsInCore @@ -53,6 +56,11 @@ from tests.lint.violation import Violation +# CE062 IS DELIBERATELY UNUSED and must stay that way — the ids above jump 061 +# to 063. It was claimed during the turn-timing work and then folded into CE063 +# rather than shipped. An id is a permanent documentation anchor: a suppression +# comment carrying 062 in an older branch, review or commit message must never +# start meaning something new. Claim 064 next. type RuleClass = type[BaseRule] ALL_RULES: list[RuleClass] = [ @@ -97,6 +105,9 @@ SidecarShimStdlibOnly, NoTimingLiteral, GenerationWindowIsTwoReads, + MessageIdDeclared, + WindowViaCloseWindow, + NoBusyMsInAgents, ] # Anti-shadow invariant (mirrors AgentRegistry / register_pricing): every CE rule diff --git a/tests/test_agent_golden_master.py b/tests/test_agent_golden_master.py index 4c54d0a2..c250bb0e 100644 --- a/tests/test_agent_golden_master.py +++ b/tests/test_agent_golden_master.py @@ -62,6 +62,26 @@ # subtraction in codex_agent._flush_message. "codex_d_cross_flush_is_error", # flush lands before the tool completes: zero-width window "codex_e_orphan_tool", # the tool never completes, so the window never opens + # Same shape, reached from the opposite direction. This scenario injects + # a 5 ms CLI tool interval into a replay whose whole turn is well under + # one millisecond, so the tool spans BOTH windows entirely and the + # central subtraction takes each down to a measured 0.0. It is the tool + # interval that is fictional, not the subtraction — which is why the + # scenario is in FICTIONAL_DURATIONS too. + # + # BE HONEST ABOUT WHAT IS LEFT. With both exemptions on, this snapshot + # asserts neither the identity nor a positive window, and it does NOT + # record the tiling the scenario is named for — `SCRUB_KEYS` masks + # `started_at`, `completed_at` and `generation_duration_ms`, so nothing + # about where a window opened survives into the JSON. What it still + # pins is the STRUCTURE: two assistant messages, their content blocks, + # their token buckets, and one resolved command. OpenCode's tiling is + # asserted where it can be — `tests/test_timing_identity_contract.py` + # (scripted clock, ms-exact) and + # `tests/test_opencode_agent.py::TestGenerationWindowsTileTheTurn`. + # `pi_c_multi_turn_tiling` is the same scenario shape on a harness whose + # stamps come from its own clock, and it needs neither exemption. + "opencode_c_multi_step_tiling", } ) @@ -70,6 +90,50 @@ def _expect_window(harness: str, scenario_name: str) -> bool: return f"{harness}_{scenario_name}" not in NO_GENERATION_WINDOW +# Scenarios that inject their own SDK timestamps, so their recorded durations +# are FICTIONAL and cannot be reconciled against the replay's real wall clock. +# `_rebase_notifications` / `_rebase_lines` put those stamps on the replay's +# clock, which fixes the era — but the SDK's stamps are integer MILLISECONDS +# and these scenarios declare 17-900 ms of item time, while the replay itself +# runs in well under one. No rebasing closes that; the agent's own clock would +# have to be faked too. Everything else — every claude, antigravity and pi +# scenario, and the codex/opencode ones that inject nothing — is checked. +# +# The last two entries were ADDED to buy stability, and the trade is worth +# stating. They previously injected NO stamps at all, so `_flush_message` took +# `_ms_to_dt(None)` for both window bounds — two adjacent `datetime.now()` +# reads, which collide at microsecond resolution often enough that +# `assert_timing_captured`'s `completed_at > started_at` failed roughly one run +# in twenty under parallel load, naming a different scenario each time. Their +# identity check was near-vacuous anyway (a window of width zero reconciles +# trivially), so giving them real bounds trades that for a stable, meaningful +# bounds-span assertion. +FICTIONAL_DURATIONS: frozenset[str] = frozenset( + { + "codex_b_command_execution", # 250 ms command + 150 ms generation + "codex_c_reasoning_placeholder", # 300 ms of item time — see below + "codex_d_cross_flush_is_error", # 400 ms command + "codex_e_orphan_tool", # command started, never completed + "codex_f_collab_fallback", # 900 ms collab wait + "codex_h_no_turn_completed_crash", # 200 ms of item time — see below + "opencode_b_tool_call_resolved", # 17 ms tool interval + # 5 ms tool interval, injected as CLI epoch stamps. OpenCode takes its + # tool bounds from the CLI payload rather than from its own clock, so + # every scenario of this harness that resolves a tool injects them — + # there is no version of this scenario that stays commensurable with a + # sub-millisecond replay. Its TILING property (the second window opens + # at the first `step_finish`) is what the scenario is for, and that is + # still snapshotted; the identity is asserted for this harness by + # tests/test_timing_identity_contract.py, on a scripted clock. + "opencode_c_multi_step_tiling", + } +) + + +def _check_identity(harness: str, scenario_name: str) -> bool: + return f"{harness}_{scenario_name}" not in FICTIONAL_DURATIONS + + _EXPECTED_DIR = Path(__file__).parent / "_fixtures" / "golden_streams" / "expected" _REGEN = os.environ.get("GOLDEN_REGEN", "").strip().lower() in {"1", "true", "yes", "on"} @@ -101,7 +165,11 @@ async def test_claude_golden(scenario, tmp_path): # Reconciliation is asserted on the UNscrubbed dump (token buckets are never # scrubbed, but cost/timestamps are — assert before masking to be explicit). assert_reconciliation(raw) - assert_timing_captured(raw, expect_generation_window=_expect_window("claude", scenario.name)) + assert_timing_captured( + raw, + expect_generation_window=_expect_window("claude", scenario.name), + check_identity=_check_identity("claude", scenario.name), + ) _compare_or_regen(f"claude_{scenario.name}", scrub(raw)) @@ -111,7 +179,11 @@ async def test_claude_golden(scenario, tmp_path): async def test_codex_golden(scenario, tmp_path): raw = await run_codex_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) - assert_timing_captured(raw, expect_generation_window=_expect_window("codex", scenario.name)) + assert_timing_captured( + raw, + expect_generation_window=_expect_window("codex", scenario.name), + check_identity=_check_identity("codex", scenario.name), + ) _compare_or_regen(f"codex_{scenario.name}", scrub(raw)) @@ -137,7 +209,11 @@ async def test_codex_reconciliation_invariant(scenario, tmp_path): async def test_antigravity_golden(scenario, tmp_path): raw = await run_antigravity_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) - assert_timing_captured(raw, expect_generation_window=_expect_window("antigravity", scenario.name)) + assert_timing_captured( + raw, + expect_generation_window=_expect_window("antigravity", scenario.name), + check_identity=_check_identity("antigravity", scenario.name), + ) _compare_or_regen(f"antigravity_{scenario.name}", scrub(raw)) @@ -153,7 +229,11 @@ async def test_antigravity_reconciliation_invariant(scenario, tmp_path): async def test_opencode_golden(scenario, tmp_path): raw = await run_opencode_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) - assert_timing_captured(raw, expect_generation_window=_expect_window("opencode", scenario.name)) + assert_timing_captured( + raw, + expect_generation_window=_expect_window("opencode", scenario.name), + check_identity=_check_identity("opencode", scenario.name), + ) _compare_or_regen(f"opencode_{scenario.name}", scrub(raw)) @@ -169,7 +249,11 @@ async def test_opencode_reconciliation_invariant(scenario, tmp_path): async def test_pi_golden(scenario, tmp_path): raw = await run_pi_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) - assert_timing_captured(raw, expect_generation_window=_expect_window("pi", scenario.name)) + assert_timing_captured( + raw, + expect_generation_window=_expect_window("pi", scenario.name), + check_identity=_check_identity("pi", scenario.name), + ) _compare_or_regen(f"pi_{scenario.name}", scrub(raw)) @@ -265,9 +349,22 @@ def _record( windows: list[float | None] = (), commands: list[dict[str, Any]] = (), bounds_collapse: bool = False, + overhead: tuple[float | None, float | None] = (0.0, 3.5), + duration_seconds: float = 10.0, ) -> dict[str, Any]: - """A record whose bounds span each window, unless `bounds_collapse`.""" + """A record whose bounds span each window, unless `bounds_collapse`. + + `overhead` is the (head, tail) pair. It defaults to a MEASURED pair — + a 0.0 head is antigravity's real answer — because every record here + carries an assistant message unless a test says otherwise, and the + sensor requires both buckets on such a turn. + + `duration_seconds` defaults to a turn long enough that the four-bucket + identity is trivially satisfied, so these cases constrain only what + each is about; the identity has its own cases below. + """ return { + "duration_seconds": duration_seconds, "messages": [ { "role": "assistant", @@ -278,14 +375,18 @@ def _record( for w in windows ], "commands": list(commands), + "harness_startup_ms": overhead[0], + "harness_teardown_ms": overhead[1], } def test_a_positive_window_passes(self): assert_timing_captured(self._record(windows=[12.5]), expect_generation_window=True) def test_a_none_window_raises_when_one_is_expected(self): + # overhead=(None, None) because a turn with no measurable window has no + # head or tail either; this isolates the generation-window assertion. with pytest.raises(AssertionError, match="positive generation window"): - assert_timing_captured(self._record(windows=[None]), expect_generation_window=True) + assert_timing_captured(self._record(windows=[None], overhead=(None, None)), expect_generation_window=True) def test_exactly_zero_raises_too(self): # The Antigravity defect's exact signature: a value that is present, @@ -303,7 +404,7 @@ def test_collapsed_bounds_raise_even_with_a_healthy_duration(self): assert_timing_captured(self._record(windows=[500.0], bounds_collapse=True), expect_generation_window=True) def test_a_none_window_passes_when_none_is_expected(self): - assert_timing_captured(self._record(windows=[None]), expect_generation_window=False) + assert_timing_captured(self._record(windows=[None], overhead=(None, None)), expect_generation_window=False) def test_one_positive_among_several_passes(self): # The FLOOR, not a per-entry rule. claude_d_subagent_terminal holds two @@ -362,3 +463,75 @@ def test_an_unresolved_command_is_exempt(self): def test_a_scenario_with_no_commands_is_vacuously_fine(self): assert_timing_captured(self._record(windows=[5.0]), expect_generation_window=True) + + # The turn's head and tail. Presence only — the replays run in ~0.3 ms of + # synthetic wall clock, so any bound check here would be noise. + def test_a_generating_turn_must_report_a_head(self): + with pytest.raises(AssertionError, match="harness_startup_ms is None"): + assert_timing_captured(self._record(windows=[5.0], overhead=(None, 3.5)), expect_generation_window=True) + + def test_a_generating_turn_must_report_a_tail(self): + with pytest.raises(AssertionError, match="harness_teardown_ms is None"): + assert_timing_captured(self._record(windows=[5.0], overhead=(0.0, None)), expect_generation_window=True) + + def test_a_turn_with_no_generation_must_report_neither(self): + # A number here claims a measurement nobody could have taken: the + # collector measures both against the messages that report a window. + with pytest.raises(AssertionError, match=r"harness_startup_ms is 0\.0"): + assert_timing_captured(self._record(windows=[], overhead=(0.0, 3.5)), expect_generation_window=False) + assert_timing_captured(self._record(windows=[], overhead=(None, None)), expect_generation_window=False) + + def test_an_unmeasurable_window_is_not_something_to_measure_against(self): + # codex_g_items_rebuild's shape: an assistant message exists, but it was + # rebuilt after the turn ended with placeholder now() bounds and says so + # via generation_duration_ms=None. Those stamps are not window bounds, so + # the honest head and tail are None — keying on "any assistant message" + # would have demanded a number derived from a placeholder. + with pytest.raises(AssertionError, match=r"harness_startup_ms is 0\.0"): + assert_timing_captured(self._record(windows=[None], overhead=(0.0, 3.5)), expect_generation_window=False) + + # The four-bucket identity: generation + tool union + head + tail cannot + # exceed the turn, because the four are disjoint. + def test_buckets_summing_past_the_turn_raise(self): + # 4s generation + a 3.5ms tail on a 1s turn. + with pytest.raises(AssertionError, match="booked twice"): + assert_timing_captured(self._record(windows=[4000.0], duration_seconds=1.0), expect_generation_window=True) + + def test_a_tool_double_booked_into_the_tail_is_caught(self): + """The exact defect: an orphan force-closed inside the tail, counted + both in the tool union and in harness_teardown_ms.""" + record = self._record( + windows=[40.0], + duration_seconds=0.1, # 100 ms turn + overhead=(0.0, 50.0), + commands=[ + { + "tool_id": "orphan", + "result_status": "success", + "duration_ms": 50.0, + "execution_started_at": "2026-01-01T00:00:00.020000", + "execution_completed_at": "2026-01-01T00:00:00.070000", + } + ], + ) + with pytest.raises(AssertionError, match="booked twice"): + assert_timing_captured(record, expect_generation_window=True) + + def test_the_identity_can_be_waived_for_a_fictional_clock(self): + # codex/opencode scenarios declare integer-millisecond item durations + # that a sub-millisecond replay can never contain. + assert_timing_captured( + self._record(windows=[4000.0], duration_seconds=1.0), + expect_generation_window=True, + check_identity=False, + ) + + def test_buckets_well_inside_the_turn_pass(self): + assert_timing_captured(self._record(windows=[40.0], duration_seconds=1.0), expect_generation_window=True) + + def test_the_buckets_are_checked_even_when_no_window_is_expected(self): + # codex_e_orphan_tool clears the flag (its window subtracts to zero) + # while still having a head and a tail — so the flag is the wrong key + # for this half of the sensor, and the early return must not skip it. + with pytest.raises(AssertionError, match="harness_teardown_ms is None"): + assert_timing_captured(self._record(windows=[5.0], overhead=(0.0, None)), expect_generation_window=False) diff --git a/tests/test_agent_telemetry.py b/tests/test_agent_telemetry.py index 12c8c171..90f013c4 100644 --- a/tests/test_agent_telemetry.py +++ b/tests/test_agent_telemetry.py @@ -1,6 +1,8 @@ """Tests for command telemetry status tracking (V2 fix).""" import time +from datetime import datetime, timedelta +from types import SimpleNamespace import pytest @@ -1278,3 +1280,241 @@ async def mock_query(prompt, options): assert second.cache_read_tokens == 50 finally: agent_module.query = original_query + + +class TestClaudeHeadIsMeasuredAtFirstOutput: + """claude-code's head is the wall clock up to the first observed model output. + + `_ClaudeTurnState.__init__` stamps `last_event_wall`, and + `_seed_first_generation_window` re-stamps it at the first `message_start`. + So the first window opens where the model first spoke, and the CLI spawn, + provider resolution and time to first token before it are the head. + + `_build_claude_query` is NOT in the head: it runs at `communicate`'s + `:1095`, before `AgentStartEvent` is emitted at `:1106`, so it precedes the + head's own start stamp. It used to sit inside msg0's generation window + (`last_event_wall` was stamped at state construction, ahead of the build); + it now sits inside `duration_seconds` but outside all four buckets, as + unexplained residual. That is why the budget below still matters and why it + is not the same guard it was: at 0.03-0.10 ms the residual is noise, and + the two tests keep it that way. + + It used to be `0.0`, and that was a CLAMPED NEGATIVE rather than a + measurement: both marks were stamped before `AgentStartEvent` was emitted, + so `decompose_turn`'s `max(..., 0.0)` produced it. The rejection rested on + claude-code running the model in-process. It does not — `claude-agent-sdk` + spawns the `claude` CLI over `anyio.open_process` and `_pump_messages` + calls `query()` once per `communicate()`, a fresh CLI per turn. + + The two budget tests below survive the rewrite with their meaning INVERTED. + `_build_claude_query`'s cost now lands in the head rather than inside msg0's + generation, so they no longer guard "the build is cheap enough to leave + hidden by the clamp" — they guard "our own setup is a negligible part of a + head that is now published", which is what makes the head readable as the + harness's latency rather than as ours. + """ + + # Measured at 0.03 ms bare and 0.10 ms with four plugin roots. The bound is + # ~300x that: generous enough that a loaded CI box cannot trip it, tight + # enough to catch a regression that would make the reasoning above wrong. + BUDGET_MS = 50.0 + + @staticmethod + def _build_ms(**config_kwargs) -> float: + from pathlib import Path + + from coder_eval.agents.claude_code_agent import ClaudeCodeAgent + + config = parse_agent_config(type=AgentKind.CLAUDE_CODE, model="claude-haiku-4-5-20251001", **config_kwargs) + agent = ClaudeCodeAgent(config) + agent.working_directory = Path(".") + # Best of N: the claim is about the work the call does, not about the + # worst scheduling slice a shared runner happens to hand it. + samples = [] + for _ in range(5): + started = time.perf_counter() + agent._build_claude_query("hi", 60, 10, lambda _line: None) + samples.append((time.perf_counter() - started) * 1000.0) + return min(samples) + + def test_the_query_build_is_a_negligible_part_of_the_published_head(self): + elapsed = self._build_ms() + assert elapsed < self.BUDGET_MS, ( + f"_build_claude_query took {elapsed:.2f} ms, over the {self.BUDGET_MS} ms budget. It runs " + "BEFORE the AgentStartEvent, so it is inside the turn's duration_seconds but outside " + "all four buckets — unexplained residual that no bucket accounts for. At a few hundred " + "microseconds that is noise; at this size the four buckets would visibly stop summing " + "to the turn and the gap would be ours, not the harness's." + ) + + def test_plugin_resolution_does_not_change_that(self, tmp_path): + """A plugin-heavy task is where our own setup could plausibly dominate.""" + (tmp_path / "skills").mkdir() + roots = [{"type": "local", "path": str(tmp_path)} for _ in range(4)] + elapsed = self._build_ms(plugins=roots) + assert elapsed < self.BUDGET_MS, ( + f"_build_claude_query with 4 plugin roots took {elapsed:.2f} ms, over the " + f"{self.BUDGET_MS} ms budget — see the sibling test for why that matters." + ) + + +class TestClaudeFirstWindowReseed: + """The first `message_start` moves the window mark; a later one must not. + + Driven at `_ClaudeTurnState` with both clocks patched off one counter. + claude-code derives the window's DURATION from `time.monotonic()` and its + BOUNDS from `datetime.now()`, so patching one leaves the other real and + these tests would measure nothing while still passing. + """ + + BASE = datetime(2026, 9, 11, 9, 0, 0) + + class _Stepped(datetime): + at_ms = 0.0 + + @staticmethod + def now(tz=None): # type: ignore[override] + return TestClaudeFirstWindowReseed.BASE + timedelta(milliseconds=TestClaudeFirstWindowReseed._Stepped.at_ms) + + def _state(self, monkeypatch): + from coder_eval.agents import claude_code_agent as claude_module + from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeTurnState + from coder_eval.streaming.callbacks import CompositeStreamCallback + from coder_eval.streaming.collector import EventCollector + + stepped = self._Stepped + stepped.at_ms = 0.0 + monkeypatch.setattr(claude_module, "datetime", stepped) + monkeypatch.setattr(claude_module, "time", SimpleNamespace(monotonic=lambda: stepped.at_ms / 1000.0)) + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + collector = EventCollector() + return stepped, _ClaudeTurnState( + agent, + emit=CompositeStreamCallback([collector]), + collector=collector, + task_id="t", + user_input="go", + iteration=1, + max_turns=None, + log=agent._log, + turn_start_time=0.0, + deadline=None, + ) + + @staticmethod + def _assistant(mid: str): + from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage as SdkAssistantMessage + + return SdkAssistantMessage([], usage={"input_tokens": 10, "output_tokens": 5}, message_id=mid) + + def test_cli_boot_before_the_first_message_start_is_not_msg0_generation(self, monkeypatch): + """The interval the CLI spent booting is head, not model time. + + Before the re-seed the window opened when the turn state was built, so + this whole interval was published as msg0's `generation_duration_ms` — + ~3.6 s per turn on the measured corpus. + """ + clock, state = self._state(monkeypatch) + clock.at_ms = 800 # CLI spawn + provider resolution + TTFT + state.on_stream_event(_message_start("m1")) + clock.at_ms = 1000 + state.on_assistant_message(self._assistant("m1")) + + message = state.sdk_messages[0] + assert message.started_at == self.BASE + timedelta(milliseconds=800) + assert message.generation_duration_ms == pytest.approx(200.0) + + def test_only_the_first_message_start_reseeds_so_the_windows_still_tile(self, monkeypatch): + """A second re-seed would drop the gap before the next emission. + + That gap — a tool result landing, then the next request going out — is + real model time, and falling into no bucket at all is the defect pi + shipped with. + """ + clock, state = self._state(monkeypatch) + clock.at_ms = 800 + state.on_stream_event(_message_start("m1")) + clock.at_ms = 1000 + state.on_assistant_message(self._assistant("m1")) + clock.at_ms = 1500 + state.on_stream_event(_message_start("m2")) + clock.at_ms = 2000 + state.on_assistant_message(self._assistant("m2")) + + first, second = state.sdk_messages[0], state.sdk_messages[1] + assert second.started_at == first.completed_at, "the second window must tile from the first" + assert second.generation_duration_ms == pytest.approx(1000.0) + + def test_seeding_twice_by_hand_is_a_no_op_the_second_time(self, monkeypatch): + """The once-per-turn guard, stated outright rather than inferred. + + The sibling test above would also fail if the guard were removed, but + only via the tiling it implies. This says the property directly, so a + reviewer does not have to reproduce a mutation to see it. + """ + clock, state = self._state(monkeypatch) + clock.at_ms = 800 + state._seed_first_generation_window() + seeded_wall = state.last_event_wall + + clock.at_ms = 5000 + state._seed_first_generation_window() + + assert state.last_event_wall == seeded_wall + + def test_a_stream_with_no_message_start_still_clamps_to_zero(self, monkeypatch): + """Partial streaming off, a mocked query(), or a crash before the first event. + + The re-seed never fires, the turn-entry mark stands, and the head + clamps exactly as it did before. That is the correct degradation, and + asserting it is what keeps it from becoming an untested branch. + """ + clock, state = self._state(monkeypatch) + clock.at_ms = 1000 + state.on_assistant_message(self._assistant("m1")) + + assert state.first_output_seen is False, "nothing latched, so the turn-entry mark stands" + # The window still opens at turn entry, which PRECEDES the + # AgentStartEvent — so the head is a negative that decompose_turn + # clamps, exactly as it did before this phase. Asserted on the mark + # rather than by re-deriving `max(elapsed, 0.0)` from hand-built + # arguments, which would restate the implementation and could not fail. + assert state.sdk_messages[0].started_at == self.BASE + + def test_the_four_buckets_account_for_a_tool_free_turn(self, monkeypatch): + """head + generation + tail == the turn, with the head read DIRECTLY. + + The sibling tests assert the window's `started_at`, which pins the mark + but never the published `harness_startup_ms` itself — so nothing here + read the field this phase exists to change. With no tool calls the tool + bucket is empty and the other three must tile the turn exactly. + """ + from coder_eval.models import TokenUsage + from coder_eval.streaming.collector import EventCollector + from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, AgentStartEvent + + clock, state = self._state(monkeypatch) + clock.at_ms = 800 + state.on_stream_event(_message_start("m1")) + clock.at_ms = 1000 + state.on_assistant_message(self._assistant("m1")) + + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=self.BASE)) + collector.on_event( + AgentEndEvent( + task_id="t", + status=AgentEndStatus.COMPLETED, + messages=list(state.sdk_messages), + usage=TokenUsage(), + timestamp=self.BASE + timedelta(milliseconds=1500), + ) + ) + record = collector.build_turn_record() + + assert record.harness_startup_ms == pytest.approx(800.0), "the CLI boot is the head, published" + assert record.harness_teardown_ms == pytest.approx(500.0) + generation = sum(m.generation_duration_ms or 0.0 for m in record.messages if m.role == "assistant") + assert generation == pytest.approx(200.0) + assert record.harness_startup_ms + generation + record.harness_teardown_ms == pytest.approx(1500.0) diff --git a/tests/test_antigravity_agent.py b/tests/test_antigravity_agent.py index ef6f3587..54763f2e 100644 --- a/tests/test_antigravity_agent.py +++ b/tests/test_antigravity_agent.py @@ -5,6 +5,7 @@ """ import asyncio +import inspect import os import sys from collections.abc import Callable @@ -24,7 +25,7 @@ _to_token_usage, ) from coder_eval.agents.registry import AgentRegistry -from coder_eval.models import AgentKind, AntigravityAgentConfig, parse_agent_config +from coder_eval.models import AgentKind, AntigravityAgentConfig, AssistantMessage, parse_agent_config from coder_eval.plugins import ensure_plugins_loaded from coder_eval.pricing import calculate_cost from tests._fixtures.golden_streams._scrub import assert_reconciliation @@ -335,6 +336,13 @@ async def test_communicate_maps_steps_to_turn_record(): assert sum(m.output_tokens for m in bucketed) == tr.token_usage.output_tokens assert sum(m.cache_creation_tokens for m in bucketed) == tr.token_usage.cache_creation_input_tokens assert sum(m.cache_read_tokens for m in bucketed) == tr.token_usage.cache_read_input_tokens + # Every generation carries its own identity. Filter explicitly: `tr.messages` + # is list[TranscriptMessage] and ReconciliationMessage has no `message_id`, + # so a bare comprehension would raise the moment a residual is booked. + # The literal strings pin the 0-based Codex-parity scheme, which mere + # distinctness (a uuid would pass) does not. + ids = [m.message_id for m in tr.messages if isinstance(m, AssistantMessage)] + assert ids == ["antigravity-1-msg-0", "antigravity-1-msg-1", "antigravity-1-msg-2"] assert agent.pending_turn is None # success path leaves no partial @@ -1493,16 +1501,26 @@ async def test_cap_reached_on_a_poll_redrain_stops_polling(monkeypatch): class _Clock: - """Controlled stand-in for the two clocks the reducer reads. - - ONE monotonically advancing counter, read by both clocks: every read — - `time.monotonic()` or `datetime.now()` — costs TICK_MS. So the fixture's - timeline is driven by read ORDER, not by elapsed time, and the two clocks - are deliberately coupled rather than independent. That is enough to pin - the arithmetic exactly; it is NOT a cross-check that the reducer keeps the - two clocks in their proper roles (a variant deriving the span from the - wall stamps would pass every test here). The module's only clock uses are - `time.monotonic` and `datetime.now`, so patching these two covers it. + """Controlled stand-in for the reducer's clocks — a `TurnClock` and `time`. + + ONE monotonically advancing counter, read by both: every read — the turn + clock's `now()` or `time.monotonic()` — costs TICK_MS. So the fixture's + timeline is driven by read ORDER, not by elapsed time, and the two are + deliberately coupled rather than independent. That is enough to pin the + arithmetic exactly. + + Every WALL stamp the reducer records now derives from its per-turn + `TurnClock`, so this stands in for that object rather than for the + module's `datetime`. That distinction is load-bearing, not cosmetic: a + derived stamp does not read `datetime.now()`, so the old patch would no + longer reach it and these tests would quietly measure the real clock and + pass by accident. `time` is still patched because `duration_seconds` and + the poll deadlines read `time.monotonic()` directly, and must — a deadline + may not move when the wall clock steps. + + What it still does NOT prove is that the reducer keeps the two in their + proper roles; with one basis for every wall stamp there is no longer a + second role to confuse it with. """ TICK_MS = 100.0 @@ -1522,8 +1540,15 @@ def now(self) -> datetime: def _install_clock(monkeypatch, clock: _Clock) -> None: + """Hand the reducer this clock for the turn it is about to build. + + `TurnClock` is replaced by a factory rather than the fake being passed + positionally, because the state — and therefore its clock — is built + inside `communicate()`, out of the caller's reach. One typed seam, and the + stand-in has to satisfy `now()`. + """ monkeypatch.setattr(agent_module, "time", SimpleNamespace(monotonic=clock.monotonic)) - monkeypatch.setattr(agent_module, "datetime", SimpleNamespace(now=clock.now)) + monkeypatch.setattr(agent_module, "TurnClock", lambda: clock) def _assistant(record): @@ -1546,7 +1571,7 @@ def _at(ms: float) -> datetime: return _CLOCK_BASE + timedelta(milliseconds=ms) def _busy(self, spans, lo=0, hi=10_000) -> float: - from coder_eval.agents._timing import busy_ms + from coder_eval.timing import busy_ms return busy_ms([(self._at(s), self._at(e)) for s, e in spans], self._at(lo), self._at(hi)) @@ -1697,14 +1722,22 @@ async def test_tool_execution_is_subtracted_from_the_window(monkeypatch): second = messages[1] bash = next(c for c in record.commands if c.tool_name == "Bash") - # The window spans 400ms of wall clock and contains a 100ms tool call, so - # 300ms of it was the model generating. Cross-checked against the recorded - # bounds, which come from the OTHER clock the reducer reads. + # The window contains a 100ms tool call, so what is left of it was the + # model generating. That relation is the assertion that matters, and it is + # independent of the fixture's tick size. span_ms = (second.completed_at - second.started_at).total_seconds() * 1000.0 assert bash.duration_ms == pytest.approx(100.0) - assert span_ms == pytest.approx(400.0) assert second.generation_duration_ms == pytest.approx(span_ms - bash.duration_ms) - assert second.generation_duration_ms == pytest.approx(300.0) + + # The absolute figures are artifacts of `_Clock`, which charges one TICK_MS + # per clock READ. They moved from 400/300 to 300/200 when the reducer + # stopped taking a monotonic reading it no longer needs: a flush now reads + # the turn clock once where it used to read two clocks, so each window is + # one tick shorter on this fixture's read-driven timeline. Nothing about + # real elapsed time changed — the 100ms tool, which is still two reads + # apart, is unmoved. + assert span_ms == pytest.approx(300.0) + assert second.generation_duration_ms == pytest.approx(200.0) async def test_a_straddling_tool_is_charged_only_for_its_in_window_part(monkeypatch): @@ -1834,10 +1867,17 @@ async def test_a_no_op_flush_does_not_move_the_mark(monkeypatch): async def test_generation_and_tool_time_account_for_the_turn(): - """Σ generation + Σ tool execution lands inside the turn's own duration. + """Σ generation + Σ tool + head + tail lands inside the turn's own duration. Bounds, not equality: the fake conversation's own overhead sits in the - residual. Before this change the generation half was identically 0. + residual. Before the window existed the generation half was identically 0. + + The HEAD is part of the sum, and has to be: the first window now opens at + the first observed `Step` rather than at turn entry, so the dispatch before + it is a measured bucket instead of time hidden inside msg0's generation. + Asserting `generation + tool` alone against a share of the turn was an + assertion that the head stays empty — which is what this phase deliberately + stopped being true. """ steps = [ _step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5)), @@ -1861,11 +1901,28 @@ async def test_generation_and_tool_time_account_for_the_turn(): gen_ms = sum(m.generation_duration_ms or 0.0 for m in _assistant(record)) tool_ms = sum(c.duration_ms or 0.0 for c in record.commands) + head_ms = record.harness_startup_ms or 0.0 + tail_ms = record.harness_teardown_ms or 0.0 turn_ms = record.duration_seconds * 1000.0 assert gen_ms > 0 - assert gen_ms + tool_ms <= turn_ms - assert gen_ms + tool_ms >= 0.5 * turn_ms + assert head_ms > 0, "the dispatch before the first Step is now a measured bucket, not 0.0" + assert gen_ms + tool_ms + head_ms + tail_ms <= turn_ms + + # NO relative LOWER bound. This case runs on the REAL clock, and the fake + # conversation's own overhead is the residual — under parallel load the + # denominator (`duration_seconds`, the agent's monotonic span) inflates + # while the measured buckets do not, so any `>= share * turn_ms` assertion + # is a scheduler-noise detector. It was one: a `>= 0.5 *` bound survived + # here only while the sum excluded the head, and failed under `-n auto` + # once the head joined it. + # + # The share this test was reaching for IS asserted, exactly, in + # tests/test_timing_identity_contract.py — on a scripted clock, where the + # magnitudes are real and the identity closes to the millisecond. What is + # left here is what an end-to-end run can honestly claim: the buckets are + # measured, the head is no longer the clamped 0.0, and nothing overflows + # the turn. async def test_timing_change_moves_no_token_bucket(): @@ -1902,3 +1959,233 @@ async def test_timing_change_moves_no_token_bucket(): # local re-implementation of two of its four buckets. assert_reconciliation(record.model_dump(mode="json")) assert all(m.generation_duration_ms is not None for m in _assistant(record)) + + +async def test_the_published_window_reconciles_to_its_own_bounds(monkeypatch): + """The reducer subtracted exactly the spans the record carries. + + The per-migrated-reducer check its three siblings gained when they moved + onto `close_window`; antigravity could not have it until its span stopped + being monotonic while these intervals were wall. `decompose_run.py` and the + evalboard's Unaccounted cell both recompute the tool UNION from the + recorded command spans and subtract it from the recorded window bounds, so + this asserts the reducer fed the window that same set. + """ + from coder_eval.timing import busy_ms + + _install_clock(monkeypatch, _Clock()) + steps = [ + _step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5)), + _step( + "TOOL_CALL", + "ACTIVE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "t1", {"command_line": "ls"})], + ), + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "t1", {"command_line": "ls", "exit_code": 0})], + ), + _step("THINKING", "DONE", thinking="second", usage=_usage(100, 0, 5, 5)), + ] + record = await _agent_with_steps(steps).communicate("go") + + second = _assistant(record)[1] + spans = [ + (c.execution_started_at, c.execution_completed_at) + for c in record.commands + if c.execution_started_at is not None and c.execution_completed_at is not None + ] + span_ms = (second.completed_at - second.started_at).total_seconds() * 1000.0 + expected = span_ms - busy_ms(spans, second.started_at, second.completed_at) + assert second.generation_duration_ms == pytest.approx(expected) + + +async def test_the_window_is_measured_without_relying_on_the_negative_clamp(monkeypatch): + """A positive window, and no clamp underneath it. + + The span used to be read off `time.monotonic()` while the tool intervals + were wall, so the two could disagree and drive the result negative; the + clamp that caught it published a `0.0` indistinguishable from a real + instant generation, and a debug line was the only trace. One basis makes + that unrepresentable: `busy_ms` clips to the window and unions overlaps, so + it cannot exceed a span derived from the same clock. + """ + _install_clock(monkeypatch, _Clock()) + steps = [ + _step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5)), + _step( + "TOOL_CALL", + "ACTIVE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "t1", {"command_line": "ls"})], + ), + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "t1", {"command_line": "ls", "exit_code": 0})], + ), + _step("THINKING", "DONE", thinking="second", usage=_usage(100, 0, 5, 5)), + ] + record = await _agent_with_steps(steps).communicate("go") + + second = _assistant(record)[1] + assert second.generation_duration_ms > 0.0 + assert second.completed_at > second.started_at + # The branch and its debug line are deleted, not merely unreachable. + source = inspect.getsource(agent_module) + assert "Generation window went negative" not in source + assert "_gen_mark_monotonic" not in source + + +async def test_each_turn_gets_a_fresh_clock(): + """A second turn on the same agent re-anchors rather than inheriting. + + One clock per turn is the rule: a clock outliving its turn would stamp the + next one with the previous turn's wall origin, and over a long run would + accumulate drift against real wall time. + """ + step = _step("THINKING", "DONE", thinking="a", usage=_usage(100, 0, 5, 5)) + agent = _agent_with_steps([step]) + first = _assistant(await agent.communicate("go")) + # The fake conversation yields one batch and is then spent, so borrow a + # fresh one. The agent INSTANCE is deliberately the same: what is under + # test is that its second turn builds its own clock rather than inheriting + # the first turn's origin. + agent._sdk_agent = _agent_with_steps([step])._sdk_agent + second = _assistant(await agent.communicate("again")) + + assert first and second + # Re-anchored: the later turn's window opens after the earlier one closed. + assert second[0].started_at >= first[0].completed_at + assert second[0].completed_at > second[0].started_at + + +class TestAntigravityFirstWindowReseed: + """The first `Step` moves `_gen_mark_wall`; a later one must not. + + Driven at `_AntigravityTurnState` with an injected clock, NOT through + `communicate()`: the fake conversation yields with no delay, so an + end-to-end run cannot pin the MAGNITUDE — the two stamps land within + microseconds of each other, so no assertion there could say the mark moved + by the right amount. + + It can detect the mark moving at all, and does: + `test_generation_and_tool_time_account_for_the_turn` asserts `head_ms > 0` + and fails if the re-seed call is removed. These tests are the ones that say + WHERE it moved to and that it moves only once. + """ + + BASE = datetime(2026, 9, 11, 9, 0, 0) + + class _Clock: + def __init__(self, at_ms: float = 0.0) -> None: + self.at_ms = at_ms + + def now(self) -> datetime: + return TestAntigravityFirstWindowReseed.BASE + timedelta(milliseconds=self.at_ms) + + def _state(self, clock): + from coder_eval.agents.antigravity_agent import _AntigravityTurnState + from coder_eval.streaming.callbacks import CompositeStreamCallback + from coder_eval.streaming.collector import EventCollector + + agent = AntigravityAgent(parse_agent_config(type="antigravity", model="gemini-3.5-flash")) + collector = EventCollector() + return _AntigravityTurnState( + agent=agent, + emit=CompositeStreamCallback([collector]), + task_id="t", + turn_id="turn", + collector=collector, + user_input="go", + iteration=1, + model="gemini-3.5-flash", + turn_start_time=0.0, + clock=clock, + ) + + def test_the_first_step_moves_the_mark_off_the_turn_entry_stamp(self): + """Dispatch before the first Step is head, not the first generation. + + Before the re-seed the mark was stamped when the turn state was built, + so this interval was published as generation — ~4.7 s per turn against + a later-window median of 3.3 s. + """ + clock = self._Clock() + state = self._state(clock) + assert state._gen_mark_wall == self.BASE + + clock.at_ms = 900 # dispatch + TTFT + state.process_step(_step("THINKING", "ACTIVE", thinking="...")) + + assert state._gen_mark_wall == self.BASE + timedelta(milliseconds=900) + + def test_a_later_step_does_not_move_it(self): + """Re-seeding more than once per turn is the defect, not the feature.""" + clock = self._Clock() + state = self._state(clock) + clock.at_ms = 900 + state.process_step(_step("THINKING", "ACTIVE", thinking="...")) + seeded = state._gen_mark_wall + + clock.at_ms = 5000 + state.process_step(_step("THINKING", "ACTIVE", thinking="more")) + + assert state._gen_mark_wall == seeded + + def test_seeding_twice_by_hand_is_a_no_op_the_second_time(self): + """The once-per-turn guard, stated outright rather than inferred.""" + clock = self._Clock() + state = self._state(clock) + clock.at_ms = 900 + state._seed_first_generation_window("MODEL") + seeded = state._gen_mark_wall + + clock.at_ms = 5000 + state._seed_first_generation_window("MODEL") + + assert state._gen_mark_wall == seeded + + def test_a_flush_still_advances_the_mark_and_opens_at_the_reseeded_one(self): + """The re-seed must not break the tiling it sits in front of.""" + clock = self._Clock() + state = self._state(clock) + clock.at_ms = 900 + state.process_step(_step("THINKING", "ACTIVE", thinking="plan")) + clock.at_ms = 2000 + state.process_step(_step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5))) + + message = _assistant(state)[0] + assert message.started_at == self.BASE + timedelta(milliseconds=900), "opens at the RE-SEEDED mark" + assert message.generation_duration_ms == pytest.approx(1100.0) + assert state._gen_mark_wall == self.BASE + timedelta(milliseconds=2000), "and the flush advances it" + + def test_a_non_model_step_does_not_seed_the_window(self): + """The field is MODEL output, and the SDK streams Steps that are not. + + `StepSource` carries SYSTEM and USER besides MODEL, and the SDK's event + processor queues every `step_update` verbatim, so a turn can open with + one. Seeding on it would put the mark before the model spoke and hand + the remainder back to msg0's generation — the defect being fixed. + """ + clock = self._Clock() + state = self._state(clock) + + clock.at_ms = 400 + state.process_step(_step("SYSTEM_MESSAGE", "DONE", source="SYSTEM", content="compacting")) + assert state._first_output_seen is False + assert state._gen_mark_wall == self.BASE, "a system Step must not open the generation window" + + clock.at_ms = 900 + state.process_step(_step("THINKING", "ACTIVE", thinking="...")) + assert state._gen_mark_wall == self.BASE + timedelta(milliseconds=900), "the first MODEL Step does" + + def test_a_turn_that_streams_no_step_keeps_the_turn_entry_mark(self): + clock = self._Clock() + state = self._state(clock) + assert state._first_output_seen is False + assert state._gen_mark_wall == self.BASE diff --git a/tests/test_codex_agent.py b/tests/test_codex_agent.py index 342f7297..e15b1dee 100644 --- a/tests/test_codex_agent.py +++ b/tests/test_codex_agent.py @@ -2301,6 +2301,47 @@ async def test_generation_plus_tool_exec_does_not_exceed_the_window(self): assert gen_ms == pytest.approx(10.0) assert gen_ms + tool_ms == pytest.approx(window_ms) + async def test_the_published_window_reconciles_to_its_own_bounds(self): + """The collector subtracted exactly the spans the record carries. + + `scripts/timing/decompose_run.py` and the evalboard's Unaccounted cell + both recompute the tool UNION from the recorded command spans and + subtract it from the recorded window bounds. The cases above pin + arithmetic results against known fixture constants; this one asserts + the reducer fed the window the same span set the record publishes. + + The narrow half by design: `expected` comes from the PUBLISHED bounds, + so it cannot see a wrong mark (TestFlushMessageWindowBounds does), and + both sides call `busy_ms`, so it cannot see a union bug. + """ + from coder_eval.timing import busy_ms + + first = _bounds_command_item("cmd_a") + second = _bounds_command_item("cmd_b") + notifications = [ + _item_notification("item/started", first, started_at_ms=_BOUNDS_EPOCH_MS), + _item_notification("item/completed", first, completed_at_ms=_BOUNDS_EPOCH_MS + 120), + _item_notification("item/started", second, started_at_ms=_BOUNDS_EPOCH_MS + 130), + _item_notification("item/completed", second, completed_at_ms=_BOUNDS_EPOCH_MS + 900), + _token_usage(inp=10, out=5, cached=0), + _turn_completed(), + ] + agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) + record = await agent.communicate("go") + + assistant = [m for m in record.messages if m.role == "assistant"] + spans = [ + (c.execution_started_at, c.execution_completed_at) + for c in record.commands + if c.execution_started_at is not None and c.execution_completed_at is not None + ] + # Codex splits one window's gen_ms across its sub-messages by output + # share, so the reconciliation is against their SUM, not any one row. + lo = min(m.started_at for m in assistant) + hi = max(m.completed_at for m in assistant) + expected = (hi - lo).total_seconds() * 1000.0 - busy_ms(spans, lo, hi) + assert sum(m.generation_duration_ms or 0.0 for m in assistant) == pytest.approx(expected) + class TestGenerationWindowsTileTheTurn: """Each generation window runs from the PREVIOUS one's end, not its own first item. @@ -2364,6 +2405,97 @@ async def test_tool_time_is_still_excluded_from_a_tiled_window(self): assert gen_ms + tool_ms == pytest.approx(2050.0) +class TestFlushMessageWindowBounds: + """Where `_flush_message`'s window OPENS, driven at the reducer. + + The end-to-end cases above all describe a stream whose stamps advance, so + they cannot reach the awkward case the reducer still hands `close_window`: + the emission's own first stamp (`item_start`), whose `min()` against the + mark is the backwards-clock defence. The tool-span arguments this class + also used to cover are gone — the subtraction moved to + `EventCollector.subtract_tool_time`, and + `tests/test_event_collector.py::TestSubtractToolTime` pins it there. + """ + + @staticmethod + def _flush(*, gen_mark_ms, open_start_ms, open_end_ms, open_tool_started_ms=None): + from coder_eval.agents.codex_agent import _CodexTurnState, _ms_to_dt + from coder_eval.models import CommandTelemetry, ContentBlock + from coder_eval.streaming.callbacks import CompositeStreamCallback + from coder_eval.streaming.collector import EventCollector + + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) + collector = EventCollector() + st = _CodexTurnState( + agent, + emit=CompositeStreamCallback([collector]), + task_id="codex", + turn_id="codex-1", + collector=collector, + commands=[], + messages=[], + user_input="go", + iteration=1, + turn_start_time=0.0, + ) + st.open_blocks = [ContentBlock(block_type="text", sequence=0, text="answer")] + st.gen_mark_ms = gen_mark_ms + st.open_start_ms = open_start_ms + st.open_end_ms = open_end_ms + if open_tool_started_ms is not None: + st.open_tools["open-1"] = CommandTelemetry( + tool_name="bash", + tool_id="open-1", + timestamp=_ms_to_dt(open_tool_started_ms), + execution_started_at=_ms_to_dt(open_tool_started_ms), + ) + st._flush_message(SimpleNamespace(input_tokens=10, cached_input_tokens=0, output_tokens=5)) + return st.messages[0] + + def test_a_mark_later_than_the_first_item_does_not_invert_the_window(self): + # A backwards SDK stamp: the previous flush closed at +2000 while this + # emission's first item claims +500. The window must cover the item. + # Without `item_start` it opens at +2000, past its own end, and clamps + # to a fabricated instant generation. + message = self._flush( + gen_mark_ms=_BOUNDS_EPOCH_MS + 2000, + open_start_ms=_BOUNDS_EPOCH_MS + 500, + open_end_ms=_BOUNDS_EPOCH_MS + 1100, + ) + from coder_eval.agents.codex_agent import _ms_to_dt + + assert message.started_at == _ms_to_dt(_BOUNDS_EPOCH_MS + 500) + assert message.generation_duration_ms == pytest.approx(600.0) + + def test_the_published_window_is_raw_and_ignores_a_call_still_open(self): + """The reducer publishes the RAW span; the collector subtracts. + + It used to bound a still-open call at the window's end and take that + slice out here. `EventCollector.subtract_tool_time` sees every span at + once, so a call is subtracted from the windows its REAL interval + overlaps once it resolves — no boundary approximation, and nothing for + this reducer to remember. A call that never resolves has no + `execution_completed_at` and contributes nothing, which is what "never + timed" should cost. + """ + message = self._flush( + gen_mark_ms=_BOUNDS_EPOCH_MS, + open_start_ms=_BOUNDS_EPOCH_MS, + open_end_ms=_BOUNDS_EPOCH_MS + 1000, + open_tool_started_ms=_BOUNDS_EPOCH_MS + 700, + ) + assert message.generation_duration_ms == pytest.approx(1000.0) + + def test_a_call_opening_after_the_window_closes_is_ignored(self): + message = self._flush( + gen_mark_ms=_BOUNDS_EPOCH_MS, + open_start_ms=_BOUNDS_EPOCH_MS, + open_end_ms=_BOUNDS_EPOCH_MS + 1000, + open_tool_started_ms=_BOUNDS_EPOCH_MS + 1500, + ) + assert message.generation_duration_ms == pytest.approx(1000.0) + + class TestFlushMessageGenTimeSplit: """`gen_ms` is apportioned across sub-messages by their output share. @@ -2485,3 +2617,99 @@ async def test_the_split_survives_end_to_end_through_communicate(self): # sub-message produced. assert [m.generation_duration_ms for m in assistant] == [400.0, 600.0] assert sum(m.generation_duration_ms or 0.0 for m in assistant) == 1000.0 + + +class TestTwoSpecGenerationContainingATool: + """A thinking+action window holding a tool: the case the split and the + subtraction have to survive TOGETHER. + + Codex is the only harness that cuts one window into several messages, and + a generation that calls a tool is a two-spec window by construction — the + thinking block plus the tool_use. `TestFlushMessageGenTimeSplit` drives two + specs with no tool; `TestGenerationWindowExcludesToolExecution` on the + other harnesses drives a tool into a single-spec window. Neither reaches + the interaction, which is where grouping by bounds earns its keep: subtract + per message and the overlap comes out twice, and the parts stop summing to + the window. + """ + + @staticmethod + def _published(*, window_ms: int, tool_from_ms: int, tool_to_ms: int, think_out: int, action_out: int): + from coder_eval.agents.codex_agent import _CodexTurnState, _ms_to_dt + from coder_eval.models import CommandTelemetry, ContentBlock, TokenUsage + from coder_eval.streaming.callbacks import CompositeStreamCallback + from coder_eval.streaming.collector import EventCollector + from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, AgentStartEvent, ToolEndEvent + + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) + collector = EventCollector() + state = _CodexTurnState( + agent, + emit=CompositeStreamCallback([collector]), + task_id="codex", + turn_id="codex-1", + collector=collector, + commands=[], + messages=[], + user_input="go", + iteration=1, + turn_start_time=0.0, + ) + command = CommandTelemetry( + tool_name="bash", + tool_id="c1", + timestamp=_ms_to_dt(_BOUNDS_EPOCH_MS + tool_from_ms), + execution_started_at=_ms_to_dt(_BOUNDS_EPOCH_MS + tool_from_ms), + execution_completed_at=_ms_to_dt(_BOUNDS_EPOCH_MS + tool_to_ms), + result_status="success", + ) + state.commands.append(command) + state.open_blocks = [ + ContentBlock(block_type="thinking", sequence=0, thinking="plan"), + ContentBlock(block_type="tool_use", sequence=0, tool_use_id="c1"), + ] + state.open_start_ms = _BOUNDS_EPOCH_MS + state.open_end_ms = _BOUNDS_EPOCH_MS + window_ms + state._flush_message( + SimpleNamespace( + input_tokens=100, + cached_input_tokens=0, + output_tokens=think_out + action_out, + reasoning_output_tokens=think_out, + ) + ) + + collector.on_event( + AgentStartEvent(task_id="codex", prompt="go", iteration=1, timestamp=_ms_to_dt(_BOUNDS_EPOCH_MS)) + ) + collector.on_event(ToolEndEvent(task_id="codex", turn_id="codex-1", tool=command)) + collector.on_event( + AgentEndEvent( + task_id="codex", + status=AgentEndStatus.COMPLETED, + messages=list(state.messages), + usage=TokenUsage(), + timestamp=_ms_to_dt(_BOUNDS_EPOCH_MS + window_ms), + ) + ) + record = collector.build_turn_record() + return [m for m in record.messages if m.role == "assistant"] + + def test_the_group_is_subtracted_once_and_the_parts_still_sum(self): + # A 1000 ms window, split 80/20 by output share, holding a 250 ms tool. + published = self._published(window_ms=1000, tool_from_ms=300, tool_to_ms=550, think_out=800, action_out=200) + assert len(published) == 2, "a thinking + tool_use window is two sub-messages" + total = sum(m.generation_duration_ms or 0.0 for m in published) + # ONCE: 1000 - 250. Subtracting per message would give 500. + assert total == pytest.approx(750.0) + assert [m.generation_duration_ms for m in published] == [pytest.approx(600.0), pytest.approx(150.0)] + + def test_both_sub_messages_still_share_one_window(self): + """The bounds are what the grouping keys on, so they must stay identical.""" + published = self._published(window_ms=1000, tool_from_ms=300, tool_to_ms=550, think_out=800, action_out=200) + assert published[0].started_at == published[1].started_at + assert published[0].completed_at == published[1].completed_at + + def test_a_window_entirely_covered_by_its_tool_splits_zero_two_ways(self): + published = self._published(window_ms=1000, tool_from_ms=0, tool_to_ms=1000, think_out=800, action_out=200) + assert [m.generation_duration_ms for m in published] == [0.0, 0.0] diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py index c2852353..2183098c 100644 --- a/tests/test_custom_lint.py +++ b/tests/test_custom_lint.py @@ -3935,6 +3935,130 @@ def test_the_real_module_is_clean(self): assert "_os._exit(137)" in source, "the guarded call must still exist" +class TestCE061WindowViaCloseWindow: + """CE061 flags a reducer that computes a generation window of its own. + + Every source string carries its own import line: the rule derives its + constructor set from the module's own `coder_eval.models` imports (shared + with CE060 via `_model_ctor`), so a bare `AssistantMessage(...)` with no + import is correctly invisible to it. + """ + + _IMPORT = "from coder_eval.models import AssistantMessage\n" + _HELPER = "from coder_eval.timing import close_window\n" + + @staticmethod + def _run(src: str, filepath: str = "src/coder_eval/agents/pi_agent.py"): + import ast + + from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow + + return WindowViaCloseWindow(filepath).check(ast.parse(src)) + + def test_flags_a_measured_window_without_the_helper(self): + assert len(self._run(self._IMPORT + "m = AssistantMessage(generation_duration_ms=x)")) == 1 + + def test_allows_a_measured_window_when_the_helper_is_imported(self): + assert not self._run(self._IMPORT + self._HELPER + "m = AssistantMessage(generation_duration_ms=x)") + + def test_allows_an_explicit_none(self): + # "Never measured" is an honest claim and needs no window arithmetic — + # codex's rollout rebuild and claude-code's sub-agent synthesis. + assert not self._run(self._IMPORT + "m = AssistantMessage(generation_duration_ms=None)") + + def test_allows_the_kwarg_absent(self): + # Defaults to None, which is the same honest claim. + assert not self._run(self._IMPORT + "m = AssistantMessage(model=model)") + + def test_flags_an_arbitrary_alias(self): + # The gap CE058 concedes: a name list guards the in-tree spelling by + # coincidence and misses `as Msg` outright. + assert ( + len(self._run("from coder_eval.models import AssistantMessage as Msg\nm = Msg(generation_duration_ms=x)")) + == 1 + ) + + def test_flags_the_module_attribute_spelling(self): + assert ( + len(self._run("import coder_eval.models as models\nm = models.AssistantMessage(generation_duration_ms=x)")) + == 1 + ) + + def test_accepts_a_relative_helper_import(self): + # `agents/` uses relative imports; matching only the absolute path + # would leave the rule blind for a whole file. + assert not self._run( + self._IMPORT + "from ..timing import close_window\nm = AssistantMessage(generation_duration_ms=x)" + ) + + def test_accepts_the_module_import_spelling_of_the_helper(self): + # `timing.close_window(...)` is a working call site; a rule that saw + # only the from-import would tell its author to change it. + assert not self._run( + self._IMPORT + "from coder_eval import timing\nm = AssistantMessage(generation_duration_ms=x)" + ) + + def test_an_unrelated_timing_import_does_not_disarm_the_rule(self): + # `from somewhere.else import timing` is not this module; accepting any + # name spelled `timing` would switch the rule off for a whole file. + assert ( + len( + self._run( + self._IMPORT + "from vendor.sdk import timing\nm = AssistantMessage(generation_duration_ms=x)" + ) + ) + == 1 + ) + + def test_ignores_a_file_outside_agents(self): + assert not self._run( + self._IMPORT + "m = AssistantMessage(generation_duration_ms=x)", + filepath="src/coder_eval/streaming/collector.py", + ) + + def test_keys_on_the_helper_name_rather_than_a_literal(self): + from coder_eval.timing import close_window as _helper + from tests.lint.rules import ce061_window_via_close_window as rule_mod + + assert _helper.__name__ == rule_mod._HELPER + + def test_the_real_agents_tree_is_clean(self): + # After claude-code's single permanent suppression. Antigravity + # carried a temporary one until it moved onto `close_window`. + import pathlib + + from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow + from tests.lint.runner import check_file + + root = pathlib.Path(__file__).resolve().parent.parent / "src" / "coder_eval" / "agents" + found = [v for path in sorted(root.glob("*.py")) for v in check_file(path, [WindowViaCloseWindow])] + assert not found, found + + def test_the_rule_is_now_exemption_free(self): + """No reducer needs a `# noqa: CE061` any more, and the set is PINNED empty. + + A noqa nobody needs is a noqa that outlives its reason, so this asserts + the exact set rather than merely that it shrank. It has earned that + twice: antigravity carried a TEMPORARY suppression until it moved onto + `close_window`, and claude-code carried a permanent one until the tool + subtraction moved to `EventCollector.subtract_tool_time` — at which + point it could call the same shrunken helper as the other four. This + test is what failed each time the reason expired. + """ + import ast + import pathlib + + from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow + + root = pathlib.Path(__file__).resolve().parent.parent / "src" / "coder_eval" / "agents" + suppressed = { + path.name + for path in sorted(root.glob("*.py")) + if WindowViaCloseWindow(str(path)).check(ast.parse(path.read_text(encoding="utf-8"))) + } + assert suppressed == set() + + class TestRuffExternalCoversEveryRule: """Every CE rule's documented `# noqa` must be accepted by ruff. @@ -4355,6 +4479,28 @@ def test_ignores_a_dict_literal_that_is_not_an_update_kwarg(self): # Scoped to `update=` so an unrelated fixture dict cannot fire. assert not self._run('row = {"duration_ms": 0.0}') + # The head/tail family — the turn-level buckets on TurnRecord. + def test_flags_a_zero_harness_startup(self): + assert self._run("rec = TurnRecord(iteration=0, harness_startup_ms=0.0)") + + def test_flags_a_zero_harness_teardown(self): + assert self._run("rec = TurnRecord(iteration=0, harness_teardown_ms=0)") + + def test_allows_an_unmeasured_harness_startup(self): + assert not self._run("rec = TurnRecord(iteration=0, harness_startup_ms=None)") + + def test_allows_a_measured_harness_startup(self): + assert not self._run("rec = TurnRecord(iteration=0, harness_startup_ms=head_ms)") + + def test_flags_the_head_coalesce(self): + assert self._run("x = rec.harness_startup_ms or 0") + + def test_ignores_a_name_that_merely_starts_with_startup(self): + # Anchored at both ends, and the family needs a leading segment: a + # limit is not a measurement, and a bare `startup_ms` is not ours. + assert not self._run("cfg = TurnRecord(startup_ms_limit=0)") + assert not self._run("x = startup_ms_limit or 0") + # Scope + suppression. def test_is_out_of_scope_outside_src(self): assert not self._run( @@ -4424,3 +4570,168 @@ def test_noqa_suppresses(self): path = SRC / "coder_eval/agents/antigravity_agent.py" assert path.is_file(), "the noqa fixture file must exist or this test passes vacuously" assert not [v for v in check_file(path) if v.rule_id == "CE059"] + + +class TestCE060MessageIdDeclared: + """CE060 flags an assistant message built without an identity. + + Every source string carries its own import line: the rule derives its + constructor set from the module's own `coder_eval.models` imports, so a + bare `AssistantMessage(...)` with no import is correctly invisible to it. + """ + + _IMPORT = "from coder_eval.models import AssistantMessage\n" + + @staticmethod + def _run(src: str, filepath: str = "src/coder_eval/agents/antigravity_agent.py"): + import ast + + from tests.lint.rules.ce060_message_id_declared import MessageIdDeclared + + return MessageIdDeclared(filepath).check(ast.parse(src)) + + def test_flags_an_omitted_message_id(self): + assert self._run(self._IMPORT + "m = AssistantMessage(model=model, output_tokens=3)") + + def test_flags_an_explicit_none(self): + # Passing None is a claim that no id exists, which is never true for a + # harness that can synthesize one. + assert self._run(self._IMPORT + "m = AssistantMessage(model=model, message_id=None)") + + def test_flags_the_in_tree_alias_spelling(self): + assert self._run( + "from coder_eval.models import AssistantMessage as AssistantMessageTelemetry\n" + "m = AssistantMessageTelemetry(model=model)" + ) + + def test_flags_an_arbitrary_alias(self): + # The case a hardcoded name list misses entirely — the whole reason + # CE060 resolves aliases instead. + assert self._run("from coder_eval.models import AssistantMessage as Msg\nm = Msg(model=model)") + + def test_flags_the_module_alias_spelling(self): + # The realistic way to write `models.AssistantMessage(...)`: the class + # itself is never bound, so only the attribute is left to match on. + assert self._run("import coder_eval.models as models\nm = models.AssistantMessage(model=model)") + + def test_flags_the_attribute_spelling_beside_a_direct_import(self): + assert self._run(self._IMPORT + "m = models.AssistantMessage(model=model)") + + def test_flags_a_relative_import(self): + # `agents/` does use relative imports, and the absolute path test alone + # left the rule silently blind for a whole file. + assert self._run("from ..models import AssistantMessage\nm = AssistantMessage(model=model)") + + def test_keys_on_the_model_name_rather_than_a_literal(self): + # The constant moved into the shared resolver when CE061 was added; it + # is still derived from the model, which is the property under test. + from coder_eval.models import AssistantMessage as _Model + from tests.lint.rules import _model_ctor + + assert _Model.__name__ == _model_ctor.ASSISTANT_MESSAGE + + def test_flags_a_star_expanded_call(self): + # `**fields` has not declared the field at the site. + assert self._run(self._IMPORT + "m = AssistantMessage(**fields)") + + def test_allows_a_literal_id(self): + assert not self._run(self._IMPORT + 'm = AssistantMessage(message_id="x")') + + def test_allows_an_fstring_id(self): + assert not self._run(self._IMPORT + 'm = AssistantMessage(message_id=f"{turn_id}-msg-{i}")') + + def test_allows_a_fallback_expression(self): + # The runtime-None blind spot, exempted deliberately: passing a + # fallback expression IS deciding what the id is. + assert not self._run(self._IMPORT + "m = AssistantMessage(message_id=str(x) or None)") + + def test_allows_a_star_expanded_call_that_also_passes_the_field(self): + assert not self._run(self._IMPORT + "m = AssistantMessage(**fields, message_id=mid)") + + def test_ignores_an_unrelated_constructor(self): + assert not self._run(self._IMPORT + "s = Span(model=model)") + + def test_ignores_a_module_with_no_matching_import(self): + # Nothing is bound, so the rule claims nothing here. A construction + # site has to import the class to reach it. + assert not self._run("m = AssistantMessage(model=model)") + + def test_is_out_of_scope_outside_agents(self): + assert not self._run( + self._IMPORT + "m = AssistantMessage(model=model)", + filepath="src/coder_eval/orchestrator.py", + ) + + def test_the_real_antigravity_flush_declares_its_id(self): + from tests.lint.runner import check_file + + path = SRC / "coder_eval/agents/antigravity_agent.py" + assert path.is_file(), "the fixture file must exist or this test passes vacuously" + assert not [v for v in check_file(path) if v.rule_id == "CE060"] + + +class TestCE063NoBusyMsInAgents: + """CE063 flags a reducer that would subtract tool time itself. + + The subtraction lives once, in + `coder_eval.streaming.collector.subtract_tool_time`. A reducer that also + does it has its tool time taken out TWICE — once by itself, once by the + collector — which under-reports generation on that harness alone. + """ + + @staticmethod + def _run(src: str, filepath: str = "src/coder_eval/agents/pi_agent.py"): + import ast + + from tests.lint.rules.ce063_no_busy_ms_in_agents import NoBusyMsInAgents + + return NoBusyMsInAgents(filepath).check(ast.parse(src)) + + def test_flags_the_bare_name_import(self): + assert len(self._run("from coder_eval.timing import busy_ms")) == 1 + + def test_flags_it_under_an_alias(self): + # The import is what is banned, whatever it is bound to. + assert len(self._run("from coder_eval.timing import busy_ms as union")) == 1 + + def test_flags_it_alongside_an_allowed_import(self): + assert len(self._run("from coder_eval.timing import busy_ms, close_window")) == 1 + + def test_flags_a_relative_import(self): + # `agents/` uses relative imports; matching only the absolute path + # would leave the rule blind for a whole file. + assert len(self._run("from ..timing import busy_ms")) == 1 + + def test_flags_the_module_attribute_spelling(self): + assert len(self._run("from coder_eval import timing\nx = timing.busy_ms(s, lo, hi)")) == 1 + + def test_does_not_fire_on_close_window_through_the_module(self): + """The exact false positive a naive inversion of CE061's resolver gives. + + `_imports_the_helper` returns True for a bare module import so that + `timing.close_window(...)` counts as reaching the helper. Inverted into + a ban, that branch flags every reducer importing the module — which + after the migration is four of the five. + """ + assert not self._run("from coder_eval import timing\nx = timing.close_window(mark=m, now=n)") + + def test_does_not_fire_on_close_window_by_name(self): + assert not self._run("from coder_eval.timing import close_window\nx = close_window(mark=m, now=n)") + + def test_does_not_fire_on_an_unrelated_attribute_named_busy_ms(self): + # `self.busy_ms` is not `timing.busy_ms`; only the module spelling counts. + assert not self._run("x = self.busy_ms") + + def test_does_not_fire_outside_agents(self): + # The collector is where the subtraction belongs, so it must import it. + assert not self._run("from coder_eval.timing import busy_ms", filepath="src/coder_eval/streaming/collector.py") + + def test_is_suppressible(self, tmp_path): + from tests.lint.rules.ce063_no_busy_ms_in_agents import NoBusyMsInAgents + from tests.lint.runner import check_file + + agents = tmp_path / "src" / "coder_eval" / "agents" + agents.mkdir(parents=True) + target = agents / "pi_agent.py" + target.write_text("from coder_eval.timing import busy_ms # noqa: CE063\n", encoding="utf-8") + assert not check_file(target, [NoBusyMsInAgents]) diff --git a/tests/test_event_collector.py b/tests/test_event_collector.py index 76e49e49..6072c970 100644 --- a/tests/test_event_collector.py +++ b/tests/test_event_collector.py @@ -5,9 +5,11 @@ rules in ``coder_eval/streaming/collector.py``. """ -from datetime import datetime +from datetime import datetime, timedelta from typing import ClassVar +import pytest + from coder_eval.models import ( AssistantMessage, CommandTelemetry, @@ -16,13 +18,15 @@ TokenUsage, TurnRecord, ) -from coder_eval.streaming.collector import EventCollector +from coder_eval.streaming.collector import EventCollector, subtract_tool_time from coder_eval.streaming.events import ( AgentEndEvent, + AgentEndStatus, AgentStartEvent, ToolEndEvent, TurnStartEvent, ) +from coder_eval.timing import union_ms TASK_ID = "collector-test" @@ -202,7 +206,17 @@ class TestFullFieldParity: # provider_call_costs -> joined in post-run by the orchestrator from the # LiteLLM proxy cost log (litellm_cost.apply_actual_cost), # not emitted by the agent/EventCollector. - _DERIVED: ClassVar[set[str]] = {"commands", "token_usage", "timestamp", "provider_call_costs"} + _DERIVED: ClassVar[set[str]] = { + "commands", + "token_usage", + "timestamp", + "provider_call_costs", + # Measured by the collector between the agent's own start/end event + # stamps and the first/last generation window — not carried on + # AgentEndEvent, because no agent computes them. + "harness_startup_ms", + "harness_teardown_ms", + } def _full_agent_end(self) -> AgentEndEvent: """An AgentEndEvent with every verbatim field set to a non-default sentinel.""" @@ -458,3 +472,564 @@ def test_minimal_record_without_agent_end(self): assert record.model_used == "gpt-x" assert record.assistant_turn_count == 1 assert [c.tool_id for c in record.commands] == ["a"] + + +class TestHarnessOverheadBuckets: + """The turn's two unexplained ends: before the first generation, after the last. + + Measured live across all five harnesses, these two plus generation plus tool + execution account for the turn to within 0.1 ms — so what the evalboard shows + as "Unaccounted" is fully explained rather than merely displayed. The head is + where the harnesses differ most — every one of them now measures it up to + its first observed model output, but what that interval CONTAINS ranges from + ~0.23 s on Pi to ~4.7 s on Antigravity, depending on whether the harness + spawns its process per turn and how long the provider takes to first token. + That spread is exactly why it is booked as its own bucket instead of being + folded into generation. + """ + + @staticmethod + def _msg(started: datetime, completed: datetime, *, measurable: bool = True) -> AssistantMessage: + """A generation window. ``measurable=False`` is the placeholder shape + every fabricated-bounds producer writes — a rollout rebuild or a + sub-agent recovery — which stamps one instant on both bounds and says + so with ``generation_duration_ms=None``.""" + return AssistantMessage( + started_at=started, + completed_at=completed, + generation_duration_ms=1.0 if measurable else None, + ) + + @staticmethod + def _subagent_msg(started: datetime, completed: datetime) -> AssistantMessage: + """A sub-agent generation: same shape, tagged with the spawning Agent + call's tool_use_id. Its time is already inside that call's interval.""" + return AssistantMessage( + started_at=started, + completed_at=completed, + generation_duration_ms=1.0, + parent_tool_use_id="toolu_agent", + ) + + @staticmethod + def _tool(started: datetime, completed: datetime, tool_id: str = "t1") -> ToolEndEvent: + return ToolEndEvent( + task_id=TASK_ID, + tool=CommandTelemetry( + tool_id=tool_id, + tool_name="Bash", + timestamp=started, + sequence_number=0, + execution_started_at=started, + execution_completed_at=completed, + result_status="success", + ), + ) + + def _record(self, messages, *, start: datetime, end: datetime, tools=()) -> TurnRecord: + collector = EventCollector() + _feed( + collector, + [ + AgentStartEvent(task_id=TASK_ID, prompt="go", iteration=1, timestamp=start), + *tools, + AgentEndEvent( + task_id=TASK_ID, + usage=TokenUsage(output_tokens=1), + messages=messages, + timestamp=end, + ), + ], + ) + return collector.build_turn_record() + + def test_head_and_tail_are_measured_from_the_agent_event_stamps(self): + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [self._msg(t0.replace(second=2), t0.replace(second=5))], + start=t0, + end=t0.replace(second=9), + ) + assert rec.harness_startup_ms == pytest.approx(2000.0) + assert rec.harness_teardown_ms == pytest.approx(4000.0) + + def test_a_sub_agent_generation_does_not_move_the_bracket(self): + """MAIN THREAD ONLY, the rule the two sibling call sites already apply. + + The identity these buckets complete sums generation over the main + thread only — a sub-agent's run is already inside its parent Agent + call's interval. Letting a sub-agent message bracket the span shrinks + the head or the tail by time no bucket then claims, and Codex's + recovered child messages carry the CHILD's clock, so the bracket can + move either way. + """ + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [ + self._msg(t0.replace(second=2), t0.replace(second=5)), + # Stamps outside the main thread's own span, in both directions. + self._subagent_msg(t0.replace(second=1), t0.replace(second=8)), + ], + start=t0, + end=t0.replace(second=9), + ) + assert rec.harness_startup_ms == pytest.approx(2000.0) + assert rec.harness_teardown_ms == pytest.approx(4000.0) + + def test_a_turn_whose_only_generations_are_sub_agent_reports_no_overhead(self): + """No main-thread window means nothing was measured — None, not 0.0.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [self._subagent_msg(t0.replace(second=2), t0.replace(second=5))], + start=t0, + end=t0.replace(second=9), + ) + assert rec.harness_startup_ms is None + assert rec.harness_teardown_ms is None + + def test_a_turn_with_no_generation_says_so_rather_than_claiming_zero(self): + """None means never measured; 0.0 would mean measured-and-instant (CE058).""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record([], start=t0, end=t0.replace(second=9)) + assert rec.harness_startup_ms is None + assert rec.harness_teardown_ms is None + + def test_a_measured_zero_head_is_zero_not_none(self): + """claude-code and Antigravity really do open their first window at turn + start, so their head is a genuine 0.0 — the distinction from None is the + whole point of the field.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record([self._msg(t0, t0.replace(second=5))], start=t0, end=t0.replace(second=5)) + assert rec.harness_startup_ms == 0.0 + assert rec.harness_teardown_ms == 0.0 + + def test_the_tail_ignores_a_trailing_reconciliation_entry(self): + """It is always last when present and carries no timestamps at all, so + indexing messages[-1] would raise rather than measure.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [ + self._msg(t0.replace(second=1), t0.replace(second=4)), + ReconciliationMessage(input_tokens=5, note="residual"), + ], + start=t0, + end=t0.replace(second=6), + ) + assert rec.harness_teardown_ms == pytest.approx(2000.0) + + def test_a_clock_inversion_clamps_rather_than_going_negative(self): + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [self._msg(t0.replace(minute=59, hour=11), t0.replace(second=5))], + start=t0, + end=t0.replace(second=1), + ) + assert rec.harness_startup_ms == 0.0 + + def test_a_snapshot_before_the_terminal_event_measures_nothing(self): + collector = EventCollector() + _feed(collector, [AgentStartEvent(task_id=TASK_ID, prompt="go", iteration=1)]) + rec = collector.build_turn_record() + assert rec.harness_startup_ms is None + assert rec.harness_teardown_ms is None + + def test_a_placeholder_message_does_not_supply_the_bounds(self): + """A Codex turn rebuilt from its rollout stamps every message at turn + END and marks them generation_duration_ms=None. Reading those stamps as + window bounds books the WHOLE TURN as harness startup.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [ + self._msg(t0.replace(second=2), t0.replace(second=5)), + self._msg(t0.replace(second=9), t0.replace(second=9), measurable=False), + ], + start=t0, + end=t0.replace(second=9), + ) + assert rec.harness_startup_ms == pytest.approx(2000.0) + # 9s - 5s, measured off the real window, not off the placeholder's stamp. + assert rec.harness_teardown_ms == pytest.approx(4000.0) + + def test_a_turn_of_only_placeholders_measures_nothing(self): + """codex_g_items_rebuild's shape: an assistant message exists, but + nothing in it was timed, so there is no end to measure against.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [self._msg(t0.replace(second=9), t0.replace(second=9), measurable=False)], + start=t0, + end=t0.replace(second=9), + ) + assert rec.harness_startup_ms is None + assert rec.harness_teardown_ms is None + + def test_the_bounds_do_not_depend_on_append_order(self): + """Codex appends recovered sub-agent messages after the parent's last + flush, so the list is not ordered by time.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [ + self._msg(t0.replace(second=6), t0.replace(second=8)), + self._msg(t0.replace(second=2), t0.replace(second=4)), + ], + start=t0, + end=t0.replace(second=9), + ) + assert rec.harness_startup_ms == pytest.approx(2000.0) + assert rec.harness_teardown_ms == pytest.approx(1000.0) + + def test_a_tool_running_past_the_last_window_is_not_counted_twice(self): + """Antigravity force-closes an orphan at finalization, stamping its + completion inside the tail, and backgrounds anything over ten seconds. + Such a span is already in the tool bucket, so leaving it in the tail + books it twice and drives the residual sharply negative.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [self._msg(t0.replace(second=1), t0.replace(second=4))], + start=t0, + end=t0.replace(second=9), + tools=[self._tool(t0.replace(second=3), t0.replace(second=7))], + ) + # Tail spans 4s->9s = 5s, of which 4s->7s = 3s was the tool still running. + assert rec.harness_teardown_ms == pytest.approx(2000.0) + + def test_a_tool_running_before_the_first_window_is_not_counted_twice(self): + t0 = datetime(2026, 1, 1, 12, 0, 0) + rec = self._record( + [self._msg(t0.replace(second=5), t0.replace(second=8))], + start=t0, + end=t0.replace(second=8), + tools=[self._tool(t0.replace(second=1), t0.replace(second=3))], + ) + # Head spans 0s->5s = 5s, of which 1s->3s = 2s was tool execution. + assert rec.harness_startup_ms == pytest.approx(3000.0) + + def test_a_new_turn_clears_the_previous_turn_terminal_event(self): + """EarlyStopWatcher keeps ONE collector across retries. Left stale, the + next attempt's start pairs with the last attempt's end and the clamped + inversion publishes as a measured 0.0.""" + t0 = datetime(2026, 1, 1, 12, 0, 0) + collector = EventCollector() + _feed( + collector, + [ + AgentStartEvent(task_id=TASK_ID, prompt="go", iteration=1, timestamp=t0), + AgentEndEvent( + task_id=TASK_ID, + usage=TokenUsage(output_tokens=1), + messages=[self._msg(t0.replace(second=1), t0.replace(second=2))], + timestamp=t0.replace(second=3), + crashed=True, + ), + # Retry, a minute later, with no terminal event of its own yet. + AgentStartEvent(task_id=TASK_ID, prompt="go", iteration=1, timestamp=t0.replace(minute=1)), + ], + ) + rec = collector.build_turn_record() + assert rec.harness_startup_ms is None + assert rec.harness_teardown_ms is None + + +class TestSubtractToolTime: + """The ONE tool subtraction, moved here from five reducers. + + Four of them did it inside `close_window` as they flushed; claude-code did + it once at finalization. Head and tail were already computed centrally, in + this module — that asymmetry was the complexity, and every timing defect + this branch fixed lived in the per-reducer bookkeeping around the + subtraction rather than in the subtraction itself. + """ + + BASE: ClassVar[datetime] = datetime(2026, 9, 11, 9, 0, 0) + + @classmethod + def _at(cls, ms: float) -> datetime: + return cls.BASE + timedelta(milliseconds=ms) + + @classmethod + def _msg(cls, lo: float, hi: float, gen: float | None, **kwargs) -> AssistantMessage: + return AssistantMessage(started_at=cls._at(lo), completed_at=cls._at(hi), generation_duration_ms=gen, **kwargs) + + def test_a_contained_tool_is_subtracted_exactly_once(self): + out = subtract_tool_time([self._msg(0, 1000, 1000.0)], [(self._at(200), self._at(700))]) + assert out[0].generation_duration_ms == pytest.approx(500.0) + + def test_the_input_messages_are_not_mutated(self): + """Non-mutating because of ALIASING, not because of repeated calls. + + Every agent builds its terminal event as + `AgentEndEvent(messages=list(...))`, which copies the LIST and not the + message objects — so an in-place write would reach back into the + agent's own live state from the collector. + """ + messages = [self._msg(0, 1000, 1000.0)] + subtract_tool_time(messages, [(self._at(200), self._at(700))]) + assert messages[0].generation_duration_ms == pytest.approx(1000.0) + + def test_a_group_sharing_bounds_is_subtracted_once_and_the_parts_still_sum(self): + """Codex splits one window across two sub-messages by output share. + + Subtracting the group's overlap from each part separately would take it + twice and stop the parts summing to the window. Grouping is on the + BOUNDS, not on `message_id` — OpenCode and Pi can carry `None` there. + """ + # A 1000 ms window split 25/75, with a 250 ms tool inside it. + out = subtract_tool_time( + [self._msg(0, 1000, 250.0, message_id="m"), self._msg(0, 1000, 750.0, message_id="m")], + [(self._at(300), self._at(550))], + ) + assert [m.generation_duration_ms for m in out] == [pytest.approx(187.5), pytest.approx(562.5)] + assert sum(m.generation_duration_ms or 0.0 for m in out) == pytest.approx(750.0) + + def test_a_group_with_no_message_id_is_still_grouped_by_its_bounds(self): + """The case keying on `message_id` would break. + + Two id-less messages sharing a window must be one group; keying on the + id would instead collapse every id-less message of the turn into one. + """ + out = subtract_tool_time( + [self._msg(0, 1000, 500.0), self._msg(0, 1000, 500.0), self._msg(2000, 3000, 1000.0)], + [(self._at(200), self._at(400))], + ) + assert sum(m.generation_duration_ms or 0.0 for m in out[:2]) == pytest.approx(800.0) + assert out[2].generation_duration_ms == pytest.approx(1000.0), "a different window is a different group" + + def test_concurrent_tools_subtract_their_union_not_their_sum(self): + """Summing would clamp a real generation to zero. + + The expectation is DERIVED from `union_ms` rather than written as a + literal, so this cannot drift from the rule the rest of the codebase + applies — and the sum is asserted separately to be the wrong answer. + """ + spans = [ + (self._at(100), self._at(500)), + (self._at(150), self._at(550)), + (self._at(200), self._at(600)), + (self._at(250), self._at(650)), + ] + out = subtract_tool_time([self._msg(0, 1000, 1000.0)], spans) + assert out[0].generation_duration_ms == pytest.approx(1000.0 - union_ms(spans)) + assert sum((e - s).total_seconds() * 1000.0 for s, e in spans) > 1000.0, ( + "the fixture must actually over-subtract when summed, or this proves nothing" + ) + assert out[0].generation_duration_ms > 0.0 + + def test_a_window_entirely_covered_by_tools_is_a_measured_zero(self): + out = subtract_tool_time([self._msg(0, 1000, 1000.0)], [(self._at(0), self._at(1000))]) + assert out[0].generation_duration_ms == 0.0, "a measurement, not an absence" + + def test_a_none_duration_stays_none(self): + """`None` means no window was ever measured, and CE058 keeps it distinct.""" + out = subtract_tool_time([self._msg(0, 1000, None)], [(self._at(0), self._at(500))]) + assert out[0].generation_duration_ms is None + + def test_a_zero_group_does_not_divide_by_zero(self): + out = subtract_tool_time([self._msg(0, 1000, 0.0)], [(self._at(0), self._at(500))]) + assert out[0].generation_duration_ms == 0.0 + + def test_a_sub_agent_generation_is_skipped(self): + """Its own tools are not in this span set, and the spawning Agent call + already covers its whole run.""" + out = subtract_tool_time([self._msg(0, 1000, 900.0, parent_tool_use_id="t1")], [(self._at(0), self._at(500))]) + assert out[0].generation_duration_ms == pytest.approx(900.0) + + def test_a_crash_partial_with_no_messages_and_live_spans_is_safe(self): + """The shape a crashed turn actually produces. + + `Agent._finalize` builds a record from whatever the collector saw, and + a turn that died before its first emission has resolved tool calls but + NO messages. Nothing to group, nothing to subtract — and no + ZeroDivisionError, no IndexError, and no invented entry. + """ + out = subtract_tool_time([], [(self._at(0), self._at(500))]) + assert out == [] + + def test_non_assistant_entries_pass_through_by_identity(self): + reconciliation = ReconciliationMessage( + input_tokens=1, output_tokens=1, cache_creation_tokens=0, cache_read_tokens=0, note="n" + ) + out = subtract_tool_time([self._msg(0, 1000, 1000.0), reconciliation], [(self._at(0), self._at(200))]) + assert out[1] is reconciliation + + +class TestBuildTurnRecordIsIdempotent: + """Building the record twice must give the same numbers. + + `EventCollector` is not built once and read once. `EarlyStopWatcher` holds + ONE across a turn's tool-call rounds and calls `build_turn_record()` on + every one, and the crash path builds it again from `Agent._finalize`. The + tool subtraction now happens inside that method, so a version of it that + mutated would subtract again on every call — and the numbers would depend + on how many times something happened to look. + """ + + BASE: ClassVar[datetime] = datetime(2026, 9, 11, 9, 0, 0) + + def _collector(self) -> EventCollector: + at = lambda ms: self.BASE + timedelta(milliseconds=ms) # noqa: E731 + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=at(0))) + collector.on_event( + ToolEndEvent( + task_id="t", + turn_id="t1", + tool=CommandTelemetry( + tool_name="bash", + tool_id="c1", + timestamp=at(700), + execution_started_at=at(700), + execution_completed_at=at(1200), + result_status="success", + ), + ) + ) + collector.on_event( + AgentEndEvent( + task_id="t", + status=AgentEndStatus.COMPLETED, + messages=[ + AssistantMessage( + started_at=at(500), completed_at=at(2000), generation_duration_ms=1500.0, output_tokens=5 + ) + ], + usage=TokenUsage(output_tokens=5), + timestamp=at(2500), + ) + ) + return collector + + def test_two_builds_agree_on_every_timing_figure(self): + collector = self._collector() + first, second = collector.build_turn_record(), collector.build_turn_record() + + assert [m.generation_duration_ms for m in first.messages if m.role == "assistant"] == [ + m.generation_duration_ms for m in second.messages if m.role == "assistant" + ] + assert first.harness_startup_ms == second.harness_startup_ms + assert first.harness_teardown_ms == second.harness_teardown_ms + + def test_the_first_build_already_subtracted_once(self): + """Guards the other direction: identical-but-wrong would also pass above.""" + record = self._collector().build_turn_record() + generation = [m.generation_duration_ms for m in record.messages if m.role == "assistant"] + # A 1500 ms window holding a 500 ms tool. + assert generation == [pytest.approx(1000.0)] + + def test_the_agents_own_message_objects_are_not_written_through(self): + """The aliasing case, which is the real reason for `model_copy`. + + `AgentEndEvent(messages=list(...))` copies the LIST, not the messages, + so the objects the collector receives are the agent's own live state. + """ + at = lambda ms: self.BASE + timedelta(milliseconds=ms) # noqa: E731 + message = AssistantMessage(started_at=at(0), completed_at=at(1000), generation_duration_ms=1000.0) + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=at(0))) + collector.on_event( + ToolEndEvent( + task_id="t", + turn_id="t1", + tool=CommandTelemetry( + tool_name="bash", + tool_id="c1", + timestamp=at(200), + execution_started_at=at(200), + execution_completed_at=at(700), + result_status="success", + ), + ) + ) + collector.on_event( + AgentEndEvent(task_id="t", status=AgentEndStatus.COMPLETED, messages=[message], timestamp=at(1000)) + ) + collector.build_turn_record() + + assert message.generation_duration_ms == pytest.approx(1000.0), "the agent's own object must be untouched" + + +class TestOverheadExcludesSubAgentTools: + """The head and tail are bracketed on the MAIN thread, commands included. + + `_overhead_ms` filtered its GENERATIONS to the main thread and then passed + EVERY command as a tool span, so its own claim to keep all four buckets + measuring one thread was true only by luck: a child nests inside the parent + Agent call, whose interval the union already covers. Codex's recovered + child tools carry the CHILD's clock, so nothing made it true by + construction — and the evalboard's twin DOES filter, so the two agreed by + accident. + """ + + BASE: ClassVar[datetime] = datetime(2026, 9, 11, 9, 0, 0) + + def test_a_sub_agent_tool_inside_the_head_does_not_shrink_it(self): + at = lambda ms: self.BASE + timedelta(milliseconds=ms) # noqa: E731 + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=at(0))) + # A sub-agent's own tool call, sitting inside what is otherwise head. + collector.on_event( + ToolEndEvent( + task_id="t", + turn_id="t1", + tool=CommandTelemetry( + tool_name="Bash", + tool_id="child-1", + timestamp=at(100), + execution_started_at=at(100), + execution_completed_at=at(400), + result_status="success", + ), + ) + ) + collector.on_event( + AgentEndEvent( + task_id="t", + status=AgentEndStatus.COMPLETED, + messages=[ + AssistantMessage(started_at=at(500), completed_at=at(1000), generation_duration_ms=500.0), + # The child generation that OWNS child-1. + AssistantMessage( + started_at=at(100), + completed_at=at(400), + generation_duration_ms=300.0, + parent_tool_use_id="agent-call", + tool_use_ids=["child-1"], + ), + ], + timestamp=at(1500), + ) + ) + record = collector.build_turn_record() + + # 500 ms of head, all of it. Counting the child's tool would book 300 ms + # of it as tool execution that no main-thread bucket claims. + assert record.harness_startup_ms == pytest.approx(500.0) + assert record.harness_teardown_ms == pytest.approx(500.0) + + def test_a_main_thread_tool_inside_the_head_still_shrinks_it(self): + """The control: the filter must exclude children, not all commands.""" + at = lambda ms: self.BASE + timedelta(milliseconds=ms) # noqa: E731 + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=at(0))) + collector.on_event( + ToolEndEvent( + task_id="t", + turn_id="t1", + tool=CommandTelemetry( + tool_name="Bash", + tool_id="main-1", + timestamp=at(100), + execution_started_at=at(100), + execution_completed_at=at(400), + result_status="success", + ), + ) + ) + collector.on_event( + AgentEndEvent( + task_id="t", + status=AgentEndStatus.COMPLETED, + messages=[AssistantMessage(started_at=at(500), completed_at=at(1000), generation_duration_ms=500.0)], + timestamp=at(1500), + ) + ) + record = collector.build_turn_record() + assert record.harness_startup_ms == pytest.approx(200.0), "500 ms of head minus a 300 ms tool" diff --git a/tests/test_opencode_agent.py b/tests/test_opencode_agent.py index 6d62bbcd..605563fa 100644 --- a/tests/test_opencode_agent.py +++ b/tests/test_opencode_agent.py @@ -32,8 +32,9 @@ _unwrap, ) from coder_eval.errors import AgentCrashError, TurnTimeoutError -from coder_eval.models import AssistantMessage, CommandTelemetry, OpenCodeAgentConfig, PermissionMode +from coder_eval.models import AssistantMessage, CommandTelemetry, OpenCodeAgentConfig, PermissionMode, TokenUsage from coder_eval.pricing import calculate_cost +from coder_eval.streaming.collector import EventCollector from coder_eval.streaming.events import ( AgentEndEvent, AgentEndStatus, @@ -1784,24 +1785,35 @@ def test_orphan_result_is_never_dropped(self): class TestGenerationWindowExcludesToolExecution: - """A tool running inside a step is not model time. - - OpenCode marks the window at `step_start` and closes it at - `step_finish`, and every tool call executes INSIDE it while also - publishing its own measured `duration_ms`. Publishing the raw span as - generation time counted the same milliseconds twice, which the task - page's Unaccounted cell renders as a ~-100% residual. - - Driven at the reducer rather than through `communicate()`: the window is - two `datetime.now()` reads and the tool interval comes from the event - payload, so only setting both explicitly makes the arithmetic - deterministic. + """A tool running inside a step is not model time — asserted where it is now DECIDED. + + The reducer no longer subtracts anything. It publishes the RAW window, and + `EventCollector.subtract_tool_time` takes the tool union back out of it + once, for all five harnesses. So these cases drive the reducer and then a + real collector, and assert the PUBLISHED number — the one that reaches + `task.json` — rather than an intermediate the reducer used to own. + + They are not duplicates of + `tests/test_event_collector.py::TestSubtractToolTime`: those pin the + arithmetic, these pin that THIS reducer hands the collector a window and a + span set the arithmetic can be right about. """ WINDOW_START = datetime(2026, 1, 1, 12, 0, 0) WINDOW_END = datetime(2026, 1, 1, 12, 0, 1) # a 1000ms step def _finish_step(self, monkeypatch, spans, open_starts=()): + """Drive the reducer, then publish through a real collector. + + `spans` are RESOLVED calls (both bounds); `open_starts` are calls that + never returned. An unresolved call now contributes NO span — it has no + `execution_completed_at`, and inventing one is what `None` exists to + prevent — where the reducer used to bound it at the window's end. That + is a real change and a better one: the collector sees every span at + once, so a call straddling a boundary is clipped to each window it + actually overlapped instead of approximated at the boundary. + """ + class _Clock(datetime): @staticmethod def now(tz=None): @@ -1809,37 +1821,56 @@ def now(tz=None): state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="do it", model="deepseek/deepseek-v4-pro") state.step_started_at = self.WINDOW_START - state.step_tool_spans = list(spans) - for i, started in enumerate(open_starts): - state.open_tools[f"open-{i}"] = CommandTelemetry( + commands = [ + CommandTelemetry( tool_name="bash", - tool_id=f"open-{i}", + tool_id=f"closed-{i}", timestamp=started, execution_started_at=started, + execution_completed_at=completed, + result_status="success", ) + for i, (started, completed) in enumerate(spans) + ] + commands += [ + CommandTelemetry(tool_name="bash", tool_id=f"open-{i}", timestamp=st, execution_started_at=st) + for i, st in enumerate(open_starts) + ] monkeypatch.setattr(agent_module, "datetime", _Clock) state.on_step_finish({"reason": "stop", "tokens": {"input": 100, "output": 20}}) - assistant = [m for m in state.messages if m.role == "assistant"] - assert len(assistant) == 1 - return assistant[0] + + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t1", prompt="do it", iteration=1, timestamp=self.WINDOW_START)) + for command in commands: + collector.on_event(ToolEndEvent(task_id="t1", turn_id="s1", tool=command)) + collector.on_event( + AgentEndEvent( + task_id="t1", + status=AgentEndStatus.COMPLETED, + messages=list(state.messages), + usage=TokenUsage(), + timestamp=self.WINDOW_END, + ) + ) + published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] + assert len(published) == 1 + return published[0] def test_tool_time_inside_the_step_is_subtracted(self, monkeypatch): - # A 500ms tool squarely inside the 1000ms step. message = self._finish_step( monkeypatch, [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], ) span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 - assert span_ms == pytest.approx(1000.0) + assert span_ms == pytest.approx(1000.0), "the reducer still publishes the whole window as its bounds" assert message.generation_duration_ms == pytest.approx(500.0) def test_a_step_with_no_tools_keeps_its_whole_window(self, monkeypatch): - message = self._finish_step(monkeypatch, []) - assert message.generation_duration_ms == pytest.approx(1000.0) + assert self._finish_step(monkeypatch, []).generation_duration_ms == pytest.approx(1000.0) def test_concurrent_tools_are_subtracted_once(self, monkeypatch): - # Two overlapping 500ms tools occupy 600ms of wall clock, not 1000ms. - # Summing them would leave 0 generation for a step that generated 400. + # Two overlapping 500ms tools occupy 600ms, not 1000ms. Summing them + # would leave 0 generation for a step that generated 400. message = self._finish_step( monkeypatch, [ @@ -1850,34 +1881,76 @@ def test_concurrent_tools_are_subtracted_once(self, monkeypatch): assert message.generation_duration_ms == pytest.approx(400.0) def test_the_window_never_goes_negative(self, monkeypatch): - # A tool whose recorded interval straddles the step is clipped to it. message = self._finish_step( monkeypatch, [(self.WINDOW_START - timedelta(seconds=30), self.WINDOW_END + timedelta(seconds=30))], ) assert message.generation_duration_ms == 0.0 - def test_a_tool_still_open_at_the_boundary_is_subtracted(self, monkeypatch): - # The windows tile from the previous step's finish, so a call that - # opens inside this step and closes inside the NEXT one straddles the - # boundary. Counting only closed intervals published the pre-boundary - # 400ms as generation while the call's own duration_ms counted it - # again — the exact double-count `busy_ms` exists to prevent. - message = self._finish_step( - monkeypatch, - [], - open_starts=[self.WINDOW_START + timedelta(milliseconds=600)], - ) - assert message.generation_duration_ms == pytest.approx(600.0) + def test_a_tool_still_open_at_the_boundary_contributes_no_span(self, monkeypatch): + """The behaviour that CHANGED with the move, stated rather than implied. + + The reducer used to bound a still-open call at the window's end and + subtract that slice. The collector cannot: a call with no + `execution_completed_at` was never timed. Its time is subtracted when it + RESOLVES, from whichever windows its real interval overlaps. + """ + message = self._finish_step(monkeypatch, [], open_starts=[self.WINDOW_START + timedelta(milliseconds=600)]) + assert message.generation_duration_ms == pytest.approx(1000.0) - def test_an_open_tool_overlapping_a_closed_one_is_counted_once(self, monkeypatch): - # Union, not sum, across the closed and still-open sets alike. + def test_a_resolved_tool_overlapping_an_unresolved_one_counts_only_the_resolved(self, monkeypatch): message = self._finish_step( monkeypatch, [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], open_starts=[self.WINDOW_START + timedelta(milliseconds=500)], ) - assert message.generation_duration_ms == pytest.approx(200.0) + assert message.generation_duration_ms == pytest.approx(500.0) + + def test_a_mark_later_than_the_step_start_does_not_invert_the_window(self, monkeypatch): + """The backwards-clock defence, pinned at the reducer, not in isolation. + + `close_window`'s `min()` only fires if the reducer actually passes the + step's own start as `item_start`. Drop that argument and the window + opens at the (later) mark instead, so the span shrinks — or inverts and + clamps to 0.0, publishing a fabricated instant generation. Nothing else + in this file fails when it is dropped, which is the whole reason it is + here: the mark is what the reducer still owns after the tool + subtraction moved to the collector. + """ + state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="do it", model="m") + state.step_started_at = self.WINDOW_START + # A mark 400ms AFTER this step began: the CLI's step_finish for the + # previous step landed late, or the clock stepped. + state.gen_mark = self.WINDOW_START + timedelta(milliseconds=400) + + class _Clock(datetime): + @staticmethod + def now(tz=None): + return TestGenerationWindowExcludesToolExecution.WINDOW_END + + monkeypatch.setattr(agent_module, "datetime", _Clock) + state.on_step_finish({"reason": "stop", "tokens": {"input": 100, "output": 20}}) + + message = next(m for m in state.messages if m.role == "assistant") + assert message.started_at == self.WINDOW_START + assert message.generation_duration_ms == pytest.approx(1000.0) + + def test_the_published_window_reconciles_to_its_own_bounds(self, monkeypatch): + """The collector subtracted exactly the spans the record carries. + + `scripts/timing/decompose_run.py` and the evalboard's Unaccounted cell + both recompute the tool UNION from the recorded command spans and + subtract it from the recorded window bounds. This asserts the published + record is internally consistent under that recomputation, so a span + silently added or dropped on the way in shows up here. + """ + from coder_eval.timing import busy_ms + + closed = [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))] + message = self._finish_step(monkeypatch, closed) + span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 + expected = span_ms - busy_ms(closed, message.started_at, message.completed_at) + assert message.generation_duration_ms == pytest.approx(expected) class TestGenerationWindowsTileTheTurn: @@ -1905,7 +1978,6 @@ def now(tz=None): return now state.step_started_at = step_start - state.step_tool_spans = [] monkeypatch.setattr(agent_module, "datetime", _Clock) state.on_step_finish({"reason": "stop", "tokens": {"input": 100, "output": 20}}) @@ -1944,3 +2016,181 @@ def test_the_steps_leave_no_gap_between_them(self, monkeypatch): covered = (second.completed_at - first.started_at).total_seconds() * 1000.0 gen = sum(m.generation_duration_ms or 0.0 for m in (first, second)) assert gen == pytest.approx(covered) + + +_SPAN_EPOCH_MS = 1_800_000_000_000 +_SPAN_BASE = datetime.fromtimestamp(_SPAN_EPOCH_MS / 1000) + + +class _SteppedClock(datetime): + """A clock the test moves by hand, in ms from `_SPAN_BASE`. + + Subclasses `datetime` rather than stubbing it, because `_epoch_ms_to_dt` + calls `datetime.fromtimestamp` through the same module global and must keep + resolving to the real implementation — the CLI's epoch stamps and the + reducer's own `now()` reads have to land on ONE timeline for the span + arithmetic under test to mean anything. + """ + + at_ms = 0.0 + + @staticmethod + def now(tz=None): + return _SPAN_BASE + timedelta(milliseconds=_SteppedClock.at_ms) + + +class TestToolSpansSurviveTheStepBoundary: + """A tool that closes BETWEEN two steps still belongs to the next window. + + This used to be a bookkeeping problem: a per-step span list, cleared at + `step_start` — after the window it feeds had already opened at `gen_mark` — + so a call closing in the gap had its span wiped before the next + `step_finish` could subtract it. That list is gone. + `EventCollector.subtract_tool_time` sees every span at once and clips each + to the windows it overlaps, so the property now holds by construction + rather than by a reset rule. Kept, and re-pointed at the collector, because + the property is what matters: a future reducer change could still break it + by moving a mark or failing to emit the ToolEnd the collector reduces. + + It needs the NON-TERMINAL tool path to reach: the CLI normally emits one + already-`completed` event per call, which closes inside the step that + opened it. That is why the measured corpus reads 0.00% and a reproduction + has to drive the state object. + """ + + def _run(self, monkeypatch): + monkeypatch.setattr(agent_module, "datetime", _SteppedClock) + state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="go", model="m") + # The resolved telemetry leaves the state via ToolEnd; the identity + # case below reconciles against what was RECORDED, not against the + # clock the test scripted. + resolved: list[Any] = [] + state.bind(lambda e: resolved.append(e.tool) if isinstance(e, ToolEndEvent) else None) + + def tool(status, *, end_ms=None): + times = {"start": _SPAN_EPOCH_MS + 100} + if end_ms is not None: + times["end"] = _SPAN_EPOCH_MS + end_ms + state.on_tool_use({"callID": "c1", "tool": "bash", "state": {"status": status, "time": times}}) + + _SteppedClock.at_ms = 0 + state.on_step_start({"messageID": "m1"}) + _SteppedClock.at_ms = 100 + tool("running") # non-terminal: stays open across the boundary + _SteppedClock.at_ms = 1000 + state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) + _SteppedClock.at_ms = 1500 + tool("completed", end_ms=1500) # closes in the GAP between the steps + _SteppedClock.at_ms = 1600 + state.on_step_start({"messageID": "m2"}) + _SteppedClock.at_ms = 2000 + state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) + + # Published through the real collector: the reducer hands over raw + # windows, and the tool subtraction happens once, there. + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t1", prompt="go", iteration=1, timestamp=_SPAN_BASE)) + for command in resolved: + collector.on_event(ToolEndEvent(task_id="t1", turn_id="s1", tool=command)) + collector.on_event( + AgentEndEvent( + task_id="t1", + status=AgentEndStatus.COMPLETED, + messages=list(state.messages), + usage=TokenUsage(), + timestamp=_SPAN_BASE + timedelta(milliseconds=2000), + ) + ) + published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] + return resolved, published + + def test_the_gap_slice_of_a_straddling_call_is_not_published_as_generation(self, monkeypatch): + _, messages = self._run(monkeypatch) + assert len(messages) == 2 + # Window 2 tiles 1000 -> 2000. c1 ran for 1000 -> 1500 of it, so 500ms + # is model time. Before the reset moved, this published 1000.0 — a 100% + # overstatement, with c1's own duration_ms counting the same 500ms. + assert messages[1].generation_duration_ms == pytest.approx(500.0) + + def test_the_call_is_subtracted_from_exactly_one_window(self, monkeypatch): + # Window 1 owns c1's 100 -> 1000 slice (it was open at that boundary + # and bounded there); window 2 owns 1000 -> 1500. Neither owns both. + _, messages = self._run(monkeypatch) + assert messages[0].generation_duration_ms == pytest.approx(100.0) + assert messages[1].generation_duration_ms == pytest.approx(500.0) + + def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self, monkeypatch): + """generation + UNION(tool) accounts for the whole span, to the ms. + + The assertion the golden corpus CANNOT make: `_scrub.py` masks + `generation_duration_ms` and both bounds to a placeholder, so a + snapshot records that a window was measured and never what it + measured, and its identity check is an upper bound besides — so + under-accounting, the defect this phase fixes, passes it silently. + """ + from coder_eval.timing import busy_ms + + resolved, messages = self._run(monkeypatch) + lo, hi = messages[0].started_at, messages[1].completed_at + generation_ms = sum(m.generation_duration_ms or 0.0 for m in messages) + command = next(c for c in resolved if c.tool_id == "c1") + tool_ms = busy_ms([(command.execution_started_at, command.execution_completed_at)], lo, hi) + + assert generation_ms + tool_ms == pytest.approx((hi - lo).total_seconds() * 1000.0) + + def test_a_duplicate_step_finish_does_not_republish_the_previous_window(self, monkeypatch): + """A spent `step_started_at` must not seed the next window. + + `close_window`'s `min(mark, item_start)` pulls the window open to cover + the item's own start. That is the backwards-clock defence — which this + reducer genuinely needs, since its stamps are raw `datetime.now()` and + not on a `TurnClock`. But a start stamp left in place after its step was + published is not a backwards clock: it is a stale value BEFORE the mark, + so the guard reopens the next window at the previous step's start and + publishes that whole span again. Reproduced on Pi's identical twin + before the fix: 3000 ms of generation for a 2000 ms turn. + """ + monkeypatch.setattr(agent_module, "datetime", _SteppedClock) + state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="go", model="m") + _SteppedClock.at_ms = 0 + state.on_step_start({"messageID": "m1"}) + _SteppedClock.at_ms = 1000 + state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) + _SteppedClock.at_ms = 2000 + state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) + + messages = [m for m in state.messages if m.role == "assistant"] + assert len(messages) == 2 + assert messages[1].started_at == messages[0].completed_at + assert sum(m.generation_duration_ms or 0.0 for m in messages) == pytest.approx(2000.0) + + def test_a_step_that_never_finishes_does_not_advance_the_mark(self, monkeypatch): + """The half of this that is still the reducer's job. + + There is no span list to preserve any more — the collector reduces the + ToolEnd stream itself. What the reducer still owns is the MARK: a step + that published nothing must not advance it, or its time is handed to + whichever step finishes next. + """ + monkeypatch.setattr(agent_module, "datetime", _SteppedClock) + state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="go", model="m") + _SteppedClock.at_ms = 0 + state.on_step_start({"messageID": "m1"}) + _SteppedClock.at_ms = 1000 + state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) + mark_after_flush = state.gen_mark + + _SteppedClock.at_ms = 1600 + state.on_step_start({"messageID": "m2"}) + _SteppedClock.at_ms = 1700 + state.on_tool_use( + { + "callID": "c2", + "tool": "bash", + "state": {"status": "running", "time": {"start": _SPAN_EPOCH_MS + 1700}}, + } + ) + _SteppedClock.at_ms = 1900 + state.close_open_tools() # crash/timeout orphan sweep — no message appended + + assert state.gen_mark == mark_after_flush diff --git a/tests/test_pi_agent.py b/tests/test_pi_agent.py index dcc1c3f7..f9e6f5dd 100644 --- a/tests/test_pi_agent.py +++ b/tests/test_pi_agent.py @@ -23,11 +23,11 @@ import pytest -from coder_eval.agents import pi_agent as agent_module from coder_eval.agents.pi_agent import PiAgent, _PiTurnState, _result_text from coder_eval.errors import AgentCrashError, TurnTimeoutError -from coder_eval.models import AgentKind, AssistantMessage, CommandTelemetry, PiAgentConfig +from coder_eval.models import AgentKind, AssistantMessage, CommandTelemetry, PiAgentConfig, TokenUsage from coder_eval.pricing import calculate_cost +from coder_eval.streaming.collector import EventCollector from coder_eval.streaming.events import ( AgentEndEvent, AgentEndStatus, @@ -39,6 +39,7 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.timing import TurnClock from tests._fixtures.golden_streams.pi_fixtures import ( EXPECTED_CACHE_READ, EXPECTED_COST, @@ -1093,64 +1094,98 @@ async def test_zero_reported_cost_on_an_unpriced_model_stays_zero(self, patch_ex assert record.token_usage.total_cost_usd == 0.0 -class TestGenerationWindowExcludesToolExecution: - """A tool running inside a turn is not model time. +class _FixedClock: + """A `TurnClock` stand-in frozen at one instant, injected into the state.""" + + def __init__(self, at: datetime) -> None: + self.at = at - Pi marks the window at `turn_start` and closes it at `turn_end`, and - every tool call executes INSIDE it while also publishing its own - measured `duration_ms`. Publishing the raw span as generation time - counted the same milliseconds twice, which the task page's Unaccounted - cell renders as a ~-100% residual. + def now(self) -> datetime: + return self.at - Driven at the reducer: the window is two `datetime.now()` reads and the - tool interval comes from the event payload, so only setting both - explicitly makes the arithmetic deterministic. + +class TestGenerationWindowExcludesToolExecution: + """A tool running inside a turn is not model time — asserted where it is now DECIDED. + + The reducer no longer subtracts anything. It publishes the RAW window, and + `EventCollector.subtract_tool_time` takes the tool union back out of it + once, for all five harnesses. So these cases drive the reducer and then a + real collector, and assert the PUBLISHED number — the one that reaches + `task.json` — rather than an intermediate the reducer used to own. + + They are not duplicates of + `tests/test_event_collector.py::TestSubtractToolTime`: those pin the + arithmetic, these pin that THIS reducer hands the collector a window and a + span set the arithmetic can be right about. """ WINDOW_START = datetime(2026, 1, 1, 12, 0, 0) WINDOW_END = datetime(2026, 1, 1, 12, 0, 1) # a 1000ms turn - def _finish_turn(self, monkeypatch, spans, open_starts=()): - class _Clock(datetime): - @staticmethod - def now(tz=None): - return TestGenerationWindowExcludesToolExecution.WINDOW_END - - state = _PiTurnState(task_id="t", iteration=1, user_input="x", model="m") + def _finish_turn(self, spans, open_starts=()): + """Drive the reducer, then publish through a real collector. + + `spans` are RESOLVED calls (both bounds); `open_starts` are calls that + never returned. An unresolved call now contributes NO span — it has no + `execution_completed_at`, and inventing one is what `None` exists to + prevent — where the reducer used to bound it at the window's end. That + is a real change and a better one: the collector sees every span at + once, so a call straddling a boundary is clipped to each window it + actually overlapped instead of approximated at the boundary. + """ + state = _PiTurnState(task_id="t", iteration=1, user_input="x", model="m", clock=_FixedClock(self.WINDOW_END)) state.turn_started_at = self.WINDOW_START - state.turn_tool_spans = list(spans) - for i, started in enumerate(open_starts): - state.open_tools[f"open-{i}"] = CommandTelemetry( + commands = [ + CommandTelemetry( tool_name="bash", - tool_id=f"open-{i}", + tool_id=f"closed-{i}", timestamp=started, execution_started_at=started, + execution_completed_at=completed, + result_status="success", ) - monkeypatch.setattr(agent_module, "datetime", _Clock) + for i, (started, completed) in enumerate(spans) + ] + commands += [ + CommandTelemetry(tool_name="bash", tool_id=f"open-{i}", timestamp=s, execution_started_at=s) + for i, s in enumerate(open_starts) + ] state.on_turn_end( {"message": {"role": "assistant", "usage": {"input": 100, "output": 20}, "stopReason": "stop"}} ) - assistant = [m for m in state.messages if m.role == "assistant"] - assert len(assistant) == 1 - return assistant[0] - def test_tool_time_inside_the_turn_is_subtracted(self, monkeypatch): + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="x", iteration=1, timestamp=self.WINDOW_START)) + for command in commands: + collector.on_event(ToolEndEvent(task_id="t", turn_id="t1", tool=command)) + collector.on_event( + AgentEndEvent( + task_id="t", + status=AgentEndStatus.COMPLETED, + messages=list(state.messages), + usage=TokenUsage(), + timestamp=self.WINDOW_END, + ) + ) + published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] + assert len(published) == 1 + return published[0] + + def test_tool_time_inside_the_turn_is_subtracted(self): message = self._finish_turn( - monkeypatch, [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], ) span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 - assert span_ms == pytest.approx(1000.0) + assert span_ms == pytest.approx(1000.0), "the reducer still publishes the whole window as its bounds" assert message.generation_duration_ms == pytest.approx(500.0) - def test_a_turn_with_no_tools_keeps_its_whole_window(self, monkeypatch): - assert self._finish_turn(monkeypatch, []).generation_duration_ms == pytest.approx(1000.0) + def test_a_turn_with_no_tools_keeps_its_whole_window(self): + assert self._finish_turn([]).generation_duration_ms == pytest.approx(1000.0) - def test_concurrent_tools_are_subtracted_once(self, monkeypatch): + def test_concurrent_tools_are_subtracted_once(self): # Two overlapping 500ms tools occupy 600ms, not 1000ms. Summing them # would leave 0 generation for a turn that generated 400. message = self._finish_turn( - monkeypatch, [ (self.WINDOW_START + timedelta(milliseconds=100), self.WINDOW_START + timedelta(milliseconds=600)), (self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700)), @@ -1158,30 +1193,290 @@ def test_concurrent_tools_are_subtracted_once(self, monkeypatch): ) assert message.generation_duration_ms == pytest.approx(400.0) - def test_the_window_never_goes_negative(self, monkeypatch): + def test_the_window_never_goes_negative(self): message = self._finish_turn( - monkeypatch, [(self.WINDOW_START - timedelta(seconds=30), self.WINDOW_END + timedelta(seconds=30))], ) assert message.generation_duration_ms == 0.0 - def test_a_tool_still_open_at_the_boundary_is_subtracted(self, monkeypatch): - # A call that opens inside this turn and closes inside the NEXT one - # straddles the boundary. Counting only closed intervals published the - # pre-boundary 400ms as generation while the call's own duration_ms - # counted it again. - message = self._finish_turn( - monkeypatch, - [], - open_starts=[self.WINDOW_START + timedelta(milliseconds=600)], - ) - assert message.generation_duration_ms == pytest.approx(600.0) + def test_a_tool_still_open_at_the_boundary_contributes_no_span(self): + """The behaviour that CHANGED with the move, stated rather than implied. - def test_an_open_tool_overlapping_a_closed_one_is_counted_once(self, monkeypatch): - # Union, not sum, across the closed and still-open sets alike. + The reducer used to bound a still-open call at the window's end and + subtract that slice. The collector cannot: a call with no + `execution_completed_at` was never timed. Its time is subtracted when it + RESOLVES, from whichever windows its real interval overlaps. + """ + message = self._finish_turn([], open_starts=[self.WINDOW_START + timedelta(milliseconds=600)]) + assert message.generation_duration_ms == pytest.approx(1000.0) + + def test_a_resolved_tool_overlapping_an_unresolved_one_counts_only_the_resolved(self): message = self._finish_turn( - monkeypatch, [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], open_starts=[self.WINDOW_START + timedelta(milliseconds=500)], ) - assert message.generation_duration_ms == pytest.approx(200.0) + assert message.generation_duration_ms == pytest.approx(500.0) + + def test_the_published_window_reconciles_to_its_own_bounds(self): + """The collector subtracted exactly the spans the record carries. + + `scripts/timing/decompose_run.py` and the evalboard's Unaccounted cell + both recompute the tool UNION from the recorded command spans and + subtract it from the recorded window bounds. This asserts the published + record is internally consistent under that recomputation, so a span + silently added or dropped on the way in shows up here. + """ + from coder_eval.timing import busy_ms + + closed = [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))] + message = self._finish_turn(closed) + span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 + expected = span_ms - busy_ms(closed, message.started_at, message.completed_at) + assert message.generation_duration_ms == pytest.approx(expected) + + +_SPAN_BASE = datetime(2026, 3, 1, 9, 0, 0) + + +class _SteppedClock: + """A `TurnClock` stand-in the test moves by hand, in ms from `_SPAN_BASE`. + + INJECTED, never monkeypatched onto the module. Pi derives every wall stamp + from its turn clock now, so patching `agent_module.datetime` would no + longer reach it: the tests would quietly start measuring the real clock and + pass by accident instead of failing. Injection also puts the "one clock per + turn" lifetime in the constructor signature where it can be read. + """ + + def __init__(self, at_ms: float = 0.0) -> None: + self.at_ms = at_ms + + def now(self) -> datetime: + return _SPAN_BASE + timedelta(milliseconds=self.at_ms) + + +def _turn_end_payload(): + return {"message": {"role": "assistant", "usage": {"input": 10, "output": 5}, "stopReason": "stop"}} + + +class TestGenerationWindowsTileTheTurn: + """Each window runs from the PREVIOUS `turn_end`, not from its own `turn_start`. + + Pi was the only harness measuring from its own turn start, so the wall + clock between one `turn_end` and the next `turn_start` — the model time + that PRODUCED the next turn — fell into no bucket at all. The four-bucket + identity is asserted only as an upper bound, so nothing failed. + + The gap is small in practice (measured across 25 real window pairs: median + 0.25 ms, max 0.75 ms). The value here is that it closes, and that the tool + spans keep working once it does — see TestToolSpansSurviveTheTurnBoundary, + which is the half that carries the weight. + """ + + def _two_turns(self): + clock = _SteppedClock() + state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) + state.on_turn_start() + clock.at_ms = 1000 + state.on_turn_end(_turn_end_payload()) + clock.at_ms = 1600 + state.on_turn_start() + clock.at_ms = 2000 + state.on_turn_end(_turn_end_payload()) + return [m for m in state.messages if m.role == "assistant"] + + def test_the_second_window_abuts_the_first(self): + messages = self._two_turns() + assert len(messages) == 2 + assert messages[1].started_at == messages[0].completed_at + + def test_the_inter_turn_gap_is_inside_a_window_rather_than_unaccounted(self): + messages = self._two_turns() + # 1000 -> 2000, which includes the 600ms between `turn_end` and the + # next `turn_start`. Untiled this reported 400ms and lost the 600. + assert messages[1].generation_duration_ms == pytest.approx(1000.0) + + +class TestToolSpansSurviveTheTurnBoundary: + """A tool that closes BETWEEN two turns still belongs to the next window. + + This used to be a bookkeeping problem: a per-turn span list, cleared at + `turn_start` — after the window it feeds had already opened at the mark — + so a call closing in the gap had its span wiped before the flush could + subtract it. That list is gone. `EventCollector.subtract_tool_time` sees + every span at once and clips each to the windows it overlaps, so the + property now holds by construction rather than by a reset rule. + + Kept, and re-pointed at the collector, because the property itself is what + matters and a future reducer change could still break it — by moving a + mark, or by failing to emit the ToolEnd the collector reduces. + """ + + def _run(self): + clock = _SteppedClock() + state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) + # The resolved telemetry leaves the state via ToolEnd; the identity + # case below reconciles against what was RECORDED, not against the + # clock the test scripted. + resolved: list[Any] = [] + state.bind(lambda e: resolved.append(e.tool) if isinstance(e, ToolEndEvent) else None) + state.on_turn_start() + clock.at_ms = 100 + state.on_tool_execution_start({"toolCallId": "c1", "toolName": "bash", "args": {}}) + clock.at_ms = 1000 + state.on_turn_end(_turn_end_payload()) + clock.at_ms = 1500 + state.on_tool_execution_end({"toolCallId": "c1", "result": "ok"}) # closes in the GAP + clock.at_ms = 1600 + state.on_turn_start() + clock.at_ms = 2000 + state.on_turn_end(_turn_end_payload()) + + # Published through the real collector: the reducer hands over raw + # windows, and the tool subtraction happens once, there. + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=_SPAN_BASE)) + for command in resolved: + collector.on_event(ToolEndEvent(task_id="t", turn_id="t1", tool=command)) + collector.on_event( + AgentEndEvent( + task_id="t", + status=AgentEndStatus.COMPLETED, + messages=list(state.messages), + usage=TokenUsage(), + timestamp=_SPAN_BASE + timedelta(milliseconds=2000), + ) + ) + published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] + return resolved, published + + def test_the_gap_slice_of_a_straddling_call_is_not_published_as_generation(self): + _, messages = self._run() + # Window 2 tiles 1000 -> 2000. c1 ran for 1000 -> 1500 of it, so 500ms + # is model time. With the reset left at `turn_start` this reads 1000.0. + assert messages[1].generation_duration_ms == pytest.approx(500.0) + + def test_the_call_is_subtracted_from_exactly_one_window(self): + _, messages = self._run() + # Window 1 bounded c1 at its own close (100 -> 1000); window 2 takes + # only the remainder. + assert messages[0].generation_duration_ms == pytest.approx(100.0) + assert messages[1].generation_duration_ms == pytest.approx(500.0) + + def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self): + """generation + UNION(tool) accounts for the whole span, to the ms. + + This is the assertion the golden corpus CANNOT make: `_scrub.py` masks + `generation_duration_ms` and both bounds to a placeholder, so a + snapshot records that a window was measured and never what it measured. + Its identity check (`_scrub.py`) is an upper bound besides, so + under-accounting — the defect this phase fixes — passes it silently. + `scripts/timing/decompose_run.py --max-residual-pct` is the two-sided + check on live runs; this is the committed one. + """ + from coder_eval.timing import busy_ms + + resolved, messages = self._run() + lo, hi = messages[0].started_at, messages[1].completed_at + generation_ms = sum(m.generation_duration_ms or 0.0 for m in messages) + command = next(c for c in resolved if c.tool_id == "c1") + tool_ms = busy_ms([(command.execution_started_at, command.execution_completed_at)], lo, hi) + + assert generation_ms + tool_ms == pytest.approx((hi - lo).total_seconds() * 1000.0) + + def test_a_duplicate_turn_end_does_not_republish_the_previous_window(self): + """A spent `turn_started_at` must not seed the next window. + + `close_window`'s `min(mark, item_start)` pulls the window open to cover + the item's own start. That is the backwards-clock defence, but a start + stamp left in place after its turn was published is not a backwards + clock — it is a stale value BEFORE the mark, so the guard reopens the + next window at the previous turn's start and publishes that whole span + again. Reproduced before the fix: 3000 ms of generation for a 2000 ms + turn. This reducer promises to survive a malformed stream, and Pi's CLI + retries internally, so a duplicate or replayed `turn_end` is a transport + hiccup rather than a hypothetical. + """ + clock = _SteppedClock() + state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) + state.on_turn_start() + clock.at_ms = 1000 + state.on_turn_end(_turn_end_payload()) + clock.at_ms = 2000 + state.on_turn_end(_turn_end_payload()) # no intervening `turn_start` + + messages = [m for m in state.messages if m.role == "assistant"] + assert len(messages) == 2 + assert messages[1].started_at == messages[0].completed_at + assert sum(m.generation_duration_ms or 0.0 for m in messages) == pytest.approx(2000.0) + + def test_a_turn_that_never_finishes_does_not_advance_the_mark(self): + """The half of this that is still the reducer's job. + + There is no span list to preserve any more — the collector reduces the + ToolEnd stream itself. What the reducer still owns is the MARK: a turn + that published nothing must not advance it, or its time is handed to + whichever turn finishes next. + """ + clock = _SteppedClock() + state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) + state.on_turn_start() + clock.at_ms = 1000 + state.on_turn_end(_turn_end_payload()) + mark_after_flush = state.gen_mark + + clock.at_ms = 1600 + state.on_turn_start() + clock.at_ms = 1700 + state.on_tool_execution_start({"toolCallId": "c2", "toolName": "bash", "args": {}}) + clock.at_ms = 1900 + state.close_open_tools() # crash/timeout orphan sweep — no message appended + + assert state.gen_mark == mark_after_flush + + +class TestClockIsFreshPerTurn: + """A retried turn must not inherit the crashed turn's clock. + + `TurnClock` anchors once and derives every later stamp from that anchor, so + one surviving a retry would stamp the new turn against the old turn's wall + origin — and over a long run accumulate drift against real wall time. The + lifetime is structural (the clock is built with the turn state, and the + state is built per `communicate()`), which is exactly the kind of property + that stays true only while someone is checking. + """ + + async def test_a_turn_after_a_crash_is_anchored_to_a_fresh_clock(self, patch_exec, tmp_path): + agent = _agent() + patch_exec(_FakeProcess([], returncode=1, stderr=b"boom: bad model")) + with pytest.raises(AgentCrashError): + await _run(agent, tmp_path) + crashed_clock = agent # the state is gone; only the agent survives a crash + + patch_exec(_FakeProcess(HAPPY_STREAM)) + record = await crashed_clock.communicate("try again") + + # The recovered turn measured a real window of its own, rather than one + # anchored before the crash — which a stale clock would have produced + # as an inflated first generation. + windows = [m for m in record.messages if m.role == "assistant" and m.generation_duration_ms is not None] + assert windows + for message in windows: + assert message.completed_at >= message.started_at + assert message.generation_duration_ms < 60_000, "a window spanning the crashed turn means a stale clock" + + async def test_the_agent_retains_no_clock_between_turns(self, patch_exec, tmp_path): + """Nothing to reset, because nothing survives — the structural half. + + The clock is reachable only through the turn state, and the turn state + is a local of `communicate()`. If either were ever hoisted onto the + agent (a plausible refactor — several other fields are), the next turn + would silently inherit the previous turn's anchor and no assertion + about a single turn's numbers would notice. + """ + agent = _agent() + patch_exec(_FakeProcess(HAPPY_STREAM)) + await _run(agent, tmp_path) + + leaked = [name for name, value in vars(agent).items() if isinstance(value, _PiTurnState | TurnClock)] + assert not leaked, f"a turn's clock outlived its turn via {leaked}" diff --git a/tests/test_reports_html.py b/tests/test_reports_html.py index b468d8a0..fcc2ade0 100644 --- a/tests/test_reports_html.py +++ b/tests/test_reports_html.py @@ -2,13 +2,14 @@ from __future__ import annotations -from datetime import datetime +from datetime import datetime, timedelta from pathlib import Path import pytest from coder_eval.models import ( AgentKind, + AssistantMessage, CommandStatistics, CommandTelemetry, CriterionResult, @@ -1337,3 +1338,253 @@ def test_no_surface_publishes_a_fabricated_zero(self): html = HTMLReportGenerator.generate_experiment_html(self._all_ungraded(["v1"]), None) assert "n/a" in html assert "0.0%" not in html + + +class TestGenerationMetricsBuckets: + """The offline report carries the same four buckets as the evalboard. + + `reports_html` is described in CLAUDE.md as the evalboard's static twin, and + it rendered only Total Latency / Turns / Avg Turn Latency — so anyone + reading the artifact rather than the dashboard got none of the wall-clock + accounting. The arithmetic lives in `reports_stats.turn_time_buckets`; this + asserts the rendering AND, through it, that arithmetic. + """ + + BASE = datetime(2026, 1, 1, 12, 0, 0) + + @classmethod + def _at(cls, ms: float) -> datetime: + return cls.BASE + timedelta(milliseconds=ms) + + @staticmethod + def _stat(html: str, label: str) -> str: + """The rendered VALUE of one stat card, by its label.""" + import re + + match = re.search(rf'
{re.escape(label)}[^<]*
\s*
([^<]*)
', html) + assert match is not None, f"no stat card labelled {label!r}" + return match.group(1) + + @classmethod + def _turn( + cls, + *, + startup: float | None, + teardown: float | None, + generations: list[tuple[float, float, float | None]], + tools: tuple[float, float] | None = None, + sub_agent: tuple[float, float, float] | None = None, + ) -> TurnRecord: + messages: list = [ + AssistantMessage(started_at=cls._at(lo), completed_at=cls._at(hi), generation_duration_ms=gen) + for lo, hi, gen in generations + ] + commands: list[CommandTelemetry] = [] + if tools is not None: + lo, hi = tools + commands.append( + CommandTelemetry( + tool_name="Bash", + tool_id="main-1", + timestamp=cls._at(lo), + execution_started_at=cls._at(lo), + execution_completed_at=cls._at(hi), + result_status="success", + ) + ) + if sub_agent is not None: + lo, hi, gen = sub_agent + messages.append( + AssistantMessage( + started_at=cls._at(lo), + completed_at=cls._at(hi), + generation_duration_ms=gen, + parent_tool_use_id="agent-call", + tool_use_ids=["child-1"], + ) + ) + commands.append( + CommandTelemetry( + tool_name="Bash", + tool_id="child-1", + timestamp=cls._at(lo), + execution_started_at=cls._at(lo), + execution_completed_at=cls._at(hi), + result_status="success", + ) + ) + return TurnRecord( + iteration=1, + user_input="go", + agent_output="done", + commands=commands, + messages=messages, + harness_startup_ms=startup, + harness_teardown_ms=teardown, + ) + + def test_each_bucket_is_summed_across_turns(self): + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result( + iterations=[ + self._turn(startup=500.0, teardown=100.0, generations=[(500, 1500, 800.0)], tools=(600, 800)), + self._turn(startup=300.0, teardown=50.0, generations=[(2000, 3000, 1000.0)], tools=(2100, 2400)), + ] + ) + buckets = turn_time_buckets(result) + assert buckets.startup_ms == pytest.approx(800.0) + assert buckets.teardown_ms == pytest.approx(150.0) + assert buckets.generation_ms == pytest.approx(1800.0) + assert buckets.tool_ms == pytest.approx(500.0), "200ms + 300ms, each turn's own union" + + def test_each_label_renders_its_own_value(self): + """Pins the label-to-value WIRING, not just that five cards exist. + + Asserting presence alone would pass if `Startup` rendered + `buckets.teardown_ms` — and the unmeasured-bucket test below compares + two `None`s, so a swap is invisible there too. These are five distinct + numbers precisely so a mix-up cannot hide. + """ + result = _make_result( + iterations=[self._turn(startup=500.0, teardown=100.0, generations=[(500, 1500, 800.0)], tools=(600, 800))] + ) + html = HTMLReportGenerator().generate_task_html(result) + assert self._stat(html, "Startup") == "500ms" + assert self._stat(html, "Generation") == "800ms" + assert self._stat(html, "Tool exec") == "200ms" + assert self._stat(html, "Teardown") == "100ms" + # 90s task minus 1.6s of measured buckets. + assert self._stat(html, "Unaccounted") == "88.40s" + + def test_an_unmeasured_bucket_renders_a_dash_not_zero(self): + """A run recorded before the head/tail existed measured nothing. + + `0ms` would claim a measurement nobody took — the same distinction + CE058 enforces in `src/`, and the reason the evalboard's `sumMeasured` + returns null. + """ + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result(iterations=[self._turn(startup=None, teardown=None, generations=[(500, 1500, 800.0)])]) + buckets = turn_time_buckets(result) + assert buckets.startup_ms is None + assert buckets.teardown_ms is None + + html = HTMLReportGenerator().generate_task_html(result) + assert self._stat(html, "Startup") == "—" + assert self._stat(html, "Teardown") == "—" + + def test_a_measured_zero_still_renders_as_zero(self): + """The control for the dash: `None` and `0.0` must stay distinguishable. + + Asserted through the RENDERER, not just the arithmetic — the dash is a + rendering decision, so its counterexample has to be one too. + """ + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result(iterations=[self._turn(startup=0.0, teardown=0.0, generations=[(500, 1500, 800.0)])]) + assert turn_time_buckets(result).startup_ms == 0.0 + + html = HTMLReportGenerator().generate_task_html(result) + assert self._stat(html, "Startup") == "0ms" + assert self._stat(html, "Teardown") == "0ms" + + def test_an_unmeasured_bucket_still_counts_as_zero_in_the_residual(self): + """Display and arithmetic differ on purpose. + + A bucket nobody measured shows as a dash but sums as 0.0, so the + missing time surfaces in Unaccounted rather than vanishing. That is the + rule `scripts/timing/decompose_run.py::_turn_buckets` already applies. + """ + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result(iterations=[self._turn(startup=None, teardown=None, generations=[(500, 1500, 800.0)])]) + # 90s task, 800ms of generation, nothing else measured. + assert turn_time_buckets(result).unaccounted_ms == pytest.approx(90_000.0 - 800.0) + + def test_a_negative_residual_is_rendered_signed_not_clamped(self): + """Real, and it means generation and tool execution OVERLAPPED. + + The fixture has to PRODUCE the overlap rather than manufacture the sign + some other way, or it tests the formatter and not the condition the + message names: a 60 s generation and a 60 s tool inside a 90 s task sum + past the task's own wall clock, which is what overlapping looks like in + the buckets. + """ + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result( + iterations=[self._turn(startup=0.0, teardown=0.0, generations=[(0, 60_000, 60_000.0)], tools=(0, 60_000))] + ) + buckets = turn_time_buckets(result) + assert buckets.generation_ms == pytest.approx(60_000.0) + assert buckets.tool_ms == pytest.approx(60_000.0), "the two overlap in wall clock" + assert buckets.unaccounted_ms is not None and buckets.unaccounted_ms < 0 + + html = HTMLReportGenerator().generate_task_html(result) + assert self._stat(html, "Unaccounted").startswith("-"), "a negative residual must keep its sign" + + def test_sub_agent_generations_and_their_tools_are_excluded(self): + """The same main-thread filter the collector and the evalboard apply. + + The spawning Agent call's own interval already spans the child's run, + so counting either books it twice. + """ + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result( + iterations=[ + self._turn( + startup=500.0, + teardown=100.0, + generations=[(500, 1500, 800.0)], + tools=(600, 800), + sub_agent=(3000, 3400, 400.0), + ) + ] + ) + buckets = turn_time_buckets(result) + assert buckets.generation_ms == pytest.approx(800.0), "the child's 400ms is not main-thread generation" + assert buckets.tool_ms == pytest.approx(200.0), "and its tool is not a main-thread span" + + def test_the_existing_four_stats_are_unchanged(self): + result = _make_result(iterations=[self._turn(startup=500.0, teardown=100.0, generations=[(500, 1500, 800.0)])]) + html = HTMLReportGenerator().generate_task_html(result) + for label in ("Total Latency", "Turns", "Assistant Turns", "Avg Turn Latency"): + assert self._stat(html, label), f"missing or empty {label} stat" + + def test_a_turn_that_recorded_no_tool_span_has_no_tool_total(self): + """`0ms` would claim the tools were measured and took no time. + + A turn that ran tools none of which were timed is indistinguishable + from one that ran none, so the presence of a SPAN decides — the same + None-vs-0 distinction CE058 enforces in `src/`. + """ + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result(iterations=[self._turn(startup=500.0, teardown=100.0, generations=[(500, 1500, 800.0)])]) + assert turn_time_buckets(result).tool_ms is None + assert self._stat(HTMLReportGenerator().generate_task_html(result), "Tool exec") == "—" + + def test_a_run_with_no_duration_has_no_residual(self): + """Subtracting real buckets from an untimed run fabricates a negative. + + `duration_seconds` defaults to 0.0 rather than None, so the guard has to + be on the value. The evalboard keeps its own residual null for exactly + this case. + """ + from coder_eval.reports_stats import turn_time_buckets + + result = _make_result(iterations=[self._turn(startup=500.0, teardown=100.0, generations=[(500, 1500, 800.0)])]) + result.duration_seconds = 0.0 + assert turn_time_buckets(result).unaccounted_ms is None + assert self._stat(HTMLReportGenerator().generate_task_html(result), "Unaccounted") == "—" + + def test_a_run_with_no_turns_does_not_raise(self): + from coder_eval.reports_stats import turn_time_buckets + + buckets = turn_time_buckets(_make_result(iterations=[])) + assert buckets.generation_ms is None + assert buckets.tool_ms is None + HTMLReportGenerator().generate_task_html(_make_result(iterations=[])) diff --git a/tests/test_timing_close_window.py b/tests/test_timing_close_window.py new file mode 100644 index 00000000..31dc5dc2 --- /dev/null +++ b/tests/test_timing_close_window.py @@ -0,0 +1,382 @@ +"""`close_window` — the one generation-window arithmetic the tiling reducers share. + +Each of codex, opencode and pi had to get these cases right independently while +the body was copy-pasted; this file proves them once, against the helper. It is +the sensor for the arithmetic ITSELF, as distinct from the per-reducer tests, +which pin that a given reducer feeds it the right bounds and spans. +""" + +import importlib.util +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + +from coder_eval.timing import busy_ms, close_window, decompose_turn, union_ms + + +MARK = datetime(2026, 9, 11, 12, 0, 0) + + +def _at(ms: int) -> datetime: + return MARK + timedelta(milliseconds=ms) + + +def _load_decompose_run(): + """Import `scripts/timing/decompose_run.py`, which is not an importable package.""" + path = Path(__file__).parents[1] / "scripts" / "timing" / "decompose_run.py" + spec = importlib.util.spec_from_file_location("decompose_run_for_test", path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class TestCloseWindow: + """The RAW window: where it opens, where it ends, and the clamp. + + The tool subtraction these cases used to cover moved to + `streaming/collector.py::subtract_tool_time`, where it happens once for all + five harnesses instead of five times in five reducers — see + `tests/test_event_collector.py::TestSubtractToolTime`, which carries the + union, grouping, clamping and non-mutation cases. What is left here is the + part that is genuinely per-reducer: the mark. + """ + + def test_the_window_is_the_whole_span_from_the_mark(self): + started, span_ms = close_window(mark=MARK, now=_at(1000)) + assert started == MARK + assert span_ms == pytest.approx(1000.0) + + def test_item_start_before_the_mark_wins(self): + # A stamp that went backwards: the window must cover the item, so the + # min() moves the start back rather than inverting the span. + started, span_ms = close_window(mark=MARK, now=_at(1000), item_start=_at(-200)) + assert started == _at(-200) + assert span_ms == pytest.approx(1200.0) + + def test_item_start_after_the_mark_keeps_the_mark(self): + # The normal tiling case: the gap between the previous close and this + # item's first stamp IS model time and belongs inside the window. + started, span_ms = close_window(mark=MARK, now=_at(1000), item_start=_at(400)) + assert started == MARK + assert span_ms == pytest.approx(1000.0) + + def test_an_inverted_window_clamps_to_zero_rather_than_going_negative(self): + started, span_ms = close_window(mark=_at(1000), now=MARK) + assert started == _at(1000) + assert span_ms == 0.0 + + def test_mark_is_keyword_only_and_has_no_default(self): + # A reducer cannot open a window without STATING what it tiles from. + # The value is still the caller's to get right — see the docstring. + with pytest.raises(TypeError): + close_window(MARK, _at(1000)) # type: ignore[misc] + with pytest.raises(TypeError): + close_window(now=_at(1000)) # type: ignore[call-arg] + + def test_it_no_longer_accepts_the_span_arguments_that_moved(self): + """The subtraction moved; the parameters must not linger as no-ops. + + A reducer still passing `closed_spans=` would otherwise keep compiling + while its tool time was silently subtracted a second time centrally. + """ + with pytest.raises(TypeError): + close_window(mark=MARK, now=_at(1000), closed_spans=[]) # type: ignore[call-arg] + with pytest.raises(TypeError): + close_window(mark=MARK, now=_at(1000), open_started_ats=[]) # type: ignore[call-arg] + + +class TestNaiveAwareMix: + """A mixed naive/aware pair fails loudly at the seam, not cryptically inside it. + + `decompose_turn`'s bare arithmetic raised `TypeError: can't subtract + offset-naive and offset-aware datetimes` straight out of + `EventCollector.build_turn_record`, killing the turn with a message naming + neither the field nor the harness. + + Unreachable from this repo — every stamp in `agents/` and `streaming/` is a + naive `datetime.now()`. The exposure is a THIRD-PARTY agent registered + through the `coder_eval.plugins` SPI, which lives outside + `src/coder_eval/agents/` and which a lint rule scoped to that directory + could never see. That is why this is a guard and not a rule. + """ + + AWARE = MARK.replace(tzinfo=UTC) + + def test_a_mixed_head_names_the_field(self): + with pytest.raises(TypeError, match="harness_startup_ms"): + decompose_turn(self.AWARE, None, MARK, None) + + def test_a_mixed_tail_names_the_field(self): + with pytest.raises(TypeError, match="harness_teardown_ms"): + decompose_turn(None, self.AWARE, None, MARK) + + def test_a_mixed_busy_ms_window_names_the_field(self): + with pytest.raises(TypeError, match="busy_ms window"): + busy_ms([(MARK, _at(500))], MARK, self.AWARE) + + def test_a_mixed_span_start_is_caught_too_and_not_by_the_bare_comparison(self): + """The clipping compares each span against the window. + + Left unguarded that raises "can't compare offset-naive and offset-aware + datetimes" — the exact message this replaces — so checking only the + bounds would leave the guard not covering its own function. + """ + with pytest.raises(TypeError, match="tool span's start"): + busy_ms([(self.AWARE, self.AWARE)], MARK, _at(1000)) + + def test_a_mixed_span_end_is_caught_by_its_own_branch(self): + """The end is a separate check against `hi`, so it needs its own case. + + A span whose START matches the window and whose END does not passes the + previous branch and must still raise — otherwise that branch is live, + reachable and unexercised. + """ + with pytest.raises(TypeError, match="tool span's end"): + busy_ms([(MARK, self.AWARE)], MARK, _at(1000)) + + def test_one_wording_for_every_call_site(self): + """One helper, so one template — checked across ALL FIVE call sites. + + Two inline guards would drift, and a test asserting the text would then + pin only whichever one it happened to call. What varies between sites + is deliberate and only that: the field name, and which side is aware. + Everything after that clause is the advice, and it must be identical or + the sites are no longer sharing a helper. + """ + advice = set() + for call in ( + lambda: decompose_turn(self.AWARE, None, MARK, None), # head + lambda: decompose_turn(None, self.AWARE, None, MARK), # tail + lambda: busy_ms([(MARK, _at(500))], MARK, self.AWARE), # window bounds + lambda: busy_ms([(self.AWARE, self.AWARE)], MARK, _at(1000)), # span start + lambda: busy_ms([(MARK, self.AWARE)], MARK, _at(1000)), # span end + ): + with pytest.raises(TypeError) as excinfo: + call() + message = str(excinfo.value) + assert "is timezone-aware and the other is naive" in message + advice.add(message.split("), ", 1)[1]) + assert len(advice) == 1, advice + + def test_all_naive_is_unchanged(self): + assert decompose_turn(_at(1000), _at(2000), MARK, _at(3000)) == (1000.0, 1000.0) + + def test_all_aware_works_because_the_guard_is_about_the_mix(self): + def aware(ms: int) -> datetime: + return _at(ms).replace(tzinfo=UTC) + + assert decompose_turn(aware(1000), aware(2000), self.AWARE, aware(3000)) == (1000.0, 1000.0) + + def test_an_empty_span_list_is_not_checked_at_all(self): + """Not even the bounds, and the MIXED case is the one that proves it. + + With no spans the comprehension never runs: nothing is compared and + nothing is subtracted, so there is no pair for the guard to be about. + Checking the bounds anyway rejected a call that has always returned + `0.0` — asserted here on mixed bounds, because the naive case would + pass either way and so could not tell the two behaviours apart. + """ + assert busy_ms([], MARK, _at(1000)) == 0.0 + assert busy_ms([], MARK, self.AWARE) == 0.0 + + +class TestUnionMs: + """`union_ms` is `busy_ms` over the spans' own extent. + + Extracted because the golden sensor (`tests/_fixtures/golden_streams/_scrub.py`) + and the live residual gate (`scripts/timing/decompose_run.py`) had copied + that same `min`/`max`/`busy_ms` tail. Both answer the same question about + the same recorded commands, so the two copies could only ever agree by + hand — `test_the_two_recorded_command_readers_agree` below is the half + that pins them together. + """ + + def test_no_spans_is_zero_not_a_min_of_an_empty_sequence(self): + assert union_ms([]) == 0.0 + + def test_overlapping_spans_are_the_union_not_the_sum(self): + # Two 500 ms calls overlapping by 400 ms occupy 600 ms of wall clock. + assert union_ms([(_at(100), _at(600)), (_at(200), _at(700))]) == pytest.approx(600.0) + + def test_disjoint_spans_add(self): + assert union_ms([(_at(100), _at(200)), (_at(400), _at(900))]) == pytest.approx(600.0) + + def test_the_extent_is_the_spans_own_bounds(self): + # No window is passed, so nothing clips: a span far from the origin is + # measured in full rather than dropped as out of range. + assert union_ms([(_at(10_000), _at(10_250))]) == pytest.approx(250.0) + + def test_the_two_recorded_command_readers_agree(self): + """`_scrub.py` and `decompose_run.py` must report one tool total. + + They read the SAME `task.json` shape — the golden sensor from a dumped + record, the gate from the file on disk — and a divergence would let one + pass while the other failed on identical bytes. They keep their own + stamp parsing (the inputs differ in how they are reached); the union + tail is what this pins. + """ + from tests._fixtures.golden_streams._scrub import _tool_union_ms + + # Loaded by path: `scripts/` is deliberately not a package (it sits + # outside the Makefile's LINT_PATHS), so there is no import to make. + _tool_ms = _load_decompose_run()._tool_ms + + turn = { + "commands": [ + {"execution_started_at": _at(100).isoformat(), "execution_completed_at": _at(600).isoformat()}, + {"execution_started_at": _at(200).isoformat(), "execution_completed_at": _at(700).isoformat()}, + # Never timed: contributes nothing on either side. + {"execution_started_at": None, "execution_completed_at": None}, + # Inverted bounds: both readers drop these while BUILDING their + # span list, which is why `union_ms` does not filter them. + {"execution_started_at": _at(900).isoformat(), "execution_completed_at": _at(800).isoformat()}, + ] + } + assert _tool_union_ms(turn) == pytest.approx(600.0) + assert _tool_ms(turn) == _tool_union_ms(turn) + + +class TestTurnClock: + """One (wall, monotonic) pair per turn, every later stamp derived from it.""" + + def test_successive_reads_never_go_backwards(self): + from coder_eval.timing import TurnClock + + clock = TurnClock() + stamps = [clock.now() for _ in range(50)] + assert stamps == sorted(stamps) + + def test_a_derived_stamp_advances_by_the_monotonic_delta(self): + import time as _time + + from coder_eval.timing import TurnClock + + clock = TurnClock() + before = clock.now() + mono_before = _time.monotonic() + while _time.monotonic() - mono_before < 0.01: + pass + elapsed_ms = (_time.monotonic() - mono_before) * 1000.0 + derived_ms = (clock.now() - before).total_seconds() * 1000.0 + assert derived_ms == pytest.approx(elapsed_ms, abs=5.0) + + def test_a_fresh_clock_anchors_on_its_own_pair(self): + """Each clock holds its OWN (wall, monotonic) origin — the per-turn part. + + Note what this deliberately does NOT assert: that two clocks report + different times. They should AGREE, and closely, because both derive + from the same monotonic source — re-anchoring exists to correct drift + against real wall time, not to introduce an offset. An earlier version + of this test asserted `second.now() != first.now()`; that passed only + on sub-microsecond skew between the two constructors' reads, so it was + flaky under load and asserted the opposite of the design. + """ + import time as _time + + from coder_eval.timing import TurnClock + + first = TurnClock() + mono = _time.monotonic() + while _time.monotonic() - mono < 0.005: + pass + second = TurnClock() + + assert second._mono0 > first._mono0 + assert second._wall0 >= first._wall0 + + +class TestTheThreeToolUnionsAgree: + """Three implementations recompute the turn's tool union. They must agree. + + * `EventCollector._main_thread_tool_spans` — what the harness subtracts + from the generation windows and measures the head and tail against. + * `tests/_fixtures/golden_streams/_scrub.py::_tool_union_ms` — the golden + corpus's identity check. + * `scripts/timing/decompose_run.py::_tool_ms` — the LIVE two-sided residual + gate, which `.github/workflows/pr-checks.yml` runs against a real run. + + They agreed by luck once and it cost a defect: the collector filtered its + GENERATIONS to the main thread and then passed EVERY command as a tool + span. A child nests inside the parent Agent call, whose interval the union + already covers, so nothing failed — but Codex's recovered child tools carry + the CHILD's clock, so the nesting is not guaranteed. When the collector + started filtering, the other two did not, and a gate computing a different + tool total than the harness reports a residual that is an artifact of the + disagreement rather than a bucket error. That is the worst possible place + for a divergence, because this is the only live sensor for the identity. + """ + + @staticmethod + def _record() -> dict: + """A turn with a sub-agent whose own tool sits OUTSIDE the parent call. + + Inside, the three agree whatever they filter, so the fixture has to put + the child's tool where the parent's interval does not cover it. + """ + return { + "duration_seconds": 3.0, + "commands": [ + { + "tool_id": "agent-call", + "execution_started_at": _at(1000).isoformat(), + "execution_completed_at": _at(1500).isoformat(), + }, + { + "tool_id": "child-tool", + "execution_started_at": _at(2000).isoformat(), + "execution_completed_at": _at(2400).isoformat(), + }, + ], + "messages": [ + {"role": "assistant", "parent_tool_use_id": None, "tool_use_ids": ["agent-call"]}, + {"role": "assistant", "parent_tool_use_id": "agent-call", "tool_use_ids": ["child-tool"]}, + ], + } + + def test_the_two_recomputing_readers_exclude_the_sub_agent_tool(self): + from tests._fixtures.golden_streams._scrub import _tool_union_ms + + tool_ms = _load_decompose_run()._tool_ms + record = self._record() + # Only the parent Agent call's own 500 ms. Counting the child's 400 ms + # books time no main-thread bucket claims. + assert _tool_union_ms(record) == pytest.approx(500.0) + assert tool_ms(record) == pytest.approx(500.0) + + def test_the_collector_excludes_it_too(self): + from coder_eval.models import AssistantMessage, CommandTelemetry + from coder_eval.streaming.collector import EventCollector + from coder_eval.streaming.events import ToolEndEvent + + collector = EventCollector() + for tool_id, lo, hi in (("agent-call", 1000, 1500), ("child-tool", 2000, 2400)): + collector.on_event( + ToolEndEvent( + task_id="t", + turn_id="t1", + tool=CommandTelemetry( + tool_name="Agent", + tool_id=tool_id, + timestamp=_at(lo), + execution_started_at=_at(lo), + execution_completed_at=_at(hi), + result_status="success", + ), + ) + ) + messages = [ + AssistantMessage( + started_at=_at(0), completed_at=_at(1000), generation_duration_ms=1000.0, tool_use_ids=["agent-call"] + ), + AssistantMessage( + started_at=_at(2000), + completed_at=_at(2400), + generation_duration_ms=400.0, + parent_tool_use_id="agent-call", + tool_use_ids=["child-tool"], + ), + ] + spans = collector._main_thread_tool_spans(messages) + assert union_ms(spans) == pytest.approx(500.0), "the same 500 ms the other two report" diff --git a/tests/test_timing_identity_contract.py b/tests/test_timing_identity_contract.py new file mode 100644 index 00000000..027a44cf --- /dev/null +++ b/tests/test_timing_identity_contract.py @@ -0,0 +1,574 @@ +"""The four-bucket identity, to the millisecond, on every harness. + + head + Σ generation + UNION(tool) + tail == the turn's own span + +This is the committed MAGNITUDE sensor, and it exists because nothing else in +the suite is one: + +* the golden corpus masks ``generation_duration_ms``, both window bounds, both + ``execution_*_at`` stamps and both head/tail fields to a placeholder + (``_scrub.py::SCRUB_KEYS``), so a snapshot records that a window was measured + and never what it measured — a timing value can move by seconds with every + golden test still green; +* ``_scrub.py::assert_timing_captured``'s own identity check is ONE-SIDED + (``overshoot <= ...``), so an UNDERCOUNT — a bucket claiming less time than + it should, which is the defect class this whole area keeps producing — passes + it silently. It cannot be made two-sided either: the replays run in ~0.3 ms of + synthetic wall clock, where a relative bound is vacuous; +* ``scripts/timing/decompose_run.py --max-residual-pct`` IS two-sided, but needs + live ``task.json`` files. + +Magnitudes are only real where a scripted clock makes them real, so each case +drives the harness's own REDUCER with a clock it moves by hand, then feeds the +messages and commands it produced through a real ``EventCollector`` — the same +seam production uses to compute the head and the tail. Every number asserted is +therefore one the harness computed, against a span the test declared. + +Three clock-injection styles are needed, and all three already exist in the +per-harness suites (this module reuses their idiom rather than inventing a +fourth): + +* an injected ``TurnClock`` — pi and antigravity take ``clock=`` / build one + through a patched ``TurnClock`` factory; +* a ``datetime`` SUBCLASS monkeypatched onto the module — opencode, which also + calls ``datetime.fromtimestamp`` through the same global (see + ``tests/test_opencode_agent.py``'s ``_SteppedClock`` for why a stub breaks); +* ``time.monotonic`` AND ``datetime`` both patched — claude-code. Its window is + wall-derived now, but ``turn_start_time`` and the turn deadline still read + ``time.monotonic()``, so patching only one leaves the reducer straddling a + real clock and a scripted one. + +Codex is the fifth and takes its stamps from SDK epoch milliseconds rather than +from any host clock, so its case scripts those stamps directly. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timedelta +from types import SimpleNamespace +from typing import Any + +import pytest + +from coder_eval.models import ( + AgentKind, + AssistantMessage, + CommandTelemetry, + TokenUsage, + TranscriptMessage, + TurnRecord, + parse_agent_config, +) +from coder_eval.streaming.callbacks import CompositeStreamCallback +from coder_eval.streaming.collector import EventCollector, main_thread_tool_spans +from coder_eval.streaming.events import ( + AgentEndEvent, + AgentEndStatus, + AgentStartEvent, + ToolEndEvent, + ToolEndStatus, +) +from coder_eval.timing import union_ms + + +# The two CLI harnesses (opencode, codex) report their stamps as epoch +# milliseconds and convert them with the real ``datetime.fromtimestamp``. So +# the shared origin is DERIVED from an epoch value rather than written as a +# wall time: that is what puts a scripted ``now()`` read and a converted CLI +# stamp on ONE timeline, without patching the conversion itself. +EPOCH_MS = 1_800_000_000_000 +BASE = datetime.fromtimestamp(EPOCH_MS / 1000.0) + + +def at(ms: float) -> datetime: + return BASE + timedelta(milliseconds=ms) + + +@dataclass(frozen=True) +class Turn: + """What one harness case produces: a scripted span plus what it recorded. + + ``started_ms`` / ``ended_ms`` are the turn's own bounds — where the + ``AgentStartEvent`` and ``AgentEndEvent`` land. Everything else came out of + the reducer. + """ + + started_ms: float + ended_ms: float + messages: list[TranscriptMessage] + commands: list[CommandTelemetry] + + +def _record(turn: Turn) -> TurnRecord: + """Reduce a scripted turn through the production collector seam. + + Deliberately the real ``EventCollector`` rather than a direct + ``decompose_turn`` call: the head and the tail are only as correct as the + arguments that seam builds for them, and those (the main-thread generation + filter, the command span set) are half of what this file is asserting. + """ + collector = EventCollector() + collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=at(turn.started_ms))) + for command in turn.commands: + collector.on_event(ToolEndEvent(task_id="t", turn_id="turn", tool=command, status=ToolEndStatus.OK)) + collector.on_event( + AgentEndEvent( + task_id="t", + status=AgentEndStatus.COMPLETED, + iteration=1, + user_input="go", + messages=turn.messages, + usage=TokenUsage(), + duration_seconds=(turn.ended_ms - turn.started_ms) / 1000.0, + timestamp=at(turn.ended_ms), + ) + ) + return collector.build_turn_record() + + +def assert_identity_closes(turn: Turn) -> None: + """head + Σ generation + UNION(tool) + tail == the scripted span, EXACTLY. + + ``pytest.approx`` rather than an order-of-magnitude bound: every input is + scripted, so the only slack is float representation. A bound wide enough to + absorb a real defect is the sensor this module exists to replace. + + MAIN THREAD ONLY on both sides, and both through production's own helpers: + a sub-agent's generations bubble into the same stream, and the spawning + Agent call's own interval already spans them and their tools. + """ + record = _record(turn) + span_ms = turn.ended_ms - turn.started_ms + + generation_ms = sum( + m.generation_duration_ms or 0.0 + for m in record.messages + if isinstance(m, AssistantMessage) and m.parent_tool_use_id is None + ) + # The PRODUCTION selector, not a re-derivation of it. Unioning every command + # would assert a different identity than the collector computes: production, + # the golden sensor, the live residual gate and the HTML report all exclude + # a sub-agent's own tools (the spawning Agent call's interval already spans + # them). No case here has a child command yet, so a local copy stayed green + # while quietly testing something else — and the first sub-agent case added + # would have reported a false regression. + tool_ms = union_ms(main_thread_tool_spans(record.messages, record.commands)) + assert record.harness_startup_ms is not None, "a turn that generated has a measured head" + assert record.harness_teardown_ms is not None, "a turn that generated has a measured tail" + bucket_sum = record.harness_startup_ms + generation_ms + tool_ms + record.harness_teardown_ms + + assert bucket_sum == pytest.approx(span_ms), ( + f"the four buckets sum to {bucket_sum:.4f} ms against a {span_ms:.4f} ms turn " + f"(off by {bucket_sum - span_ms:+.4f} ms): head={record.harness_startup_ms:.4f}, " + f"generation={generation_ms:.4f}, tool_union={tool_ms:.4f}, tail={record.harness_teardown_ms:.4f}. " + "They tile the turn, so a sum UNDER it means some interval is booked nowhere — the " + "defect class the golden corpus cannot see — and a sum OVER it means one is booked twice." + ) + + +# -------------------------------------------------------------------------- +# pi — an injected TurnClock +# -------------------------------------------------------------------------- + + +class _InjectedClock: + """A ``TurnClock`` stand-in the test moves by hand, in ms from ``BASE``. + + Injected rather than monkeypatched: pi and antigravity derive every wall + stamp from their per-turn clock, so patching the module's ``datetime`` + would no longer reach them and the case would quietly measure the real + clock and pass by accident. + """ + + def __init__(self, at_ms: float = 0.0) -> None: + self.at_ms = at_ms + + def now(self) -> datetime: + return at(self.at_ms) + + +def _pi_turn(*, untile: bool = False) -> Turn: + """Two tiled windows around a tool, with a real head and a real tail. + + The tool closes INSIDE the first window rather than across the boundary — + that case is pinned by ``tests/test_pi_agent.py``. What this adds is the + two ends: pi's first window opens at its first ``turn_start``, so the CLI + boot before it is head, and the turn runs on past the last ``turn_end``. + + ``untile`` reproduces the defect pi actually shipped with — each window + measured from its OWN ``turn_start`` instead of from the previous flush — + so that the sensor can be shown to catch it. See + ``test_the_sensor_sees_a_window_that_stops_tiling``. + """ + from coder_eval.agents.pi_agent import _PiTurnState + + payload = {"message": {"role": "assistant", "usage": {"input": 10, "output": 5}, "stopReason": "stop"}} + clock = _InjectedClock() + state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) + commands: list[CommandTelemetry] = [] + state.bind(lambda e: commands.append(e.tool) if isinstance(e, ToolEndEvent) else None) + + clock.at_ms = 500 # CLI boot: head + state.on_turn_start() + clock.at_ms = 700 + state.on_tool_execution_start({"toolCallId": "c1", "toolName": "bash", "args": {}}) + clock.at_ms = 1200 + state.on_tool_execution_end({"toolCallId": "c1", "result": "ok"}) + clock.at_ms = 2000 + state.on_turn_end(payload) + clock.at_ms = 2600 # the inter-turn gap, which window 2 tiles back over + state.on_turn_start() + if untile: + state.gen_mark = None + clock.at_ms = 3000 + state.on_turn_end(payload) + + return Turn( + started_ms=0.0, + ended_ms=3500.0, # process teardown after the last turn: tail + messages=list(state.messages), + commands=commands, + ) + + +# -------------------------------------------------------------------------- +# opencode — a datetime subclass on the module +# -------------------------------------------------------------------------- + + +class _SteppedDatetime(datetime): + """A clock the test moves by hand, in ms from ``BASE``. + + Subclasses ``datetime`` rather than stubbing it, because the reducer also + calls ``datetime.fromtimestamp`` through the same module global to convert + the CLI's epoch stamps, and that must keep resolving to the real + implementation — the CLI's stamps and the reducer's own ``now()`` reads + have to land on ONE timeline for the arithmetic to mean anything. + """ + + at_ms = 0.0 + + @staticmethod + def now(tz: Any = None) -> datetime: # type: ignore[override] + return at(_SteppedDatetime.at_ms) + + +def _opencode_turn(monkeypatch: pytest.MonkeyPatch) -> Turn: + """The same two-window shape, driven through OpenCode's step stream.""" + from coder_eval.agents import opencode_agent as opencode_module + from coder_eval.agents.opencode_agent import _OpenCodeTurnState + + monkeypatch.setattr(opencode_module, "datetime", _SteppedDatetime) + state = _OpenCodeTurnState(task_id="t", iteration=1, user_input="go", model="m") + commands: list[CommandTelemetry] = [] + state.bind(lambda e: commands.append(e.tool) if isinstance(e, ToolEndEvent) else None) + finish = {"reason": "stop", "tokens": {"input": 10, "output": 5}} + + _SteppedDatetime.at_ms = 500 # Node boot: head + state.on_step_start({"messageID": "m1"}) + _SteppedDatetime.at_ms = 1200 + state.on_tool_use( + { + "callID": "c1", + "tool": "bash", + "state": {"status": "completed", "time": {"start": EPOCH_MS + 700, "end": EPOCH_MS + 1200}}, + } + ) + _SteppedDatetime.at_ms = 2000 + state.on_step_finish(finish) + _SteppedDatetime.at_ms = 2600 + state.on_step_start({"messageID": "m2"}) + _SteppedDatetime.at_ms = 3000 + state.on_step_finish(finish) + + return Turn(started_ms=0.0, ended_ms=3500.0, messages=list(state.messages), commands=commands) + + +# -------------------------------------------------------------------------- +# antigravity — an injected TurnClock, at the reducer +# -------------------------------------------------------------------------- + + +def _antigravity_turn() -> Turn: + """One interleaved window per generation, with a tool inside the first. + + Driven at ``_AntigravityTurnState`` rather than through ``communicate()``: + the fake conversation yields with no delay, so an end-to-end run cannot + distinguish a window that opened at the turn's start from one that opened + later. The state's own ``_gen_mark_wall`` is stamped at construction, so + constructing it AFTER the scripted agent start is what gives this harness a + measurable head at all. + """ + from coder_eval.agents.antigravity_agent import AntigravityAgent, _AntigravityTurnState + from tests._fixtures.golden_streams.antigravity_fixtures import _step, _tc, _usage + + agent = AntigravityAgent(parse_agent_config(type=AgentKind.ANTIGRAVITY, model="gemini-3.5-flash")) + collector = EventCollector() + clock = _InjectedClock(at_ms=500) # dispatch before the first Step: head + state = _AntigravityTurnState( + agent=agent, + emit=CompositeStreamCallback([collector]), + task_id="t", + turn_id="turn", + collector=collector, + user_input="go", + iteration=1, + model="gemini-3.5-flash", + turn_start_time=0.0, + clock=clock, + ) + + clock.at_ms = 700 + state.process_step( + _step( + "TOOL_CALL", + "ACTIVE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "c1", {"command_line": "ls"})], + ) + ) + clock.at_ms = 1200 + state.process_step( + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "c1", {"command_line": "ls", "exit_code": 0})], + ) + ) + clock.at_ms = 2000 + state.process_step(_step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5))) + clock.at_ms = 3000 + state.process_step(_step("TEXT_RESPONSE", "DONE", content="done", complete=True, usage=_usage(200, 0, 10, 0))) + + return Turn(started_ms=0.0, ended_ms=3500.0, messages=list(state.messages), commands=list(state.commands)) + + +# -------------------------------------------------------------------------- +# codex — scripted SDK epoch-millisecond stamps +# -------------------------------------------------------------------------- + + +def _codex_turn() -> Turn: + """Two tiled windows, the first SPLIT across two sub-messages. + + Codex is the only harness that cuts one window into several messages + (thinking and action, apportioned by output-token share), and they share + one pair of bounds. The identity has to close over the GROUP, so this case + drives that split deliberately rather than the simpler one-message shape. + + Its stamps are the SDK's own epoch milliseconds, unreachable from any host + clock, so ``_flush_message`` is driven with them set by hand — the idiom + ``tests/test_codex_agent.py::TestFlushMessageWindowBounds`` already uses. + """ + from coder_eval.agents.codex_agent import CodexAgent, _CodexTurnState, _ms_to_dt + from coder_eval.models import ContentBlock + + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) + collector = EventCollector() + state = _CodexTurnState( + agent, + emit=CompositeStreamCallback([collector]), + task_id="t", + turn_id="turn", + collector=collector, + commands=[], + messages=[], + user_input="go", + iteration=1, + turn_start_time=0.0, + ) + command = CommandTelemetry( + tool_name="bash", + tool_id="c1", + timestamp=_ms_to_dt(EPOCH_MS + 700), + execution_started_at=_ms_to_dt(EPOCH_MS + 700), + execution_completed_at=_ms_to_dt(EPOCH_MS + 1200), + duration_ms=500.0, + result_status="success", + ) + state.commands.append(command) + + # Window 1 — no mark yet, so it opens at its own first item (+500): the CLI + # boot before that is head. Thinking + text, so the flush cuts two + # sub-messages sharing the window. + state.open_blocks = [ + ContentBlock(block_type="thinking", sequence=0, thinking="plan"), + ContentBlock(block_type="text", sequence=0, text="answer"), + ] + state.open_start_ms = EPOCH_MS + 500 + state.open_end_ms = EPOCH_MS + 2000 + state._flush_message( + SimpleNamespace(input_tokens=500, cached_input_tokens=0, output_tokens=100, reasoning_output_tokens=80) + ) + + # Window 2 — tiles back from the mark (+2000), covering the gap before its + # own first item at +2600. + state.open_blocks = [ContentBlock(block_type="text", sequence=0, text="more")] + state.open_start_ms = EPOCH_MS + 2600 + state.open_end_ms = EPOCH_MS + 3000 + state._flush_message(SimpleNamespace(input_tokens=10, cached_input_tokens=0, output_tokens=5)) + + return Turn(started_ms=0.0, ended_ms=3500.0, messages=list(state.messages), commands=[command]) + + +# -------------------------------------------------------------------------- +# claude-code — BOTH the monotonic and the wall clock patched +# -------------------------------------------------------------------------- + + +def _claude_turn(monkeypatch: pytest.MonkeyPatch) -> Turn: + """A tool call between two emissions, with a real head and a real tail. + + Both module globals are patched off one counter. The window itself is + wall-derived, but ``turn_start_time`` and the deadline still read + ``time.monotonic()``, so patching only one leaves the reducer straddling a + real clock and a scripted one. + + The first `message_start` re-seeds the window, so the CLI spawn and the + query build before it are head rather than msg0's generation. That a LATER + one must not re-seed is asserted directly in + `tests/test_agent_telemetry.py`; here it shows up as the windows still + tiling. + + Note where its windows do NOT tile: the tool result resets both marks, so + the interval between the emission that ISSUED the call and the result is + left outside every window. That gap is the tool's own execution, which is + exactly what the tool bucket claims — which is why the identity still + closes to the millisecond. + """ + from coder_eval.agents import claude_code_agent as claude_module + from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeTurnState + from coder_eval.streaming.events import AgentEndStatus as _AgentEndStatus + from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage as SdkAssistantMessage + from tests._fixtures.golden_streams.claude_fixtures import ToolUseBlock, UserMessage, message_start + + class _Stepped(datetime): + at_ms = 0.0 + + @staticmethod + def now(tz: Any = None) -> datetime: # type: ignore[override] + return at(_Stepped.at_ms) + + def _monotonic() -> float: + return _Stepped.at_ms / 1000.0 + + monkeypatch.setattr(claude_module, "datetime", _Stepped) + monkeypatch.setattr(claude_module, "time", SimpleNamespace(monotonic=_monotonic)) + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + collector = EventCollector() + commands: list[CommandTelemetry] = [] + + _Stepped.at_ms = 500 # the turn state is built here; the head runs past it + state = _ClaudeTurnState( + agent, + emit=CompositeStreamCallback( + [ + collector, + SimpleNamespace(on_event=lambda e: commands.append(e.tool) if isinstance(e, ToolEndEvent) else None), + ] + ), + collector=collector, + task_id="t", + user_input="go", + iteration=1, + max_turns=None, + log=agent._log, + turn_start_time=_monotonic(), + deadline=None, + ) + + # The stream really does put `message_start` before the emission it + # announces — the recorded corpus shows it and the SDK guarantees it — and + # the FIRST one is what re-seeds the window, so an ordering this case got + # wrong would silently stop exercising the re-seed at all. + _Stepped.at_ms = 800 + state.on_stream_event(message_start("m1")) + _Stepped.at_ms = 1000 + state.on_assistant_message( + SdkAssistantMessage( + [ToolUseBlock("c1", "Bash", {"command": "ls"})], + usage={"input_tokens": 10, "output_tokens": 5}, + message_id="m1", + ) + ) + _Stepped.at_ms = 1800 # the tool ran for the whole gap + state.on_user_message(UserMessage("c1", False, "ok")) + _Stepped.at_ms = 2000 + state.on_stream_event(message_start("m2")) # does NOT re-seed: once per turn + _Stepped.at_ms = 2500 + state.on_assistant_message(SdkAssistantMessage([], usage={"input_tokens": 10, "output_tokens": 5}, message_id="m2")) + state.finalize(_AgentEndStatus.COMPLETED) + + return Turn(started_ms=0.0, ended_ms=3000.0, messages=list(state.sdk_messages), commands=commands) + + +# -------------------------------------------------------------------------- +# The contract +# -------------------------------------------------------------------------- + + +def test_pi_buckets_tile_the_turn(): + assert_identity_closes(_pi_turn()) + + +def test_opencode_buckets_tile_the_turn(monkeypatch: pytest.MonkeyPatch): + assert_identity_closes(_opencode_turn(monkeypatch)) + + +def test_antigravity_buckets_tile_the_turn(): + assert_identity_closes(_antigravity_turn()) + + +def test_codex_buckets_tile_the_turn(): + assert_identity_closes(_codex_turn()) + + +def test_claude_code_buckets_tile_the_turn(monkeypatch: pytest.MonkeyPatch): + assert_identity_closes(_claude_turn(monkeypatch)) + + +def test_every_built_in_harness_has_a_case(): + """A sensor that silently covers four of five is worse than one naming the gap. + + Derived from ``AgentKind`` rather than from a hand-written list, so a sixth + built-in harness fails here instead of shipping unmeasured. A harness whose + reducer genuinely cannot be driven without a live process belongs in an + exemption set carrying its reason — not in a weaker end-to-end assertion. + + From the ENUM and not from ``AgentRegistry``, which is open: a third-party + plugin agent registers there too, and an out-of-tree harness is not this + repo's to cover (``coder_eval_uipath``'s delegate-sdk is the live example). + """ + covered = {name for name in globals() if name.startswith("test_") and name.endswith("_buckets_tile_the_turn")} + # NONE is the agentless task double — no reducer, no generation window at + # all; UNKNOWN is a load-failure placeholder that never runs. + built_in = set(AgentKind) - {AgentKind.NONE, AgentKind.UNKNOWN} + missing = {kind for kind in built_in if f"test_{kind.value.replace('-', '_')}_buckets_tile_the_turn" not in covered} + assert not missing, f"no ms-exact identity case for {sorted(m.value for m in missing)}" + + +def test_the_sensor_sees_a_window_that_stops_tiling(): + """The gating mutation check, as a committed test rather than an attestation. + + A window seeded from its own turn start instead of from the previous + flush's close is the defect pi shipped with, and the whole point of this + module is that the SUITE notices it rather than a reviewer reproducing it + by hand. The golden corpus cannot: it masks every value involved. + + Asserted on the MAGNITUDE as well as on the failure, because "it raised" + would also pass if the mutation broke the case in some unrelated way. The + 600 ms is the scripted gap between one ``turn_end`` and the next + ``turn_start`` — real model time, which untiling books to nothing. + """ + healthy = _pi_turn() + mutated = _pi_turn(untile=True) + + def _generation_ms(turn: Turn) -> float: + return sum(m.generation_duration_ms or 0.0 for m in turn.messages if isinstance(m, AssistantMessage)) + + assert _generation_ms(healthy) - _generation_ms(mutated) == pytest.approx(600.0) + with pytest.raises(AssertionError, match="booked nowhere"): + assert_identity_closes(mutated) diff --git a/tests/test_timing_union_parity.py b/tests/test_timing_union_parity.py index 1fac0c97..7579a39b 100644 --- a/tests/test_timing_union_parity.py +++ b/tests/test_timing_union_parity.py @@ -1,7 +1,7 @@ """The Python and TypeScript tool-execution unions must agree. -``coder_eval.agents._timing.busy_ms`` subtracts tool time from an agent's -generation window; ``evalboard/lib/runs.ts::busyMs`` subtracts tool time from a +``coder_eval.timing.busy_ms`` subtracts tool time from an agent's +generation window; ``evalboard/lib/timing.ts::busyMs`` subtracts tool time from a task's wall clock to produce the task page's ``Unaccounted`` residual. They answer the same question about the same ``task.json``, so a divergence is not a style difference — it is the harness and the evalboard reporting two different @@ -20,14 +20,16 @@ import pytest -from coder_eval.agents._timing import busy_ms +from coder_eval.timing import busy_ms, union_ms _FIXTURE = Path(__file__).parent / "_fixtures" / "timing_union_cases.json" _TS_TEST = Path(__file__).parents[1] / "evalboard" / "lib" / "__tests__" / "timing-union-parity.test.ts" _BASE = datetime(2026, 1, 1, 12, 0, 0) -_CASES = json.loads(_FIXTURE.read_text())["cases"] +_CORPUS = json.loads(_FIXTURE.read_text()) +_CASES = _CORPUS["cases"] +_UNION_CASES = _CORPUS["union_cases"] def _at(offset_ms: float) -> datetime: @@ -41,6 +43,20 @@ def test_busy_ms_matches_the_shared_corpus(case: dict) -> None: assert busy_ms(spans, _at(lo), _at(hi)) == pytest.approx(case["expected_ms"]) +@pytest.mark.parametrize("case", _UNION_CASES, ids=[c["name"] for c in _UNION_CASES]) +def test_union_ms_matches_the_shared_corpus(case: dict) -> None: + """The same union, with the extent derived rather than handed in. + + ``union_ms`` is what the golden sensor and the live residual gate both + call; the TypeScript side of these cases is ``toolExecutionMs``, which + derives the extent with its own ``min``/``max`` rather than being given + one. That derivation is the only part of the union rule the ``cases`` + array above cannot reach. + """ + spans = [(_at(s), _at(e)) for s, e in case["spans"]] + assert union_ms(spans) == pytest.approx(case["expected_ms"]) + + def test_the_typescript_half_replays_the_same_file() -> None: """A parity corpus only one side reads is not a parity corpus. @@ -53,3 +69,7 @@ def test_the_typescript_half_replays_the_same_file() -> None: # The TS side must exercise every case, not a hand-picked subset — it reads # the array rather than restating it. assert re.search(r"\.cases\b", source), "the TS test must iterate the corpus, not inline cases" + assert re.search(r"\.union_cases\b", source), ( + "the TS test must also iterate `union_cases`, the half that pins toolExecutionMs's " + "own min/max extent against union_ms's" + )