Skip to content

test: run the m1 tests audit - #179

Merged
kixelated merged 6 commits into
mainfrom
quest/m1/audit-tests
Nov 21, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m1/audit-tests

Conversation

@kixelated

@kixelated kixelated commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Runs the m1 tests audit (quest/m1/audit-tests.md) and plans its m2 run.

Summary

  • Pricing bug fixed. prices::prefix_spans priced a Codex write whose lifetime split names only one lifetime (say ttl_30m_tokens without ttl_12h_tokens) as part write, part ordinary input. It also priced 12h writes outside the Responses API. price_components leaves both of those requests unpriced, so the context check's figures for them were invented. prefix_spans now agrees with price_components.
  • T3 schema fixture fixed. test/fixtures/t3-schema.sql claimed T3 c4ce7b0's indexes but mis-copied two (provider_turns_thread_ordinal_idx is UNIQUE (provider_thread_id, ordinal), and turn_items_active_run_idx covers (thread_id, run_id)). It also left out 13 more indexes over the columns it keeps. It now holds all 24 verbatim from the pinned migrations, plus the ordinal column the unique index needs. Its header states the copy rule. The query-plan test's scan counts hold against the real index set.
  • Codex fixture reshaped. test/fixtures/codex.jsonl had the requested tier in turn_context. Codex never writes it there: none of 6,097 turn_context records on this host has one, and upstream's TurnContextItem has no such field. The tier lives in thread_settings_applied, and every turn starts with task_started, neither of which the fixture had. It now records both, using the tier label Codex actually writes (default). The reader's task_started and thread_settings effort paths now run in tests.
  • New tests for gaps ranked by what breaks.
    • The OpenAI premium-tier multipliers on gpt-5.4 and gpt-5.5, at the 272k context boundary.
    • Codex prefix spans for each write lifetime.
    • A doctor write that fails after another file was already written: it names that file, keeps the earlier backup and cleans up its staged one.
    • The one-by-one walk through turned-off MCP servers (yes, no, one by one, quit).
    • A Codex total from before the window starting the first delta inside it, and a malformed earlier total leaving that delta unknown.
    • The SSH coordinator rejecting a destination that changes its window or result kind, sends more after its result, or fails after sending it.
  • Next run planned. quest/m1/audit-tests.md becomes quest/m2/audit-tests.md, carrying the baseline, how to measure it, the mirrors to recheck and what to check first. quest/m2/README.md matches the one the other m1 audit PRs add.

Findings by kind

  • Gaps. Coverage was 96.32% of lines (56,447 of 58,605) and is now 96.44% (56,723 of 58,815); functions went from 96.01% to 96.07%. Ranked by consequence, the gaps were billing math in prices.rs, settings writes in doctor.rs, Codex cumulative totals in reader.rs, and untrusted SSH streams in ssh.rs. All are covered above. Left for the next run, as its Plan says: report.rs and idle.rs rendering lines, and two guards no test can reach without new seams.
  • Unreachable tests. None. No #[ignore] and no env-gated suites. Every cfg gate matches a CI platform, and every test file compiles into a target CI runs. One platform gap, a follow-up below: every --ssh test is cfg(unix), so a Windows coordinator has none.
  • Missing regression tests. None. Every fix PR on main, and every PR listing a ## Caused by, added or changed tests, except kixel-ai/token-max#45, which only edits a workflow.
  • Wall-clock tests. None. Reports take now as a value. The SSH and pty integration tests use deadlines only as hang guards around real child processes.
  • Drifted mirrors. The T3 schema and Codex transcript fixtures, both fixed above. prices.json matches models.dev's live shape, including tiers beside context_over_200k. claude.jsonl's one key absent locally, compactMetadata.precomputed, is a real (feature-gated) Claude Code 2.1.325 field.

Choices made without asking

  • prefix_spans returns None, as price_components does, rather than guessing a lifetime for a partial split.
  • The T3 fixture copies every pinned index over its columns, not only the ones the adapter's queries use, so the plans it checks are the plans a real database picks.
  • The Codex fixture uses the tier label real records carry (default). One fast test now expects that label instead of standard; both price as standard.
  • Removing the first table of a Codex config.toml leaves the file starting with a blank line (TOML decor from the next table). It is cosmetic and valid TOML, so I left it alone.
  • The SSH guard for a result arriving before its request depends on timing, so I left it untested rather than add a racy test.

Caused by

  • kixel-ai/token-max#26 - prefix_spans' Codex arm priced a write with a partial lifetime split, and 12h writes off the Responses API, which price_components leaves unknown (found by check)

Public API

None. prefix_spans returns None in two more cases; no signature or wire format changes.

Validation

  • just check passes: fmt, clippy, all suites and quest check.
  • The new prefix_spans test fails on the old code: it prices 10,000 of 30,000 written tokens as writes and the rest as ordinary input, where the request total is unknown.
  • I measured coverage before and after with cargo-llvm-cov 0.9.1, folding in the profiles of the binary the integration tests spawn (they env_clear() it). The next stub's Plan records the method.
  • I checked the T3 fixture against this host's live statev2.sqlite schema (schema only) and the pinned migrations. I checked the transcript fixtures' key paths and types against this host's 2,703 Claude and 4,144 Codex transcripts, comparing key names and types only, never values. I checked prices.json against models.dev's live api.json.
  • The SSH suite's wall time is unchanged within noise (11.8s to 13.1s across runs, with and without the new test).

Follow-ups

  • quest/m1/ssh-from-windows.md - test --ssh run from Windows. Settled in the merge pick (comment): ✅ keep it (rejected: drop it); ✅ port the SSH test harness to the Windows CI runner using Git for Windows' sh, size M (rejected: refuse --ssh on Windows with a WSL hint). No open decisions.

None of the other m1 audit PRs (kixel-ai/token-max#174, #175, #176, #177, #178) adds an overlapping quest.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm B)
Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS, arm B)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: e01769a

  • Base: origin/main at 309b526

  • Risk: high, The change affects billing calculations, so incorrect whole-request pricing could misstate users’ costs; reverting the code would not correct cost decisions already made.

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1fa51-d45f-7a83-9285-816765e32f28, round 1

  • Score: 9/10, Static review found no actionable regressions in the pricing fix, fixtures, or added tests; builds and tests were not run, as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, medium effort, shadow

  • Session: 01a1fa51-d45f-7d91-9751-f953d8b4cabc, round 1

  • Score: 9/10, No actionable regressions were found: the pricing correction matches whole-request billing logic and the added coverage is coherent, but builds and tests were not run as instructed.

  • Raised the risk to high: The change affects billing calculations, so incorrect whole-request pricing could misstate users’ costs; reverting the code would not correct cost decisions already made.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

Commit 7a4d54a, after the reviewed head e01769a, only records the merge pick's follow-up answers in quest/m1/ssh-from-windows.md, so no new round ran.

The duplicate lines quest status prints for quest/m2/README.md stay on purpose: every m1 audit PR adds the same m2 milestone README for its next stub, with identical content, so the merges resolve cleanly.

Summary by CodeRabbit

  • Bug Fixes
    • Cache-write price breakdowns now handle lifetime counts consistently, and 12-hour write pricing is limited to the Responses API.
  • Tests
    • Expanded checks for usage totals across date ranges, file-writing errors, setup diagnostics, remote SSH audit results, and pricing tier boundaries.
    • Updated test data to reflect current service-tier values and database schemas.
  • Documentation
    • Added planning notes for Windows SSH testing and test-coverage audits, and removed an earlier audit note.

Quest-Claim: c24e65f2dac09dc1
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The pull request updates quest planning documents and adds or adjusts tests for doctor operations, usage and pricing behavior, SSH audit results, and T3 schema fixtures. It also changes prefix_spans handling for OpenAI cache-write lifetimes.

Changes: Quest planning documents

Layer / File(s) Summary
Audit plans and milestone notes
quest/m1/audit-tests.md, quest/m1/ssh-from-windows.md, quest/m2/README.md, quest/m2/audit-tests.md
The m1 test-audit document is removed. The m2 README and test-audit document describe the audit goal, coverage baseline, checks, and schedule. A separate quest describes Windows SSH audit testing.

Doctor operation tests

Layer / File(s) Summary
Settings writes and MCP cleanup
src/doctor.rs
Tests check file and backup state after a later settings-file write fails, and removal outcomes for bulk, declined, individual, and quit choices.

Usage and pricing tests

Layer / File(s) Summary
Codex service-tier fixture and expectation
test/fixtures/codex.jsonl, src/fast.rs
The Codex fixture records the default service tier in a thread-settings event, and the fast-tier test expects that value.
Cache-write and speed-tier pricing
src/prices.rs
prefix_spans applies lifetime-count rules and limits 12-hour spans to Responses API writes. Tests cover these cases and speed-tier pricing thresholds.
Cumulative totals at the window boundary
src/reader.rs
A test covers legacy cumulative totals before and within a time window, including malformed earlier totals. The settings-event insertion index is updated.

T3 schema fixtures

Layer / File(s) Summary
Schema fixture indexes
test/fixtures/t3-schema.sql
The fixture documents its migration pin and updates event, provider-turn, run, runtime-request, and turn-item indexes.
Provider-turn fixture rows
src/t3.rs, src/t3/guards.rs
T3 tests and guards add ordinal values to provider-turn fixture inserts.

SSH audit tests

Layer / File(s) Summary
Transcript setup and remote-result failures
tests/ssh.rs
The transcript setup uses task, turn, and usage records. New cases check changed scan windows, unexpected result types, trailing data, and nonzero exits after a result.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Other

Merge Risk: Minimal · up to 7a4d5

This PR mainly adds tests, corrects fixtures, and updates planning notes. It also makes one small fix so that Codex cache-write pricing in prefix_spans agrees with price_components for partial or unsupported write-lifetime splits. No actionable merge-blocking issues were found.

Security architecture risk: Minimal · up to e0176

The changes improve cost-estimate accuracy and test coverage. The inspected changes establish no introduced or worsened risk to security boundaries, production data ownership, failure containment, or rollback guarantees.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "test: run the m1 tests audit" matches the main change. The PR runs the m1 tests audit, and the diff deletes quest/m1/audit-tests.md. The PR also contains many other changes, including prici…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "test: run the m1 tests audit" matches the main change. The PR runs the m1 tests audit, and the diff deletes quest/m1/audit-tests.md. The PR also contains many other changes, including pricing fixes, fixture corrections, new tests, and an SSH-on-Windows plan. The title covers only part of that work, but it names the primary intent. It is specific, so it is not vague. No search was needed for this check.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a24ccc66-8b0a-4c89-93ce-4e3481a2d034
Commits

Reviewing files that changed from the base of the PR and between e01769a and 7a4d54a.

Files selected for processing: 1

  • quest/m1/ssh-from-windows.md
Files skipped from review as they are similar to previous changes: 1

  • quest/m1/ssh-from-windows.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

kixelated and others added 4 commits November 20, 2026 10:42
…pans as requests are

The T3 schema fixture mis-copied two indexes from the pinned c4ce7b0 and
left out 13 more over the columns it keeps, so the query-plan test ran
against a schema no real database has. It now holds every index that
revision creates over those columns, verbatim.

prefix_spans priced a Codex write whose lifetime split names one
lifetime as part write and part ordinary input, and priced 12h writes
outside the Responses API, where price_components leaves both unknown.
It now agrees with price_components. New tests cover it and the
premium-tier multipliers on gpt-5.4 and gpt-5.5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…otals and SSH protocol guards

These paths had no test: a write failing after another file was already
written, the one-by-one walk through turned-off MCP servers, a Codex
total before the window starting the first delta inside it, and the
coordinator rejecting a destination that changes its window or result,
sends more after it, or fails after sending it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex writes its requested service tier in thread_settings_applied, never
in turn_context, and starts each turn with task_started; none of 6,097
turn_context records on a heavy user's host carries a tier, and upstream's
TurnContextItem has no such field. The fixture now records the turn's
start, its context without a tier, and its applied settings with the
model, provider, effort and the tier label Codex writes, so the tests
read Codex's real paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Audit test coverage test: run the m1 tests audit Nov 20, 2026
@kixelated
kixelated marked this pull request as ready for review November 20, 2026 18:58
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 20, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. The only behavior change makes prefix_spans follow the same rules as price_components: a partial lifetime split stays unknown, and so does a 12h write outside the Responses API. The rest brings fixtures in line with what they copy and adds tests for the gaps that would cost the most if they broke.

Reviewed changes

I reviewed all 13 files. I also ran the new and changed lib tests locally (codex_prefix_spans_*, premium_tiers_*, a_write_that_fails_after_*, turned_off_mcp_*, legacy_totals_*, nested_codex_settings_*, codex_response_usage_*) and they all pass. I did not run tests/ssh.rs.

  • prefix_spans Codex arm: it now returns None in exactly the cases where price_components (src/prices.rs ~L1424–1452) marks the write split or the 12h lifetime unknown. An unsplit count still takes the default 30m lifetime.
  • T3 schema fixture: it gains ordinal INTEGER NOT NULL and the full set of pinned indexes, including the corrected UNIQUE (provider_thread_id, ordinal). Every provider_turns INSERT in t3.rs and guards.rs now supplies an ordinal, and the duplicate-row test uses ordinal+1 so the unique index still accepts it.
  • Codex fixture: the tier moves out of turn_context into thread_settings_applied ("default"), and a task_started record is added. All three consumers are re-indexed: the reader.rs insert moves to index 4, tests/ssh.rs uses fixture[4] for usage, and fast.rs now expects "default". Both labels price as standard (prices.rs:1175).
  • Doctor write tests: the blocker path .config.toml.token-max-{pid} matches Target::write's staging name. The early ? in create_copy correctly leaves the blocker in place, and the uncommitted Staged backup is dropped.
  • SSH protocol guards and pre-window totals: these are new coverage for the window/kind mismatch, trailing output, failure after the result, and a malformed earlier cumulative total.
  • Quest files: the m1 stub becomes the m2 stub with a baseline and a measurement method, and adds the ssh-from-windows follow-up with its open decision.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision, in the merge pick round for this PR:

  • Pick: ✅ /quest-merge (rejected: /quest-iterate, skip)
  • Follow-up quest/m1/ssh-from-windows.md: ✅ keep (rejected: drop)
  • Its approach: ✅ port the SSH test harness to the Windows CI runner using Git for Windows' sh, size M (rejected: refuse --ssh on Windows with a WSL hint)

(Written by Claude Opus 5.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

The pick's follow-up answers (decision) are now recorded at 7a4d54a. quest/m1/ssh-from-windows.md's Plan records the decision to port the SSH test harness to the Windows runner using Git for Windows' sh, size M. Its Waiting item is removed, and ## Follow-ups is updated.

(Written by Claude Opus 5.5)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the one commit since the prior pullfrog review (docs only).

  • Windows SSH decision recorded: quest/m1/ssh-from-windows.md now states the maintainer's pick (port the harness) and drops the resolved ## Waiting section.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit d512cf8 Nov 21, 2026
7 checks passed
@kixelated
kixelated deleted the quest/m1/audit-tests branch November 21, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant