From 2102538e79e25ed1e5a1415c42e81ddb52e839ea Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 13:24:22 -0800 Subject: [PATCH 1/6] chore(quest): claim quest/m0/routing-evidence/usage-token-max.md Quest-Claim: 54484a76d642b96e From e0d8b3377484a90b83be45625718b2f34117eab9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 13:37:49 -0800 Subject: [PATCH 2/6] feat(usage)!: price sessions through token-max, across SSH hosts `quest usage` reads each session's harness, model, effort, branches, repositories, and cost from `token-max sessions`, which treats each sub-agent as a session of its own, and drops ccusage and quest's own transcript readers. `--ssh HOST` (repeatable) adds other hosts' sessions, matched through the repositories resolved on their own host; a failing host fails the report. A session found on several hosts counts once, at its costliest copy, and the report says how many copies it folded. Sessions whose cost is a lower bound are listed. `fixedBy` counts only merged PRs, and only a `## Caused by` bullet that opens with the culprit's link. Co-Authored-By: Claude Opus 5.5 --- README.md | 53 +- flake.lock | 27 +- flake.nix | 13 +- quest/m0/mistake-audit.md | 9 +- quest/m0/routing-evidence/README.md | 2 +- quest/m0/routing-evidence/charts.md | 5 - quest/m0/routing-evidence/usage-token-max.md | 86 --- quest/m0/routing-evidence/writer-ab.md | 5 - src/cli/github.ts | 2 +- src/cli/main.ts | 36 +- src/cli/messages.ts | 58 ++- src/cli/usage.ts | 517 +++++++++---------- test/start.test.ts | 3 + test/usage.test.ts | 419 ++++++++------- 14 files changed, 604 insertions(+), 631 deletions(-) delete mode 100644 quest/m0/routing-evidence/usage-token-max.md diff --git a/README.md b/README.md index 1028a651c..348f2d082 100644 --- a/README.md +++ b/README.md @@ -1568,9 +1568,9 @@ own retries, which also counts as the limit there. ## Audit cost and accuracy `quest usage` reports `origin`'s PRs updated since `--since` (a local -`YYYY-MM-DD`, by default 7 days ago) and the agent sessions this host ran -since then: which harness, model, and effort worked on each PR, in which role, -at what token cost, and how the work went. +`YYYY-MM-DD`, by default 7 days ago) and the agent sessions this host, and +each host `--ssh` names, ran since then: which harness, model, and effort +worked on each PR, in which role, at what token cost, and how the work went. ```console $ quest usage --pr 569 @@ -1582,27 +1582,32 @@ moq-dev/quest, PRs updated and sessions active since 2026-11-06; costs (~$) esti online review: review (OpenAI), not at the final head; 0 findings: 0 accepted, 0 declined, 0 unclassified, 0 unanswered land fixes: codex, gpt-6.1-sol, medium effort sessions: - implementing claude claude-opus-5-5 low in 50, cache 36.2k written 1.1M read, out 6.1k, ~$0.63 - reviewing codex gpt-6.1-sol medium in 14.7k, cache 0 written 39.8k read, out 421, ~$0.04 + implementing local claude claude-opus-5-5 low in 50, cache 36.2k written 1.1M read, out 6.1k, ~$0.63 + reviewing local codex gpt-6.1-sol medium in 14.7k, cache 0 written 39.8k read, out 421, ~$0.04 ``` -[ccusage](https://github.com/ccusage/ccusage)'s `session --json` reads and +[token-max](https://github.com/kixel-ai/token-max)'s `sessions` reads and prices the sessions, at each model's API list price, which subscriptions do -not bill, so the cost is an estimate. `quest usage` ties each Claude Code or -Codex session to the PR whose head branch its log last names, reading only the -branch, directory, remote, start time, and effort from `$CLAUDE_CONFIG_DIR` (or -`~/.config/claude` and `~/.claude`) and `$CODEX_HOME` (or `~/.codex`), never a +not bill, so the cost is an estimate. A Claude sub-agent or a Codex review or +spawned thread is a session of its own, so a quest a sub-agent worked on +carries its own cost, not its coordinator's. `quest usage` ties each Claude +Code or Codex session to the PR whose head branch it last names, from the +branches, directory, repositories, and start token-max exports for it, never a prompt or reply. Since another repository may use the same branch name, the -session must also be this repository's, as its Codex log, its checkout's -remotes, or, once the checkout is deleted, a Codex log for that exact directory -names (never a parent's, which a nested clone may not share); otherwise it stays unattributed. A session is reviewing when the PR's `## Review` record names +session must also be this repository's, as the repositories token-max resolved +for it on its own host name, or, once its checkout is deleted, those resolved +there for another session in that exact directory, as a Codex log records +(never a parent's, which a nested clone may not share); otherwise it stays +unattributed. A session is reviewing when the PR's `## Review` record names it or Codex marks it a review, implementing when it began before the PR opened, and iterating otherwise. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the `quest review` record names, with its latest round and score and its tally of findings across every round, marked `shadow` for the shadow, where a merged PR whose last round left only low findings open counts them settled, since `/quest-review` fixes or refutes those -before merge without another round; the land fixer's fixes; the later PRs that list it under -`## Caused by`; and each online reviewer's record, read from GitHub. +before merge without another round; the land fixer's fixes; the later merged PRs +whose `## Caused by` names it on a bullet that opens with its link, as +`- [owner/repo#N](url) - why`, never a link elsewhere on a line; and each +online reviewer's record, read from GitHub. An online reviewer is any bot account whose review has a body or opens a thread, as a reply-only review does not, or a review whose body opens with @@ -1630,11 +1635,21 @@ counts as unreviewed. An `Implementer:` line's own size, as `(iterate, XS)`, counts over the PR's; and, over the merged PRs, each online reviewer's coverage, final heads reviewed, findings per PR reviewed, and answers. Sessions on no PR here, from any -repository or agent ccusage reads, total on an `Unattributed` line. `--pr` and +repository or agent token-max reads, total on an `Unattributed` line. `--pr` and `--quest` narrow the report to one PR or one quest's branch, and `--json` -prints it as one object. Without `ccusage` on `PATH`, it says how to install it -and reports the PRs' records alone. Each host reports its own sessions; the -land fixer's container keeps no session log, so its fixes show without tokens. +prints it as one object. Without `token-max` on `PATH`, it says how to install +it and reports the PRs' records alone. + +`--ssh HOST`, repeatable, also reads that SSH destination's sessions through +the token-max installed there, matching each through the repositories resolved +on its own host. A host that fails fails the report, naming it; drop it from +`--ssh` to report without it. A session found on several hosts, such as a +history copied between them, counts once at its costliest copy, and the report +says how many copies it folded from which hosts. A session with a request +token-max could not price, or a transcript line it could not read or that was +unfinished, is listed under `Lower bounds` with its unreadable lines, and every +cost it reaches reads `or more`. The land fixer's container keeps no session log, so +its fixes show without tokens. ### Typed messages diff --git a/flake.lock b/flake.lock index 4037523d3..463113dc1 100644 --- a/flake.lock +++ b/flake.lock @@ -71,7 +71,8 @@ "flake-utils": "flake-utils", "nixpkgs": "nixpkgs", "nixpkgs-bun": "nixpkgs-bun", - "quest-pinned": "quest-pinned" + "quest-pinned": "quest-pinned", + "token-max": "token-max" } }, "systems": { @@ -88,6 +89,30 @@ "repo": "default", "type": "github" } + }, + "token-max": { + "inputs": { + "flake-utils": [ + "flake-utils" + ], + "nixpkgs": [ + "nixpkgs" + ], + "quest": [] + }, + "locked": { + "lastModified": 1794593870, + "narHash": "sha256-6qw7DHhjqGi51kqLHgzWFTg7RLlImW8jnall3cpsB8Q=", + "owner": "kixel-ai", + "repo": "token-max", + "rev": "83ab6167e232f073e2188f1f6c8d37bae13b763c", + "type": "github" + }, + "original": { + "owner": "kixel-ai", + "repo": "token-max", + "type": "github" + } } }, "root": "root", diff --git a/flake.nix b/flake.nix index 33bfa3d5e..8e49961eb 100644 --- a/flake.nix +++ b/flake.nix @@ -17,6 +17,14 @@ url = "github:moq-dev/quest"; flake = false; }; + # token-max, which reads and prices `quest usage`'s sessions; the lock pins it so its JSON stays what the + # parser reads. Its own quest input follows this flake, so the lock never chains older quest pins. + token-max = { + url = "github:kixel-ai/token-max"; + inputs.nixpkgs.follows = "nixpkgs"; + inputs.flake-utils.follows = "flake-utils"; + inputs.quest.follows = ""; + }; }; outputs = @@ -26,6 +34,7 @@ nixpkgs-bun, flake-utils, quest-pinned, + token-max, ... }: # The pinned nixpkgs no longer evaluates on x86_64-darwin; Intel Macs use the release binary. @@ -66,8 +75,8 @@ jq git gh - # Prices `quest usage`'s sessions, pinned with the lock so its JSON stays what the parser reads. - ccusage + # Exports and prices `quest usage`'s sessions. + token-max.packages.${system}.default direnv actionlint shellcheck diff --git a/quest/m0/mistake-audit.md b/quest/m0/mistake-audit.md index 4e4f792c4..b51c5b501 100644 --- a/quest/m0/mistake-audit.md +++ b/quest/m0/mistake-audit.md @@ -47,11 +47,10 @@ Agent's choices, for the implementer to revisit. Each run: of those culprits and their fixes; and the typed messages from `quest messages --since --json`. - Read each culprit pair's `## Caused by` line before counting it. On - 2026-11-11 only 7 of 20 pairs were escapes: `fixedBy` takes every link - in the section, including "No single PR" lines and unmerged PRs, until - [the token-max attribution](/quest/m0/routing-evidence/usage-token-max.md) - lands; the rest named a PR that left the work to a planned follow-up - or kept older behavior. + 2026-11-11 only 7 of 20 pairs were escapes. `fixedBy` now counts only a + merged fix's bullet that opens with the culprit's link, but a culprit + can still name a PR that left the work to a planned follow-up or kept + older behavior. - `quest messages` returns every human turn as a `kind: "message"` entry, beside the `kind: "answer"` entries tallied below, so set aside machine-written ones yourself: bare skill invocations, pasted relays and logs, accepted diff --git a/quest/m0/routing-evidence/README.md b/quest/m0/routing-evidence/README.md index 4bb9e80b2..650d210f2 100644 --- a/quest/m0/routing-evidence/README.md +++ b/quest/m0/routing-evidence/README.md @@ -31,7 +31,7 @@ Decided 2026-11-11 (user, /quest-plan, from an audit of PRs merged only; none). - Records? - ✅ Cumulative review tallies and online reviewers in usage - - ✅ [Attribution through token-max](/quest/m0/routing-evidence/usage-token-max.md), + - ✅ Attribution through token-max, kept after the maintainer first left it out, because the writer A/B compares cost and `quest usage` crashes on this host - Rejected: a PR-body hook that validates `## Models` and `## Caused by` diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md index 580c174f7..0f60984fa 100644 --- a/quest/m0/routing-evidence/charts.md +++ b/quest/m0/routing-evidence/charts.md @@ -74,8 +74,3 @@ Agent's choices, for the implementer to revisit: findings on their own, record them beside the tally. - Make the first charts from the data on main when this lands, link them in this quest's PR, and say which cells are still too thin. - -## Required - -- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - - the cost axis, which today misses sub-agent transcripts, 72% of host spend diff --git a/quest/m0/routing-evidence/usage-token-max.md b/quest/m0/routing-evidence/usage-token-max.md deleted file mode 100644 index f8020cc0c..000000000 --- a/quest/m0/routing-evidence/usage-token-max.md +++ /dev/null @@ -1,86 +0,0 @@ -# [M] quest usage prices sessions through token-max - -## Goal - -`quest usage` takes each session's harness, model, effort, branch, working -directory, and cost from token-max's priced-session export. Sub-agent -transcripts are included, so a quest that a `/quest-spawn` or `/quest-run` -sub-agent worked on carries its own cost, not its coordinator's. quest -drops ccusage and its own transcript readers and keeps only the GitHub -records and the attribution of sessions to PRs. Fix PRs count toward -`fixedBy` only once merged, and only through a bulleted -`- [owner/repo#N](url) - why` culprit line, not a context link. - -`quest usage --ssh HOST` (repeatable) also attributes the sessions of -other hosts the user names, through `token-max sessions --ssh`. Every -session, local or remote, matches its PR through the repositories token-max -exports for it, read on the host where it ran, never through a checkout on -the host running the report. A host that fails fails the report, naming it. -A session found on several hosts, such as a history copied between them, -counts once, and the report says how many such copies it folded and from -which hosts. - -## Plan - -Decided 2026-11-11 (user, /quest-plan, -[routing evidence](/quest/m0/routing-evidence/README.md)): - -- Where does transcript pricing live? - - ✅ token-max prices, quest attributes: one transcript reader for both - tools - - Rejected: moving all of `quest usage` into token-max, which would widen - it into quest-specific reporting; fixing quest's own reader and keeping - ccusage -- Pin token-max as a flake input, the way the other consumers pin quest, - until its 0.1.0 release. - -Decided 2026-11-12 (user, /quest-plan, asked whether the CLIProxyAPI -gateway's request history could replace reading transcripts on each host): - -- Source? ✅ transcripts, read on each host by token-max (rejected: the - gateway's usage records, which v8 drops once read or after at most an - hour, which carry no session, branch or working directory, and which - cover only Claude traffic routed through the gateway). -- Other hosts? ✅ a `--ssh HOST` flag passed through to token-max, so - hosts stay the user's to name (rejected: this host only). -- A session on several hosts? ✅ count it once, keeping its costliest - copy, and report the fold (rejected: failing on it, which one copied - history would turn into a blocked report; counting every copy, which - inflates the cost per PR the writer A/B compares). The costliest copy - rather than the one with the most requests, since copies continued on - both hosts diverge and requests differ in price. -- An unreachable host fails the report rather than leaving its spend out, - since the user named it and a report missing a host would pass for - complete; drop the host from `--ssh` to report without it. -- Matching a remote session to its repository: today's matcher in - `src/cli/usage.ts` reads the git remotes of the session's working - directory on this host, which for another host's path is missing or a - different checkout. token-max exports each session's repositories, - resolved on its own host, and quest matches only through them. - -The audit found today's failures: - -- `quest usage` stops at a Claude Code transcript with an interleaved, - malformed line in the middle of the file, such as line 1385 of a moq - session on 2026-11-04. On 2026-11-11 that one line made `quest usage` - exit 1 in all three repositories, since every report reads every session - on the host, and the file is Claude Code's, not quest's to repair. The - reader skips each such line, prints how many it skipped in which - sessions, and marks the cost totals as lower bounds, so the report never - passes for complete. Report the corruption upstream to Claude Code. -- ccusage folds sub-agent records into their parent's session ID, and - quest reads only the parent's log, so 72% of host spend was unattributed. -- `fixedBy` counted context links, such as moq-dev/quest#558 linking #494, - and fixes still open or closed unmerged, such as moq-dev/moq#8476. - The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 9 of - 20 culprit pairs that week were such artifacts: links on a "No single - PR" or "None" line, a "(fixed by #N)" aside, a note after the bullets, - a closed duplicate fix, and an unmerged culprit. Test against - moq-dev/moq#8471, #8426, #8483, #8427, #8371, and #8493. - -## Waiting - -- kixel-ai/token-max: `token-max sessions` accepts `--ssh` and exports - each session's repositories, as - [its quest](https://github.com/kixel-ai/token-max/blob/main/quest/m0/sessions-ssh.md) - plans; check that `token-max sessions --help` documents both diff --git a/quest/m0/routing-evidence/writer-ab.md b/quest/m0/routing-evidence/writer-ab.md index 11ab96230..e47dbdf6b 100644 --- a/quest/m0/routing-evidence/writer-ab.md +++ b/quest/m0/routing-evidence/writer-ab.md @@ -55,8 +55,3 @@ Agent's choices, for the implementer to revisit: arm on the PR's first Implementer line: `--writer` also takes that line's effort, or a new option names the arm. - [The readout](/quest/m0/routing-evidence/README.md) ends the split. - -## Required - -- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - - counts sub-agent writers' cost, without which an arm's cost is undercounted diff --git a/src/cli/github.ts b/src/cli/github.ts index 993a486ae..883d74158 100644 --- a/src/cli/github.ts +++ b/src/cli/github.ts @@ -180,7 +180,7 @@ export function remoteRepository(remote: Remote, push = false): { url: string; r */ export function githubRepository(url: string): Repository | null { const match = - /^(?:https:\/\/(?:[^@/]+@)?github\.com\/|ssh:\/\/(?:[^@/]+@)?github\.com(?::\d+)?\/|[^@/:]+@github\.com:)([^/]+)\/([^/]+?)(?:\.git)?\/?$/.exec( + /^(?:https:\/\/(?:[^@/]+@)?github\.com\/|ssh:\/\/(?:[^@/]+@)?github\.com(?::\d+)?\/|(?:[^@/:]+@)?github\.com:)([^/]+)\/([^/]+?)(?:\.git)?\/?$/.exec( url, ); return match ? { owner: match[1], repo: match[2] } : null; diff --git a/src/cli/main.ts b/src/cli/main.ts index 743bbe239..a2271dd38 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -84,7 +84,7 @@ import { CONTEXT, GUIDE, GUIDE_SECTIONS, SKILLS, guideSections, sectionHeading, import { type Terminal, Theme } from "./theme"; import { type Snapshot, checkout, collect, load, snapshot, under } from "./tree"; import { pullRequestStatus, statusLines } from "./status"; -import { CCUSAGE_MISSING, daysBefore, usage, usageDependencies, usageLines } from "./usage"; +import { TOKEN_MAX_MISSING, daysBefore, usage, usageDependencies, usageLines } from "./usage"; import { waitForActivity, WaitTimeout } from "./wait"; import { Worktrees } from "./worktree"; import { HOOK_EVENTS, claudeHook } from "./worktree-hook"; @@ -1155,22 +1155,25 @@ export function main(argv: string[], out: Output): number | Promise { .command("usage") .summary("Report which models worked on each PR, at what token cost, and how its reviews went") .description( - "Report origin's PRs updated since --since and this host's agent sessions active since then. Each " + - "Claude Code or Codex session belongs to the PR whose head branch its log last names, when its Codex log, its " + - "checkout's remotes, or a Codex log for its directory names this repository, read from " + - "$CLAUDE_CONFIG_DIR (or ~/.config/claude and ~/.claude) and $CODEX_HOME (or ~/.codex); no prompt or reply is read out. Its role is " + + "Report origin's PRs updated since --since and the agent sessions active since then on this host and each " + + "--ssh host, as token-max sessions exports and prices them, the cost an estimate at each model's API list " + + "price; its sub-agents are sessions of their own. Each Claude Code or Codex session belongs to the PR whose " + + "head branch it last names, when a repository token-max resolved for it on its own host, or for another " + + "session in the same directory there, is this one; no prompt or reply is read out. Its role is " + "reviewing when the PR's ## Review record names it or Codex marks it a review, implementing when it began " + - "before the PR opened, and iterating otherwise. Tokens and cost come from ccusage (session --json), the " + - "cost an estimate at each model's API list price. Each PR also shows its ## Models size and implementers, and as unread each Implementer line it cannot read, " + + "before the PR opened, and iterating otherwise. A session found on several hosts counts once, at its " + + "costliest copy, and the report says how many copies it folded from which hosts; a session with a request " + + "token-max could not price or a line it could not read is listed, its cost a lower bound. A host that fails fails the report. " + + "Each PR also shows its ## Models size and implementers, and as unread each Implementer line it cannot read, " + "each reviewer the ## Review record names, with its latest round and score and its tally of findings across every round, marked when it is the shadow, " + "where a merged PR whose last round left only low findings open counts them settled, since the author fixes or refutes those before merge, " + - "the later PRs that list it under ## Caused by, and each online reviewer's record from GitHub: whether it " + + "the later merged PRs whose ## Caused by bullets open with its link, and each online reviewer's record from GitHub: whether it " + "reviewed the last head, and its findings accepted, declined, unclassified, or unanswered. An online reviewer " + "is a bot whose review has a body or opens a thread, or a review whose body opens with Automated review by ; a finding is a " + "review thread it opens or a [P] line in its review body, and its first answer's first word takes or " + "declines it. Rollups follow: spend by size, role, harness, model, and " + "effort, accuracy by size and implementer, summing each branch's tallies but the shadow's once, where an Implementer line's own size, as (iterate, XS), wins " + - "over the PR's, and each online reviewer's coverage and findings over the merged PRs. Without ccusage on PATH, it says how to install it and reports the PRs' records alone. " + + "over the PR's, and each online reviewer's coverage and findings over the merged PRs. Without token-max on PATH, it says how to install it and reports the PRs' records alone. " + "--json prints the same report as one object.\n\n" + "Uses GH_TOKEN, GITHUB_TOKEN, or gh auth token. Exits 0 when done, 2 for invalid arguments, and 1 for " + "other failures.", @@ -1178,14 +1181,23 @@ export function main(argv: string[], out: Output): number | Promise { .option("--since ", "First local date to report, as YYYY-MM-DD (default: 7 days ago)", date) .option("--quest ", "Report only the PRs of this quest's branch") .option("--pr ", "Report only this PR", positiveInteger("--pr")) + .option( + "--ssh ", + "Also attribute this SSH destination's sessions, read by the token-max installed there (repeatable)", + (host: string, hosts: string[] = []) => [...hosts, host], + ) .option("--json", "Print one JSON object") - .action((options: { since?: string; quest?: string; pr?: number; json?: boolean }) => { + .action((options: { since?: string; quest?: string; pr?: number; ssh?: string[]; json?: boolean }) => { const github = { github: () => gitHubDependencies(warn), warn }; const since = options.since ?? daysBefore(new Date(), 7); settle( - usage(root(), { since, quest: options.quest, pr: options.pr }, usageDependencies(github)), + usage( + root(), + { since, quest: options.quest, pr: options.pr, ssh: options.ssh }, + usageDependencies(github), + ), (report) => { - if (!report.priced) warn(CCUSAGE_MISSING); + if (!report.priced) warn(TOKEN_MAX_MISSING); if (options.json) print(JSON.stringify(report)); else for (const line of usageLines(report)) print(line); }, diff --git a/src/cli/messages.ts b/src/cli/messages.ts index a870f2a04..91e58babb 100644 --- a/src/cli/messages.ts +++ b/src/cli/messages.ts @@ -3,12 +3,66 @@ // drives, Claude Code's `origin` for the rest of its sessions, and Codex's content provenance for its interactive ones. // Answers come from each harness's own log, which keeps the question's options. Output goes to stdout only. import { spawnSync } from "node:child_process"; -import { existsSync, readdirSync, statSync } from "node:fs"; +import { closeSync, existsSync, openSync, readdirSync, readSync, statSync } from "node:fs"; import { join } from "node:path"; import { DatabaseSync } from "node:sqlite"; import { discoveryEnv } from "./git"; import { githubRepository } from "./github"; -import { claudeLogs, eachLine, parseLine } from "./usage"; + +/** + * Visit each line of the file at `path`, reading it in bounded chunks since logs reach gigabytes. The last line is + * marked incomplete when no newline ends it, as while its session is still writing it. Reading stops once `visit` + * returns false. + */ +function eachLine(path: string, visit: (line: string, complete: boolean) => boolean | void): void { + const fd = openSync(path, "r"); + try { + const chunk = Buffer.alloc(1 << 22); + let rest = Buffer.alloc(0); + for (let read = readSync(fd, chunk); read > 0; read = readSync(fd, chunk)) { + const data = rest.length ? Buffer.concat([rest, chunk.subarray(0, read)]) : chunk.subarray(0, read); + let start = 0; + for (let end = data.indexOf(10); end >= 0; end = data.indexOf(10, start)) { + if (end > start && visit(data.toString("utf8", start, end), true) === false) return; + start = end + 1; + } + rest = Buffer.from(data.subarray(start)); + } + if (rest.length) visit(rest.toString("utf8"), false); + } finally { + closeSync(fd); + } +} + +/** One JSON record of a log line, or null for an incomplete last line. */ +function parseLine(path: string, line: string, complete: boolean): Record | null { + try { + return JSON.parse(line) as Record; + } catch (error) { + if (!complete) return null; + throw new Error(`${path} has a line that is not JSON: ${(error as Error).message}`); + } +} + +/** Claude Code's session logs by session ID, from each config directory Claude Code uses, once each. */ +function claudeLogs(deps: { env: NodeJS.ProcessEnv; home: string }): Map { + const configured = deps.env.CLAUDE_CONFIG_DIR?.split(",") + .map((dir) => dir.trim()) + .filter(Boolean); + const dirs = configured?.length ? configured : [join(deps.home, ".config", "claude"), join(deps.home, ".claude")]; + const logs = new Map(); + for (const dir of dirs) { + const projects = join(dir, "projects"); + if (!existsSync(projects)) continue; + for (const project of readdirSync(projects, { withFileTypes: true })) { + if (!project.isDirectory()) continue; + for (const file of readdirSync(join(projects, project.name))) + if (file.endsWith(".jsonl") && !logs.has(file.slice(0, -6))) + logs.set(file.slice(0, -6), join(projects, project.name, file)); + } + } + return logs; +} /** One message a person typed. */ export interface Message { diff --git a/src/cli/usage.ts b/src/cli/usage.ts index 30f0511e7..f54b59132 100644 --- a/src/cli/usage.ts +++ b/src/cli/usage.ts @@ -1,14 +1,11 @@ // `quest usage`: which harness, model, and effort worked on each PR, in which role and at what token cost, and how -// the work went by the PR's own record. ccusage reads and prices the sessions; quest ties each to a PR by the -// branch its log names, reading only that metadata and never a prompt or reply. +// the work went by the PR's own record. token-max reads and prices the sessions, on this host and any SSH host +// named; quest ties each to a PR by the branches and repositories token-max exports for it. import { spawnSync } from "node:child_process"; -import { closeSync, existsSync, openSync, readdirSync, readSync } from "node:fs"; -import { homedir } from "node:os"; -import { join } from "node:path"; import { type Size, SIZES, handedBack, questBranch } from "../core"; import { recorded } from "../github/models"; import { type ClaimDependencies, repositoryPath, upstream } from "./claims"; -import { discoveryEnv, git } from "./git"; +import { git } from "./git"; import { fixes } from "./land-fixer"; import { MERGER } from "./land-watch-fix"; import { type GitHub, type ObjectValue, type Repository, githubRepository, link, requireRemote } from "./github"; @@ -23,39 +20,46 @@ export interface Tokens { cacheRead: number; output: number; cost: number; + /** False when the cost is a lower bound: a request went unpriced or a transcript line was unreadable. */ + complete: boolean; } -/** One session as ccusage prices it: its harness (ccusage's agent), its ID there, and its spend per model. */ -export interface Priced { - harness: string; - id: string; - models: ({ model: string } & Tokens)[]; -} +/** One model and effort's spend in a session; token-max names neither when the transcript does not. */ +export type Spent = { model: string | null; effort: string | null } & Tokens; -/** Where a session spent its tokens, as far as its log tells. */ -export interface SessionLog { - /** Epoch milliseconds of its first record. */ - started: number; - /** Each branch its log names, with the first and last time it does, and the directory it last ran in there. */ - branches: Map; - /** The repository its checkout fetched from, when the log names it; Claude Code's does not. */ - repository: Repository | null; - effort: string | null; - /** Codex marks the sessions `codex exec review` starts. */ +/** One session as `token-max sessions` exports and prices it. */ +export interface Session { + /** The `--ssh` destination it was read on, or null for this host. */ + host: string | null; + harness: string; + /** The harness's own ID, as `quest review` records it; null when the transcript names none. */ + id: string | null; review: boolean; + cwd: string | null; + /** The GitHub repositories, as lowercase `owner/repo`, its host resolved for it; null when it could not. */ + repositories: string[] | null; + /** Each branch it named, with the first and last time, in epoch milliseconds. */ + branches: { name: string; first: number; last: number }[]; + /** Epoch milliseconds of its first request in the window. */ + start: number; + models: Spent[]; + cost: number; + /** False when its cost is a lower bound: a request went unpriced, or a line was unreadable or unfinished. */ + complete: boolean; + /** Transcript lines token-max could not read, which its cost leaves out. */ + skippedLines: number; } export type Role = "implementing" | "iterating" | "reviewing"; /** One session attributed to a PR. */ export interface SessionRow { + host: string | null; harness: string; - session: string; + session: string | null; role: Role; - effort: string | null; - models: ({ model: string } & Tokens)[]; + models: Spent[]; } - /** One PR's record and the sessions that worked on it. */ export interface PullRow { number: number; @@ -87,7 +91,7 @@ export interface SpendRow extends Tokens { size: Size | null; role: Role; harness: string; - model: string; + model: string | null; effort: string | null; sessions: number; } @@ -132,14 +136,18 @@ export interface OnlineRow { export interface Usage { repository: string; since: string; - /** Whether ccusage ran; without it, the report holds only the PRs' records. */ + /** Whether token-max ran; without it, the report holds only the PRs' records. */ priced: boolean; pulls: PullRow[]; spend: SpendRow[]; accuracy: AccuracyRow[]; online: OnlineRow[]; - /** The sessions on no PR reported here, from every repository and agent ccusage reads; null when narrowed. */ + /** The sessions on no PR reported here, from every repository and agent token-max reads; null when narrowed. */ unattributed: ({ sessions: number } & Tokens) | null; + /** Copies of a session found on several hosts, left out for its costliest copy, and the hosts they came from. */ + folded: { copies: number; hosts: (string | null)[] }; + /** Each session reported whose cost is a lower bound, with the transcript lines token-max could not read. */ + incomplete: { host: string | null; harness: string; session: string | null; skippedLines: number }[]; } export interface UsageOptions { @@ -147,18 +155,18 @@ export interface UsageOptions { since: string; quest?: string; pr?: number; + /** SSH destinations whose sessions are attributed too, beside this host's. */ + ssh?: string[]; } export interface UsageDependencies extends ClaimDependencies { - /** ccusage's sessions since a `YYYY-MM-DD` date, or null when it is not installed. */ - ccusage(since: string): Priced[] | null; - env: NodeJS.ProcessEnv; - home: string; + /** token-max's sessions since an instant on this host and each SSH destination, or null when it is not installed. */ + sessions(since: Date, ssh: string[]): Session[] | null; } -/** The install hint printed when ccusage is missing. */ -export const CCUSAGE_MISSING = - "ccusage is not installed, so no session tokens or cost are reported; install it with `npm install -g ccusage`"; +/** The install hint printed when token-max is missing. */ +export const TOKEN_MAX_MISSING = + "token-max is not installed, so no session tokens or cost are reported; install it with `cargo install --locked --git https://github.com/kixel-ai/token-max`"; /** The local date `days` before `now`, as `YYYY-MM-DD`. */ export function daysBefore(now: Date, days: number): string { @@ -166,180 +174,102 @@ export function daysBefore(now: Date, days: number): string { return `${at.getFullYear()}-${String(at.getMonth() + 1).padStart(2, "0")}-${String(at.getDate()).padStart(2, "0")}`; } -/** Run `ccusage session --json` from PATH, or return null when it is not installed. */ -export function runCcusage(since: string, env: NodeJS.ProcessEnv): Priced[] | null { - const result = spawnSync("ccusage", ["session", "--json", "--since", since.replaceAll("-", "")], { - encoding: "utf8", - env, - maxBuffer: 1 << 30, - }); - if ((result.error as NodeJS.ErrnoException | undefined)?.code === "ENOENT") return null; +/** Run `token-max sessions` from PATH, or return null when it is not installed. */ +export function runTokenMax(since: Date, ssh: string[]): Session[] | null { + const hosts = ssh.flatMap((host) => ["--ssh", host]); + const args = ["sessions", "--since", since.toISOString(), ...hosts, ...(ssh.length ? ["--include-local"] : [])]; + const result = spawnSync("token-max", args, { encoding: "utf8", maxBuffer: 1 << 30 }); + if ((result.error as NodeJS.ErrnoException | undefined)?.code === "ENOENT") { + if (ssh.length) throw new Error("token-max is not installed, so --ssh cannot read other hosts' sessions"); + return null; + } if (result.error) throw result.error; - if (result.status !== 0) throw new Error(`ccusage session --json exited ${result.status}: ${result.stderr.trim()}`); - return parseCcusage(result.stdout); + // token-max prints its error as one JSON object, naming the host that failed. + if (result.status !== 0) throw new Error(`token-max sessions exited ${result.status}: ${result.stderr.trim()}`); + return parseSessions(result.stdout); } -/** The sessions `ccusage session --json` printed, failing on any field it lacks. */ -export function parseCcusage(text: string): Priced[] { - const rows: unknown = (JSON.parse(text) as { session?: unknown } | null)?.session; - if (!Array.isArray(rows)) throw new Error("ccusage session --json printed no session list"); - const count = (value: unknown, what: string): number => { - if (typeof value !== "number" || !Number.isFinite(value) || value < 0) - throw new Error(`ccusage printed a session without a ${what}`); - return value; +/** The sessions `token-max sessions` printed, one JSON object per line, failing on any field it lacks. */ +export function parseSessions(text: string): Session[] { + const fail = (what: string): never => { + throw new Error(`token-max printed a session without ${what}`); }; - const word = (value: unknown, what: string): string => { - if (typeof value !== "string" || value === "") throw new Error(`ccusage printed a session without a ${what}`); - return value; + const count = (value: unknown, what: string): number => + typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : fail(what); + const word = (value: unknown, what: string): string => + typeof value === "string" && value !== "" ? value : fail(what); + const optional = (value: unknown, what: string): string | null => (value === null ? null : word(value, what)); + const flag = (value: unknown, what: string): boolean => (typeof value === "boolean" ? value : fail(what)); + const time = (value: unknown, what: string): number => { + const at = Date.parse(word(value, what)); + return Number.isNaN(at) ? fail(what) : at; }; - return rows.map((row: Record) => { - const models = row?.modelBreakdowns; - if (!Array.isArray(models)) throw new Error("ccusage printed a session without modelBreakdowns"); - return { - harness: word(row.agent, "agent"), - id: word(row.period, "session ID"), - models: models.map((m: Record) => ({ - model: word(m?.modelName, "model name"), - input: count(m.inputTokens, "input token count"), - cacheWrite: count(m.cacheCreationTokens, "cache write token count"), - cacheRead: count(m.cacheReadTokens, "cache read token count"), - output: count(m.outputTokens, "output token count"), - cost: count(m.cost, "cost"), - })), - }; - }); + return text + .split("\n") + .filter((line) => line !== "") + .map((line) => { + const row = JSON.parse(line) as Record | null; + if (row === null || typeof row !== "object") return fail("a JSON object"); + const repositories = row.repositories; + if (repositories !== null && !Array.isArray(repositories)) fail("a repository list"); + const branches = Array.isArray(row.branches) ? row.branches : fail("a branch list"); + const models = Array.isArray(row.models) ? row.models : fail("a model list"); + return { + host: optional(row.host, "a host"), + harness: word(row.harness, "a harness"), + id: optional(row.session, "a session ID"), + review: flag(row.review, "a review mark"), + cwd: optional(row.cwd, "a directory"), + // A repository that is not on GitHub can head no PR here, so only GitHub's count. + repositories: + repositories === null + ? null + : (repositories as unknown[]).flatMap((url) => { + const named = githubRepository(word(url, "a repository URL")); + return named ? [key(named)] : []; + }), + branches: branches.map((b: Record) => ({ + name: word(b?.name, "a branch name"), + first: time(b.first, "a branch's first time"), + last: time(b.last, "a branch's last time"), + })), + start: time(row.start, "a start"), + models: models.map((m: Record) => ({ + model: optional(m?.model, "a model"), + effort: optional(m.effort, "an effort"), + input: count(m.inputTokens, "an input token count"), + cacheWrite: count(m.cacheWriteTokens, "a cache write token count"), + cacheRead: count(m.cacheReadTokens, "a cache read token count"), + output: count(m.outputTokens, "an output token count"), + cost: count(m.costUsd, "a model's cost"), + complete: flag(m.complete, "a model's completeness"), + })), + cost: count(row.costUsd, "a cost"), + complete: flag(row.complete, "its completeness"), + skippedLines: count(row.skippedLines, "a skipped line count"), + }; + }); } /** - * Visit each line of the file at `path`, reading it in bounded chunks since logs reach gigabytes. The last line is - * marked incomplete when no newline ends it, as while its session is still writing it. Reading stops once `visit` - * returns false. + * Each session once, keeping its costliest copy where several hosts hold it, as when history was copied between + * them, since counting every copy would inflate the cost per PR. Returns the copies it left out too. */ -export function eachLine(path: string, visit: (line: string, complete: boolean) => boolean | void): void { - const fd = openSync(path, "r"); - try { - const chunk = Buffer.alloc(1 << 22); - let rest = Buffer.alloc(0); - for (let read = readSync(fd, chunk); read > 0; read = readSync(fd, chunk)) { - const data = rest.length ? Buffer.concat([rest, chunk.subarray(0, read)]) : chunk.subarray(0, read); - let start = 0; - for (let end = data.indexOf(10); end >= 0; end = data.indexOf(10, start)) { - if (end > start && visit(data.toString("utf8", start, end), true) === false) return; - start = end + 1; - } - rest = Buffer.from(data.subarray(start)); - } - if (rest.length) visit(rest.toString("utf8"), false); - } finally { - closeSync(fd); - } -} - -/** One JSON record of a log line, or null for an incomplete last line. */ -export function parseLine(path: string, line: string, complete: boolean): Record | null { - try { - return JSON.parse(line) as Record; - } catch (error) { - if (!complete) return null; - throw new Error(`${path} has a line that is not JSON: ${(error as Error).message}`); - } -} - -/** Claude Code's session logs by session ID, from each config directory ccusage reads, once each. */ -export function claudeLogs(deps: { env: NodeJS.ProcessEnv; home: string }): Map { - const configured = deps.env.CLAUDE_CONFIG_DIR?.split(",") - .map((dir) => dir.trim()) - .filter(Boolean); - const dirs = configured?.length ? configured : [join(deps.home, ".config", "claude"), join(deps.home, ".claude")]; - const logs = new Map(); - for (const dir of dirs) { - const projects = join(dir, "projects"); - if (!existsSync(projects)) continue; - for (const project of readdirSync(projects, { withFileTypes: true })) { - if (!project.isDirectory()) continue; - for (const file of readdirSync(join(projects, project.name))) - if (file.endsWith(".jsonl") && !logs.has(file.slice(0, -6))) - logs.set(file.slice(0, -6), join(projects, project.name, file)); - } +export function fold(sessions: Session[]): { kept: Session[]; dropped: Session[] } { + const best = new Map(); + const dropped: Session[] = []; + const id = (session: Session) => JSON.stringify([session.harness, session.id]); + for (const session of sessions) { + if (session.id === null) continue; + const other = best.get(id(session)); + if (other === undefined) best.set(id(session), session); + else if (session.cost > other.cost) { + dropped.push(other); + best.set(id(session), session); + } else dropped.push(session); } - return logs; -} - -/** What a Claude Code log tells: every record carries its checkout's branch, and each reply its effort. */ -export function readClaudeLog(path: string): SessionLog { - const log: SessionLog = { started: NaN, branches: new Map(), repository: null, effort: null, review: false }; - eachLine(path, (line, complete) => { - const record = parseLine(path, line, complete); - const at = typeof record?.timestamp === "string" ? Date.parse(record.timestamp) : NaN; - if (record === null || Number.isNaN(at)) return; - if (!(at >= log.started)) log.started = at; - const branch = record.gitBranch; - if (typeof branch === "string" && branch !== "" && branch !== "HEAD") { - const seen = log.branches.get(branch); - const cwd = typeof record.cwd === "string" ? record.cwd : null; - if (seen) { - seen.last = Math.max(seen.last, at); - seen.cwd = cwd ?? seen.cwd; - } else log.branches.set(branch, { first: at, last: at, cwd }); - } - if (record.type === "assistant" && typeof record.effort === "string") log.effort = record.effort; - }); - return log; -} - -/** What a Codex rollout tells: its first record names the checkout's branch and remote, and each turn its effort. */ -export function readCodexLog(path: string): SessionLog { - let log: SessionLog | undefined; - eachLine(path, (line, complete) => { - if (log === undefined) { - const meta = parseLine(path, line, complete); - const payload = meta?.payload as Record | undefined; - const at = Date.parse(String(payload?.timestamp ?? meta?.timestamp)); - if (meta?.type !== "session_meta" || Number.isNaN(at)) - throw new Error(`${path} does not start with a Codex session_meta record`); - const repo = payload?.git as Record | undefined; - const branch = typeof repo?.branch === "string" && repo.branch !== "" ? repo.branch : null; - const url = typeof repo?.repository_url === "string" ? repo.repository_url : null; - const cwd = typeof payload?.cwd === "string" ? payload.cwd : null; - log = { - started: at, - branches: new Map(branch ? [[branch, { first: at, last: at, cwd }]] : []), - repository: url === null ? null : githubRepository(url), - effort: null, - review: (payload?.source as Record | undefined)?.subagent === "review", - }; - return; - } - // A turn context is small, unlike the replies and tool output, so only its lines are parsed. - if (!line.includes('"type":"turn_context"')) return; - const effort = (parseLine(path, line, complete)?.payload as Record | undefined)?.effort; - if (typeof effort === "string") log.effort = effort; - }); - if (log === undefined) throw new Error(`${path} is empty`); - return log; -} - -/** The log of each priced session, found where its harness keeps it, or null for one quest cannot read. */ -function sessionLogs(sessions: Priced[], deps: UsageDependencies): (SessionLog | null)[] { - let claude: Map | undefined; - const codex = deps.env.CODEX_HOME || join(deps.home, ".codex"); - return sessions.map((session) => { - if (session.harness === "claude") { - claude ??= claudeLogs(deps); - const path = claude.get(session.id); - return path === undefined ? null : readClaudeLog(path); - } - if (session.harness === "codex") { - // ccusage names a rollout by its path under the sessions directory, or under the flat archive. - if (session.id.split("/").includes("..")) - throw new Error(`ccusage named a Codex session outside its home: ${session.id}`); - const path = ["sessions", "archived_sessions"] - .map((dir) => join(codex, dir, `${session.id}.jsonl`)) - .find((each) => existsSync(each)); - return path === undefined ? null : readCodexLog(path); - } - return null; - }); + const kept = sessions.filter((session) => session.id === null || best.get(id(session)) === session); + return { kept, dropped }; } interface Pull { @@ -392,18 +322,20 @@ async function updatedPulls(client: GitHub, repository: Repository, since: numbe return pulls; } -/** The PR numbers of `repository` that `body`'s `## Caused by` section links. */ -function causes(body: string, repository: Repository): number[] { +/** + * The PR numbers of `repository` that `body`'s `## Caused by` section names as culprits: each bullet that opens with + * `[owner/repo#N](url) - why`. A link anywhere else on a line, as context or a "fixed by" aside, names no culprit. + */ +export function causes(body: string, repository: Repository): number[] { const heading = /^## Caused by[ \t]*\r?$/m.exec(body); if (!heading) return []; const rest = body.slice(heading.index + heading[0].length); const section = rest.slice(0, /^## /m.exec(rest)?.index ?? rest.length); const escape = (text: string) => text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); - const url = new RegExp( - `https://github\\.com/${escape(repository.owner)}/${escape(repository.repo)}/pull/(\\d+)`, - "gi", - ); - return [...new Set([...section.matchAll(url)].map((match) => Number(match[1])))]; + const name = `${escape(repository.owner)}/${escape(repository.repo)}`; + const culprit = new RegExp(`^- \\[${name}#(\\d+)\\]\\(https://github\\.com/${name}/pull/(\\d+)\\) - \\S`, "gim"); + const found = [...section.matchAll(culprit)].filter((match) => match[1] === match[2]); + return [...new Set(found.map((match) => Number(match[1])))]; } const sized = (value: string | null | undefined): Size | null => @@ -430,7 +362,8 @@ function settledOnMerge(reviews: ReviewSummary[], merged: boolean): ReviewSummar /** Each PR's record, without its sessions yet. */ function pullRows(pulls: Pull[], repository: Repository): PullRow[] { const fixedBy = new Map(); - for (const each of pulls) + // A fix counts once it lands: an open or abandoned one, such as a closed duplicate, fixed nothing. + for (const each of pulls.filter((pull) => pull.state === "merged")) for (const cause of causes(each.body, repository)) if (cause !== each.number) fixedBy.set(cause, [...(fixedBy.get(cause) ?? []), each.number]); return pulls.map((each) => { @@ -464,79 +397,58 @@ const add = (to: Tokens, from: Tokens) => { to.cacheRead += from.cacheRead; to.output += from.output; to.cost += from.cost; + to.complete &&= from.complete; }; -const zero = (): Tokens => ({ input: 0, cacheWrite: 0, cacheRead: 0, output: 0, cost: 0 }); - -/** - * The ID a harness gives a session, as `quest review` records it: Claude Code's is ccusage's, and Codex's is the - * thread ID that ends the rollout's file name. - */ -function sessionId(session: Priced): string { - if (session.harness !== "codex") return session.id; - return /([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})$/.exec(session.id)?.[1] ?? session.id; -} - -/** The GitHub repositories the checkout at `cwd` has remotes for, or null once it is no checkout to ask. */ -function checkoutRepositories(cwd: string): string[] | null { - if (!existsSync(cwd)) return null; - const remotes = spawnSync("git", ["-C", cwd, "remote", "-v"], { encoding: "utf8", env: discoveryEnv() }); - if (remotes.status !== 0) return null; - return remotes.stdout.split("\n").flatMap((line) => { - const named = githubRepository(line.split(/\s+/)[1] ?? ""); - return named ? [key(named)] : []; - }); -} +const zero = (): Tokens => ({ input: 0, cacheWrite: 0, cacheRead: 0, output: 0, cost: 0, complete: true }); /** - * Which repositories the directory `cwd` belongs to: its checkout's remotes while it exists, else the remote a - * Codex log names for that exact directory, as a reviewer run in the same checkout records; null when nothing - * tells, since a branch name alone may be another repository's. A directory above it never tells, since a - * nested clone may be another repository's. + * Which repositories the directory `cwd` on `host` belongs to: those token-max resolved for the session, else those + * it resolved for another session in that exact directory on the same host, as a Codex log records after the + * checkout is deleted; null when nothing tells, since a branch name alone may be another repository's. A directory + * above it never tells, since a nested clone may be another repository's. */ -function owners(logs: (SessionLog | null)[]): (cwd: string) => string[] | null { - const logged = new Map>(); - for (const log of logs) - if (log?.repository) - for (const seen of log.branches.values()) - if (seen.cwd) logged.set(seen.cwd, (logged.get(seen.cwd) ?? new Set()).add(key(log.repository))); - const cache = new Map(); - return (cwd) => { - if (!cache.has(cwd)) { - const names = logged.get(cwd); - cache.set(cwd, checkoutRepositories(cwd) ?? (names ? [...names] : null)); +function owners(sessions: Session[]): (session: Session) => string[] | null { + const where = (session: Session) => JSON.stringify([session.host, session.cwd]); + const known = new Map>(); + for (const session of sessions) + if (session.cwd !== null && session.repositories?.length) { + const names = known.get(where(session)) ?? new Set(); + for (const name of session.repositories) names.add(name); + known.set(where(session), names); } - return cache.get(cwd)!; + return (session) => { + if (session.repositories?.length) return session.repositories; + const names = session.cwd === null ? undefined : known.get(where(session)); + return names ? [...names] : null; }; } /** - * Attribute each priced session to the PR whose head branch its log last named, among those still open when the - * session reached the branch and headed from the repository its log, checkout, or directory names, and give it a - * role: reviewing when a PR's review record names it or Codex marks it a review, implementing when it began before - * its PR opened, and iterating otherwise. Returns the sessions on no PR here. + * Attribute each session to the PR whose head branch it last named, among those still open when the session + * reached the branch and headed from a repository its host resolved for it, and give it a role: reviewing when a + * PR's review record names it or Codex marks it a review, implementing when it began before its PR opened, and + * iterating otherwise. Returns the sessions it gave a PR in `rows`, and those on no PR here. */ function attribute( - sessions: Priced[], - logs: (SessionLog | null)[], + sessions: Session[], pulls: Pull[], rows: Map, reviewers: Set, -): Priced[] { +): { attributed: Session[]; unattributed: Session[] } { const byBranch = new Map(); for (const each of [...pulls].sort((a, b) => a.created - b.created)) byBranch.set(each.branch, [...(byBranch.get(each.branch) ?? []), each]); - const unattributed: Priced[] = []; - const owned = owners(logs); - sessions.forEach((session, index) => { - const log = logs[index]; - const branches = log ? [...log.branches].sort((a, b) => b[1].last - a[1].last) : []; + const attributed: Session[] = []; + const unattributed: Session[] = []; + const owned = owners(sessions); + for (const session of sessions) { + // Another repository may use the same branch name, so a session counts only where its repository is known. + const known = owned(session); let found: Pull | undefined; - for (const [branch, seen] of branches) { - // Another repository may use the same branch name, so a session counts only where its repository is known. - const known = log!.repository ? [key(log!.repository)] : seen.cwd ? owned(seen.cwd) : null; - if (known === null) continue; + for (const seen of [...session.branches].sort((a, b) => b.last - a.last)) { + if (known === null) break; // A branch can head a new PR after an earlier one closed: take the one open when the session reached it. - const open = (byBranch.get(branch) ?? []).filter( + const open = (byBranch.get(seen.name) ?? []).filter( (each) => (each.closed === null || each.closed >= seen.first) && each.repository !== null && @@ -547,26 +459,27 @@ function attribute( found = open[0]; if (found) break; } - const row = found && rows.get(found.number); - if (!found || !row || !log) { + if (!found) { unattributed.push(session); - return; + continue; } - const id = sessionId(session); + const row = rows.get(found.number); + if (!row) continue; + attributed.push(session); row.sessions.push({ + host: session.host, harness: session.harness, - session: id, + session: session.id, role: - log.review || reviewers.has(id) + session.review || (session.id !== null && reviewers.has(session.id)) ? "reviewing" - : log.started < found.created + : session.start < found.created ? "implementing" : "iterating", - effort: log.effort, models: session.models, }); - }); - return unattributed; + } + return { attributed, unattributed }; } function spendRows(pulls: PullRow[]): SpendRow[] { @@ -574,7 +487,7 @@ function spendRows(pulls: PullRow[]): SpendRow[] { for (const each of pulls) for (const session of each.sessions) for (const spent of session.models) { - const id = JSON.stringify([each.size, session.role, session.harness, spent.model, session.effort]); + const id = JSON.stringify([each.size, session.role, session.harness, spent.model, spent.effort]); let row = rows.get(id); if (!row) { row = { @@ -582,7 +495,7 @@ function spendRows(pulls: PullRow[]): SpendRow[] { role: session.role, harness: session.harness, model: spent.model, - effort: session.effort, + effort: spent.effort, sessions: 0, ...zero(), }; @@ -754,16 +667,20 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe const allRows = pullRows(all, repository); const rows = allRows.filter((row) => pulls.some((each) => each.number === row.number)); const byNumber = new Map(rows.map((row) => [row.number, row])); - const priced = deps.ccusage(options.since); + const exported = deps.sessions(new Date(since), options.ssh ?? []); + const { kept, dropped } = fold(exported ?? []); let unattributed: Usage["unattributed"] = null; - if (priced !== null) { + let incomplete: Session[] = []; + if (exported !== null) { // Every PR in the window competes for a session, so narrowing never hands one PR another's sessions. const reviewers = new Set(allRows.flatMap((row) => row.review.flatMap((review) => review.session ?? []))); - const left = attribute(priced, sessionLogs(priced, deps), all, byNumber, reviewers); + const { attributed, unattributed: left } = attribute(kept, all, byNumber, reviewers); + const reported = narrowed ? attributed : [...attributed, ...left]; if (!narrowed) { unattributed = { sessions: left.length, ...zero() }; for (const session of left) for (const spent of session.models) add(unattributed, spent); } + incomplete = reported.filter((session) => !session.complete); } const online = await readOnlineReviews( client, @@ -774,12 +691,19 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe return { repository: key(repository), since: options.since, - priced: priced !== null, + priced: exported !== null, pulls: rows, spend: spendRows(rows), accuracy: accuracyRows(rows), online: onlineRows(rows), unattributed, + folded: { copies: dropped.length, hosts: [...new Set(dropped.map((session) => session.host))] }, + incomplete: incomplete.map((session) => ({ + host: session.host, + harness: session.harness, + session: session.id, + skippedLines: session.skippedLines, + })), }; } @@ -792,7 +716,7 @@ const compact = (n: number) => ? `${(n / 1e3).toFixed(1)}k` : String(n); const tokens = (t: Tokens) => - `in ${compact(t.input)}, cache ${compact(t.cacheWrite)} written ${compact(t.cacheRead)} read, out ${compact(t.output)}, ~$${t.cost.toFixed(2)}`; + `in ${compact(t.input)}, cache ${compact(t.cacheWrite)} written ${compact(t.cacheRead)} read, out ${compact(t.output)}, ~$${t.cost.toFixed(2)}${t.complete ? "" : " or more"}`; const mean = (sum: number, count: number) => (count ? (sum / count).toFixed(1) : "-"); const percent = (part: number, whole: number) => (whole ? `${Math.round((100 * part) / whole)}%` : "-"); const answers = (r: Omit) => @@ -829,7 +753,15 @@ export function usageLines(report: Usage): string[] { if (row.landFixes.length) lines.push(` land fixes: ${row.landFixes.join("; ")}`); if (row.fixedBy.length) lines.push(` fixed later by: ${row.fixedBy.map((n) => `#${n}`).join(", ")}`); const sessions = row.sessions.flatMap((s) => - s.models.map((m) => [" ", s.role, s.harness, m.model, s.effort ?? "-", tokens(m)]), + s.models.map((m) => [ + " ", + s.role, + s.host ?? "local", + s.harness, + m.model ?? "-", + m.effort ?? "-", + tokens(m), + ]), ); if (sessions.length) lines.push(" sessions:", ...table(sessions)); } @@ -843,7 +775,7 @@ export function usageLines(report: Usage): string[] { s.size ?? "-", s.role, s.harness, - s.model, + s.model ?? "-", s.effort ?? "-", `${s.sessions}x`, tokens(s), @@ -879,10 +811,31 @@ export function usageLines(report: Usage): string[] { ); const u = report.unattributed; if (u) lines.push("", `Unattributed: ${u.sessions} sessions on no PR here, any repository or agent: ${tokens(u)}`); + const host = (name: string | null) => name ?? "local"; + const { folded, incomplete } = report; + if (folded.copies) + lines.push( + "", + `Folded ${folded.copies} ${folded.copies === 1 ? "copy" : "copies"} of sessions found on several hosts, keeping each one's costliest; left out from ${folded.hosts.map(host).join(", ")}`, + ); + if (incomplete.length) + lines.push( + "", + "Lower bounds: these sessions had a request token-max could not price, or a line it could not read or that was unfinished:", + ...table( + incomplete.map((s) => [ + " ", + host(s.host), + s.harness, + s.session ?? "-", + `${s.skippedLines} unreadable lines`, + ]), + ), + ); return lines; } /** The dependencies `quest usage` runs with. */ export function usageDependencies(github: ClaimDependencies): UsageDependencies { - return { ...github, ccusage: (since) => runCcusage(since, process.env), env: process.env, home: homedir() }; + return { ...github, sessions: runTokenMax }; } diff --git a/test/start.test.ts b/test/start.test.ts index 76c0eee30..91d77cd9a 100644 --- a/test/start.test.ts +++ b/test/start.test.ts @@ -840,6 +840,8 @@ test("githubRepository reads HTTPS, ssh://, and scp-like remote URLs", () => { "ssh://git@github.com:22/example/project", "git@github.com:example/project.git", "org-123@github.com:example/project.git", + // An scp-like remote without its user, as token-max exports one after stripping userinfo. + "github.com:example/project.git", ]) { expect(githubRepository(url), url).toEqual(repository); } @@ -847,6 +849,7 @@ test("githubRepository reads HTTPS, ssh://, and scp-like remote URLs", () => { "https://gitlab.com/example/project.git", "https://github.com.evil.example/example/project", "git@github.com:example", + "notgithub.com:example/project", "/srv/git/project.git", ]) { expect(githubRepository(url), url).toBeNull(); diff --git a/test/usage.test.ts b/test/usage.test.ts index f04208a5e..d8cc7d228 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -1,5 +1,5 @@ import { execFileSync } from "node:child_process"; -import { mkdirSync, writeFileSync } from "node:fs"; +import { mkdirSync } from "node:fs"; import { join } from "node:path"; import { describe, expect, test } from "vitest"; @@ -8,7 +8,7 @@ import { implementer } from "../src/github/models"; import { fixes } from "../src/cli/land-fixer"; import type { Round, Tally } from "../src/cli/review"; import { parseReview, reviewRecord } from "../src/cli/review-record"; -import { type Priced, parseCcusage, readClaudeLog, readCodexLog, usage, usageLines } from "../src/cli/usage"; +import { type Session, causes, fold, parseSessions, usage, usageLines } from "../src/cli/usage"; import { tempDir } from "./fixture"; const THREAD = "01a1b8e8-8fca-7793-99d1-062f41f8a7b0"; @@ -17,138 +17,139 @@ const OTHER = "961c731d-399c-46c5-81ef-069105f87cc4"; const OWNED = "f1913a12-44c4-438c-a9d9-06e5b1192c04"; const GONE = "83a08f0e-6ef1-4b41-ad0c-58b37fc993f8"; const BOTH = "5d2c7f3e-8a41-4b6e-9f0d-2c6a1e7b4d93"; +const REMOTE = "a1c2e3f4-0000-4000-8000-000000000001"; +const ITERATE = "11111111-1111-1111-1111-111111111111"; -const lines = (...records: unknown[]) => `${records.map((r) => JSON.stringify(r)).join("\n")}\n`; - -/** A Claude Code log line; the message stands in for a prompt the report must never read out. */ -const claude = (timestamp: string, gitBranch: string, extra: object = {}) => ({ - type: "user", - timestamp, - gitBranch, - message: { content: "SECRET PROMPT" }, - ...extra, -}); - -const codexLog = ( - timestamp: string, - branch: string | null, - extra: { source?: unknown; url?: string; cwd?: string } = {}, -) => - lines( +/** A session as `token-max sessions` prints it, on one model and effort. */ +const exported = (session: string, model: string, effort: string, cost: number, extra: object = {}) => ({ + host: null, + harness: model.startsWith("gpt") ? "codex" : "claude", + session, + parent: null, + review: false, + cwd: "/work", + repositories: ["https://github.com/acme/proj.git"], + repositoriesSource: "checkout", + branches: [{ name: "quest/m0/a", first: "2026-11-06T10:05:00Z", last: "2026-11-06T10:06:00Z" }], + start: "2026-11-06T10:05:00Z", + end: "2026-11-06T10:06:00Z", + models: [ { - type: "session_meta", - timestamp, - payload: { - id: THREAD, - timestamp, - cwd: extra.cwd ?? "/work", - source: extra.source ?? "exec", - git: branch ? { branch, repository_url: extra.url ?? "https://github.com/acme/proj.git" } : undefined, - }, + model, + effort, + requests: 3, + inputTokens: 10, + cacheWriteTokens: 20, + cacheReadTokens: 30, + outputTokens: 40, + costUsd: cost, + complete: true, }, - { type: "response_item", payload: { type: "message", content: "SECRET PROMPT" } }, - { type: "turn_context", payload: { model: "gpt-6.1-sol", effort: "medium" } }, - { type: "turn_context", payload: { model: "gpt-6.1-sol", effort: "high" } }, - ); - -const spent = (model: string, cost: number) => ({ - model, - input: 10, - cacheWrite: 20, - cacheRead: 30, - output: 40, - cost, + ], + costUsd: cost, + complete: true, + skippedLines: 0, + files: ["/home/me/.claude/projects/-work/x.jsonl"], + ...extra, }); -describe("parseCcusage", () => { - test("reads each session's agent, ID, and spend per model", () => { - const text = JSON.stringify({ - session: [ +const parse = (...rows: object[]) => parseSessions(rows.map((row) => JSON.stringify(row)).join("\n") + "\n"); + +describe("parseSessions", () => { + test("reads each session's host, ID, repositories, branches, and spend per model and effort", () => { + const [session] = parse( + exported(THREAD, "gpt-6.1-sol", "high", 0.5, { + host: "box", + repositories: ["https://github.com/Acme/Proj.git", "https://gitlab.com/acme/proj.git"], + }), + ); + expect(session).toEqual({ + host: "box", + harness: "codex", + id: THREAD, + review: false, + cwd: "/work", + repositories: ["acme/proj"], + branches: [ { - agent: "codex", - period: `2026/11/06/rollout-2026-11-06T20-23-41-${THREAD}`, - metadata: { lastActivity: "2026-11-07T00:00:00Z" }, - modelBreakdowns: [ - { - modelName: "gpt-6.1-sol", - inputTokens: 1, - cacheCreationTokens: 0, - cacheReadTokens: 2, - outputTokens: 3, - cost: 0.5, - }, - ], + name: "quest/m0/a", + first: Date.parse("2026-11-06T10:05:00Z"), + last: Date.parse("2026-11-06T10:06:00Z"), }, ], - totals: {}, + start: Date.parse("2026-11-06T10:05:00Z"), + models: [ + { + model: "gpt-6.1-sol", + effort: "high", + input: 10, + cacheWrite: 20, + cacheRead: 30, + output: 40, + cost: 0.5, + complete: true, + }, + ], + cost: 0.5, + complete: true, + skippedLines: 0, }); - expect(parseCcusage(text)).toEqual([ - { - harness: "codex", - id: `2026/11/06/rollout-2026-11-06T20-23-41-${THREAD}`, - models: [{ model: "gpt-6.1-sol", input: 1, cacheWrite: 0, cacheRead: 2, output: 3, cost: 0.5 }], - }, - ]); + // A session token-max could not resolve a repository, model, or ID for keeps the gap as null. + const [unknown] = parse({ + ...exported(THREAD, "gpt-6.1-sol", "high", 1, { repositories: null, session: null }), + models: [{ ...exported(THREAD, "x", "y", 1).models[0], model: null, effort: null }], + }); + expect([unknown!.id, unknown!.repositories, unknown!.models[0]!.model]).toEqual([null, null, null]); }); - test("fails on a session missing a field", () => { - expect(() => parseCcusage(JSON.stringify({ session: [{ agent: "claude", period: "x" }] }))).toThrow( - /modelBreakdowns/, - ); - expect(() => parseCcusage(JSON.stringify({ daily: [] }))).toThrow(/no session list/); + test("fails on a session missing a field, and on a line that is not JSON", () => { + const { skippedLines: _, ...missing } = exported(THREAD, "gpt-6.1-sol", "high", 1); + expect(() => parse(missing)).toThrow(/skipped line count/); + expect(() => parse(exported(THREAD, "gpt-6.1-sol", "high", 1, { start: "soon" }))).toThrow(/a start/); + expect(() => parseSessions('{"host"\n')).toThrow(); }); }); -describe("session logs", () => { - test("a Claude Code log gives each branch's first and last time, and the last reply's effort", () => { - const path = join(tempDir(), "log.jsonl"); - writeFileSync( - path, - lines( - { type: "custom-title", customTitle: "no timestamp" }, - claude("2026-11-06T10:00:00Z", "t3/scratch"), - claude("2026-11-06T10:05:00Z", "quest/m0/a"), - { type: "assistant", timestamp: "2026-11-06T10:06:00Z", gitBranch: "quest/m0/a", effort: "medium" }, - claude("2026-11-06T10:07:00Z", "HEAD"), - { type: "assistant", timestamp: "2026-11-06T10:09:00Z", gitBranch: "quest/m0/a", effort: "high" }, - ) + '{"type":"user","timestamp":"2026-11-06T11:00', +describe("fold", () => { + test("keeps each session's costliest copy across hosts, and leaves alone sessions without an ID", () => { + const [here, there, alone, anonymous, again] = parse( + exported(CLAUDE, "claude-opus-5-5", "high", 2), + exported(CLAUDE, "claude-opus-5-5", "high", 5, { host: "box" }), + exported(OTHER, "claude-opus-5-5", "high", 1, { host: "box" }), + exported(THREAD, "gpt-6.1-sol", "high", 1, { session: null }), + exported(THREAD, "gpt-6.1-sol", "high", 1, { session: null, host: "box" }), ); - const log = readClaudeLog(path); - expect(log.started).toBe(Date.parse("2026-11-06T10:00:00Z")); - expect([...log.branches]).toEqual([ - [ - "t3/scratch", - { first: Date.parse("2026-11-06T10:00:00Z"), last: Date.parse("2026-11-06T10:00:00Z"), cwd: null }, - ], - [ - "quest/m0/a", - { first: Date.parse("2026-11-06T10:05:00Z"), last: Date.parse("2026-11-06T10:09:00Z"), cwd: null }, - ], - ]); - expect(log.effort).toBe("high"); - expect(log.repository).toBeNull(); - }); - - test("a malformed line before the last fails", () => { - const path = join(tempDir(), "log.jsonl"); - writeFileSync(path, `{"oops"\n${lines(claude("2026-11-06T10:00:00Z", "main"))}`); - expect(() => readClaudeLog(path)).toThrow(/not JSON/); + const { kept, dropped } = fold([here!, there!, alone!, anonymous!, again!]); + expect(kept).toEqual([there, alone, anonymous, again]); + expect(dropped).toEqual([here]); }); +}); - test("a Codex rollout gives its branch, remote, review mark, and last turn's effort", () => { - const path = join(tempDir(), "rollout.jsonl"); - writeFileSync(path, codexLog("2026-11-06T10:00:00Z", "quest/m0/a", { source: { subagent: "review" } })); - const log = readCodexLog(path); - expect([...log.branches.keys()]).toEqual(["quest/m0/a"]); - expect(log.repository).toEqual({ owner: "acme", repo: "proj" }); - expect(log.review).toBe(true); - expect(log.effort).toBe("high"); - }); +describe("causes", () => { + const repository = { owner: "moq-dev", repo: "moq" }; + const link = (n: number) => `[moq-dev/moq#${n}](https://github.com/moq-dev/moq/pull/${n})`; + const body = (section: string) => `Fix.\n\n## Caused by\n\n${section}\n\n## Models\n\n- Size: S\n`; - test("a Codex rollout that does not start with its session_meta fails", () => { - const path = join(tempDir(), "rollout.jsonl"); - writeFileSync(path, lines({ type: "turn_context", payload: {} })); - expect(() => readCodexLog(path)).toThrow(/session_meta/); + test("counts only a bullet that opens with the culprit's link and says why", () => { + // The shapes the 2026-11-11 mistake audit found counted by mistake, from moq-dev/moq's PRs. + expect(causes(body(`No bug on \`main\`; the \`release\` fix is ${link(8469)}.`), repository)).toEqual([]); + expect( + causes(body(`- No single PR: see ${link(8441)}.\n\nMerged \`main\`: ${link(8441)}.`), repository), + ).toEqual([]); + expect(causes(body(`None: new coverage for the models ${link(8447)} added.`), repository)).toEqual([]); + expect(causes(body(`- ${link(8054)} - \`ready\` waits (fixed by ${link(8351)})`), repository)).toEqual([8054]); + expect(causes(body(`- ${link(8459)} put a shaper profile among the fixtures.`), repository)).toEqual([]); + // Another repository's culprit, or a link whose text names another PR, is no culprit here. + expect( + causes(body(`- [moq-dev/quest#494](https://github.com/moq-dev/quest/pull/494) - a gap`), repository), + ).toEqual([]); + expect(causes(body(`- [moq-dev/moq#1](https://github.com/moq-dev/moq/pull/2) - mismatch`), repository)).toEqual( + [], + ); + expect(causes(body(`- ${link(7)} - one\r\n- ${link(8)} - two\n- ${link(7)} - again`), repository)).toEqual([ + 7, 8, + ]); + expect(causes(`- ${link(7)} - outside the section`, repository)).toEqual([]); }); }); @@ -381,58 +382,53 @@ describe("usage", () => { execFileSync("git", ["init", "-q"], { cwd: root }); execFileSync("git", ["remote", "add", "origin", "https://github.com/acme/proj.git"], { cwd: root }); - const claudeHome = join(dir, "claude"); - mkdirSync(join(claudeHome, "projects", "-work"), { recursive: true }); - writeFileSync( - join(claudeHome, "projects", "-work", `${CLAUDE}.jsonl`), - lines( - claude("2026-11-06T09:00:00Z", "t3/scratch"), - claude("2026-11-06T10:05:00Z", "quest/m0/a", { cwd: root }), - { type: "assistant", timestamp: "2026-11-06T10:06:00Z", gitBranch: "quest/m0/a", effort: "high" }, - ), - ); - // A session in a checkout of another repository, on a branch of the same name. - const other = join(dir, "other"); - mkdirSync(other); - execFileSync("git", ["init", "-q"], { cwd: other }); - execFileSync("git", ["remote", "add", "origin", "git@github.com:other/repo.git"], { cwd: other }); - writeFileSync( - join(claudeHome, "projects", "-work", `${OTHER}.jsonl`), - lines(claude("2026-11-06T10:05:00Z", "quest/m0/a", { cwd: join(other, "sub") })), - ); - mkdirSync(join(other, "sub")); - // A session in a checkout with remotes for both the fork and its upstream, on a branch each has a PR from. - const both = join(dir, "both"); - mkdirSync(both); - execFileSync("git", ["init", "-q"], { cwd: both }); - execFileSync("git", ["remote", "add", "origin", "https://github.com/fork/proj.git"], { cwd: both }); - execFileSync("git", ["remote", "add", "upstream", "https://github.com/acme/proj.git"], { cwd: both }); - writeFileSync( - join(claudeHome, "projects", "-work", `${BOTH}.jsonl`), - lines(claude("2026-11-06T10:05:00Z", "quest/m0/a", { cwd: both })), - ); - // Two sessions in checkouts since deleted: one in the directory a Codex log names, and one in a clone nested - // below it, which may be another repository's and so is not charged to this one. - writeFileSync( - join(claudeHome, "projects", "-work", `${OWNED}.jsonl`), - lines(claude("2026-11-06T14:00:00Z", "quest/m0/a", { cwd: "/work" })), - ); - writeFileSync( - join(claudeHome, "projects", "-work", `${GONE}.jsonl`), - lines(claude("2026-11-06T14:00:00Z", "quest/m0/a", { cwd: "/work/nested" })), - ); - const codexHome = join(dir, "codex"); - const day = join(codexHome, "sessions", "2026", "11", "06"); - mkdirSync(day, { recursive: true }); - // The reviewer, named by #10's record; an iterate on #10; one on another repository's branch of the same name. - writeFileSync(join(day, `rollout-a-${THREAD}.jsonl`), codexLog("2026-11-06T12:00:00Z", "quest/m0/a")); - writeFileSync( - join(day, "rollout-b-11111111-1111-1111-1111-111111111111.jsonl"), - codexLog("2026-11-06T13:00:00Z", "quest/m0/a"), - ); - writeFileSync( - join(day, "rollout-c-22222222-2222-2222-2222-222222222222.jsonl"), - codexLog("2026-11-06T13:00:00Z", "quest/m0/a", { url: "https://github.com/other/repo.git", cwd: "/other" }), + const sessions: Session[] = parse( + exported(CLAUDE, "claude-opus-5-5", "high", 5, { + branches: [ + { name: "t3/scratch", first: "2026-11-06T09:00:00Z", last: "2026-11-06T09:00:00Z" }, + { name: "quest/m0/a", first: "2026-11-06T10:05:00Z", last: "2026-11-06T10:06:00Z" }, + ], + start: "2026-11-06T09:00:00Z", + }), + // A session in a checkout of another repository, on a branch of the same name. + exported(OTHER, "claude-opus-5-5", "high", 6, { + cwd: "/other/sub", + repositories: ["git@github.com:other/repo.git"], + }), + // The reviewer, named by #10's record; an iterate on #10, with a cheaper copy on another host; one on + // another repository's branch of the same name. + exported(THREAD, "gpt-6.1-sol", "high", 1, { start: "2026-11-06T12:00:00Z" }), + exported(ITERATE, "gpt-6.1-sol", "high", 2, { start: "2026-11-06T13:00:00Z" }), + exported(ITERATE, "gpt-6.1-sol", "high", 1, { start: "2026-11-06T13:00:00Z", host: "box" }), + exported("22222222-2222-2222-2222-222222222222", "gpt-6.1-sol", "high", 3, { + cwd: "/other", + repositories: ["https://github.com/other/repo.git"], + start: "2026-11-06T13:00:00Z", + }), + // Two sessions in checkouts since deleted: one in the directory the Codex sessions name, and one in a + // clone nested below it, which may be another repository's and so is not charged to this one. The + // nested one had lines token-max could not read, so its cost is a lower bound. + exported(OWNED, "claude-opus-5-5", "unknown", 7, { + repositories: null, + start: "2026-11-06T14:00:00Z", + }), + { + ...exported(GONE, "claude-opus-5-5", "high", 8, { + cwd: "/work/nested", + repositories: null, + start: "2026-11-06T14:00:00Z", + complete: false, + skippedLines: 2, + }), + models: [{ ...exported(GONE, "claude-opus-5-5", "high", 8).models[0], complete: false }], + }, + // Another host's session in a directory of the same path: this host's sessions there tell nothing of it. + exported(REMOTE, "claude-opus-5-5", "high", 4, { host: "box", repositories: null }), + // A checkout with remotes for both the fork and its upstream, on a branch each has a PR from. + exported(BOTH, "claude-opus-5-5", "high", 9, { + cwd: "/both", + repositories: ["https://github.com/fork/proj.git", "https://github.com/acme/proj.git"], + }), ); const body10 = [ @@ -479,6 +475,7 @@ describe("usage", () => { const fork13 = body10.slice(body10.indexOf("## Review")).replaceAll(THREAD, "thread-fork"); const body11 = "Fix.\n\n## Caused by\n\n- [acme/proj#10](https://github.com/acme/proj/pull/10) - missed a case\n"; + // A closed duplicate of #11's fix: it never landed, so it fixed nothing. const pulls = [ pr(11, "fix/a", body11, "2026-11-06T20:00:00Z"), pr(10, "quest/m0/a", body10, "2026-11-06T10:00:00Z"), @@ -538,6 +535,7 @@ describe("usage", () => { updated_at: "2026-11-06T00:00:00Z", }), ), + pr(12, "fix/a-again", body11, "2026-11-06T21:00:00Z", { merged_at: null }), pr(8, "old", "", "2026-10-01T00:00:00Z", { updated_at: "2026-10-01T00:00:00Z" }), ]; const requests: string[] = []; @@ -622,25 +620,8 @@ describe("usage", () => { if (url.pathname === "/repos/acme/proj/pulls") return json(pulls); return new Response("not found", { status: 404 }); }; - const priced: Priced[] = [ - { harness: "claude", id: CLAUDE, models: [spent("claude-opus-5-5", 5)] }, - { harness: "claude", id: OTHER, models: [spent("claude-opus-5-5", 6)] }, - { harness: "codex", id: `2026/11/06/rollout-a-${THREAD}`, models: [spent("gpt-6.1-sol", 1)] }, - { - harness: "codex", - id: "2026/11/06/rollout-b-11111111-1111-1111-1111-111111111111", - models: [spent("gpt-6.1-sol", 2)], - }, - { - harness: "codex", - id: "2026/11/06/rollout-c-22222222-2222-2222-2222-222222222222", - models: [spent("gpt-6.1-sol", 3)], - }, - { harness: "claude", id: OWNED, models: [spent("claude-opus-5-5", 7)] }, - { harness: "claude", id: GONE, models: [spent("claude-opus-5-5", 8)] }, - { harness: "claude", id: BOTH, models: [spent("claude-opus-5-5", 9)] }, - ]; - const deps = (ccusage: Priced[] | null) => ({ + const calls: [Date, string[]][] = []; + const deps = (exported: Session[] | null) => ({ github: () => ({ fetch: fetch as typeof globalThis.fetch, now: Date.now, @@ -649,25 +630,30 @@ describe("usage", () => { warn: () => {}, }), warn: () => {}, - ccusage: () => ccusage, - env: { CLAUDE_CONFIG_DIR: claudeHome, CODEX_HOME: codexHome }, - home: dir, + sessions: (since: Date, ssh: string[]) => { + calls.push([since, ssh]); + return exported; + }, }); - return { root, deps, priced, requests }; + return { root, deps, sessions, calls, requests }; } test("ties each session to its PR and role, and rolls up spend and accuracy", async () => { - const { root, deps, priced } = setup(); - const report = await usage(root, { since: "2026-11-05" }, deps(priced)); - expect(report.pulls.map((p) => p.number)).toEqual([11, 10, 9, 7, 13, 5, 6, 3, 4]); + const { root, deps, sessions, calls } = setup(); + const report = await usage(root, { since: "2026-11-05", ssh: ["box"] }, deps(sessions)); + expect(calls).toEqual([[new Date("2026-11-05T00:00:00"), ["box"]]]); + expect(report.pulls.map((p) => p.number)).toEqual([11, 10, 9, 7, 13, 5, 6, 3, 4, 12]); const ten = report.pulls.find((p) => p.number === 10)!; - expect(ten.sessions.map((s) => [s.harness, s.session, s.role, s.effort])).toEqual([ - ["claude", CLAUDE, "implementing", "high"], - ["codex", THREAD, "reviewing", "high"], - ["codex", "11111111-1111-1111-1111-111111111111", "iterating", "high"], - ["claude", OWNED, "iterating", null], + expect(ten.sessions.map((s) => [s.host, s.harness, s.session, s.role, s.models[0]!.effort])).toEqual([ + [null, "claude", CLAUDE, "implementing", "high"], + [null, "codex", THREAD, "reviewing", "high"], + [null, "codex", ITERATE, "iterating", "high"], + [null, "claude", OWNED, "iterating", "unknown"], ]); + // #12 names #10 too, but closed without merging. expect(ten.fixedBy).toEqual([11]); + expect(report.folded).toEqual({ copies: 1, hosts: ["box"] }); + expect(report.incomplete).toEqual([{ host: null, harness: "claude", session: GONE, skippedLines: 2 }]); expect(ten.implementers).toEqual([ { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: null }, { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: "L" }, @@ -678,12 +664,13 @@ describe("usage", () => { expect(report.pulls.find((p) => p.number === 9)!.sessions).toEqual([]); expect(report.pulls.find((p) => p.number === 7)!.sessions).toEqual([]); expect(report.unattributed).toEqual({ - sessions: 4, - input: 40, - cacheWrite: 80, - cacheRead: 120, - output: 160, - cost: 26, + sessions: 5, + input: 50, + cacheWrite: 100, + cacheRead: 150, + output: 200, + cost: 30, + complete: false, }); expect(report.spend.map((s) => [s.size, s.role, s.harness, s.cost])).toEqual([ ["L", "implementing", "claude", 5], @@ -764,19 +751,31 @@ describe("usage", () => { expect(text).toContain( "rabbit[bot] reviewed 1/4 (25%), final head 1 (25%), 1.0 findings per PR reviewed, 100% accepted;", ); - expect(text).toContain("fixed later by: #11"); + expect(text).toContain("fixed later by: #11\n"); + expect(text).toContain( + " implementing local claude claude-opus-5-5 high in 10, cache 20 written 30 read, out 40, ~$5.00\n", + ); + expect(text).toContain( + "Unattributed: 5 sessions on no PR here, any repository or agent: in 50, cache 100 written 150 read, out 200, ~$30.00 or more\n", + ); + expect(text).toContain( + "Folded 1 copy of sessions found on several hosts, keeping each one's costliest; left out from box\n", + ); + expect(text).toContain(` local claude ${GONE} 2 unreadable lines`); expect(text).not.toContain("SECRET"); expect(JSON.stringify(report)).not.toContain("SECRET"); }); - test("narrows to one PR, and reports the records alone without ccusage", async () => { - const { root, deps, priced } = setup(); - const one = await usage(root, { since: "2026-11-05", pr: 10 }, deps(priced)); + test("narrows to one PR, and reports the records alone without token-max", async () => { + const { root, deps, sessions } = setup(); + const one = await usage(root, { since: "2026-11-05", pr: 10 }, deps(sessions)); expect(one.pulls.map((p) => p.number)).toEqual([10]); expect(one.pulls[0]!.fixedBy).toEqual([11]); expect(one.unattributed).toBeNull(); + // The unreadable lines were an unattributed session's, which a narrowed report leaves out. + expect(one.incomplete).toEqual([]); // #9 shares #10's branch but closed before any session reached it, so narrowing to it hands it none. - const nine = await usage(root, { since: "2026-11-05", pr: 9 }, deps(priced)); + const nine = await usage(root, { since: "2026-11-05", pr: 9 }, deps(sessions)); expect(nine.pulls.map((p) => [p.number, p.sessions.length])).toEqual([[9, 0]]); const quest = await usage(root, { since: "2026-11-05", quest: "quest/m0/a.md" }, deps(null)); expect(quest.priced).toBe(false); From f04ec8c77ade3f96ce7a3aa151eda12703b27f77 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 14:25:49 -0800 Subject: [PATCH 3/6] quest: plan quest messages reading the transcripts it fails on today Co-Authored-By: Claude Opus 5.5 --- quest/m0/messages-robustness.md | 38 +++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 quest/m0/messages-robustness.md diff --git a/quest/m0/messages-robustness.md b/quest/m0/messages-robustness.md new file mode 100644 index 000000000..b900afdb1 --- /dev/null +++ b/quest/m0/messages-robustness.md @@ -0,0 +1,38 @@ +# [S] quest messages reads transcripts it fails on today + +## Goal + +`quest messages` reads every Claude Code and Codex transcript the +maintainer's hosts write today, without failing on two shapes it refuses +now, while still failing on a shape it does not know. + +- A malformed line in a Claude Code transcript, such as an interleaved + partial record in the middle of the file, is skipped and counted, as + token-max does, and the output says how many lines it skipped in which + sessions. Today `parseLine` in `src/cli/messages.ts` throws on any + complete line that is not JSON, the failure `quest usage` had before it + read sessions through token-max. +- A Codex 0.162 TUI turn that carries `content_item_kinds` with no + `content_item_metadata` provenance counts its `user.text` parts as + typed and sets aside injected kinds, such as `agents_md.instructions` + and `environments.environment_context`. A kind it does not know still + fails, and the refusal stays for parts whose kind says nothing: all + `unknown`, except a compaction replay. + +## Plan + +Decided 2026-11-13 (user, /quest-run, answering +[moq-dev/quest#704](https://github.com/moq-dev/quest/pull/704)): one quest +for both causes (rejected: a quest each; only the Codex one). + +- Cause B was salvaged from the closed + [moq-dev/quest#696](https://github.com/moq-dev/quest/pull/696). It failed + `quest messages` on two of the maintainer's three hosts, in 4 and 21 + sessions, all from codex-tui 0.162.1. `readCodexMessages` accepts kinds + only when all are `unknown`, and `test/messages.test.ts` expects + `["user.text", "unknown"]` to throw; that expectation flips. +- Its PR lists under `## Caused by` + [moq-dev/quest#626](https://github.com/moq-dev/quest/pull/626), which + required provenance on every typed Codex turn. Cause A is as old as the + reader. +- Reproduce each on a real transcript before fixing it. From 133eeb6160bad5d22572e162bc24f55a1f4697d8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 14:29:55 -0800 Subject: [PATCH 4/6] fix(usage): a session's gaps reach each model, and folded copies still name repositories Co-Authored-By: Claude Opus 5.5 --- README.md | 5 +++-- src/cli/main.ts | 4 ++-- src/cli/usage.ts | 18 ++++++++++++------ test/usage.test.ts | 29 ++++++++++++++++++++++++++++- 4 files changed, 45 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 364591f40..96f407782 100644 --- a/README.md +++ b/README.md @@ -1637,8 +1637,9 @@ for it on its own host name, or, once its checkout is deleted, those resolved there for another session in that exact directory, as a Codex log records (never a parent's, which a nested clone may not share); otherwise it stays unattributed. A session is reviewing when the PR's `## Review` record names -it or Codex marks it a review, implementing when it began before the PR opened, -and iterating otherwise. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the +it or Codex marks it a review, implementing when its first request since +`--since` came before the PR opened, and iterating otherwise, so a session +that opened a PR before `--since` and kept working on it reads as iterating. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the `quest review` record names, with its latest round and score and its tally of findings across every round, marked `shadow` for the shadow, where a merged PR whose last round left only low findings open counts them settled, since `/quest-review` fixes or refutes those diff --git a/src/cli/main.ts b/src/cli/main.ts index ff68f199a..8078869b4 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1162,8 +1162,8 @@ export function main(argv: string[], out: Output): number | Promise { "price; its sub-agents are sessions of their own. Each Claude Code or Codex session belongs to the PR whose " + "head branch it last names, when a repository token-max resolved for it on its own host, or for another " + "session in the same directory there, is this one; no prompt or reply is read out. Its role is " + - "reviewing when the PR's ## Review record names it or Codex marks it a review, implementing when it began " + - "before the PR opened, and iterating otherwise. A session found on several hosts counts once, at its " + + "reviewing when the PR's ## Review record names it or Codex marks it a review, implementing when its first " + + "request since --since came before the PR opened, and iterating otherwise. A session found on several hosts counts once, at its " + "costliest copy, and the report says how many copies it folded from which hosts; a session with a request " + "token-max could not price or a line it could not read is listed, its cost a lower bound. A host that fails fails the report. " + "Each PR also shows its ## Models size and implementers, and as unread each Implementer line it cannot read, " + diff --git a/src/cli/usage.ts b/src/cli/usage.ts index f54b59132..b8e520286 100644 --- a/src/cli/usage.ts +++ b/src/cli/usage.ts @@ -214,6 +214,9 @@ export function parseSessions(text: string): Session[] { if (repositories !== null && !Array.isArray(repositories)) fail("a repository list"); const branches = Array.isArray(row.branches) ? row.branches : fail("a branch list"); const models = Array.isArray(row.models) ? row.models : fail("a model list"); + // A session's unreadable or unfinished lines may hide any model's requests, so each model's cost is then + // a lower bound too, though token-max marks only the session. + const complete = flag(row.complete, "its completeness"); return { host: optional(row.host, "a host"), harness: word(row.harness, "a harness"), @@ -242,10 +245,10 @@ export function parseSessions(text: string): Session[] { cacheRead: count(m.cacheReadTokens, "a cache read token count"), output: count(m.outputTokens, "an output token count"), cost: count(m.costUsd, "a model's cost"), - complete: flag(m.complete, "a model's completeness"), + complete: flag(m.complete, "a model's completeness") && complete, })), cost: count(row.costUsd, "a cost"), - complete: flag(row.complete, "its completeness"), + complete, skippedLines: count(row.skippedLines, "a skipped line count"), }; }); @@ -426,11 +429,14 @@ function owners(sessions: Session[]): (session: Session) => string[] | null { /** * Attribute each session to the PR whose head branch it last named, among those still open when the session * reached the branch and headed from a repository its host resolved for it, and give it a role: reviewing when a - * PR's review record names it or Codex marks it a review, implementing when it began before its PR opened, and - * iterating otherwise. Returns the sessions it gave a PR in `rows`, and those on no PR here. + * PR's review record names it or Codex marks it a review, implementing when its first request in the window came + * before its PR opened, and iterating otherwise. `evidence` holds every exported copy, folded ones included, since a + * dropped copy may be the only one naming its directory's repository. Returns the sessions it gave a PR in `rows`, + * and those on no PR here. */ function attribute( sessions: Session[], + evidence: Session[], pulls: Pull[], rows: Map, reviewers: Set, @@ -440,7 +446,7 @@ function attribute( byBranch.set(each.branch, [...(byBranch.get(each.branch) ?? []), each]); const attributed: Session[] = []; const unattributed: Session[] = []; - const owned = owners(sessions); + const owned = owners(evidence); for (const session of sessions) { // Another repository may use the same branch name, so a session counts only where its repository is known. const known = owned(session); @@ -674,7 +680,7 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe if (exported !== null) { // Every PR in the window competes for a session, so narrowing never hands one PR another's sessions. const reviewers = new Set(allRows.flatMap((row) => row.review.flatMap((review) => review.session ?? []))); - const { attributed, unattributed: left } = attribute(kept, all, byNumber, reviewers); + const { attributed, unattributed: left } = attribute(kept, exported ?? [], all, byNumber, reviewers); const reported = narrowed ? attributed : [...attributed, ...left]; if (!narrowed) { unattributed = { sessions: left.length, ...zero() }; diff --git a/test/usage.test.ts b/test/usage.test.ts index 8b38ed057..de095db59 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -399,7 +399,11 @@ describe("usage", () => { // another repository's branch of the same name. exported(THREAD, "gpt-6.1-sol", "high", 1, { start: "2026-11-06T12:00:00Z" }), exported(ITERATE, "gpt-6.1-sol", "high", 2, { start: "2026-11-06T13:00:00Z" }), - exported(ITERATE, "gpt-6.1-sol", "high", 1, { start: "2026-11-06T13:00:00Z", host: "box" }), + exported(ITERATE, "gpt-6.1-sol", "high", 1, { + start: "2026-11-06T13:00:00Z", + host: "box", + repositories: null, + }), exported("22222222-2222-2222-2222-222222222222", "gpt-6.1-sol", "high", 3, { cwd: "/other", repositories: ["https://github.com/other/repo.git"], @@ -767,6 +771,29 @@ describe("usage", () => { expect(JSON.stringify(report)).not.toContain("SECRET"); }); + test("a folded copy still names its directory's repository, and a session's gaps reach each model", async () => { + const { root, deps } = setup(); + // On `box`, a deleted checkout's only repository evidence is a copy that folds into this host's costlier one. + const sessions = parse( + exported(THREAD, "gpt-6.1-sol", "high", 2, { cwd: "/gone", branches: [] }), + exported(THREAD, "gpt-6.1-sol", "high", 1, { cwd: "/gone", branches: [], host: "box" }), + exported(OWNED, "claude-opus-5-5", "high", 7, { + cwd: "/gone", + host: "box", + repositories: null, + start: "2026-11-06T14:00:00Z", + complete: false, + }), + ); + expect(sessions[2]!.models[0]!.complete).toBe(false); + const report = await usage(root, { since: "2026-11-05", pr: 10 }, deps(sessions)); + expect(report.folded).toEqual({ copies: 1, hosts: ["box"] }); + expect(report.pulls[0]!.sessions.map((s) => [s.host, s.session, s.role])).toEqual([ + ["box", OWNED, "iterating"], + ]); + expect(usageLines(report).join("\n")).toContain("~$7.00 or more"); + }); + test("narrows to one PR, and reports the records alone without token-max", async () => { const { root, deps, sessions } = setup(); const one = await usage(root, { since: "2026-11-05", pr: 10 }, deps(sessions)); From 4f207906706caf8a1778a626aace84f909270941 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 15:05:33 -0800 Subject: [PATCH 5/6] fix(usage): a narrowed report counts only its sessions' folded copies Co-Authored-By: Claude Opus 5.5 --- src/cli/usage.ts | 6 +++++- test/usage.test.ts | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/src/cli/usage.ts b/src/cli/usage.ts index b8e520286..4ad408df6 100644 --- a/src/cli/usage.ts +++ b/src/cli/usage.ts @@ -677,6 +677,7 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe const { kept, dropped } = fold(exported ?? []); let unattributed: Usage["unattributed"] = null; let incomplete: Session[] = []; + let folded: Session[] = dropped; if (exported !== null) { // Every PR in the window competes for a session, so narrowing never hands one PR another's sessions. const reviewers = new Set(allRows.flatMap((row) => row.review.flatMap((review) => review.session ?? []))); @@ -687,6 +688,9 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe for (const session of left) for (const spent of session.models) add(unattributed, spent); } incomplete = reported.filter((session) => !session.complete); + // A narrowed report counts only the folded copies of the sessions it reports. + const ids = new Set(reported.map((session) => `${session.harness}\0${session.id}`)); + folded = dropped.filter((session) => ids.has(`${session.harness}\0${session.id}`)); } const online = await readOnlineReviews( client, @@ -703,7 +707,7 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe accuracy: accuracyRows(rows), online: onlineRows(rows), unattributed, - folded: { copies: dropped.length, hosts: [...new Set(dropped.map((session) => session.host))] }, + folded: { copies: folded.length, hosts: [...new Set(folded.map((session) => session.host))] }, incomplete: incomplete.map((session) => ({ host: session.host, harness: session.harness, diff --git a/test/usage.test.ts b/test/usage.test.ts index de095db59..19254d2bc 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -787,7 +787,12 @@ describe("usage", () => { ); expect(sessions[2]!.models[0]!.complete).toBe(false); const report = await usage(root, { since: "2026-11-05", pr: 10 }, deps(sessions)); - expect(report.folded).toEqual({ copies: 1, hosts: ["box"] }); + // The folded copy's session is unattributed, so a report narrowed to #10 leaves it out. + expect(report.folded).toEqual({ copies: 0, hosts: [] }); + expect((await usage(root, { since: "2026-11-05" }, deps(sessions))).folded).toEqual({ + copies: 1, + hosts: ["box"], + }); expect(report.pulls[0]!.sessions.map((s) => [s.host, s.session, s.role])).toEqual([ ["box", OWNED, "iterating"], ]); From 1a70998c449ae876dbe2297a7a1bf3250f275665 Mon Sep 17 00:00:00 2001 From: quest fixer Date: Fri, 13 Nov 2026 15:48:19 -0800 Subject: [PATCH 6/6] fix(docs): point session pricing reference to its completed PR --- quest/m0/benchmark/records.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quest/m0/benchmark/records.md b/quest/m0/benchmark/records.md index e798fc2a4..7b9eb0424 100644 --- a/quest/m0/benchmark/records.md +++ b/quest/m0/benchmark/records.md @@ -49,5 +49,5 @@ Agent's choices, for the implementer to revisit: - [Development as a benchmark](/quest/m0/benchmark/README.md) - the decisions this builds on -- [Price sessions through token-max](/quest/m0/routing-evidence/usage-token-max.md) - +- [Price sessions through token-max](https://github.com/moq-dev/quest/pull/704) - the per-session pricing these records reuse