Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 37 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1476,9 +1476,9 @@ own retries, which also counts as the limit there.
## Audit cost and accuracy

`quest usage` reports `origin`'s PRs updated since `--since` (a local
`YYYY-MM-DD`, by default 7 days ago) and the agent sessions this host ran
since then: which harness, model, and effort worked on each PR, in which role,
at what token cost, and how the work went.
`YYYY-MM-DD`, by default 7 days ago) and the agent sessions this host, and
each host `--ssh` names, ran since then: which harness, model, and effort
worked on each PR, in which role, at what token cost, and how the work went.

```console
$ quest usage --pr 569
Expand All @@ -1490,27 +1490,33 @@ moq-dev/quest, PRs updated and sessions active since 2026-11-06; costs (~$) esti
online review: review (OpenAI), not at the final head; 0 findings: 0 accepted, 0 declined, 0 unclassified, 0 unanswered
land fixes: codex, gpt-6.1-sol, medium effort
sessions:
implementing claude claude-opus-5-5 low in 50, cache 36.2k written 1.1M read, out 6.1k, ~$0.63
reviewing codex gpt-6.1-sol medium in 14.7k, cache 0 written 39.8k read, out 421, ~$0.04
implementing local claude claude-opus-5-5 low in 50, cache 36.2k written 1.1M read, out 6.1k, ~$0.63
reviewing local codex gpt-6.1-sol medium in 14.7k, cache 0 written 39.8k read, out 421, ~$0.04
```

[ccusage](https://github.com/ccusage/ccusage)'s `session --json` reads and
[token-max](https://github.com/kixel-ai/token-max)'s `sessions` reads and
prices the sessions, at each model's API list price, which subscriptions do
not bill, so the cost is an estimate. `quest usage` ties each Claude Code or
Codex session to the PR whose head branch its log last names, reading only the
branch, directory, remote, start time, and effort from `$CLAUDE_CONFIG_DIR` (or
`~/.config/claude` and `~/.claude`) and `$CODEX_HOME` (or `~/.codex`), never a
not bill, so the cost is an estimate. A Claude sub-agent or a Codex review or
spawned thread is a session of its own, so a quest a sub-agent worked on
carries its own cost, not its coordinator's. `quest usage` ties each Claude
Code or Codex session to the PR whose head branch it last names, from the
branches, directory, repositories, and start token-max exports for it, never a
prompt or reply. Since another repository may use the same branch name, the
session must also be this repository's, as its Codex log, its checkout's
remotes, or, once the checkout is deleted, a Codex log for that exact directory
names (never a parent's, which a nested clone may not share); otherwise it stays unattributed. A session is reviewing when the PR's `## Review` record names
it or Codex marks it a review, implementing when it began before the PR opened,
and iterating otherwise. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the
session must also be this repository's, as the repositories token-max resolved
for it on its own host name, or, once its checkout is deleted, those resolved
there for another session in that exact directory, as a Codex log records
(never a parent's, which a nested clone may not share); otherwise it stays
unattributed. A session is reviewing when the PR's `## Review` record names
it or Codex marks it a review, implementing when its first request since
`--since` came before the PR opened, and iterating otherwise, so a session
that opened a PR before `--since` and kept working on it reads as iterating. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the
`quest review` record names, with its latest round and score and its tally of
findings across every round, marked `shadow` for the shadow, where a merged PR whose last round left only low
findings open counts them settled, since `/quest-review` fixes or refutes those
before merge without another round; the land fixer's fixes; the later PRs that list it under
`## Caused by`; and each online reviewer's record, read from GitHub.
before merge without another round; the land fixer's fixes; the later merged PRs
whose `## Caused by` names it on a bullet that opens with its link, as
`- [owner/repo#N](url) - why`, never a link elsewhere on a line; and each
online reviewer's record, read from GitHub.

An online reviewer is any bot account whose review has a body or opens a
thread, as a reply-only review does not, or a review whose body opens with
Expand Down Expand Up @@ -1538,11 +1544,21 @@ counts as unreviewed. An `Implementer:` line's own size, as
`(iterate, XS)`, counts over the PR's; and, over the merged PRs, each online
reviewer's coverage, final heads reviewed, findings per PR reviewed, and
answers. Sessions on no PR here, from any
repository or agent ccusage reads, total on an `Unattributed` line. `--pr` and
repository or agent token-max reads, total on an `Unattributed` line. `--pr` and
`--quest` narrow the report to one PR or one quest's branch, and `--json`
prints it as one object. Without `ccusage` on `PATH`, it says how to install it
and reports the PRs' records alone. Each host reports its own sessions; the
land fixer's container keeps no session log, so its fixes show without tokens.
prints it as one object. Without `token-max` on `PATH`, it says how to install
it and reports the PRs' records alone.

`--ssh HOST`, repeatable, also reads that SSH destination's sessions through
the token-max installed there, matching each through the repositories resolved
on its own host. A host that fails fails the report, naming it; drop it from
`--ssh` to report without it. A session found on several hosts, such as a
history copied between them, counts once at its costliest copy, and the report
says how many copies it folded from which hosts. A session with a request
token-max could not price, or a transcript line it could not read or that was
unfinished, is listed under `Lower bounds` with its unreadable lines, and every
cost it reaches reads `or more`. The land fixer's container keeps no session log, so
its fixes show without tokens.

### Typed messages

Expand Down
27 changes: 26 additions & 1 deletion flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 11 additions & 2 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,14 @@
url = "github:moq-dev/quest";
flake = false;
};
# token-max, which reads and prices `quest usage`'s sessions; the lock pins it so its JSON stays what the
# parser reads. Its own quest input follows this flake, so the lock never chains older quest pins.
token-max = {
url = "github:kixel-ai/token-max";
inputs.nixpkgs.follows = "nixpkgs";
inputs.flake-utils.follows = "flake-utils";
inputs.quest.follows = "";
};
};

outputs =
Expand All @@ -26,6 +34,7 @@
nixpkgs-bun,
flake-utils,
quest-pinned,
token-max,
...
}:
# The pinned nixpkgs no longer evaluates on x86_64-darwin; Intel Macs use the release binary.
Expand Down Expand Up @@ -73,8 +82,8 @@
jq
git
gh
# Prices `quest usage`'s sessions, pinned with the lock so its JSON stays what the parser reads.
ccusage
# Exports and prices `quest usage`'s sessions.
token-max.packages.${system}.default
direnv
actionlint
shellcheck
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/benchmark/records.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,5 @@ Agent's choices, for the implementer to revisit:

- [Development as a benchmark](/quest/m0/benchmark/README.md) - the
decisions this builds on
- [Price sessions through token-max](/quest/m0/routing-evidence/usage-token-max.md) -
- [Price sessions through token-max](https://github.com/moq-dev/quest/pull/704) -
the per-session pricing these records reuse
38 changes: 38 additions & 0 deletions quest/m0/messages-robustness.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# [S] quest messages reads transcripts it fails on today

## Goal

`quest messages` reads every Claude Code and Codex transcript the
maintainer's hosts write today, without failing on two shapes it refuses
now, while still failing on a shape it does not know.

- A malformed line in a Claude Code transcript, such as an interleaved
partial record in the middle of the file, is skipped and counted, as
token-max does, and the output says how many lines it skipped in which
sessions. Today `parseLine` in `src/cli/messages.ts` throws on any
complete line that is not JSON, the failure `quest usage` had before it
read sessions through token-max.
- A Codex 0.162 TUI turn that carries `content_item_kinds` with no
`content_item_metadata` provenance counts its `user.text` parts as
typed and sets aside injected kinds, such as `agents_md.instructions`
and `environments.environment_context`. A kind it does not know still
fails, and the refusal stays for parts whose kind says nothing: all
`unknown`, except a compaction replay.

## Plan

Decided 2026-11-13 (user, /quest-run, answering
[moq-dev/quest#704](https://github.com/moq-dev/quest/pull/704)): one quest
for both causes (rejected: a quest each; only the Codex one).

- Cause B was salvaged from the closed
[moq-dev/quest#696](https://github.com/moq-dev/quest/pull/696). It failed
`quest messages` on two of the maintainer's three hosts, in 4 and 21
sessions, all from codex-tui 0.162.1. `readCodexMessages` accepts kinds
only when all are `unknown`, and `test/messages.test.ts` expects
`["user.text", "unknown"]` to throw; that expectation flips.
- Its PR lists under `## Caused by`
[moq-dev/quest#626](https://github.com/moq-dev/quest/pull/626), which
required provenance on every typed Codex turn. Cause A is as old as the
reader.
- Reproduce each on a real transcript before fixing it.
9 changes: 4 additions & 5 deletions quest/m0/mistake-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,10 @@ Agent's choices, for the implementer to revisit. Each run:
of those culprits and their fixes; and the typed messages from
`quest messages --since <date> --json`.
- Read each culprit pair's `## Caused by` line before counting it. On
2026-11-11 only 7 of 20 pairs were escapes: `fixedBy` takes every link
in the section, including "No single PR" lines and unmerged PRs, until
[the token-max attribution](/quest/m0/routing-evidence/usage-token-max.md)
lands; the rest named a PR that left the work to a planned follow-up
or kept older behavior.
2026-11-11 only 7 of 20 pairs were escapes. `fixedBy` now counts only a
merged fix's bullet that opens with the culprit's link, but a culprit
can still name a PR that left the work to a planned follow-up or kept
older behavior.
- `quest messages` returns every human turn as a `kind: "message"` entry,
beside the `kind: "answer"` entries tallied below, so set aside machine-written
ones yourself: bare skill invocations, pasted relays and logs, accepted
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/routing-evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Decided 2026-11-11 (user, /quest-plan, from an audit of PRs merged
only; none).
- Records?
- ✅ Cumulative review tallies and online reviewers in usage
- ✅ [Attribution through token-max](/quest/m0/routing-evidence/usage-token-max.md),
- ✅ Attribution through token-max,
kept after the maintainer first left it out, because the writer A/B
compares cost and `quest usage` crashes on this host
- Rejected: a PR-body hook that validates `## Models` and `## Caused by`
Expand Down
5 changes: 0 additions & 5 deletions quest/m0/routing-evidence/charts.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,3 @@ Agent's choices, for the implementer to revisit:
findings on their own, record them beside the tally.
- Make the first charts from the data on main when this lands, link them in
this quest's PR, and say which cells are still too thin.

## Required

- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) -
the cost axis, which today misses sub-agent transcripts, 72% of host spend
86 changes: 0 additions & 86 deletions quest/m0/routing-evidence/usage-token-max.md

This file was deleted.

5 changes: 0 additions & 5 deletions quest/m0/routing-evidence/writer-ab.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,3 @@ Agent's choices, for the implementer to revisit:
arm on the PR's first Implementer line: `--writer` also takes that line's
effort, or a new option names the arm.
- [The readout](/quest/m0/routing-evidence/README.md) ends the split.

## Required

- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) -
counts sub-agent writers' cost, without which an arm's cost is undercounted
2 changes: 1 addition & 1 deletion src/cli/github.ts
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ export function remoteRepository(remote: Remote, push = false): { url: string; r
*/
export function githubRepository(url: string): Repository | null {
const match =
/^(?:https:\/\/(?:[^@/]+@)?github\.com\/|ssh:\/\/(?:[^@/]+@)?github\.com(?::\d+)?\/|[^@/:]+@github\.com:)([^/]+)\/([^/]+?)(?:\.git)?\/?$/.exec(
/^(?:https:\/\/(?:[^@/]+@)?github\.com\/|ssh:\/\/(?:[^@/]+@)?github\.com(?::\d+)?\/|(?:[^@/:]+@)?github\.com:)([^/]+)\/([^/]+?)(?:\.git)?\/?$/.exec(
url,
);
return match ? { owner: match[1], repo: match[2] } : null;
Expand Down
Loading
Loading