Skip to content

Implement local TypeScript replay and Claude TTL recommendations - #3

Merged
kixelated merged 35 commits into
mainfrom
quest/m0/replay
Nov 7, 2026
Merged

kixelated merged 35 commits into
mainfrom
quest/m0/replay

Conversation

@kixelated

@kixelated kixelated commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Implement the local replay core and Claude main/outside-main cache-TTL comparison in Node-compatible TypeScript. Replace the Rust scaffold with Bun development tooling, a JavaScript build, and checks under Bun and Node. Transcripts and settings remain read-only; results retain metadata and counts, never prompt/reply text.

Implements the reviewed plan from kixelated/token-max#2, now merged into main. This PR targets main and includes main through kixelated/token-max#4. Only replay implementation and its quest/dependency cleanup belong to this delta; landing configuration, SSH and T3 integrations are separate.

Changes

  • Discover local/default/environment profiles and explicit roots; preserve aliases, deduplicate shared histories and request identities, and leave ambiguous historical costs in shared buckets.
  • Freeze file identities and lengths before streaming records. Defer unterminated final lines; complete malformed selected records fail with safe file/line diagnostics and no partial report.
  • Parse Claude message/request references, non-overlapping billing iterations, Codex individual/cumulative usage, service tiers, clients/versions, project paths, turn/tool timing, and structural compaction metadata. Missing attribution and usage remain unknown.
  • Stream and cache models.dev prices, with explicit offline/local-file modes. Keep cache/reasoning subsets disjoint, support dated OpenAI write-billing rules and context thresholds, Claude regional/fast prices and server-tool fees, and unknown costs without public-model substitution.
  • Compare aggregate estimated TTL costs per current profile and bucket, retain new-prefix writes, include summary proxies separately, and show cold-prefix sensitivity and a numeric reuse break-even. Separate invariant fees/proxies from scenario-dependent costs so unknown fixed charges do not veto a known difference. Uncertainty accompanies the winner; independent savings are not summed.
  • Add minimal text/JSON output, measured/estimated/unknown evidence, exit 0/1/2, and a shared result model for the remaining quests. Expose explicit known/estimated/unknown cache states and predecessor witnesses across cohort and window boundaries. Supported estimates carry provenance and sensitivity; unsupported affected comparisons remain unpriced. Delete the completed replay quest and all dependency links.

Validation

  • nix develop -c quest gate just check: formatting, TypeScript, JavaScript build, 62 synthetic tests under Bun 1.4.4 and Node 24, and quest validation pass.
  • The same 62 built-JavaScript tests pass under Node 22 from the pinned nixpkgs input (nix develop -c quest gate nix shell --inputs-from . nixpkgs#nodejs_22 -c node --test dist/test/*.test.js).
  • Live public-price fetch/cache validated 8,905 model entries; the local-file adapter also parsed that catalog. Synthetic tests make no network requests.
  • After the compatibility fixes and evidence-backed attempt policy, nix develop -c quest gate node dist/src/cli.js --claude-dir /home/kixelated/.claude --since 2026-10-30 --prices-file .tmp/models.json --json exits 2 with zero success-output bytes at malformed complete record agent-aac1432197b4da0d2.jsonl:194 in the selected Claude subagent history. The ignored local diagnostic .tmp/policy-host-error.json retains its full safe file location. A separately selected Codex scan also exits 2 at a pre-existing malformed complete record. No transcript is changed or skipped. This is the first failure, not the only corrupt file: independent metadata inspection found multiple malformed complete records across several histories, so repairing one location would not establish a complete scan.
  • The exploratory audit's 10-30 through 11-05 figures were 1h main/subagent $5,494/$1,906 and 5m $8,869/$2,382. A complete result cannot be produced under the strict policy on this host, so these figures are not reproduced or validated. The earlier selected/partial audits and the reviewer's recent-project subset are not complete billing totals.
  • A selected clean ~/.claude-api-credits subset since 2026-10-30 contains 11,075 selected requests and 5,954 pre-window cache-context requests. The new estimator supports 16 of 29 main attempts; 13 main and one outside-main attempt lack sufficient evidence and keep their affected comparisons unpriced. These are selected-subset coverage counts, not complete historical billing totals.
  • Metadata-only inspection verified current Claude/Codex record shapes and overlapping iteration totals, without committing observed records. No verified idle notice discriminator was found; idle attribution and final-period summary generation stay unknown.
  • git diff --check passes. No settings or transcripts are changed and no package is published.

Public API and wire impact

New CLI flags: --since, --json, repeatable --claude-dir/--codex-dir, --offline, --prices-file, and --price-cache. New JSON schema version 1 includes profiles, frozen snapshots, requests, structural events, coverage, costs, summaries, and check findings, including the explicit scan.since billing bound, scan.cacheContext for unbilled pre-window evidence, explicit cacheEvidence predecessor witnesses with independent prefix evidence/cohort completeness, cache-comparison/fixed costs, structural-event origin aliases, recorded API-attempt budgets, comparison-only usage estimates with bracket provenance, and attempt sensitivity scenarios. The private development package reserves @kixelated/token-max and the token-max executable; final npm artifact/release work remains separate.

Choices and limits

  • Keep the package private at version 0.0.0 while the npm quest owns publication packaging. Use TypeScript, Node APIs, Prettier, and synthetic tests shared by Bun/Node.
  • Parse the public catalog incrementally with @streamparser/json; use smol-toml for Codex configuration. Do not implement an SSH reader or any other check.
  • Estimate unlinked input from identical adjacent measured geometry or growing-context brackets spanning less than the documented five-minute cache lifetime. Require matching attribution, client and pricing in the same conversation/file and no recorded boundary. Growing-context estimates attribute the next measured reads beyond the preceding cached prefix as writes, exclude the next request's own new writes from estimated total input, and assume its ordinary-input count unchanged. Positive consistent downstream cache-read evidence is required; retry/content identity and absence of another warmer remain assumptions. Preserve raw usage and historical billing as unknown; show no-billed-attempt and cold-cache sensitivities. Unsupported cache inputs/prices remain unpriced.
  • Apply today's profile settings to historical sessions with explicit labels. Unset main billing defaults, unsupported recorded versions, missing tiers/prices, and ambiguous provider conditions remain advisory/unpriced as appropriate. Project/CLI settings, agent frontmatter, and server helpers can override profile defaults.
  • Attribute a unique recorded summary by labelled sequence evidence when no explicit request reference exists. Otherwise estimate one boundary summary from preTokens/postTokens, with cache/overhead and timing limits. Do not treat precomputed as proof of idle generation.
  • Unsplit recorded OpenAI writes use a labelled default-30m estimate; missing write counters remain unknown. Twelve-hour attribution requires its own split and Responses API condition.

Caused by

New requested functionality replacing an unused scaffold; no earlier PR caused a bug. Compatibility and diagnostic defects found in local review are fixed within this PR.

Models

Size: L
Implementer: Codex, gpt-6.1-sol, xhigh effort

Design change

The maintainer decision selects evidence-backed estimates with provenance and sensitivity; unsupported cases stay unknown. Attempts are never excluded or assigned observed zero usage. The policy was recorded in quest/m0/replay.md in 7afed1a. The maintainer subsequently chose B1: rethink cache tracking as explicit known, estimated and unknown states with predecessor provenance. This checkpoint is resolved and the narrow flag-only approach was rejected.

The estimator uses immediately adjacent measured requests in the same conversation/file with matching attribution/client/pricing. It copies identical geometry, or conditionally estimates a growing-context retry within the minimum documented five-minute cache lifetime using the next measured reads and the preceding warm prefix, excluding the next request's own new writes from the attempt's total input. Recorded Claude compaction boundaries interrupt estimation, entrypoints must match, and historical settings visibility remains incomplete. It does not infer output tokens or actual execution. Raw usage, missing-usage coverage and historical spend stay unknown. Cache charges form the TTL comparison; ordinary input/output/server fees and identical summary proxies cancel under fixed behavior even when unknown. Missing scenario-dependent cache inputs/rates still block the affected comparison. Conditional no-billed/cache-affecting-attempt and cold-cache sensitivities can reverse the winner; they are not guaranteed bounds or an all-scenarios veto. docs/semantics.md records the exact supported evidence and limits.

Review

Independent reviewer: Claude Opus 5.5, high effort, size L.

B1 session round 3: 9/10, clean, no unresolved findings. Reviewed head c49d3f96485c6a7882e939bfe87d839975625b7b against origin/main at 666ccb11f4e36fde6e06f593c2adcc396586a3fe. Fresh T3 delegate session e42b2808-350f-4dea-89b9-c368230aadc1 verified all round-2 and earlier fixes, the explicit state/compression model, scan billing bound, quest cleanup, and settled estimate/net-winner policies. The remaining pure-read TTL attribution limit and maintenance complexity are documented limits, with no requested code change.

The reviewer ran 62 Bun tests and TypeScript, plus 1,900 synthetic full-versus-filtered histories and 1,500 subagent-isolation cases with zero mismatches/skips. The author's full checks also cover the built JavaScript under Node 24 and Node 22 and all 13 remaining quest documents. No implementation or background work remains.

Review history: Cross-provider reviews use the required T3 delegate route. Every recheck is a new delegate. B1 starts a fresh review session against main. Round 1 (Claude Opus 5.5, high, 7/10 at 80788ff, delegate 2b03747b-bc33-46f0-abee-d49175ea4697) verified the state/compression model and every earlier fix. It found: (1) explicit --since discarded pre-window evidence, (2) unknown-kind parts were filtered before informing other buckets, (3) estimated-prefix labels lost their evidence type with incomplete cohorts. All are fixed with separate pre-window cache context, all-part replay plus agent cohorts and unknown selected-bucket comparisons, and independent prefixEvidence/cohortKnown provenance. Reader and built CLI tests cover these paths, invalid unselected usage and straddling request links. Round 2 (Claude Opus 5.5, high, 7/10 at b70a7de, fresh delegate 611ba044-8148-4119-8e37-fafad42e0422) verified those fixes and found (1) missing pre-window summary claim order, (2) copied splits being treated as observed TTL evidence for estimated attempts, and (3) unknown kinds in a known subagent unnecessarily blocking main. All three are fixed and tested: process every boundary before selecting emitted charges, apply observed-TTL guards only to measured parts, and preserve the known outside-main role of subagents. Checks also inherit the scan billing bound and reject expanding a filtered scan. Round 3 verified all three fixed, with no unresolved findings.

Earlier sessions verified the fixes for fixed-fee cancellation, source-cohort TTL transitions, copied boundary origins and pricing, managed ownership, API-key billing ambiguity, non-directory symlink discovery, empty usage iterations, not_available geography, missing-usage diagnostics and sanitized internal errors. They also verified bracket estimation, matching client provenance, growing-retry writes excluding the following request's new writes, pre-window estimated-seed sensitivity and strict audit limitations.

The last pre-B1 round (Claude Opus 5.5, high, 8/10 at c6c8252, fresh delegate session ebd67db8-97c1-4ff8-b205-0a3f0aa947cd) found pre-window unsupported cache evidence disappearing or clearing the chain depending on missing speed. It triggered the mandatory design checkpoint, now settled by B1. The author reproduced opposite priced winners and replaced the implicit map with the explicit state model. Both metadata variants now leave the affected comparison unpriced. Explicit reader windows retain predecessor requests and compaction boundaries outside selected billing/coverage, with invalid unselected usage kept unknown. Unknown kind/agent attribution spans possible buckets; selected unresolved bucket charges stay unpriced in both conditional comparisons. Missing metadata spans compatible cohorts; known touches recover their own cohort; invariant geometry can remain priced with unknown attribution. Dense-history tests compare compressed witnesses against all possible last touches at lifetime boundaries. All earlier nonpolicy fixes must also be verified in the new review.

(Written by GPT-6.1 Sol with xhigh)

@kixelated
kixelated changed the base branch from main to feat/claude-idle-compaction November 7, 2026 14:03
kixelated and others added 2 commits November 7, 2026 06:03
Replace the Rust scaffold with the local streaming reader, public price adapter, shared report, and aggregate TTL replay. Preserve unknown billing and attribution, shared-history provenance, frozen snapshots, and strict safe failures.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated kixelated changed the title Replay and the cache TTL check Implement local TypeScript replay and Claude TTL recommendations Nov 7, 2026
kixelated and others added 2 commits November 7, 2026 06:53
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Design change: The maintainer selected option 1 for API attempts without linked token usage: estimate where recorded evidence supports it, show sensitivity, and leave unsupported cases and affected comparisons unpriced. Do not silently exclude these attempts or treat missing usage as zero billing. This supersedes the unresolved policy choice from round 2. Implement defensible estimates with explicit provenance and uncertainty, retain unknowns where estimation is unsupported, then run the final independent review and checks. Recommend the estimated aggregate net winner without an all-scenarios veto; unknown charges that demonstrably cancel between scenarios need not block their known difference.

The existing strict malformed-complete-record failure policy remains in force. No malformed record is skipped and no complete real-host audit total is claimed from the partial samples.

(Written by GPT-6.1 Sol with xhigh)

kixelated and others added 5 commits November 7, 2026 07:42
Apply the maintainer policy recorded in PR 3 comment 6709040500; preserve original usage and unsupported comparisons as unknown.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Use measured neighboring input geometry within the minimum cache lifetime, require matching client provenance, and retain conditional sensitivity for estimated cache seeds outside the report window.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Exclude the following request’s own new writes from estimated input, preserve ordinary input as an explicit proxy, and reject inconsistent warm-prefix evidence. Restrict boundary inspection to Claude compaction events.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
kixelated and others added 2 commits November 7, 2026 08:15
Keep growing-context attempts unsupported when no observed cache read supports their geometry, instead of assigning a conditional zero-cache estimate.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Preserve the remaining unsupported pre-window cache-chain finding, concrete reproduction, and recommended evidence-state redesign without changing implementation after the mandatory checkpoint.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Design change: At the third-round checkpoint, the maintainer selected B1: rethink the cache replay chain as explicit known, estimated, and unknown states with predecessor provenance. Carry unsupported attempts and missing-speed ambiguity across possible cohorts and the selected window boundary instead of dropping or silently clearing their evidence. Reject the targeted flag-only patch. The evidence-backed estimation policy and aggregate estimated net winner remain settled; unsupported affected comparisons stay unpriced.

The maintainer also authorized configuring token-max's repository auto-merge like moq and continuing quest-complete/quest-spawn until done. This is not permission to merge this draft before the remaining correctness fix, checks, and independent review are complete.

(Written by GPT-6.1 Sol with xhigh)

@kixelated
kixelated changed the base branch from feat/claude-idle-compaction to main November 7, 2026 17:02
kixelated and others added 7 commits November 7, 2026 09:03
Keep replay’s current remaining work and audit limitations while integrating the squash-merged plan.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Record the maintainer's B1 design, preserve ambiguous pre-window attempts across compatible cohorts, and price only established cache geometry.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Retain unbilled pre-window requests and compaction boundaries, keep invalid unselected usage unknown, and share all part evidence before selecting comparison charges. Separate prefix evidence from cohort completeness and verify reader/CLI behavior.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Replay all boundary claims before emitting selected charges, apply observed-TTL guards only to measured parts, and keep known subagents outside main. Retain the scan billing bound for library checks.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated
kixelated marked this pull request as ready for review November 7, 2026 18:02
@kixelated
kixelated merged commit 5b503eb into main Nov 7, 2026
4 checks passed
@kixelated
kixelated deleted the quest/m0/replay branch November 7, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant