Skip to content

refactor: calibrate compaction summaries once for window, idle and T3 replays - #180

Merged
kixelated merged 3 commits into
mainfrom
quest/m1/summary-calibration
Nov 21, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m1/summary-calibration

Conversation

@kixelated

@kixelated kixelated commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Window (compaction), idle and T3 idle replays now size a compaction summary from one calibration, so a fix to how summaries are grouped or measured reaches all three. Finishes quest/m1/summary-calibration.md.

Changes

  • src/compaction.rs: CompactionSamples::new builds one Calibration per group: the samples in order, uniformity of the summary pricing cohort, and median and maximum Geometry (output, prefix, read, overhead). It groups once by SampleCohort (window replays, cohort()) and once by SampleKey, which is SampleCohort plus version and client (idle and T3, idle()).
  • src/idle.rs: ReplayIndex reads the shared calibration instead of regrouping per replay. The tuple SampleKey, sample_key, and idle's own Geometry/Calibration are gone. t3_replay looks up the same calibration instead of scanning every sample for each gap and checking uniformity field by field.
  • Repeated event ids: CompactionSample::event is now the last event recorded with the boundary's id, not just the id. Every replay reads that event's pre_tokens for summary overhead, and window replays use it as the template for a modeled boundary. Before, ReplayIndex used the last event, while T3 and the window template (through WindowEvents::by_id) used the first. Following the quest's Plan, the last record now wins everywhere.
  • WindowEvents::by_id existed only for that template lookup, so it is now a plain event list that suppress_idle filters.
  • docs/semantics.md says the three replays share one calibration, that idle and T3 also match client and version, and that a repeated boundary calibrates from its last record.
  • The quest is deleted. quest/m1/test-only-code.md now notes that Calibration::events only feeds PeriodCalibration, so it goes when the detail trace does. The trace itself is otherwise unchanged.

Net: -64 lines in src/ before the new test.

Choices made without asking

  • Idle and T3 now also reject an Agent::Unknown request, because SampleKey contains SampleCohort, which already did. The output can't change: idle and T3 replay only Claude profiles, and Claude requests are always Main or Subagent (Unknown occurs only for Codex).
  • The window template now follows the last record too. The quest named only idle and T3, but its Goal is one calibration for all three replays.
  • Left unchanged: compaction_samples still links the first CompactionCharge per event id. That decides which charge applies, not how a summary is sized, and charges come one per scanned event.

Validation

  • just check passes: quest check, fmt, clippy -D warnings, 587 lib tests plus the integration tests.
  • New test a_repeated_event_calibrates_every_replay_from_its_last_record: two records of one boundary with different pre_tokens. It checks that both the window and idle lookups give overhead 500 from the last record, and that idle also needs a client version while window replays don't. Switching last to first-wins makes it fail (tried).
  • Differential (scratch test, not committed): 300 randomized sets of 1 to 12 samples across provider, version, client, agent, model, tier, speed, summary model/provider and pre_tokens, all with unique ids. For every probe it compared the old T3 linear scan plus field-by-field uniformity and medians, and the old window cohort grouping, against idle() and cohort(). Median and max output, prefix, read and overhead, uniformity, template sample and event order were all identical. The only difference was Agent::Unknown, as intended.
  • Ran the release binary from main and from this branch, doctor --details --plain --offline --since 2026-10-20, on this host's live transcripts. Live sessions kept writing, so two main runs already differed from each other, and a byte-for-byte comparison wasn't possible. The calibration-derived figures matched exactly, e.g. added compactions "output 4168 to 5728 tokens; resumed prefix 25413 to 26120 tokens". The other idle, window and T3 lines differed only by the same small drift seen between the two main runs.

Public API

None. All the touched modules are private. No wire or report-format change.

Caused by

  • kixel-ai/token-max#36 - the T3 replay looked up a sample's event with find, so it kept the first record of a repeated id while idle's ReplayIndex kept the last (found by person: m1 code audit)

Follow-ups

None new. The edit to quest/m1/test-only-code.md adds one line to that quest's existing work.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm B)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 1995854

  • Base: origin/main at 99d2cc0

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1fbd9-11b4-7420-b6e3-c96a7799f590, round 1

  • Score: 9/10, No actionable regressions were found: shared calibration preserves replay behavior while consistently using the last event record, though builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, medium effort, shadow

  • Session: 01a1fbd9-11b3-74b0-b4fd-c013da862dde, round 1

  • Score: 9/10, The shared calibration preserves existing cohort matching and replay calculations while consistently using the last event record; code inspection found no actionable regressions, but builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

Summary by CodeRabbit

  • Improvements
    • Compaction window, idle, and T3 replays now use consistent summary and resumed-prefix calibration.
    • Idle and T3 calibration also matches the client and version, helping keep replay comparisons specific to the corresponding environment.
    • When a boundary is recorded more than once, calibration uses its latest record.
  • Documentation
    • Clarified the calibration behavior for compaction windows and idle and T3 replays.

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

Compaction, idle, and T3 replay now use shared calibration data. Idle and T3 calibration additionally match client and version. Duplicate event records use the last record for calibration.

Changes: Compaction replay calibration

Layer / File(s) Summary
Shared calibration and event sampling
src/compaction.rs, docs/semantics.md, quest/m1/summary-calibration.md, quest/m1/test-only-code.md
Calibration groups samples and derives geometry from structural events. Duplicate records use the last record. The semantics documentation and quest plans reflect these calibration details. Tests cover duplicate records and idle client/version matching.
Window replay calibration
src/compaction.rs
Added-pass simulation uses cohort calibration geometry. Added-compaction evidence uses event and resumed-request IDs from calibration samples.
Idle and T3 calibration
src/idle.rs
Native idle and T3 replay use the shared idle calibration lookup. T3 uses its median geometry and calibration events instead of local calculations.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Refactor

Possibly related PRs

Suggested reviewers: pullfrog

Merge Risk: Minimal · up to 19958

This change shares summary calibration across the window, idle and T3 replays, and repeated boundary records now use the last record. No concrete regression was established in review. The one open item is the failing CI gate job, which should be rerun and confirmed green before merge.

Security architecture risk: Minimal · up to 19958

The inspected change remains within local replay calculations. It does not introduce a new external interface or privileged action, and missing or incompatible calibration remains explicitly unpriced.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated exposure is to transcript-derived replay estimates and modeled events, including existing keepalive calculations. The changed paths do not introduce a new external caller or privileged sink. Modeled window events explicitly receive the target request's session and origins rather than inheriting the calibration event's ownership.

Trust Boundaries and Controls

  • observed — Cohort, client, and version values are calibration-matching attributes, not authorization identities. Sample construction still requires a measured positive resumed prefix and linked positive billed summary output. Missing matching attributes or ambiguous summary pricing lead to unavailable calibration or unknown cost, not additional authority.

Resilience and Maintainability Implications

  • inferred — Calibration is built before consumption and borrowed through shared references; mutable simulation state remains local to a replay and its sessions. The inspected calibration transition adds no persistent partial-commit state requiring cancellation cleanup, migration, or rollback coordination.
Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: compaction summary calibration is computed once and shared by window, idle, and T3 replays. The PR objectives and the src/compaction.rs and src/idle.rs summaries match…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: compaction summary calibration is computed once and shared by window, idle, and T3 replays. The PR objectives and the src/compaction.rs and src/idle.rs summaries match this. The title is a single concise sentence using the conventional refactor prefix.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
Fix failing CI checks

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

kixelated and others added 2 commits November 20, 2026 18:01
… replays

One Calibration per cohort, built in CompactionSamples, now gives window,
idle and T3 replays their summary geometry. Idle and T3 key it by the
window cohort plus client and version. A boundary recorded more than once
calibrates from its last record in every replay; T3 took the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for test-only-code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Calibrate compaction summaries once refactor: calibrate compaction summaries once for window, idle and T3 replays Nov 21, 2026
@kixelated
kixelated marked this pull request as ready for review November 21, 2026 02:05

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. The PR collapses three copies of the summary grouping and measuring code into one Calibration built per SampleCohort and per SampleKey. Repeated boundary ids now resolve to the last record in a single place (compaction_samples), not in three places that disagreed.

Reviewed changes

I reviewed the full diff across src/compaction.rs, src/idle.rs, docs/semantics.md and the two quest files.

  • Shared Calibration: CompactionSamples::new groups the samples twice through the generic group(). The window lookup cohort() and the idle/T3 lookup idle() both index into one calibrations vector. Median and max output, prefix, read and overhead match the old idle formulas.
  • Last record wins: CompactionSample::event now holds the last StructuralEvent recorded for the id. Idle's ReplayIndex events come from all_events, a stable sort of time-sorted events plus modeled: ids, so its last record is the same as before. Only T3 and the window template change from the first record to the last, which is the intended change.
  • T3 replay: samples.idle(..).filter(|c| c.uniform) replaces the linear scan and the field-by-field uniformity check. I checked the overhead (including input == None) and the read median against the old code, and they match.
  • WindowEvents: by_id is gone, so suppress_idle now filters every event instead of the first record per id. In practice nothing changes. The reader's add_event already dedups ids inside scan.events, and every pre-window cache_context.boundaries entry passes the e.time < since filter anyway.
  • Docs and test: docs/semantics.md describes the shared calibration and the client/version match. The new test covers a repeated id and the version requirement that applies to idle only.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional comments: 2

src/compaction.rs: 1

235-240: Trivial · Performance & Scalability · Heavy lift

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Skip idle-only calibration work for window-only callers.

compaction_checks and keepalive only use window geometry. They now build idle cohorts too, and each window cohort sorts cache-read and overhead values and allocates an event-ID list it never uses. Build the idle index only for callers that use it, and keep read, overhead, and trace data out of window cohorts. Native idle and T3 can still share one indexed calibration across their replay runs. (github.com)

docs/semantics.md-370-371: 1

370-371: Minor · Functional Correctness · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Specify which fields use the last boundary record.

The transcript reader merges copies with the same ID and retains the first copy’s token metadata. If repeated records reach replay, the last supplied record sets summary overhead and supplies the window event template. Resume matching and summary linkage still run per record. State that distinction so readers do not expect a later transcript copy to replace the earlier sample. (raw.githubusercontent.com)

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0712d450-fecf-4e6a-a983-883499dec3db
Commits

Reviewing files that changed from the base of the PR and between 99d2cc0 and 1995854.

Files selected for processing: 5

  • docs/semantics.md
  • quest/m1/summary-calibration.md
  • quest/m1/test-only-code.md
  • src/compaction.rs
  • src/idle.rs
Files with no reviewable changes: 1

  • quest/m1/summary-calibration.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

@kixelated
kixelated added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit 0154e5b Nov 21, 2026
10 of 11 checks passed
@kixelated
kixelated deleted the quest/m1/summary-calibration branch November 21, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant