Repository navigation
🤖 fix: stop the context estimate from overcounting at turn start - #7662
Conversation
The budget walker counted providerOptions on assistant parts as input text. Anthropic thinking signatures are large base64 strings that Anthropic does not bill, so a 1M Opus chat hit its 20% handoff target at about 10% real usage. Skip providerOptions on every SDK message and message part. Tool-output content items keep it because some adapters JSON-stringify output.value.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
🟢 e2e stories (Haiku): 14 passed22 agent steps · 7 replayed from cache · 15 model calls · 46.4k tokens (24% cached) All 14 tests in 9 files
e2e 0.20.0 · 1m 12s · web · run artifacts 🟢 e2e Electron stories (Haiku): 3 passed3 agent steps · 3 model calls · 7.9k tokens (22% cached) All 3 tests in 3 files
e2e 0.20.0 · 19.1s · electron · run artifacts 🟢 e2e agent repros (Haiku): no tests selectedSpend: $0.00 of the $1.00 cap (proxy ledger) e2e 0.20.0 · 5.3s · 2 targets (web, phone) · run artifacts |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69cd900a1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… usage The turn-start check and step zero now count E as the newest reply's provider input plus the messages after it, when that reply is a completed one from the same model, route and agent. Growth of the sent system text since that reply is added. Otherwise they keep the full estimate. Stage prompts are decided on the same E, so a stage published at turn start is never refused at step zero. The loopback fixture now reports provider usage that tracks the request and runs the real tokenizer.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce3ccc3e0a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…played reasoning The turn anchor now adds the last step's reasoning output, like the in-turn anchor. Claude Opus 5.5 replays that step's thinking at its full count, and the estimator sees at most the summary text. Redacted thinking and encrypted reasoning have no text at all. A last step that emitted reasoning without a reported count does not anchor. A row from another resolved model (a providers.jsonc mapping change), or from a build that did not record it, does not anchor.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 115c8c7522
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…rows in the turn anchor - replayedStepTokens returns no anchor when a persisted input, cache or reasoning count is negative or not a safe integer. - The turn anchor measures system growth against the sent system message plus its system rows, the same text the reply's systemMessageTokens counted.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ccad0f70b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
I paused the Codex review loop for a maintainer decision. The reason is the AGENTS.md early stop: the loop does not converge. Rounds 2, 3 and 4 each brought three new findings, and most of them are in the turn-start anchor. Round 4 on
Each finding is a real edge case, and each fix is a small full-estimate guard. But each round finds a new way that the rebuilt prefix differs from the prefix the provider counted. Before I add more guards, I need a decision: add the guards, or remove the turn anchor from this PR and keep only the replay-metadata fix. |
…last step - The turn anchor needs the reply's recorded thinking level to equal the request's effective level. Message preparation keeps or strips earlier Anthropic reasoning by that level. - The turn anchor is skipped when content follows a tool call in the reply's last step. The rebuild then splits the step, and a delta from the last assistant message would miss the tool call and its result.
|
@codex review Round 4 on
Known limitations that the maintainer accepted in earlier rounds (please do not raise them again):
|
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review The code review of Context from round 4, unchanged:
Known limitations that the maintainer accepted in earlier rounds (please do not raise them again):
|
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review The code review of Context from round 4, unchanged:
Known limitations that the maintainer accepted in earlier rounds (please do not raise them again):
|
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd159a55d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A thinking-level change during PREPARING, or one that races stream setup, rebuilds the first request at the new level. Step zero still counted that request against the anchor captured at the old level, so a high-to-off switch could reuse a count with replayed reasoning and refuse a request that fits. - The pre-stream fold now carries the anchor of its rebuild to step zero. - Step zero uses the full estimate when it applies a thinking override itself.
|
@codex review Round 5 on
Decisions from earlier rounds (please do not raise them again):
|
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e70fc7a07
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…saved request identity The turn anchor compared the route and agent only when both sides set them, so a row from a build that did not save them anchored a request on another route or agent. A negative saved system count read as system growth. - One identity rule: the resolved model, thinking level, route and agent a reply saved must each equal the request's, unset included (a direct route saves no route). - The saved system count goes through the same check as the usage counts.
|
@codex review Round 6 on
Decisions from earlier rounds (please do not raise them again):
|
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Summary
The context estimate (E) that decides the Token Budget stages was much higher than the provider's input count. This PR fixes two causes:
providerOptionson model output), such as Anthropic thinking signatures.Before this fix, a 1M Opus chat requested a context handoff at about 10% of real usage.
Background
A chat on
anthropic:claude-opus-5-5(1M context, 20% handoff target) showed "Context handoff requested" at 28%. The usage meter showed about 10%. The provider reported 95,548 input tokens for the next request.measureBudgetContentcounts every string in the built request, except OpenAIreasoningEncryptedContentand Xum'srawToolSearchOutputstash. It counted each Anthropic reasoning part'sproviderOptions.anthropic.signatureas text, and the Claude encoding factor then multiplied it by 1.6. In that chat, one window replayed 14 signatures (134k chars) worth about 150k budget tokens.The overcount is older than this week, but it had no effect until the stages moved to the local estimate:
Before those PRs, the estimate drove only the hard stop near 99%, so nobody saw the error.
Without signatures, the local estimate and the provider count still differ (#7337). Inside a turn, each step's estimate is already the previous step's provider input plus the new messages. The turn-start estimate was not, so the stage decision used a different count from the meter and from the steps that follow it.
Implementation
Replay metadata
The walker skips the
providerOptionskey on assistant parts (model output). This covers these fields:signatureandredactedDatareasoningEncryptedContentanditemIdthoughtSignatureon reasoning, text and tool-call partssignaturereasoning_detailsOn user and tool content, and on message-level options, the walker skips only Xum's own
muxnamespace (for example therawToolSearchOutputstash). Those options can reach the model as text:@ai-sdk/anthropiccopies a document part'sproviderOptions.anthropic.contextandtitleinto the request.Tool-output content items (
output: { type: "content", value }) still count theirproviderOptions. Theopenai-compatible,openaichat anddeepseekadapters send that value asJSON.stringify(output.value), so those bytes reach the wire. The existingtool-contentimpersonation test guards this case.These two rules replace four special-case walker kinds and their two skip blocks.
Turn-start anchor
createTurnBudgetAnchorreads the newest assistant row of the turn's provider request rows. It returns an anchor only when all of these are true:metadataModel), thinking level, route provider and agent. Each saved field must equal the request's, unset included. A direct route saves no route, and a row from a build that did not save a field gives no anchor. A route or agent change re-freezes the system prompt and tools. Aproviders.jsoncmapping change keeps the model and changes the resolved model. Message preparation keeps or strips earlier Anthropic reasoning by the thinking level, so another level replays reasoning that the provider count does not hold.web_searchdoes this). A delta from the last assistant message would miss the first two.contextUsagewith a positive input total andsystemMessageTokens, and none of these saved counts is negative or a non-integer (saved rows are unchecked JSON).The anchor is the input, cache-read and cache-write tokens of that reply's last step, plus that step's reasoning output tokens. The request replays that reasoning after the anchored prefix. A model that keeps earlier thinking counts all of it, also redacted or encrypted reasoning that has no text for the estimator. On 172 real consecutive
claude-opus-5-5reply pairs, the input growth across a turn boundary matched the previous reply's full output, reasoning included. If the last step emitted reasoning and the provider reported no reasoning count, the row does not anchor. If the sent system text grew since that reply (a live system prompt can change between turns), the growth is added. The growth compares the system message plus its system rows with the reply'ssystemMessageTokens, which counted the same text.The delta is the request messages from the last assistant message on. A tool reference in the delta, or a different model, gives the full estimate (the #5413 rule). A compaction request gets no anchor, because its trim can drop rows that the provider count holds.
assembleBudgetCheckedPromptPayloaduses the anchor for the turn-start check and returns it. The stream request carries it to step zero (contextBudgetTurnAnchor, refusal fallbacks included), so step zero enforces the same count. A stage that the turn start published is not refused at step zero. A refusal fallback that keeps the refused output continues from a partial row, so it gets no anchor and uses the full estimate.A thinking-level change can arrive after the turn start built the request. The stream then rebuilds the messages at the new level. A rebuild before the stream starts returns the anchor for the new level, and step zero uses it. A rebuild inside step zero (a change that races stream setup) gives no anchor, so step zero uses the full estimate.
The loopback fixture (
tokenBudgetStages.fixture.integration.test.ts) now reports provider usage that tracks the request size, and runs the real tokenizer, as production does. R2b now checks the fixed behavior: with provider usage at 2/3 of the full estimate, the turn start publishes the handoff, and step zero sends that request although its full estimate is past the hard ceiling.Validation
I ran the real message pipeline (
prepareProviderRequestMessages, thenprepareMessagesForProvider) on the persisted rows that the 28% notice measured. The 46 model messages, without system and tools, changed from 221,698 estimated tokens to 76,042. With the system and tool overhead from the notice, the new total is about 131k. That is under the 200k target, and the provider total was 95.5k.I also ran a live chat on an isolated
xum serverwith Token Budget on,claude-opus-5-5with the 1M window, thinkingxhigh, and a 15% handoff threshold (150,000 tokens). A temporary log, not in this PR, recorded the checked estimate at each step next to the provider's input count for that step.No notice came while the real count was under 150,000. Turn 8's tool results pushed the next step past the target, so the settled-step check ended the turn and the continuation got the handoff notice: "Context budget ~18% used (181418 of 991808 tokens)". The provider count for that request was 178,784, also 18%. Later steps inside a turn were 0.4% to 4.5% over the provider count. The full estimate without the anchor was 16% to 25% over.
Risks
Medium. The change moves the turn-start handoff and final stages, and the hard stop before the first step of a turn.
openai-compatibleadapter copiesproviderOptions.openaiCompatiblefrom assistant parts into the wire message, and the walker now skips it. Xum never sets it.[id: N](about 10 budget tokens). A reply sent with the budget off (continuous compaction) or by a build without tags holds no tags, so step zero of the next turn with the budget on does not count them. The reply of that turn has the tags, so the next turn's anchor holds them.redactedData, OpenAI ciphertext). Counting that data's JSON caused the overcount that this PR fixes. On the bug's payload (claude-opus-5-5atxhigh, which keeps earlier thinking), the full estimate is still about 131k tokens against the provider's 95.5k.Follow-ups, not in this PR:
projectRowForSizing,estimateMuxMessageTokens) also count signatures. They affect compaction input trimming and keep-recent tails, not the handoff stages.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh• Cost:$66.81