Skip to content

fix: correct underreported token totals in Codex Security scans - #1657

Merged
mldangelo-oai merged 8 commits into
mainfrom
mdangelo/codex/recover-scan-token-usage
Oct 16, 2026
Merged

mldangelo-oai merged 8 commits into
mainfrom
mdangelo/codex/recover-scan-token-usage

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Codex Security reports how many tokens a scan used, but its accounting can miss work that actually ran. A completed scan can therefore show an "at least … tokens" lower bound instead of a complete total.

This PR fixes the accounting behind that message rather than just changing the wording. It recovers recorded usage from earlier worker attempts, includes context summarization, and lets the SDK and compatible hosts finish counting the scan's final response after it ends. The goal is a trustworthy total for understanding scan usage and comparing runs—not presenting an incomplete count as exact.

Changes

  • Count earlier worker attempts. When Deep Scan restarts a background worker, include the previous attempt and its child agents without counting resumed history twice. Handle equivalent Windows path spellings correctly.
  • Include context summarization. Count individual model responses, including the work used to shorten long conversations (compaction), without also adding the older cumulative counters.
  • Finish accounting after the final response. Saving the report can happen before the last model response finishes. Record which turn needs to finish, then let the SDK or a compatible app collect its remaining usage without counting later conversation turns.
  • Report only verified totals. Update report instructions to quote a token total only when accounting is complete. Preserve incomplete or unavailable status when records genuinely cannot be recovered. Keep custom-validation workflows compatible with the updated instructions.

Testing

  • Usage collector: 67 tests passed, 2 platform-specific skips, including worker retries, resumed history, final-response accounting, and synthetic Windows/POSIX path cases.
  • SDK: 70 focused tests passed during implementation; a separate 41-test follow-up passed for custom validation, scan startup, and usage finalization. These suites overlap.
  • MCP smoke test, Python lint/format, SDK build/type/format, and plugin source compatibility checks passed; all 9 source-compatibility tests passed.
  • Local limitations: native Windows execution was not available. The full SDK package test command was blocked before execution by missing universal native payloads (native/prebuilt/COPYRIGHT-library.html).

Risk and rollout

  • No public CLI commands, flags, or defaults change. Scan results and completed report files are preserved.
  • This PR includes the plugin/SDK accounting changes and the plugin side of the app handoff. The desktop app must adopt that handoff to include the final response; desktop UI changes are separate.
  • Older hosts without scan-source metadata keep the existing immediate collection path. Older recordings may lack compaction usage, and deleted or never-recorded usage cannot be reconstructed. Those cases remain incomplete or unavailable rather than being labeled exact.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the bug Something isn't working label Oct 16, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T03:02:40.885597Z fdeef45 New commits
🔒 Security Review ✅ Completed 2026-10-16T03:02:40.813894Z fdeef45 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@mldangelo-oai mldangelo-oai changed the title fix: recover token usage from scan worker attempts fix: recover missing scan token usage Oct 16, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac1689efb1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex-security/scripts/workbench_scan_usage.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac1689efb1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex-security/scripts/workbench_scan_usage.py Outdated
Comment thread plugins/codex-security/scripts/workbench_scan_usage.py Outdated
@mldangelo-oai mldangelo-oai changed the title fix: recover missing scan token usage fix: correct underreported token totals in Codex Security scans Oct 16, 2026

@kmbroai kmbroai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed worker retry/recovery accounting, inherited child history, per-response versus legacy counters, compaction, cached input/output, and completion-turn finalization including failure and cancellation paths. No blocking findings. The implementation stays within the existing accounting and workbench boundaries; the larger diff is primarily regression coverage.

Validated locally: 69 Python usage tests; 247 SDK API/finalization tests passed with 4 platform-specific skips; 26 custom-validation tests; MCP smoke test; SDK build:ci; plugin source compatibility and its 9 tests. Latest node-ci run 38845045232 is successful on this SHA. Desktop adoption of the finalization handoff remains a separate rollout requirement, as documented.

@mldangelo-oai
mldangelo-oai merged commit 7e0ec1f into main Oct 16, 2026
127 of 160 checks passed
@mldangelo-oai
mldangelo-oai deleted the mdangelo/codex/recover-scan-token-usage branch October 16, 2026 15:30
@github-actions github-actions Bot mentioned this pull request Oct 16, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants