Skip to content

docs(mcp): publish warm search comparisons - #929

Closed
dmtrKovalenko wants to merge 7 commits into
codex/mcp-search-parityfrom
codex/monorepo-search-benchmark
Closed

dmtrKovalenko wants to merge 7 commits into
codex/mcp-search-parityfrom
codex/monorepo-search-benchmark

Conversation

@dmtrKovalenko

@dmtrKovalenko dmtrKovalenko commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Consolidated into #928. Continue review there.

Update MCP documentation and charts for the small repo and big repo buckets. Remove the comparison scripts and keep held-out results separate.


Bucket Model Shell time FFF time Wall reduction Token reduction
Small repo Opus 5.5 10.4 s 9.2 s 11.7% 8.6%
Small repo GPT-6 Astra 9.0 s 8.1 s 10.3% 6.7%
Big repo Opus 5.5 24.7 s 14.3 s 42.1% 3.4%
Big repo GPT-6 Astra 21.7 s 10.4 s 52.2% 8.2%

Six additional tasks in the small repo bucket were frozen before tuning. The selected implementation produced these results:

Model Wall reduction Token reduction
Opus 5.5 15.8% 15.7%
GPT-6 Astra -10.5% -12.3%

All 72 recorded trials returned correct answers. The 50% target remains unmet in the small repo bucket. Negative reductions indicate regressions.

Trials run sequentially with a fresh model context. Startup, indexing, and initial Git refresh occur before timing. Both arms use the same source-verification rule. The MCP client exposes tools directly. Tokens include cached input.

The repository keeps the charts and documentation. Comparison scripts, their tests, and generated measurement files are removed. Original measurements and private traces remain local.

Both charts use aligned comparison rows and a compact layout. Titles, legends, labels, and values are unchanged; subtitles and footers remain removed. Only the small repo had a separate held-out run.

Warm search comparison

Tasks held out from tuning

Stacked on #928.

Stack merge order: #924, #925, #927, #928.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9d4814f3-b216-431f-87a1-83f03a6a10bd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dmtrKovalenko
dmtrKovalenko force-pushed the codex/mcp-search-parity branch from f3d76ae to 9ccb93e Compare October 16, 2026 02:41
@dmtrKovalenko
dmtrKovalenko force-pushed the codex/monorepo-search-benchmark branch from d9514a6 to 3e95b02 Compare October 16, 2026 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants