Repository navigation
fix: Preserve native default reranker batch order - #813
Draft
buildingjoshbetter wants to merge 2 commits into
Draft
buildingjoshbetter wants to merge 2 commits into
buildingjoshbetter wants to merge 2 commits into
Conversation
This was referenced Oct 8, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Default ModernBERT batch-order correction for #735, stacked on #811. Preserve model identity, weights, inference dtype, score activation, prompts, dimensions, retrieval depth and caller/server batch ceilings.
Problem and correction
The native CrossEncoder sorts the complete request before batching. Sorting separate caller-order slices instead can put a long pair in each slice. The measured mixed100 request used
64x779, 36x779rather than the native64x779, 36x72, adding padded work and changing native batching.Alibaba-NLP/gte-reranker-modernbert-base, modern nonflattened two-string inputs, effective limit64 and more than64 pairs.Validation
fb7306ea716151f6bbef4d4e793e0ac56b45243c: 2,829 passed, 22 skipped, 3 deselected; 2,047 subtests passed, with actual sqlite-vec. Log SHA256ddc02c30b55e3727a6eb359cc698d5a365408160048c7d699f14b95696ea922e.a69a832a8a99bb1303acb89bc7f3296937f12c832164088afe1dc596f1b23565.An earlier combined candidate failed one existing admission test because an unplanned call passed a new keyword to its positional-only writer. The correction preserves the original interface on that path; the failed receipt and existing test are retained.
Native evidence and limits
The reviewed three-cell synthetic CUDA diagnostic completed 90 timed records, 3 warmups, 3,741 rows and 267 forwards, with verified cleanup and no OOM or timeout. It ran the combined source above; this publication branch has byte-identical model-server and microbatch test files.
All 30 default comparisons now have exact output arrays and ordered-feature parity, including all six default reranker comparisons. All 30 explicit-eight arrays exactly match their prior contiguous-eight outputs. Raw explicit-eight numerical acceptance is still 21/30, so combined release acceptance remains false.
Mixed100 reranking now matches the native
64x779, 36x72shapes. Instrumented medians were 429.941691 ms default versus 424.593396 ms native reference:429.941691 / 424.593396 = 1.012596275. The earlier separate run measured631.852592 / 424.322110 = 1.489087128; this context is not a paired production p95 claim.Reranker raw-report SHA256:
34c16142af8aa129d938f284f6eb17c42ef24e9a5a06e7fc072d5285c845affb. Diagnostic driver SHA256:066ab7c3800fe33b7cc3fac06970cbc3b734b59652cd9e1c1d19ac953767d1e2.The native fixture has a constant query; separate varied-query production tests cover whole-pair semantics. Serial instrumented CUDA results do not establish RPC/concurrency latency, retrieval quality, process-memory drift, MPS or thermal behavior. The old explicit-eight failures, paired runs and final soak remain open. No issue closure, main merge or release. Only synthetic test data and public model artifacts were used; no personal memories, transcripts or credentials.