Skip to content

fix: Preserve native default reranker batch order - #813

Draft
buildingjoshbetter wants to merge 2 commits into
fix/model-global-order-publish-735from
fix/reranker-global-order-publish-735
Draft

buildingjoshbetter wants to merge 2 commits into
fix/model-global-order-publish-735from
fix/reranker-global-order-publish-735

Conversation

@buildingjoshbetter

Copy link
Copy Markdown
Owner

Scope

Default ModernBERT batch-order correction for #735, stacked on #811. Preserve model identity, weights, inference dtype, score activation, prompts, dimensions, retrieval depth and caller/server batch ceilings.

Problem and correction

The native CrossEncoder sorts the complete request before batching. Sorting separate caller-order slices instead can put a long pair in each slice. The measured mixed100 request used 64x779, 36x779 rather than the native 64x779, 36x72, adding padded work and changing native batching.

  • Plan global raw-pair length order only for the known cached Alibaba-NLP/gte-reranker-modernbert-base, modern nonflattened two-string inputs, effective limit64 and more than64 pairs.
  • Use the native query-plus-document length semantics. Compensate for each inner equal-length permutation and scatter scores into original occurrence positions, including duplicate pairs.
  • Preserve once-per-request admission, output preflight, inference ownership, deadline checks and all batch limits. Lower explicit/adaptive limits and older/custom interfaces retain their previous bounded path.
  • On a planned OOM retry, retain the failed slice, its occurrence indices and actual resolved model identity. Advance only after successful score publication.
  • Keep unplanned success and retry calls compatible with positional result-writer wrappers. Existing admission regressions remain intact.

Validation

  • Root focused checks and independent review passed. Independent review ran 29 tests / 137 subtests, plus 64 cases using the extracted, hash-verified ST 6.1 prediction and length bodies with stdlib fakes.
  • Negative controls catch missing global order, absent/wrong inverse compensation, omitted scatter, document-only length and changed retry identity. Separate controls cover positional-only result writers on unplanned success and retry.
  • Combined source fb7306ea716151f6bbef4d4e793e0ac56b45243c: 2,829 passed, 22 skipped, 3 deselected; 2,047 subtests passed, with actual sqlite-vec. Log SHA256 ddc02c30b55e3727a6eb359cc698d5a365408160048c7d699f14b95696ea922e.
  • Exact isolated publication branch: 2,211 passed, 22 skipped, 3 deselected; 557 subtests passed, with actual sqlite-vec. Log SHA256 a69a832a8a99bb1303acb89bc7f3296937f12c832164088afe1dc596f1b23565.
  • Configured Ruff, Python 3.10 grammar, whitespace, new-content privacy and public commit-identity checks passed. Hosted checks are pending.

An earlier combined candidate failed one existing admission test because an unplanned call passed a new keyword to its positional-only writer. The correction preserves the original interface on that path; the failed receipt and existing test are retained.

Native evidence and limits

The reviewed three-cell synthetic CUDA diagnostic completed 90 timed records, 3 warmups, 3,741 rows and 267 forwards, with verified cleanup and no OOM or timeout. It ran the combined source above; this publication branch has byte-identical model-server and microbatch test files.

All 30 default comparisons now have exact output arrays and ordered-feature parity, including all six default reranker comparisons. All 30 explicit-eight arrays exactly match their prior contiguous-eight outputs. Raw explicit-eight numerical acceptance is still 21/30, so combined release acceptance remains false.

Mixed100 reranking now matches the native 64x779, 36x72 shapes. Instrumented medians were 429.941691 ms default versus 424.593396 ms native reference: 429.941691 / 424.593396 = 1.012596275. The earlier separate run measured 631.852592 / 424.322110 = 1.489087128; this context is not a paired production p95 claim.

Reranker raw-report SHA256: 34c16142af8aa129d938f284f6eb17c42ef24e9a5a06e7fc072d5285c845affb. Diagnostic driver SHA256: 066ab7c3800fe33b7cc3fac06970cbc3b734b59652cd9e1c1d19ac953767d1e2.

The native fixture has a constant query; separate varied-query production tests cover whole-pair semantics. Serial instrumented CUDA results do not establish RPC/concurrency latency, retrieval quality, process-memory drift, MPS or thermal behavior. The old explicit-eight failures, paired runs and final soak remain open. No issue closure, main merge or release. Only synthetic test data and public model artifacts were used; no personal memories, transcripts or credentials.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant