Repository navigation
perf: model server discards batch controls before embedding and reranking #735
Description
R01: perf: model server discards batch controls before embedding and reranking
- Priority: P1
- Evidence: reproduced
- Audited source:
063e5b8844af735a52fde886217a5d26a0f13064 - Audit date: 2026-10-08
- Disposition: new
Problem
Caller-specified batch_size is accepted by both proxy APIs but discarded from the wire request. Independently, the server calls DynamicThrottler.before_batch() and discards the returned safe batch size. It sends the entire embedding request to encode() without a batch override and always reranks with batch_size=64. A caller or throttler requesting batch=1 therefore cannot impose that allocation boundary. The prior throttler and proxy abstractions look protective but are disconnected from the operation they must control.
Advertised or expected contract
The shared-server resource budgets in docs/resource-budgets.md:8-19 and batch-sizing contract in tier_switch/throttler.py:78-95. #577 promises recovery that remains responsive under concurrent ingestion.
Source evidence
- truememory/model_client.py:494-529: Both proxy methods accept **kwargs, but serialize only texts/tier or pairs/model_name.
- truememory/model_server.py:529-543: before_batch() return value is unused; encode() receives all texts and no batch_size.
- truememory/model_server.py:580-612: Both initial rerank and CPU retry hardcode batch_size=64.
- truememory/tier_switch/throttler.py:78-95: The documented return value is the adaptive batch limit.
Reproduction
- Run
PYTHONDONTWRITEBYTECODE=1 python3 -B docs/audits/2026-10-08-v1/evidence/runtime-repro-v2.py. Read probes R01_batch_controls in the JSON output. - The extracted original EmbeddingProxy and RerankerProxy send requests with batch_size=1 to a recording transport stub.
- The original server handler receives four texts while a fake throttler returns (1, {}). Only model inference is stubbed.
Observed result
Both serialized requests omit batch_size. The fake encoder receives four texts in one call despite the throttler returning one. No real models, GPU allocations, or performance benchmark were used.
Expected result
The effective inference batch is constrained by validated caller limits, the server resource budget, and padded token lengths. Every microbatch preserves output order, dtype, dimension, and retrieval semantics.
Proposed implementation
- Define a validated, backward-compatible batch_size field for the model protocol; preserve supported proxy kwargs explicitly and reject unsupported options rather than silently discarding them.
- Have the shared server consume the adaptive limit and split embedding/reranking work into bounded microbatches. Use measured token/padding cost in addition to item counts.
- Place deadline and resource checks between microbatches, including CPU retry; retain the existing embedding models and rerankers.
Acceptance tests
- With requested batch_size=1, record every encoder/predict invocation and assert none receives more than one item.
- With requested batch_size=8 and server limit=2, record microbatches [2,2,1] for five inputs, with identical output order and contents.
- Test empty input, one input, mixed lengths, duplicate texts, CPU fallback, and explicit caller batch sizes through the actual client-server protocol.
- On the same hardware and fixed corpus, report CPU seconds, wall time, physical footprint peak, MPS peak, and thermal-pressure samples before/after; do not infer heat from latency alone.
Dependencies and sequencing
- R03
- R04
- R05
Regression risks
- Overly small batches reduce throughput; overly large token budgets reproduce memory spikes.
- Changing model identity, input truncation, or ranker choice would exceed this fix scope.
Scope and affected paths
Current main shared-model protocol and inference batching. No model substitutions, blanket input truncation, schema migration, or retrieval-scoring change.
Related history
Related closed #577; this disconnected batch-control mechanism is not described as a resolved item there. Related #334 contains unverified proposed batch estimates, not a usable acceptance baseline.
Uncertainty and measurement limits
The live model-server heap has not been attributed to a specific request. This reproduction proves loss of caller/throttler control, not the number of GiB saved by a future fix.
Release constraints
Preserve current embedding models, rerankers, internal retrieval depth, and durable source data. A synthetic control-flow reproduction establishes the stated defect only; hardware savings and retrieval quality require the separate release validation gates. Do not close this issue merely because a unit test passes.
Audit package
Full plan and catalogue | Portable reproductions and receipts
Activity
buildingjoshbetter commentedon Oct 8, 2026
Native retrieval layout investigation v1
Status
The controlled run completed in 20.854 seconds. Its six bounded-batch numerical comparisons failed; all three native32 controls passed. Exit code 2 records that failure. The container stopped with verified cleanup, no timeout and no OOM. Performance release acceptance remains held.
Reproduction and provenance
Use the frozen, wholly generated fixture with 256 documents and 32 queries. Encode the same documents with the same Qwen3 0.6B model instance, CUDA BF16, SDPA, and dimension 256. Compare native32 global sorting, global8 sorting, and the current contiguous outer8 calls. Rotate layout order over three repetitions. Encode the 32 queries once and hold them fixed. Retrieve the actual first 100 SQLite candidates for each query.
The workload completed exactly 2,336 input rows and 248 native forwards: 2,304 document rows in 216 forwards, plus 32 query rows in 32 forwards. All 288 source tokenizations matched untruncated token IDs and masks. The longest input was 569 tokens, below the 1,024-token diagnostic ceiling. No text, model, precision, dimension or candidate-depth change was made.
Raw result SHA256:
84a177e4c4b0f4956f714710d500e25e54d099045913e1f6cd9369ac6ede1945.Fixture SHA256:
e5ca8af6291cbdb9c7e641ec1d2040baf43437b602626f1d3e3b81122cc4d38c.Driver SHA256:
1c1eb05df3d48be35e264663155e33be9156409dda78fc8cda60ef2d5e451464.Runtime source anchor:
d5ca683b8f0f07c00edec2071209add5a79c42f6.Actual SQLite top-100 results
All three repetitions produced exactly the same vectors and retrieval results within each layout. Root and an independent reviewer recomputed all 288 actual-candidate Recall, MRR and NDCG values from raw SQLite IDs and fixture relevance labels without using the driver's comparison implementation. The actual SQLite and canonical rankings had no top-100 membership difference or exact boundary tie in this fixture.
Global8 changes three first results: two become worse under the generated labels and one improves. Outer8 changes two first results, both worse. All labeled relevant documents remain among the 100 candidates. The arithmetic is explicit: outer8 MRR change is
0.5870155886 - 0.6182655886 = -0.03125; NDCG change is0.5739113313 - 0.5922074674 = -0.0182961361.Both bounded layouts fail the frozen
abs(reference - candidate) <= 0.0001 + 0.001 * abs(reference)gate, with maximum absolute difference 0.375 and minimum cosine approximately 0.9997271532. Similar vector directions do not imply identical first results.Cause and boundaries
The earlier padding diagnostic held per-row tokens, masks and positions constant and still found output differences when the native batch dimension changed. This run establishes a repeatable retrieval consequence of that batching change on this fixture. It does not identify a particular CUDA kernel or reduction as the cause.
The direct encode timing compares three calls on a generated corpus. It is not RPC p95, a sustained throughput test, an MPS measurement, or an energy/thermal result. Synthetic labels are not human judgments. Neither the reranker nor the hybrid search engine was executed here. Unchanged Recall100 does not certify final answer or reranked-order equivalence.
Next checkpoint
Keep the failed numerical and ranking findings attached to issue #735 and tracker #771. Continue independently validated lifecycle, allocation and storage work. Do not ship the batch-control change as quality-preserving based on cosine or Recall100 alone. Any further native experiment must state a specific hypothesis, preserve this evidence and the original acceptance gate, and retain the owned-container limits. A separate fixed-score test using the existing reranker could isolate consequences of changed candidate membership; it would not erase this vector-stage result or certify the full hybrid engine.
buildingjoshbetter commentedon Oct 8, 2026
Batch-policy attribution clarification
The recorded native output and timing failures remain valid. Read-only review found that the frozen workload explicitly requests
batch_size=8; the previous proxy discarded that option, while the candidate now preserves it. Eight is not the production default. Current embedding and reranking ceilings remain 32 and 64.For the 32-input embedding case, the old behavior uses one native32 forward and the candidate uses four outer8 calls. For 100 rerank pairs, native64 plus36 becomes twelve8 plus4, or13 calls. The earlier comparison therefore measures repaired explicit-control behavior against formerly ignored controls. It does not, by itself, establish an unchanged-default regression.
Evidence anchors: workload
requested_model_batch_size;model_request()and proxy calls in the frozen performance harness;model_client._batch_request; server embedding/reranking limit selection. The current source preserves omitted options as omitted.A separately identified default-policy diagnostic is being planned. It will omit
batch_size, retain the same models, dtype, dimensions and abs0.0001/rel0.001 tolerance, and include mixed embedding requests larger than32 to exercise outer grouping. Explicit-eight control enforcement and comparison against native32 will remain separately reported. No existing failure, quality gate or release hold is removed, and no production batch change is justified by this clarification alone.buildingjoshbetter commentedon Oct 8, 2026
Default-policy diagnostic: completed, release gate still held
This separates omitted-default requests from explicit batch-eight requests. Source is
f660c8cb9853716efe34a743e7319e1102b6508b; embedder, reranker, dimensions, precision, model revisions and frozen numerical tolerances are unchanged. Default batch controls remain 32 embeddings and 64 rerank pairs. Actual ModelServer handlers are exercised in process; no socket/RPC timing is measured.Three separately bounded cells completed all 90 timed records, three warmups, 3,741 input rows and 267 native forwards. Each cell loaded exactly one model. Controller times were 11.561, 17.170 and 11.452 seconds. All owned containers stopped with verified cleanup, no timeout and no OOM. Exact source/artifact, request, order, output-shape and work-cap checks passed. Root and an independent reviewer recomputed all 90 numerical comparisons from raw outputs, reference first.
Every mode repeated exactly within itself. Across the ten cases and three repeats, default passes 24/30 comparisons, explicit-eight passes 21/30, and all 30 native-reference repeatability comparisons pass. Reranker candidate scores pass tolerance but are not exactly identical. No threshold was changed:
abs(candidate-reference) <= 0.0001 + 0.001 * abs(reference).For default mixed33, two of 33 rows fail and maximum absolute difference is 0.21875. For default mixed100, 69 of 100 rows fail and maximum absolute difference is 0.4375. These results are stable across all three repeats. For explicit-eight mixed32, eight of 32 rows fail and maximum absolute difference is 0.34375, preserving the prior finding.
Timing and grouping evidence
Medians of three serial, instrumented operations:
For 100 mixed embeddings, the default/reference ratio is
1000.544282 / 390.950397 = 2.55926. Both use four native forwards with row counts[32,32,32,4]. Their sequence widths differ: default[778,778,778,212], globally sorted native reference[778,70,70,70]. Equal batch counts therefore do not imply equal grouping, padding or compute. Source inspection confirms outer caller-order slicing precedes each library call's sorting. The next correction is being designed against that evidence while retaining deadline, admission and recovery guards.Faster explicit-eight cases are not accepted as improvements when their numerical gate fails. This does not evaluate retrieval ranking, concurrent traffic, production p95, whole-process memory drift, MPS or thermal behavior. Existing retrieval failures and final paired/soak requirements remain open. No issue is closed and no release is approved.
Evidence identities
d61238cd456de3b550b817b522848506f947196641e3301dc84e010cf9d20cf973019282da3a96d377cc227ac28b1cc02f9485da8458545f26313c4014ef1d50fd0585cf492a29c20e623399f808bf93e8d22e9dbd1427de4b54f7cd532936e5744a42e88ff397de0c5c17ab213ff1db188304e980614425ce189a7a0a82b5c5c8db05c5476173395c56d9682c9f1043e59f7800efba90e37f4b503fefe61ee55268f265bbb8abe80aefd58a046d3ecb7fedf74be30198ec0e28393994deb70dOnly public model artifacts and generated synthetic inputs were used. No personal memories, transcripts or credentials were included.
buildingjoshbetter commentedon Oct 8, 2026
Global request ordering: default parity restored; release gate still held
Draft #811 preserves the native Qwen request order across bounded calls. Source
2844072af8857a903ac9750a0f54026f9a33a51ckeeps models, model revisions, dtype, dimensions, defaults and result caps unchanged. Actual ModelServer handlers run in process; this is a direct serial CUDA diagnostic, without socket/RPC timing.Three separately bounded cells completed 90 timed records, three warmups, 3,741 input rows and 267 native forwards. Each loaded one model. Controller durations were 12.087, 15.176 and 11.695 seconds. Cleanup was verified for every owned container, with no timeout or OOM. Root and an independent reviewer recomputed the raw comparisons, work counts, repeatability and timing summaries.
Default passes 30/30 comparisons; explicit-eight still passes 21/30. All 24 default embedding comparisons are exact, and their ordered native feature records match the same-repeat reference: input IDs, attention masks, position IDs or explicit absence, shapes, dtypes and observed options. These recorded digests were compared; tokens were not independently regenerated. Every mode's outputs and feature records repeated exactly across its three repetitions. Reranker scores pass tolerance but are not exact; maximum difference is
4.172325134277344e-7. Its feature-parity gate is not applicable.The unchanged reference-first tolerance is
abs(candidate-reference) <= 0.0001 + 0.001 * abs(reference). Default improvement does not override explicit-eight failures.Instrumented timings
Medians of three operations, including synchronized CUDA work and CPU feature-observation overhead:
For 100 mixed embeddings, default/reference is
403.471858 / 397.146104 = 1.015928027. Both now use native shapes[32x778,32x70,32x70,4x70]. The earlier wrapper measured 1000.544282 ms against its 390.950397 ms reference in a separate diagnostic. This is evidence of the corrected grouping and measured diagnostic timing, not a controlled production p95 comparison.Mixed reranking still has default/reference
631.852592 / 424.322110 = 1.489087128. Its shapes remain[64x779,36x779]versus reference[64x779,36x72]; that performance gap remains open.Explicit-eight regression detail
Old and new native reference arrays are exact. Each finding below repeats identically three times:
The correction worsens mixed32/33 under explicit-eight, and mixed100 has a larger worst error despite fewer failed rows. Faster failed cases are not accepted as improvements. Prior padded-eight evidence already showed differences with matching token IDs, masks and effective positions, so padding alone is not established as the remedy. The specific numerical mechanism remains unproven.
Evidence and limits
9822e3503158b36043c8fa6f111d903c1f2e1b9746fbd49bedffc22d666a5d5e22451e02d3047ddd65aec9b6fbdee6021ba257415220a0cef5da5a17a4518842f923d1c44f9267f216c39bd2b1618e56b20c32cd1bbd44c9da9df8bd97c78f909d9902987844797b2dfb4950081560499d6f041205865af3bebb9c7eea1bf773a035139b3400af0e4dd18aaad4f2c1824dcfebb26a696f865c7cb82312e4af5926ec9c8ec6554a59a9de759a60beb064c226b104e25476f99ecbdda0ac73fa85Combined acceptance remains false. Retrieval rankings, concurrent traffic, production p95, whole-process memory drift, final soak, MPS and thermal behavior remain unvalidated by this diagnostic. Only public model artifacts and generated synthetic inputs were used. No personal memories, transcripts or credentials were included. No issue is closed, main merge performed or release approved.
buildingjoshbetter commentedon Oct 8, 2026
Global-order diagnostic v2: corrected default paths and explicit-eight compatibility
All three bounded cells completed on combined source
fb7306ea716151f6bbef4d4e793e0ac56b45243c: 90 timed records, 3 warmups, 3,741 rows and 267 native forwards. Every container stopped with verified cleanup; there was no OOM or timeout. The raw numerical acceptance gate remains false because prior explicit-eight embedding failures remain.Results
All modes repeat exactly across their three runs, including recorded feature digests. The unchanged reference-first raw bound is
abs(candidate-reference) <= 0.0001 + 0.001*abs(reference). No tolerance, model, dtype, dimension, prompt or retrieval limit changed.Restricting the embedding global-order plan to effective limit32 restores the previous contiguous explicit-eight arrays. The additional mixed32/33 regression introduced by global ordering at limit8 is removed. This does not solve the older explicit-eight failures. Mixed32, mixed33 and mixed100 still fail in all three repeats: 3 cases * 3 repeats = 9 failures.
The restored explicit-eight failures are fully retained:
Compared with global-order v1, mixed100 changes from 24 failed rows / 5,440 components to 83 / 18,950, while its maximum error changes from 0.5 to 0.4375. This is an exact restoration of the old contiguous policy, not an across-the-board numerical improvement or a passing gate. Mixed32/33 return from the v1 maximum error of 0.4375 to their original 0.34375.
The new ModernBERT default plan preserves the complete-request pair order at effective limit64 and scatters scores by original occurrence. Mixed100 now uses native shapes
64x779, 36x72, exactly matching the reference. The previous default used64x779, 36x779. This removes unnecessary padding from its second native forward. All six default reranker comparisons now have exact raw scores and ordered-feature parity.Instrumented medians
Three repetitions per mode; milliseconds include synchronization, feature copies and wrapper observation.
For example,
429.941691 / 424.593396 = 1.012596275. The previous, separate mixed-reranker run measured631.852592 / 424.322110 = 1.489087128. Those distinct runs provide context; they are not a paired production latency or p95 result.Evidence and prerequisites
ddc02c30b55e3727a6eb359cc698d5a365408160048c7d699f14b95696ea922e.a69a832a8a99bb1303acb89bc7f3296937f12c832164088afe1dc596f1b23565.Pinned driver SHA256
066ab7c3800fe33b7cc3fac06970cbc3b734b59652cd9e1c1d19ac953767d1e2; workload6f1c763ec4ea30de21fa2d965abc940463c2df4ee85313f3aba27d12938e6838; 77-file runtimefc6e85a9b33f109e69017948fe0120e74abef0ee7c301c7513cea8ede9d642f0.Retained private raw-report SHA256 values:
407eb4abe37fed2fa1b6f52dd8a3330ce7317d3be8aca21b1639db3a1458db1b.315f6aa7e1bf2fb43dbbdbb5d8fecd8a857379023d89d7ff8b44060980038386.34c16142af8aa129d938f284f6eb17c42ef24e9a5a06e7fc072d5285c845affb.fe444043fc11ad45b07407820889a58f12f312c598a5295ff6e9f42543f9372d.Limits and next gates
These are synthetic serial in-process CUDA diagnostics, not RPC, concurrency, retrieval quality, process-memory drift, MPS, thermal or production p95 acceptance. Feature digests are observed evidence; the controller does not regenerate token tensors. The retained reranker fixture uses a constant query, so varied-query production-body tests cover whole-pair-length semantics separately.
The explicit-eight cause investigation, paired latency runs, retrieval regression gates and final soak remain open. Prior failed results are retained. No issue is closed and no release is approved. Only generated synthetic inputs and public model artifacts were used; no personal memories, transcripts or credentials.