Repository navigation
Stage2 cpp engine - #1
Merged
Merged
Conversation
anytime_runtime is a pybind11 extension running ONNX Runtime in the caller's process. The subprocess worker and the Python wheel backend are both retained, so the new tensor path is checked against them rather than trusted. Measured on DistilBERT at batch size one, 60 requests after warm-up: backend inference p50 wall p50 transport python 13.667 ms 13.673 ms 0.006 ms extension 13.609 ms 13.628 ms 0.018 ms subprocess 13.629 ms 13.926 ms 0.296 ms All three agree bitwise on the logits and their inference times agree within 0.4%. Transport falls from 0.296 ms to 0.018 ms, which is not the reason for the change: a subprocess boundary rules out batching across requests and sharing a KV cache between them. The ONNX Runtime version is no longer written down twice. CMake reads onnxruntime.__version__ from the target interpreter and resolves an SDK to match, downloading it once into ~/.cache/anytime-inference-planner. Equality is enforced at configure time, asserted at compile time against ORT_API_VERSION read from the resolved headers, and re-checked when the extension is imported. Stage 1 built the worker against 1.20.1 while profiling against the 1.26.0 wheel and measured DistilBERT at 98.9 ms against 13.0 ms inside session->Run(); under pybind11 both copies are loaded at once, so this is a correctness gate rather than a performance one. The CI job that pinned ONNXRUNTIME_VERSION is gone, which also resolves the drift that had CI building against 1.26.0 while installing 1.28.0. The version probe clears PYTHONPATH so it reads the target environment rather than pip's build overlay, falling back to the overlay only on a first install into a clean environment, where no wheel exists yet. A clean install was verified to resolve 1.28.0 consistently for both the extension and the wheel. Tensors cross the boundary without copying. Inputs borrow the numpy buffer, whose references are held for the whole call; outputs are numpy views over ONNX Runtime's buffers, kept alive by a capsule owning the Ort::Value. A non-contiguous input is made contiguous rather than misread. The GIL is released around inference, without which a pool would serialise and the M/M/c model would describe a machine that does not exist. The backend comparison skips a backend that is not built, which would let a job compare one backend against itself and report success. ANYTIME_REQUIRE_BACKENDS turns that into a failure, and the new engine CI job builds both native backends against one SDK and requires all three.
The extension agreed bitwise with the subprocess worker on DistilBERT and their inference times agreed within 0.4%, which is what the previous commit existed to establish. Deleting it now rather than earlier meant the replacement was checked against the thing it replaces. Gone: runtime_cpp/ (the C++ worker, its hand-rolled JSON reader and base64 codecs), _SubprocessBackend and the Python-side codecs, find_runtime_binary and ANYTIME_RUNTIME_BIN, the binary= parameter on RuntimeClient and RuntimePool, and tests/test_runtime_subprocess.py. The two scripts that passed a binary explicitly no longer choose a backend, since the engine is selected automatically. Two backends remain: the extension, which is the serving path, and ONNX Runtime's Python API, which is the reference the tests compare against and the fallback where the extension has not been built. That fallback now warns that measurements taken through it do not describe the serving path. docs/runtime.md was about the JSON protocol end to end, so it is rewritten rather than patched: what survives is the version-matching rationale, which is the part that mattered, plus the measured transport comparison. The CI runtime job that only checked the worker's CLI contract is gone, and the engine job now asserts the extension links the installed wheel.
Service profiles were measured through a separate ONNX Runtime session rather than through the path that serves traffic. That is the gap that let Stage 1's 7.6x version mismatch hide: the worker and the profiler were different processes running different builds, and nothing compared them, so DistilBERT measured 98.9 ms in one and 13.0 ms in the other while every test passed. profile_variants.py now measures through the same RuntimeClient the server dispatches to, and scores accuracy through it as well, so the logits are the served ones. Closing that gap alone would only move the blind spot, so each variant is also run through a separate session and the two are required to agree within --agreement-tolerance, default 15%. A divergence beyond that aborts before anything is written to disk. Measured on this host, engine against separate session: distilbert_fp32 0.991x minilm_fp32 1.014x distilbert_int8 0.994x minilm_int8 0.999x Tightening the bound to 0.001 was used to confirm the check fires and writes nothing. Profiling also refuses to run at all when the extension is unavailable, since the Python fallback does not describe the serving path; --allow-fallback-backend records those numbers deliberately. Both paths were verified in an environment without the extension built. service_time_ms is now wall time per request through the client rather than time inside a bare session, which is what the pool actually spends. The reported profile records both, plus the agreement ratio, the backend used, and the ONNX Runtime version each side linked. configs/serving.yaml is deliberately not regenerated here: re-running moves the service times by roughly 4% under the new definition, which would leave the committed load sweep and its figure describing a different measurement. Noted in docs/benchmarks.md so the three are regenerated together. The INT8 result reproduces: on arm64 both quantised graphs are slower and no more accurate than the FP32 graphs they replace, so both remain dominated.
The service profile, configs/serving.yaml, results/load_sweep.csv, and docs/img/load_sweep.png were produced by the Stage 1 subprocess worker. They are now regenerated in one pass through the in-process engine so that all four describe the same measurement, along with the numbers quoted in README.md, benchmarks.md, quantization.md, and planner.md. The result does not move. Service times reproduce Stage 1 within their own spread, 12.89 against 12.79 ms for DistilBERT and 5.19 against 5.21 ms for MiniLM, and all four accuracies reproduce exactly: 91.06, 90.60, 90.14, 90.14. Replacing the transport did not change what is being measured. A single p50 was more precise than the measurement deserved. Measuring the same 200-request p50 eight times gave 13.84 to 14.73 ms for DistilBERT and 5.62 to 5.83 ms for MiniLM, ranges of 6.5% and 3.6%, driven by thermal state rather than by anything in the code, while the within-pass stdev is around 0.45 ms and says nothing about it. profile_variants.py now takes --repeats independent passes, default 3, reports the median of their medians, and records the range across them as service_time_spread_pct. An earlier single-pass run of this same regeneration read 13.93 ms for DistilBERT, which is how the outlier was noticed. Sweep headline at rho = 0.95: 281 rps of goodput against 43, at 49% of the compute cost, previously 287 against 37. The ratio moves from 7.8x to 6.5x because the accurate-only denominator is computed from however few requests admission lets through, which is the noisy end. Repeating the whole sweep twice more with the same seed moved goodput by at most 6% and the rho = 0.95 adaptive figure by 2%. Two things the new data shows that the old tables did not: At rho = 0.60 the attainment curves cross. Adaptive attains 84.5% against 85.1% while admitting 99.8% against 94.4%. Admitting nearly everything puts more work in the queue, so attainment among admitted requests does not improve even though goodput does, 161 against 153 rps. Recorded rather than smoothed over: it is the one point where the adaptive policy is not better on every axis. The sweep was previously run with --duration 2 while the documented reproduction command defaults to 3, so that command never reproduced the committed numbers. Regenerated at the default. Also documents that --quick numbers must not be reported: it scores accuracy on 128 of the 872 validation examples, which moves the figure by half a point.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.