[fix] wire scheduler to worker in in-process load test

[peterajhgraham]

$(cat <<'EOF'

Summary

• Root cause: InferenceWorker.__init__ creates CUDA stream objects on whichever thread calls it (the event-loop thread). Warmup was called synchronously on that same thread. When the dedicated ThreadPoolExecutor later ran _run_batch_sync from a different thread, compute_stream.synchronize() stalled indefinitely — the CUDA driver associates pending ops with the originating thread context, so the executor thread's synchronize never sees them complete.
• _ThreadedWorker adapter: wraps InferenceWorker and dispatches _run_batch_sync via asyncio.to_thread (the loop's shared default thread pool) instead of the worker's private ThreadPoolExecutor. This matches the pattern the FastAPI lifespan already uses for warmup and keeps stream creation, warmup, and inference on consistent CUDA thread contexts.
• Warmup via asyncio.to_thread: ensures CUDA is first initialised inside a thread, not on the event-loop thread — identical to loop.run_in_executor(None, worker.warmup, ...) in app.py.
• await asyncio.sleep(0) after create_task: gives the scheduler one event-loop turn to register its queue.get() waiter before request tasks flood in.

The scheduler → worker path is fully preserved: requests flow through scheduler.submit() → PriorityQueue → Scheduler.run() batch dispatch → _ThreadedWorker.run_batch() → InferenceWorker._run_batch_sync().

Test plan

• Run PYTHONPATH=. python scripts/load_test.py --concurrency 16 --requests 200 --events 256 on the Lambda A10 instance — should complete without hanging and write benchmarks/serving/results.md
• Verify p99 latency is in the expected range given ~4–5 ms warmup inference time
• Confirm --url http://localhost:8080 HTTP mode is unaffected (no changes to run_http or any server code)

https://claude.ai/code/session_01GdEH2sXv2J5QQPWZsXgHFw
EOF
)

---

Generated by Claude Code