Repository navigation
test(memory): say what the mixed-load budget caught when it expires - #12309
Conversation
test_private_store_mixed_load gives its whole task set 45 seconds and fails with a bare TimeoutError. On a loaded shard that expiry carries no information: a store that never finished, an embed queue that never drained and a starved worker all look identical, so the failure is unactionable and the usual response is a re-run. The tasks now carry store/kind/index names, and the timeout path reports what the budget caught before re-raising unchanged. It reports the CANCELLED tasks rather than the unfinished ones on purpose -- wait_for cancels the gather before it raises, so an unfinished count is always zero and says nothing. Diagnostics only. The workload, the 45-second budget, the task ordering and every assertion are untouched, and a passing run never enters this path.
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- test/test_memory_multistore_stress.py:198 -- A recall deadline reaches False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm the first real shard timeout relays the MIXED_LOAD_TIMEOUT line through xdist's captured-output on the failing worker, as #12308 expects. [DESIGN-REVIEWED] 3ce5a69 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Green at
|
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only diff in test/test_memory_multistore_stress.py -- names the asyncio tasks and prints a MIXED_LOAD_TIMEOUT diagnostic built from state the test already keeps, then re-raises the TimeoutError unchanged; workload, budget, task ordering and assertions are untouched, so a green run never takes the new path.
…irodotdev#12309) test_private_store_mixed_load gives its whole task set 45 seconds and fails with a bare TimeoutError. On a loaded shard that expiry carries no information: a store that never finished, an embed queue that never drained and a starved worker all look identical, so the failure is unactionable and the usual response is a re-run. The tasks now carry store/kind/index names, and the timeout path reports what the budget caught before re-raising unchanged. It reports the CANCELLED tasks rather than the unfinished ones on purpose -- wait_for cancels the gather before it raises, so an unfinished count is always zero and says nothing. Diagnostics only. The workload, the 45-second budget, the task ordering and every assertion are untouched, and a passing run never enters this path.
Closes the first step of #12308.
test_private_store_mixed_loadgives its whole task set 45 seconds and fails with abare
TimeoutError. On a loaded shard that expiry carries no information: a store thatnever finished, an embed queue that never drained and a starved worker all look
identical, so the failure is unactionable and the usual response is a re-run.
What changed
The operation tasks now carry
store/kind/indexnames, and the timeout path reportswhat the budget caught before re-raising the exception unchanged:
It reports the cancelled tasks rather than the unfinished ones, and that detail is
the whole point of the change.
wait_forcancels the gather before it raises, so bythe time a handler runs nothing is left "not done" -- an unfinished count is always
zero and reports nothing. The tasks the budget actually caught are the cancelled ones;
everything else finished in time. I had it the wrong way round first and the report
came out empty.
Every field comes from state the test already keeps:
stats,calls,latencies,started, and the embedder's own queue.What did not change
The workload, the 45-second budget, the task ordering and every assertion are
untouched. A passing run never enters this path. Naming a task is metadata and does not
affect scheduling.
Verification
The happy path is unchanged: all 4 tests in the file pass in 5.1s, and flake8 and black
are clean.
For the failure path, I could not force a faithful timeout inside the test. Shrinking
the budget to 1ms or 600ms breaks the test's own preconditions -- tasks get cancelled
mid-setup and a
CancelledErrorescapes fromexecutors.pybefore the budget'sTimeoutErroris the thing that propagates -- so a forced run exercises a differentfailure than the real one. Rather than claim more than I checked, I verified the
report's logic verbatim against real
asyncioTasks in a standalone harness: with 12tasks and a budget that catches 9,
t.cancelled()separates the 9 from the 3 thatfinished, and the JSON above is that harness's real output. The
except TimeoutErrorentry, the print and the re-raise were confirmed the same way.
So the mechanism is verified and the shape of the report is verified; what is not
verified is this exact handler firing inside a genuinely loaded shard, which is the
condition #12308 exists to make legible. The first real occurrence will confirm it, and
if the report comes back empty or malformed it costs a follow-up rather than a wrong
diagnosis, because the exception is re-raised either way.