Repository navigation
fix(test): stop the mixed-load queue wait betting on the scheduler - #12347
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 850123f |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
…ling bet The wait for the first embed job to reach the queue was capped at five seconds of wall clock. What it waits for is the scheduler running one of the few hundred tasks created immediately above it far enough to enqueue, so the cap bounded the SCHEDULER rather than the work. A shard running the suite in a four-worker pool on a small runner loses that bet while a slower but less contended shard wins it, which is why the failure is deterministic per branch and absent on other branches running the same test. The bound stays, at sixty seconds, and changes role: it is a lost-run ceiling rather than a scheduling budget, and expiry now raises an AssertionError naming the task count, how many finished and the queue depth. So a genuine nothing-ever-enqueues regression fails readably here instead of spinning to the per-test timeout, which under CI's --max-worker-restart=0 kills the whole job rather than one test. Polls rather than waiting on _jobs_changed because the dispatch loop CLEARS that event, making it a level that can be lowered between the enqueue and an observer, while qsize is the condition the test is about. Refs #12308
3a124ad to
850123f
Compare
GPT 5.6 Review —
|
GPT 5.6 Review —
|
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean (no Rule A-D match), security checklist all-NO, all five reviewer checks success on the latest run per name on head SHA 850123f (the earlier GPT 5.6 Review failure at 19:32:46Z is superseded by success at 19:45:36Z), readiness label (19:58:40Z) postdates the head commit (19:31:52Z). Category: test-only change to test/test_memory_multistore_stress.py -- the queue wait was betting on the scheduler with a 5s cap that a contended four-worker shard loses, so it becomes a 60s lost-run ceiling with a 10ms poll and an AssertionError naming the task/queue state it saw, instead of spinning into the per-test timeout that kills the whole job under --max-worker-restart=0. No production file touched.
SAST: Semgrep latest run success; its single annotation is GitHub's own level=notice runner-image migration notice at .github:1, not a Semgrep finding. CodeQL success; the PR-scoped open-alert query returns main's existing baseline unchanged (identical alert numbers 867-915), so this PR introduces no new alert.
…ling bet (kirodotdev#12347) The wait for the first embed job to reach the queue was capped at five seconds of wall clock. What it waits for is the scheduler running one of the few hundred tasks created immediately above it far enough to enqueue, so the cap bounded the SCHEDULER rather than the work. A shard running the suite in a four-worker pool on a small runner loses that bet while a slower but less contended shard wins it, which is why the failure is deterministic per branch and absent on other branches running the same test. The bound stays, at sixty seconds, and changes role: it is a lost-run ceiling rather than a scheduling budget, and expiry now raises an AssertionError naming the task count, how many finished and the queue depth. So a genuine nothing-ever-enqueues regression fails readably here instead of spinning to the per-test timeout, which under CI's --max-worker-restart=0 kills the whole job rather than one test. Polls rather than waiting on _jobs_changed because the dispatch loop CLEARS that event, making it a level that can be lowered between the enqueue and an observer, while qsize is the condition the test is about. Refs kirodotdev#12308
Problem / Motivation
test_private_store_mixed_load[8]fails with a bareTimeoutErroron some branches andnot others, deterministically per branch rather than intermittently. The failing line is
not the one the test's own comment prepares you for:
That wait waits for the FIRST embed job to reach
backend._jobs. The jobs it waits forcome from a few hundred tasks created immediately above it, so what the five-second cap
actually bounded was the scheduler running one of those tasks far enough to enqueue. It
bounded the scheduler, not the work.
Why it matters
The bet is lost or won by the shard, not by the change under test. Measured on
Backend Tests (3.12, 7):4 of 8 workers (14.0 GiB free, 15 GiB installed),4 workers [14383 items]-- and finished in 700.29s.SLOWER, and green.
The slower shard passed and the faster one failed, which rules out total load and leaves
contention for the scheduler. Three attempts across two heads of one PR failed on this
exact line while nine sampled controls passed, and none of that PR's test files appear in
the shard's log at all -- so it presents to a contributor as "my branch broke a memory
stress test" when the branch contributes nothing to that shard.
The failure is also opaque. #12309 added a
MIXED_LOAD_TIMEOUTreport so an expiry wouldsay which stores and operations were outstanding, but it wraps the 45-second
asyncio.wait_for(asyncio.gather(*tasks), 45)further down. This is a different wait, sothat payload can never appear for it however often it fires -- zero occurrences in the
failing logs.
How our fix solves it
The bound stays and changes ROLE. Sixty seconds is twelve times the old cap and half this
test's own
--timeout=120, so it is a lost-run ceiling rather than a scheduling budget,and expiry is now a readable failure:
An unbounded poll was the first shape of this fix and it was wrong: a genuine
nothing-ever-enqueues regression would spin to the per-test timeout, and under CI's
--max-worker-restart=0(ci.yml overriding setup.cfg's=2) that kills the whole jobrather than one test. Turning a readable assertion into a lost run is a worse trade than
the flake.
AUTOSDE.yaml'sa-ratchet-may-only-tightennames this shape directly andprescribes the remedy used here: wait on what the production path resolves, and bound it
only as a lost-run ceiling.
It polls rather than waiting on the
_jobs_changedevent that production sets one lineafter the enqueue, because the dispatch loop CLEARS that event: it is a level that can be
lowered between the enqueue and an observer, while
qsizeis the condition this test isabout.
Tests
pytest -n0 test/test_memory_multistore_stress.py-- 4 passed in 7.10s. flake8, the blackgate and the comment-history gate are clean.
The real trigger is CI-only, so the mechanism is demonstrated directly. A probe models
only what differs and reads all three paths:
Cases 1 and 2 fill the queue after six seconds, which is what a contended pool does to the
first enqueue. Case 3 never fills it, which is the regression the ceiling exists for; it
uses a short ceiling so the probe is quick, and what it demonstrates is the failure SHAPE
rather than the number.
Pattern harvest
Rule candidate: semgrep
Pattern: a test that caps a wait on work produced by tasks it created itself, so the
cap bounds the scheduler rather than the work
A timeout is not one thing. This one was doing two jobs badly: bounding how long the
scheduler may take to run a task, and bounding how long the suite may spend before
declaring a lost cause. The first is a bet whose odds are set by the runner, and losing it
looks like a defect in whichever branch drew the busy shard. The second is a real ceiling
and must exist. Separating them is the whole fix -- keep a generous ceiling, make its
expiry say what it saw, and stop treating the number as a scheduling budget.
A second, unmechanised observation: the cheapest attribution evidence here was a
comparison nobody had made. The same lane on another PR ran the same test SLOWER and
passed, which killed the load explanation in one reading, while the count of green
controls on its own only looked like the branch's fault. Two controls that differ in the
direction you expect are worth more than nine that agree.
Refs #12308
Body re-published so the hygiene check reads a payload containing the Pattern harvest section; the section itself was added before the previous push.