Skip to content

fix(test): stop the mixed-load queue wait betting on the scheduler - #12347

Merged
iamwhatever merged 1 commit into
mainfrom
fix/stress-queue-wait-scheduling
Sep 20, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/stress-queue-wait-scheduling

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

test_private_store_mixed_load[8] fails with a bare TimeoutError on some branches and
not others, deterministically per branch rather than intermittently. The failing line is
not the one the test's own comment prepares you for:

>           await asyncio.wait_for(wait_for_queue(), 5)
asyncio/timeouts.py:115: TimeoutError

That wait waits for the FIRST embed job to reach backend._jobs. The jobs it waits for
come from a few hundred tasks created immediately above it, so what the five-second cap
actually bounded was the scheduler running one of those tasks far enough to enqueue. It
bounded the scheduler, not the work.

Why it matters

The bet is lost or won by the shard, not by the change under test. Measured on
Backend Tests (3.12, 7):

  • Failing shard: the suite runs in a FOUR-worker pool -- 4 of 8 workers (14.0 GiB free, 15 GiB installed), 4 workers [14383 items] -- and finished in 700.29s.
  • Passing shard on another open PR: same lane, same test present, full scope, 808.43s.
    SLOWER, and green.

The slower shard passed and the faster one failed, which rules out total load and leaves
contention for the scheduler. Three attempts across two heads of one PR failed on this
exact line while nine sampled controls passed, and none of that PR's test files appear in
the shard's log at all -- so it presents to a contributor as "my branch broke a memory
stress test" when the branch contributes nothing to that shard.

The failure is also opaque. #12309 added a MIXED_LOAD_TIMEOUT report so an expiry would
say which stores and operations were outstanding, but it wraps the 45-second
asyncio.wait_for(asyncio.gather(*tasks), 45) further down. This is a different wait, so
that payload can never appear for it however often it fires -- zero occurrences in the
failing logs.

How our fix solves it

The bound stays and changes ROLE. Sixty seconds is twelve times the old cap and half this
test's own --timeout=120, so it is a lost-run ceiling rather than a scheduling budget,
and expiry is now a readable failure:

try:
    await asyncio.wait_for(wait_for_queue(), 60)
except TimeoutError:
    finished = sum(1 for task in tasks if task.done())
    raise AssertionError(
        "no embed job reached the queue within 60s: "
        f"{len(tasks)} operation task(s) created, {finished} already finished, "
        f"queue depth {backend._jobs.qsize()}"
    ) from None

An unbounded poll was the first shape of this fix and it was wrong: a genuine
nothing-ever-enqueues regression would spin to the per-test timeout, and under CI's
--max-worker-restart=0 (ci.yml overriding setup.cfg's =2) that kills the whole job
rather than one test. Turning a readable assertion into a lost run is a worse trade than
the flake. AUTOSDE.yaml's a-ratchet-may-only-tighten names this shape directly and
prescribes the remedy used here: wait on what the production path resolves, and bound it
only as a lost-run ceiling.

It polls rather than waiting on the _jobs_changed event that production sets one line
after the enqueue, because the dispatch loop CLEARS that event: it is a level that can be
lowered between the enqueue and an observer, while qsize is the condition this test is
about.

Tests

pytest -n0 test/test_memory_multistore_stress.py -- 4 passed in 7.10s. flake8, the black
gate and the comment-history gate are clean.

The real trigger is CI-only, so the mechanism is demonstrated directly. A probe models
only what differs and reads all three paths:

old: bare TimeoutError after 5.0s
new: reached the queue after 6.0s
new-regression: AssertionError -- no embed job reached the queue within 2s: 255 operation
                task(s) created, 0 already finished, queue depth 0

Cases 1 and 2 fill the queue after six seconds, which is what a contended pool does to the
first enqueue. Case 3 never fills it, which is the regression the ceiling exists for; it
uses a short ceiling so the probe is quick, and what it demonstrates is the failure SHAPE
rather than the number.

Pattern harvest

Rule candidate: semgrep
Pattern: a test that caps a wait on work produced by tasks it created itself, so the
cap bounds the scheduler rather than the work

A timeout is not one thing. This one was doing two jobs badly: bounding how long the
scheduler may take to run a task, and bounding how long the suite may spend before
declaring a lost cause. The first is a bet whose odds are set by the runner, and losing it
looks like a defect in whichever branch drew the busy shard. The second is a real ceiling
and must exist. Separating them is the whole fix -- keep a generous ceiling, make its
expiry say what it saw, and stop treating the number as a scheduling budget.

A second, unmechanised observation: the cheapest attribution evidence here was a
comparison nobody had made. The same lane on another PR ran the same test SLOWER and
passed, which killed the load explanation in one reading, while the count of green
controls on its own only looked like the branch's fault. Two controls that differ in the
direction you expect are worth more than nine that agree.

Refs #12308

Body re-published so the hygiene check reads a payload containing the Pattern harvest section; the section itself was added before the previous push.

@chenmingwei23
chenmingwei23 requested review from a team and CrysisDeu September 20, 2026 19:19
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 850123f4dda4917f7b0d6f433e777df991caae99 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 850123f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 850123f4dda4917f7b0d6f433e777df991caae99: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 850123f4dda4917f7b0d6f433e777df991caae99 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 850123f

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 850123f4dda4917f7b0d6f433e777df991caae99 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 850123f

Verdict parsed from the review's SHA-scoped output markers for commit 850123f4dda4917f7b0d6f433e777df991caae99.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 850123f4dda4917f7b0d6f433e777df991caae99: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
…ling bet

The wait for the first embed job to reach the queue was capped at five seconds of
wall clock. What it waits for is the scheduler running one of the few hundred
tasks created immediately above it far enough to enqueue, so the cap bounded the
SCHEDULER rather than the work. A shard running the suite in a four-worker pool
on a small runner loses that bet while a slower but less contended shard wins it,
which is why the failure is deterministic per branch and absent on other branches
running the same test.

The bound stays, at sixty seconds, and changes role: it is a lost-run ceiling
rather than a scheduling budget, and expiry now raises an AssertionError naming
the task count, how many finished and the queue depth. So a genuine
nothing-ever-enqueues regression fails readably here instead of spinning to the
per-test timeout, which under CI's --max-worker-restart=0 kills the whole job
rather than one test.

Polls rather than waiting on _jobs_changed because the dispatch loop CLEARS that
event, making it a level that can be lowered between the enqueue and an observer,
while qsize is the condition the test is about.

Refs #12308
@chenmingwei23
chenmingwei23 force-pushed the fix/stress-queue-wait-scheduling branch from 3a124ad to 850123f Compare September 20, 2026 19:32
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 3a124ad304e5c552cb187e611c09e5ca4347264b; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3a124ad304e5c552cb187e611c09e5ca4347264b: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 850123f4dda4917f7b0d6f433e777df991caae99; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 850123f4dda4917f7b0d6f433e777df991caae99: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 20, 2026 20:05

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean (no Rule A-D match), security checklist all-NO, all five reviewer checks success on the latest run per name on head SHA 850123f (the earlier GPT 5.6 Review failure at 19:32:46Z is superseded by success at 19:45:36Z), readiness label (19:58:40Z) postdates the head commit (19:31:52Z). Category: test-only change to test/test_memory_multistore_stress.py -- the queue wait was betting on the scheduler with a 5s cap that a contended four-worker shard loses, so it becomes a 60s lost-run ceiling with a 10ms poll and an AssertionError naming the task/queue state it saw, instead of spinning into the per-test timeout that kills the whole job under --max-worker-restart=0. No production file touched.

SAST: Semgrep latest run success; its single annotation is GitHub's own level=notice runner-image migration notice at .github:1, not a Semgrep finding. CodeQL success; the PR-scoped open-alert query returns main's existing baseline unchanged (identical alert numbers 867-915), so this PR introduces no new alert.

@iamwhatever
iamwhatever merged commit ad68b15 into main Sep 20, 2026
108 of 117 checks passed
@iamwhatever
iamwhatever deleted the fix/stress-queue-wait-scheduling branch September 20, 2026 20:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 20, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…ling bet (kirodotdev#12347)

The wait for the first embed job to reach the queue was capped at five seconds of
wall clock. What it waits for is the scheduler running one of the few hundred
tasks created immediately above it far enough to enqueue, so the cap bounded the
SCHEDULER rather than the work. A shard running the suite in a four-worker pool
on a small runner loses that bet while a slower but less contended shard wins it,
which is why the failure is deterministic per branch and absent on other branches
running the same test.

The bound stays, at sixty seconds, and changes role: it is a lost-run ceiling
rather than a scheduling budget, and expiry now raises an AssertionError naming
the task count, how many finished and the queue depth. So a genuine
nothing-ever-enqueues regression fails readably here instead of spinning to the
per-test timeout, which under CI's --max-worker-restart=0 kills the whole job
rather than one test.

Polls rather than waiting on _jobs_changed because the dispatch loop CLEARS that
event, making it a level that can be lowered between the enqueue and an observer,
while qsize is the condition the test is about.

Refs kirodotdev#12308
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants