Skip to content

test(memory): say what the mixed-load budget caught when it expires - #12309

Merged
iamwhatever merged 1 commit into
mainfrom
fix/stress-timeout-diagnostics
Sep 20, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/stress-timeout-diagnostics

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Closes the first step of #12308.

test_private_store_mixed_load gives its whole task set 45 seconds and fails with a
bare TimeoutError. On a loaded shard that expiry carries no information: a store that
never finished, an embed queue that never drained and a starved worker all look
identical, so the failure is unactionable and the usual response is a re-run.

What changed

The operation tasks now carry store/kind/index names, and the timeout path reports
what the budget caught before re-raising the exception unchanged:

MIXED_LOAD_TIMEOUT {"cancelled_by_kind": {"backfill": 3, "episode": 2, "fact": 2,
"recall": 2}, "cancelled_stores": 4, "concurrency": 8, "elapsed_s": 0.4,
"embed_calls": 3, "embed_queue_depth": 0, "latencies_recorded": 2, "stats": {...},
"tasks_cancelled_by_budget": 9, "tasks_total": 12}

It reports the cancelled tasks rather than the unfinished ones, and that detail is
the whole point of the change. wait_for cancels the gather before it raises, so by
the time a handler runs nothing is left "not done" -- an unfinished count is always
zero and reports nothing. The tasks the budget actually caught are the cancelled ones;
everything else finished in time. I had it the wrong way round first and the report
came out empty.

Every field comes from state the test already keeps: stats, calls, latencies,
started, and the embedder's own queue.

What did not change

The workload, the 45-second budget, the task ordering and every assertion are
untouched. A passing run never enters this path. Naming a task is metadata and does not
affect scheduling.

Verification

The happy path is unchanged: all 4 tests in the file pass in 5.1s, and flake8 and black
are clean.

For the failure path, I could not force a faithful timeout inside the test. Shrinking
the budget to 1ms or 600ms breaks the test's own preconditions -- tasks get cancelled
mid-setup and a CancelledError escapes from executors.py before the budget's
TimeoutError is the thing that propagates -- so a forced run exercises a different
failure than the real one. Rather than claim more than I checked, I verified the
report's logic verbatim against real asyncio Tasks in a standalone harness: with 12
tasks and a budget that catches 9, t.cancelled() separates the 9 from the 3 that
finished, and the JSON above is that harness's real output. The except TimeoutError
entry, the print and the re-raise were confirmed the same way.

So the mechanism is verified and the shape of the report is verified; what is not
verified is this exact handler firing inside a genuinely loaded shard, which is the
condition #12308 exists to make legible. The first real occurrence will confirm it, and
if the report comes back empty or malformed it costs a follow-up rather than a wrong
diagnosis, because the exception is re-raised either way.

test_private_store_mixed_load gives its whole task set 45 seconds and fails with a
bare TimeoutError. On a loaded shard that expiry carries no information: a store that
never finished, an embed queue that never drained and a starved worker all look
identical, so the failure is unactionable and the usual response is a re-run.

The tasks now carry store/kind/index names, and the timeout path reports what the
budget caught before re-raising unchanged. It reports the CANCELLED tasks rather than
the unfinished ones on purpose -- wait_for cancels the gather before it raises, so an
unfinished count is always zero and says nothing.

Diagnostics only. The workload, the 45-second budget, the task ordering and every
assertion are untouched, and a passing run never enters this path.
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 3ce5a698d7932105df0919b6cb60dd13ed3c421b and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- test/test_memory_multistore_stress.py:198 -- A recall deadline reaches except TimeoutError, falsely emitting MIXED_LOAD_TIMEOUT with zero budget cancellations -> Fix: re-raise without reporting when caught is empty.
[GPT-REVIEWED] 3ce5a69

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3ce5a698d7932105df0919b6cb60dd13ed3c421b: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 3ce5a698d7932105df0919b6cb60dd13ed3c421b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm the first real shard timeout relays the MIXED_LOAD_TIMEOUT line through xdist's captured-output on the failing worker, as #12308 expects.

[DESIGN-REVIEWED] 3ce5a69

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 3ce5a698d7932105df0919b6cb60dd13ed3c421b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3ce5a69

Verdict parsed from the review's SHA-scoped output markers for commit 3ce5a698d7932105df0919b6cb60dd13ed3c421b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3ce5a698d7932105df0919b6cb60dd13ed3c421b: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Green at 3ce5a698d7, and one thing the green does not mean

79 lanes success, 10 skipped, zero failures, zero in flight. All six review lanes are
green: Design, First Principles, GPT 5.6, Opus 4.8, Security Scope and UX. Readiness
published, no BLOCK-MERGE. Holding here for a human to merge.

Worth being precise about what Backend Tests (3.12, 7) proves on this PR, because the
answer is less than it looks. It reports 4 tests in 5.62s, not a full shard. That is
ci.yml's leaf-test scope doing exactly what it is designed to do: a diff touching only
a leaf test module cannot change a production code path, so the sharded suite is
replaced by the changed file run several times on independent runners.

So this PR's own CI exercised the file's four tests repeatedly and never put them under
the ~14,000-test load where the timeout actually expires. The diagnostics land
unexercised under load by construction -- no arrangement of this PR could test them,
because any diff small enough to be just this change takes the leaf path.

That is not a gap to fix here. It does mean the first real exercise comes from the next
change that triggers the full sharded suite, which is #12190's rebased head. If shard 7
expires there, the log will carry the MIXED_LOAD_TIMEOUT line with the store, the
operation kind, the embed queue depth and the completed count, and that payload goes on
#12308 as the evidence it was filed to obtain.

If instead the report comes back empty or malformed, it costs a follow-up rather than a
wrong diagnosis: the exception is re-raised unchanged either way, so the lane still
fails exactly as it does today and nothing is masked.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only diff in test/test_memory_multistore_stress.py -- names the asyncio tasks and prints a MIXED_LOAD_TIMEOUT diagnostic built from state the test already keeps, then re-raises the TimeoutError unchanged; workload, budget, task ordering and assertions are untouched, so a green run never takes the new path.

@iamwhatever
iamwhatever merged commit 10b0fb8 into main Sep 20, 2026
92 checks passed
@iamwhatever
iamwhatever deleted the fix/stress-timeout-diagnostics branch September 20, 2026 16:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 20, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…irodotdev#12309)

test_private_store_mixed_load gives its whole task set 45 seconds and fails with a
bare TimeoutError. On a loaded shard that expiry carries no information: a store that
never finished, an embed queue that never drained and a starved worker all look
identical, so the failure is unactionable and the usual response is a re-run.

The tasks now carry store/kind/index names, and the timeout path reports what the
budget caught before re-raising unchanged. It reports the CANCELLED tasks rather than
the unfinished ones on purpose -- wait_for cancels the gather before it raises, so an
unfinished count is always zero and says nothing.

Diagnostics only. The workload, the 45-second budget, the task ordering and every
assertion are untouched, and a passing run never enters this path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants