Repository navigation
fix(tests): assert the queue-depth card converges, not an exact read count - #26592
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This only changes tests: both read orders now run under the test's control, each asserting an exact count, and no assertion was weakened. [DESIGN-REVIEWED] da5f92d Description read[DESCRIPTION-READ] 023512ff9aca1b4fca6b11a5da75d5a58784fe6a5ad86b51b0c048e501786e11 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
test_a_card_converges_once_the_store_answers_again asserted exactly two chip reads after a stop whose first read fails, while the delayed re-read ran on a real 50 ms timer. The stopped rows' heal requests start a second read; when that read is still on the store as the timer fires, the re-read joins the burst, the overlapped read is discarded and a third read publishes. The count depended on the host. The test now holds the re-read's timer off and freezes the burst's clock, so the heals answer the card in exactly two reads. A new test runs the re-read's callback by hand during a held read and pins the three-read order with an exact count. Closes #23996
d9f9324 to
da5f92d
Compare
|
|
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (1 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only fix in test_queue_depth_reconcile_reads.py; the queue-depth test holds the re-read timer and the clock fixed, asserts the card converges, and adds one test for the overlapped-read order. No product code changes.
Problem / Motivation
Goal:
test_a_card_converges_once_the_store_answers_againgives the same verdict on every run, and still fails when the card never converges.test/test_queue_depth_reconcile_reads.py::test_a_card_converges_once_the_store_answers_again[writer-thread]fails intermittently withassert 3 == 2onlen(calls)(CIBackend Tests (3.12, 6)on #23967; 4 of 25 local runs on the reporter's host). The test expects exactly two chip reads after a stop whose first read fails, and sometimes sees three.Why it matters
A red that clears on re-run lands on unrelated PRs, costs a re-run and a triage each time, and teaches people to ignore reds on that shard.
Not a goal
What changed (motivation → approach → change)
cancel_for_parentstops three memory-deferred rows and the first chip read answers "unreadable", the test sometimes sees three reads instead of two._QUEUE_DEPTH_RETRY_SECS, which the helpers patch to 50 ms). Each stopped row'ssubagent_donesends a heal request (_fire_event_implinrun_lifecycle.py, because the published depth is still 3), and those start read 2. If read 2 is still on the store when the 50 ms timer fires,_fire_queue_depth_retryjoins the live burst (emit.again = True)._queue_depth_burstthen discards the overlapped read 2 and reads a third time. Whether read 2 finishes first depends on the host, so the count was 2 or 3, while the card got the same one frame either way.test_a_card_converges_once_the_store_answers_againholds the re-read's timer off (_QUEUE_DEPTH_RETRY_SECS = 60.0, the way the sibling tests intest_queue_depth_reconcile_requests.pydo) and freezes the burst's_queue_depth_clock. The heals' read answers the card. The test asserts exactly 2 reads, the one frame[{"queued": 0}], a store with no waiting rows, and that the heal read disarmed the re-read.test_a_reread_that_fires_during_a_read_still_publishes_oncepins the three-read order. A failed read arms the re-read, and a second request's read is held on an event. The test runs the re-read's callback (_fire_queue_depth_retry) by hand during that held read, then releases it. With the timer held off and the clock frozen, it asserts exactly 3 reads, one frame and no armed re-read.Backwards compatibility
Compatible: test-only change.
Removes nothing: no test, fixture or production symbol is removed. The same test ids are collected, plus the new one.
Tests
Proof: not provable -- test-only change
Reproducing it: I could not reproduce the flake on unmodified
origin/main@43a680688on this host. It passed 60/60 plain runs (-n 0, only this test's two ids), and 40/40 more pinned to a core held busy by a spin loop. To force the CI order, I used a scratch trace test (not committed). It ran the same stop with request and read logging, and slowed the second chip read by 200 ms so it outlasts the 50 ms re-read delay:READ#1(fails), threeheal=Truerequests,READ#2, thenheal=False attempt=1 join=True(the re-read joining the burst),READ#3, and one{'queued': 0}frame.assert 3 == 2, which is every slowed rep in both modes and the CI error.After the change (head
da5f92dc37e9):-n 0). Another 30/30 passed pinned to a busy core._publish_queue_depthsend no frame fails both tests.test/test_flake_pattern_ratchet.pyandtest/test_wait_discipline_ratchet.pypass againstorigin/main(compared with 43a680688268).scripts/check_comment_history.pypasses, and black 26.10.1 is clean on the file.Manual verification
N/A — test-only change; the runs above cover it.
Related Issues
Closes #23996
Sibling sweep
Swept: git grep -n "_fail_chip_reads(" -- test/ and git grep -nE "assert len(calls) ==" -- 'test/test_queue_depth_reconcile*.py'
Sibling: test/test_queue_depth_reconcile_reads.py:426 -- not affected -- every read fails, so no heal publishes and only the bounded re-reads run; the count is exact by design
Sibling: test/test_queue_depth_reconcile_requests.py:198 -- not affected -- every read fails, so the count is the requests plus the bounded re-reads with no heal joining
Sibling: test/test_queue_depth_reconcile_accepts.py:348 -- not affected -- sets _QUEUE_DEPTH_RETRY_SECS to 60.0, so the re-read cannot fire mid-test
Sibling: test/test_queue_depth_reconcile.py:306 -- not affected -- sets _QUEUE_DEPTH_RETRY_SECS to 60.0 and asserts frames, not a read count
Sibling: test/test_queue_depth_reconcile_reads.py:468 -- not affected -- already asserts a floor, len(calls) >= 2, and waits on the frame, not the count
Pattern harvest
Not generalizable: the exact count was wrong only because a sub-second re-read timer and heal requests could both answer the same failed read; the other exact counts either run with the timer at 60 s or have every read fail.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.