Skip to content

fix(tests): assert the queue-depth card converges, not an exact read count - #26592

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/23996-queue-depth-reconcile-flake
Oct 31, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/23996-queue-depth-reconcile-flake

Conversation

@JLargent42

@JLargent42 JLargent42 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_a_card_converges_once_the_store_answers_again gives the same verdict on every run, and still fails when the card never converges.

test/test_queue_depth_reconcile_reads.py::test_a_card_converges_once_the_store_answers_again[writer-thread] fails intermittently with assert 3 == 2 on len(calls) (CI Backend Tests (3.12, 6) on #23967; 4 of 25 local runs on the reporter's host). The test expects exactly two chip reads after a stop whose first read fails, and sometimes sees three.

Why it matters

A red that clears on re-run lands on unrelated PRs, costs a re-run and a triage each time, and teaches people to ignore reds on that shard.

Not a goal

  • No change to the queue-depth burst, heal or retry code. The three-read order is correct behavior: the card still gets exactly one right frame.
  • No change to the other exact read counts in the queue-depth files (see Sibling sweep).

What changed (motivation → approach → change)

  • Symptom: after cancel_for_parent stops three memory-deferred rows and the first chip read answers "unreadable", the test sometimes sees three reads instead of two.
  • Root cause: the test's docstring said "nothing else will ask" after the stop's failed read, and that is false. Read 1 fails and arms the delayed re-read on a real timer (_QUEUE_DEPTH_RETRY_SECS, which the helpers patch to 50 ms). Each stopped row's subagent_done sends a heal request (_fire_event_impl in run_lifecycle.py, because the published depth is still 3), and those start read 2. If read 2 is still on the store when the 50 ms timer fires, _fire_queue_depth_retry joins the live burst (emit.again = True). _queue_depth_burst then discards the overlapped read 2 and reads a third time. Whether read 2 finishes first depends on the host, so the count was 2 or 3, while the card got the same one frame either way.
  • Change: both orders are now driven by the test, each with an exact count. Neither depends on host timing.
    • test_a_card_converges_once_the_store_answers_again holds the re-read's timer off (_QUEUE_DEPTH_RETRY_SECS = 60.0, the way the sibling tests in test_queue_depth_reconcile_requests.py do) and freezes the burst's _queue_depth_clock. The heals' read answers the card. The test asserts exactly 2 reads, the one frame [{"queued": 0}], a store with no waiting rows, and that the heal read disarmed the re-read.
    • New test_a_reread_that_fires_during_a_read_still_publishes_once pins the three-read order. A failed read arms the re-read, and a second request's read is held on an event. The test runs the re-read's callback (_fire_queue_depth_retry) by hand during that held read, then releases it. With the timer held off and the clock frozen, it asserts exactly 3 reads, one frame and no armed re-read.

Backwards compatibility

Compatible: test-only change.

Removes nothing: no test, fixture or production symbol is removed. The same test ids are collected, plus the new one.

Tests

Proof: not provable -- test-only change

Reproducing it: I could not reproduce the flake on unmodified origin/main @ 43a680688 on this host. It passed 60/60 plain runs (-n 0, only this test's two ids), and 40/40 more pinned to a core held busy by a spin loop. To force the CI order, I used a scratch trace test (not committed). It ran the same stop with request and read logging, and slowed the second chip read by 200 ms so it outlasts the 50 ms re-read delay:

  • The trace shows READ#1 (fails), three heal=True requests, READ#2, then heal=False attempt=1 join=True (the re-read joining the burst), READ#3, and one {'queued': 0} frame.
  • Old assertion under that delay: 40/80 failed with assert 3 == 2, which is every slowed rep in both modes and the CI error.

After the change (head da5f92dc37e9):

  • The two tests, both modes, 60/60 runs passed (-n 0). Another 30/30 passed pinned to a busy core.
  • The whole file: 45 passed.
  • Mutation: making _publish_queue_depth send no frame fails both tests.
  • test/test_flake_pattern_ratchet.py and test/test_wait_discipline_ratchet.py pass against origin/main (compared with 43a680688268).
  • scripts/check_comment_history.py passes, and black 26.10.1 is clean on the file.

Manual verification

N/A — test-only change; the runs above cover it.

Related Issues

Closes #23996

Sibling sweep

Swept: git grep -n "_fail_chip_reads(" -- test/ and git grep -nE "assert len(calls) ==" -- 'test/test_queue_depth_reconcile*.py'
Sibling: test/test_queue_depth_reconcile_reads.py:426 -- not affected -- every read fails, so no heal publishes and only the bounded re-reads run; the count is exact by design
Sibling: test/test_queue_depth_reconcile_requests.py:198 -- not affected -- every read fails, so the count is the requests plus the bounded re-reads with no heal joining
Sibling: test/test_queue_depth_reconcile_accepts.py:348 -- not affected -- sets _QUEUE_DEPTH_RETRY_SECS to 60.0, so the re-read cannot fire mid-test
Sibling: test/test_queue_depth_reconcile.py:306 -- not affected -- sets _QUEUE_DEPTH_RETRY_SECS to 60.0 and asserts frames, not a read count
Sibling: test/test_queue_depth_reconcile_reads.py:468 -- not affected -- already asserts a floor, len(calls) >= 2, and waits on the frame, not the count

Pattern harvest

Not generalizable: the exact count was wrong only because a sub-second re-read timer and heal requests could both answer the same failed read; the other exact counts either run with the timer at 60 s or have every read fail.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@JLargent42 JLargent42 self-assigned this Oct 31, 2026
@JLargent42
JLargent42 requested review from a team and KiroCrewCaptain as code owners October 31, 2026 05:43
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This only changes tests: both read orders now run under the test's control, each asserting an exact count, and no assertion was weakened.

[DESIGN-REVIEWED] da5f92d

Description read

[DESCRIPTION-READ] 023512ff9aca1b4fca6b11a5da75d5a58784fe6a5ad86b51b0c048e501786e11

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] da5f92d

Verdict parsed from the review's SHA-scoped output markers for commit da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0.

False positive or not applicable? A repository writer can comment:
/ai-review override fable da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] da5f92d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt da5f92dc37e9f259abfd1b6eecd0f9e1fbe7e5a0 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 31, 2026
test_a_card_converges_once_the_store_answers_again asserted exactly two
chip reads after a stop whose first read fails, while the delayed re-read
ran on a real 50 ms timer. The stopped rows' heal requests start a second
read; when that read is still on the store as the timer fires, the re-read
joins the burst, the overlapped read is discarded and a third read
publishes. The count depended on the host.

The test now holds the re-read's timer off and freezes the burst's clock,
so the heals answer the card in exactly two reads. A new test runs the
re-read's callback by hand during a held read and pins the three-read
order with an exact count.

Closes #23996
@JLargent42
JLargent42 force-pushed the fix/23996-queue-depth-reconcile-flake branch from d9f9324 to da5f92d Compare October 31, 2026 06:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 31, 2026
@JLargent42

Copy link
Copy Markdown
Contributor Author
  • New three-read test depends on the real 1.0 s withhold clock (span=1502d9fdb806)

fixed in da5f92d: test_a_reread_that_fires_during_a_read_still_publishes_once now freezes run_mod._queue_depth_clock to 0.0 before the first emit, as the sibling in test_queue_depth_reconcile_requests.py:141 does, so a stalled runner can no longer push the overlapped read past _QUEUE_DEPTH_MAX_WITHHOLD_SECS and publish [3, 3].
The test also no longer waits on the real 50 ms timer. _QUEUE_DEPTH_RETRY_SECS is 60 s and the retry callback is run by hand inside the held read, so the frame list [3] and len(calls) == 3 follow from the test's own order. The converge test freezes the same clock. 60/60 local runs passed, plus 30/30 pinned to a busy core.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • Preserve the exact read-count constraint (span=8a904d9fc89c)

fixed in da5f92d: test_a_card_converges_once_the_store_answers_again asserts len(calls) == 2 again. The interleaving is now controlled: _QUEUE_DEPTH_RETRY_SECS is 60 s, so the re-read's timer cannot fire mid-test, and _queue_depth_clock is frozen. The stopped rows' heal read is then the one read after the failed one, so the count is exact. The convergence asserts stay alongside it: one {"queued": 0} frame, no waiting store rows, and no armed re-read.
The three-read order is its own test with its own exact count, len(calls) == 3, driven by hand. No equality is relaxed to >= on this head.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • Polling can miss the armed retry entirely (span=8a904d9fc89c)

fixed in da5f92d: the test no longer polls for the short-lived _queue_depth_retries entry. _QUEUE_DEPTH_RETRY_SECS is 60 s, so the entry stays armed. The test reads it after _settle, waits on an event set when the second read starts, then cancels the timer and calls mgr._run_events._fire_queue_depth_retry(_PARENT) itself before releasing that read. This is the fix the finding proposed: suppress the automatic firing, signal entry into the second read, and invoke the callback explicitly.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 31, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 31, 2026 08:04

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (1 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only fix in test_queue_depth_reconcile_reads.py; the queue-depth test holds the re-read timer and the clock fixed, asserts the card converges, and adds one test for the overlapped-read order. No product code changes.

@KiroCrewCaptain
KiroCrewCaptain merged commit 902bd35 into main Oct 31, 2026
98 of 105 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/23996-queue-depth-reconcile-flake branch October 31, 2026 08:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_card_converges_once_the_store_answers_again[writer-thread] sees 3 chip reads instead of 2

2 participants