Skip to content

fix(test): give the hard-stop settle double the slot-push reads - #18952

Closed
cruisercohen wants to merge 1 commit into
mainfrom
fix/hard-stop-settle-double
Closed

cruisercohen wants to merge 1 commit into
mainfrom
fix/hard-stop-settle-double

Conversation

@cruisercohen

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop test/test_session_control.py failing at random on the backend shards of every PR.

test_a_hard_stop_still_discards_and_settles_what_was_queued_before_it (and its sibling test_a_send_that_lands_while_a_hard_stop_settles_stays_queued) fail intermittently in "Backend Tests (3.12, 4)" with:

AttributeError: '_SettlingManager' object has no attribute 'published_queued_depths'
  dashboard/chat_handlers.py: stop_slot_turn -> state.push_slots_update()
  dashboard/state.py: _do_slots_broadcast -> serialize_slots -> _published_queued_by_slot

Both pass when run alone on a fast host, which is why they landed green.

Why it matters

The shard is required, so a random red here blocks merging an unrelated PR until someone notices and re-runs it. It hits whichever PR's merge ref happens to land the test on a slow worker.

Not a goal

  • No change to stop_slot_turn, the slots broadcast, or its coalescing window. Product code is untouched.
  • Not a sweep of every other hand-rolled state.subagents double in the suite. The ones found use MagicMock, which answers any read.

What changed (motivation → approach → change)

Symptom: the two hard-stop settle tests die with AttributeError on a loaded runner and pass locally.

Root cause: the tests set state.subagents to a hand-rolled _SettlingManager that implements only settle_queued_delivery. Every Stop press calls push_slots_update(). That push either serializes the slot list at once, or folds into a trailing timer when the last push was less than 0.2s ago. Creating the slot makes the first push, so:

  • on a fast host the press lands inside the window, is folded into a timer that never fires (the asyncio.run loop closes first), and nothing reads the manager;
  • on a loaded runner more than 0.2s passes between slot creation and the press, so the press serializes, serialize_slots reads published_queued_depths() and running_agents_for() off the manager, and the double has neither.

Reproduced locally by pinning the coalescing clock past the window: both tests fail 3/3 with the CI error, on unmodified main.

Change (test-only, one file):

  1. _SettlingManager gains published_queued_depths() and running_agents_for(), both empty, since this manager has no children. Its docstring now names every manager read a hard stop reaches.
  2. A _every_slots_push_broadcasts fixture puts each read of _slots_broadcast_clock (the seam state.py provides for exactly this) a second after the last, and both tests use it. The serialization then runs on every run, so the outcome no longer depends on host speed, and a future read the double lacks fails on every run instead of on a slow one.

Rejected alternatives:

  • suspend_slots_push() or a clock frozen inside the window: makes the tests pass by never serializing, which hides the incomplete double rather than fixing it.
  • MagicMock(spec=SubagentManager) for the manager: answers every read, but running_agents_for() would return a truthy mock and mark the slot as having running children, a fact the tests do not mean to assert.
  • Retrying or sleeping: does not remove the race and the sleep is flagged by Semgrep.

Backwards compatibility

Compatible: test-only change, no runtime code touched.

Removes nothing: the diff adds two methods to a test double and one fixture; no test, assertion or product path is removed.

Tests

  • Without the double's two new methods, with the fixture: both tests fail 3/3 with the CI's AttributeError.
  • With the change: test/test_session_control.py 229 passed, 8/8 runs serial, 3/3 runs under -n 4.
  • Mutation guard kept: moving slot._queue.clear() after the settle's await in stop_slot_turn still fails test_a_send_that_lands_while_a_hard_stop_settles_stays_queued.
  • black, isort, flake8 clean on the changed file; the comment-history gate passes.

Manual verification

N/A — test-only change; the unit runs above are the verification.

Related Issues

N/A — found while reading the failing backend shard on an unrelated PR; no issue filed.

Pattern harvest

Rule candidate: review-prompt
Pattern: a hand-rolled stand-in for state.subagents (or any manager the slots broadcast reads) that implements only the method under test, in a test that triggers push_slots_update(); whether the broadcast reads the stand-in depends on the 0.2s coalescing window, so it passes on a fast host and fails on a slow one.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 5f0e3ff

Description read

[DESCRIPTION-READ] 0b2dccd9c39789eb09244092c9675531d7fc10a686689ba90493b2f7d126af20

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5f0e3ff

Verdict parsed from the review's SHA-scoped output markers for commit 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5f0e3ff

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5f0e3ff9cab45508af3eeae13fb3e04e94fa3a0f: <one-sentence reason>

@cruisercohen
cruisercohen marked this pull request as ready for review October 10, 2026 21:56
@cruisercohen
cruisercohen requested review from a team and KiroCrewCaptain as code owners October 10, 2026 21:56
@cruisercohen
cruisercohen enabled auto-merge (squash) October 10, 2026 21:56
The two hard-stop settle tests in test_session_control.py stand in a
hand-rolled manager for state.subagents that implements only
settle_queued_delivery. Each Stop press calls push_slots_update, and the
slot-list serialization reads published_queued_depths and
running_agents_for off that manager. Whether the press serializes or is
folded into a trailing timer depends on the 0.2s coalescing window since
the push the slot's creation made, so on a loaded runner the press
serializes and the test dies with AttributeError.

Add the two reads to the double (both empty: it has no children) and pin
the coalescing clock past the window in both tests, so the serialization
runs on every run instead of only on a slow host.
@cruisercohen
cruisercohen force-pushed the fix/hard-stop-settle-double branch from f675fbc to 5f0e3ff Compare October 10, 2026 22:07
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@cruisercohen
cruisercohen disabled auto-merge October 10, 2026 22:48
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Closing as a duplicate: #18950 makes the same fix (the hard-stop test double answers the slot-list reads, and the slots clock always clears the coalescing window), and its checks are green.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 10, 2026
@bolichen97
bolichen97 deleted the fix/hard-stop-settle-double branch November 6, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant