Repository navigation
Restore test fixes dropped from #324; pump the agentic round in streaming tests - #332
Merged
Merged
Conversation
michaellchung
changed the title
An island with asynchronous full-parameter rounds trains until the syncer's final cut; it is not blocked between pushes, and the lead cap limits rounds on top of an unmerged push, not the absolute lead. The test harness stopped it after 20 x rounds local rounds. Tiny CPU rounds take milliseconds, while a loaded host slows the syncer's merges far more, so both islands could exhaust that budget before the final cut, raise "full-parameter sync stopped before every role acknowledged its final cut" and leave the syncer waiting until the test timed out. Async test islands now train until the final cut, bounded by a deadline inside the test's join timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PoolIslandClient.leave skipped the LEAVE frame when the client was
already shut down (the run had ended, e.g. the final cut arrived first)
and ignored send_leave's result, yet PoolAvgSync.leave always taped
elastic_leave as if the syncer had been told. The syncer tape then had
no pool_leave for an island whose own tape said it left.
PoolIslandClient.leave now returns None when the frame was written, or
why it was not ("syncer_shut_down", "not_written"). PoolAvgSync tapes
elastic_leave with sent=true, or sent=false and not_sent=<cause>.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both helpers wait only for events that must happen (a group settling, a record being logged), so the timeout bounds a failure, never a pass. A 5 s default was too tight on a 2-core CI host running the suite: the dispenser-row test timed out waiting for the fast and bad groups to settle, although neither depends on the slow gate. Raise the default to 30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AgenticRound settles finished groups only inside a round call on the caller's thread. Three streaming tests waited, or asserted right away, for a group that might end after the round call had returned: a slot frees as soon as the first trial of a group ends, so the call can return before its sibling trial ends and the group settles. On a loaded host nothing would ever settle it, and the wait timed out whatever its limit. The tests now pump empty rounds while they wait, so the condition they wait for is reached on every host; the timeout only bounds a real hang. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michaellchung
force-pushed
the
restore-lost-test-fixes
branch
from
November 3, 2026 07:09
a09ef6f to
a39a8fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restores three commits of #324 that were dropped when its branch was rebased before merging, and fixes the root cause of the flaky streaming-agentic test.
Restored from the original #324 branch
Bound async verl test islands by time, not by a round count: async verl test islands train until the final cut, bounded by a deadline, instead of a round budget a loaded host can exhaust.Tape whether an island's LEAVE reached the syncer:PoolIslandClient.leavereports whether the LEAVE frame was written;PoolAvgSynctapeselastic_leavewithsent/not_sent. Includestests/test_rl_island_leave_reason.py. This one also touchesyeto/rl/pool_client.pyandyeto/rl/engine/bridges.py.Give the streaming-agentic and wandb-tape wait helpers CI headroom: wait helper timeouts go from 5 s to 30 s.Root cause of the streaming-agentic timeout
AgenticRoundsettles finished groups only inside a round call on the caller's thread. Three tests intests/test_agentic_round_streaming.pywaited for, or asserted immediately on, a group that could end after the round call had returned. A slot frees as soon as the first trial of a group ends, so the call can return before the sibling trial ends. After that, nothing settles the group, so a longer timeout cannot help. With 0.3 s trials the dispenser-row test hung for the whole timeout every time. The tests now pump empty rounds while they wait, so the timeout only bounds a real hang.Verification
tests/test_agentic_round_streaming.py+tests/test_wandb_tape.py: 128 runs, 16 in parallel at a time with every core busy: 0 failures. Before the fix, 7 of 48 runs failed under the same kind of load.🤖 Generated with Claude Code