[michaellchung]

Problem

test_real_syncer_fast_island_stays_within_the_cap_and_both_islands_join_every_merge fails intermittently on CI (e.g. assert 40 <= 1 for _max_lead): with max_local_rounds_ahead=1 the fast island ran tens of rounds ahead of the slow island.

Root cause is in RolePairing: the lead cap only blocked on pairs in flight (lead_blockers). After a merge and before the syncer's next PULL, the island has nothing in flight, so it keeps training until the PULL arrives. Under load the syncer's PULL latency covers many fast rounds; those rounds all land in the next push (c_steps 9-13 observed) and widen the lead over the island the next merge waits for. Reproduced locally in 5 of 16 parallel runs under CPU load.

Fix

• RolePairing records the latest local round it retained a pair from (retained_round) and when.
• New lead_unpushed(): true once the island trained K rounds since then. wait_for_lead waits while either condition holds, with the same escape at half the wait timeout (measured from that retain for the no-pair case).

Tests

• Two new fake-pairing tests: the wait covers rounds trained before the next PULL arrives, and that wait gives up at half the wait timeout. Both fail without the fix.
• The real-syncer test now checks the cap's rule directly: the fast island starts round r + 1 only once every push it made from a cut <= r has the slow island's push for the same step, and only after it pushed from its round-r cut. The completed-round lead bound is 2, not 1: when the slow island itself waits under the cap, its push from an older cut can complete the fast island's merge while it has completed one round less. The absolute c_steps <= 2 bound is dropped (it depends on PULL timing).
• The test waits for the syncer to listen before the clients dial (their connect timeout is 10 s).

Verification: tests/test_sync_lead_cap.py 60/60 passing in 20 parallel runs per batch alongside 16 CPU burners; all 42 test files touching full_parameter_sync pass (810 passed), including test_verl_miles_parity.py.

🤖 Generated with Claude Code