Repository navigation
test(session-runtime): cut refresh groups without a wall-clock deadline - #4325
Open
ScriptedAlchemy wants to merge 2 commits into
Open
ScriptedAlchemy wants to merge 2 commits into
ScriptedAlchemy wants to merge 2 commits into
Conversation
|
ScriptedAlchemy
enabled auto-merge
October 18, 2026 18:07
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3908
What was broken
timed_refresh_groups_*raced real SQLite work (begin, inserts, commit) against a real 2 s deadline. Under host load the work lost the race. The cut counter is bumped after the inserts, so a cut that landed during the inserts under-counted:timed_refresh_groups_wait_at_most_one_timeout_per_item_beyond_the_groupgot 4 where it wanted 5,..._do_not_starve_items_after_a_slow_onegot 1 where it wanted 2, and an item the test meant to commit came backDeadlineExceeded. Production is correct: a deadline that covers real storage work is the contract. The tests measured wall-clock time they cannot control.Fix
Test only,
session_temporal_refresh_scheduler/worker.rs:timed_refresh_groups_*tests passDuration::MAX, which starts no timer (native_runtime::timeoutwaits unbounded when the deadline is out of range). A cut commit now ends in theDeadlineExceededthat storage returns when its own deadline passes, so no timer races any write. The tests still check the retry policy (group, then each item once) and that cut groups roll back.a_refresh_group_that_outlives_its_deadline_is_retried_item_by_itemkeeps the timer path covered, with no I/O. A commit that is ready on its first poll wins even past the deadline, so only the pending group is ever cut, whatever the load.Evidence
taskset -c 0-7, 50/50 at-c 0-1with 24 jobs, 40/40 at-c 0with 24 jobs, 30 parallel on-disk runs with 4 fsync loops, 12 whole-suite runs at 6x slowdown. A write takes under 20 ms here, so the 2 s margin is wide. Shrinking the margin to 1 ms (COMMIT_DEADLINEedit, 10 loaded runs) failed 10/10 runs with the issue's symptoms, e.g.assertion left == right failed: a pass of slow items must wait for the group and then each item once left: 4 right: 5.unit_testpasses 20/20 at--runs_per_test=20 --local_test_jobs=20undertaskset -c 0-7.commit_refresh_groups/commit_refresh_group, run against the new tests:do_not_starveandwait_at_most_one_timeoutfail (left: 2 right: 5).TimeoutError::Elapsedno longer a cut: the new deadline test fails.DeadlineExceededno longer a cut: all threetimed_refresh_groups_*tests fail.--config=clippyandrustfmt_check_2024pass for the crate.