Skip to content

test(session-runtime): cut refresh groups without a wall-clock deadline - #4325

Open
ScriptedAlchemy wants to merge 2 commits into
masterfrom
fix/3908-timed-refresh-groups
Open

ScriptedAlchemy wants to merge 2 commits into
masterfrom
fix/3908-timed-refresh-groups

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Fixes #3908

What was broken

timed_refresh_groups_* raced real SQLite work (begin, inserts, commit) against a real 2 s deadline. Under host load the work lost the race. The cut counter is bumped after the inserts, so a cut that landed during the inserts under-counted: timed_refresh_groups_wait_at_most_one_timeout_per_item_beyond_the_group got 4 where it wanted 5, ..._do_not_starve_items_after_a_slow_one got 1 where it wanted 2, and an item the test meant to commit came back DeadlineExceeded. Production is correct: a deadline that covers real storage work is the contract. The tests measured wall-clock time they cannot control.

Fix

Test only, session_temporal_refresh_scheduler/worker.rs:

  • The three timed_refresh_groups_* tests pass Duration::MAX, which starts no timer (native_runtime::timeout waits unbounded when the deadline is out of range). A cut commit now ends in the DeadlineExceeded that storage returns when its own deadline passes, so no timer races any write. The tests still check the retry policy (group, then each item once) and that cut groups roll back.
  • New a_refresh_group_that_outlives_its_deadline_is_retried_item_by_item keeps the timer path covered, with no I/O. A commit that is ready on its first poll wins even past the deadline, so only the pending group is ever cut, whatever the load.

Evidence

  • Reproduction: I could not make the old tests fail with real load. Passing runs: 20/20 filtered at taskset -c 0-7, 50/50 at -c 0-1 with 24 jobs, 40/40 at -c 0 with 24 jobs, 30 parallel on-disk runs with 4 fsync loops, 12 whole-suite runs at 6x slowdown. A write takes under 20 ms here, so the 2 s margin is wide. Shrinking the margin to 1 ms (COMMIT_DEADLINE edit, 10 loaded runs) failed 10/10 runs with the issue's symptoms, e.g. assertion left == right failed: a pass of slow items must wait for the group and then each item once left: 4 right: 5.
  • After: no deadline is left to shrink. The 4 tests pass 20/20 loaded, and the crate's full unit_test passes 20/20 at --runs_per_test=20 --local_test_jobs=20 under taskset -c 0-7.
  • Mutations of commit_refresh_groups / commit_refresh_group, run against the new tests:
    • Starve items after the first slow one: do_not_starve and wait_at_most_one_timeout fail (left: 2 right: 5).
    • TimeoutError::Elapsed no longer a cut: the new deadline test fails.
    • Storage DeadlineExceeded no longer a cut: all three timed_refresh_groups_* tests fail.
  • --config=clippy and rustfmt_check_2024 pass for the crate.

Devin Review

@changeset-bot

changeset-bot Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 6ee4cb0

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(session-temporal): timed_refresh_groups tests flake on CI (slow item commits)

1 participant