test(qwp): deflake close-lifecycle interrupt tests on saturated CI agents - #72
Merged
Merged
Conversation
…ents
facadeCloseIsBoundedUnderRepeatedInterruptsDuring{Query,Sender}Creation
raced the 100ms close budget against thread scheduling: the post-join
assert needs the interrupter to land at least two interrupts inside the
budget window, but the budget clock starts before the interrupter thread
even exists, and a freshly started, yielding thread is not guaranteed
two scheduler quanta within 100ms on a saturated agent. On a hosted
3-core mac agent (questdb macwin build 254645) the count read <= 1 and
the positive control failed while the product honored its bounded
deadline exactly.
- widen the creation-wait budget to 1s in the two interrupt tests
- hold the test until two interrupts have demonstrably landed while the
closer is still inside its bounded wait (awaitRepeatedInterrupts), so
the deadline-restart property is exercised on every run and a starved
interrupter is reported as such instead of as a count-assert failure
Verified: whole class green 3/3 on an idle machine; both interrupt
tests green 3/3 with 3 visible cores under full external CPU load.
Test-only change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test-only change. Make
facadeCloseIsBoundedUnderRepeatedInterruptsDuringQueryCreationandfacadeCloseIsBoundedUnderRepeatedInterruptsDuringSenderCreationrobust on saturated CI agents: widen the close budget the tests configure from 100ms to 1s, and turn the "storm landed repeatedly" post-join count assert into a synchronized precondition proven whileclose()is still inside its bounded creation wait.This addresses the intermittent failure in QuestDB build 254645 (
on mac-other):test did not repeatedly interrupt query close, where the interrupt counter read 0/1 while every product-side assertion held.Root cause
The tests assert, after
closer.join(), that a free-running interrupter thread incremented its counter more than once. That is a liveness bet against the scheduler:db.close()is called — before the interrupter thread even exists;awaitCreationWaiter's poll latency and the interrupter'sThread.start()-to-first-quantum latency are paid out of that same window;On a hosted 3-core mac agent running sibling surefire forks, the interrupter got at most one quantum inside the window. The failing run's log shows close honoring its deadline exactly (
QueryClientPool.close(): 1 query client creation(s) still in flight after 100ms) and returning promptly — the product invariant the test protects (repeated interrupts must not restart the bounded deadline) was intact; only the test's own precondition failed.Changes
awaitRepeatedInterrupts(count, closerStillWaiting, message)helper, called right afterinterrupter.start(): holds the test until at least two interrupts have landed while the closer is verifiably still inside its bounded wait. The deadline-restart property is now exercised on every run, and a pathologically starved interrupter fails with an explicitclose left its creation wait before the interrupt storm landed twice; interrupts landed: Ndiagnosis instead of a misleading positive-control assert.Validation
QuestDBImplCloseLifecycleTestclass green 3/3 on an idle machine (2.1s — the two tests now deliberately burn their full 1s budgets).-XX:ActiveProcessorCount=3) while all 12 physical cores were saturated by external spin load — the same starvation conditions as the failing agent.