Fix stack overflow in auto_batcher::new_id() under contention - #1492
Merged
ihsandemir merged 1 commit intoSep 16, 2026
Merged
Conversation
A caller that lost the race for an ID from a freshly fetched batch was retried by returning new_id() from a launch::sync continuation and unwrapping it. Every lost round therefore wrapped the caller's future in one more future_unwrap_shared_state, and Boost completes such a chain recursively on the thread that finishes the last fetch. Under contention a single waiter can lose hundreds of consecutive rounds, and the completion cascade overflowed the executor thread's stack (512 KiB on macOS), aborting the macOS nightly in AutoBatcherTest.concurrencySmokeTest. The retry is now flat: new_id() hands out one promise per caller and try_get_id() either completes it or registers a single void continuation on the in-flight fetch. A continuation that loses the race posts the retry back to the executor instead of nesting another future, so every retry starts from an empty stack regardless of whether the next fetch is already complete. Single-flight fetching, coalescing and error propagation are unchanged. Adds AutoBatcherTest.starvedWaiterDoesNotOverflowTheStack, which reproduces the starvation deterministically (128 waiters, 1 ms fetch) and crashed on every run before the fix. Fixes hazelcast#1491
JackPGreen
approved these changes
Sep 16, 2026
| // More concurrent waiters than the batch could serve: retry | ||
| // If a caller that cannot get an ID from the fresh batch | ||
| // transparently retries (async analogue of Java's for(;;)). | ||
| return new_id(); |
Contributor
There was a problem hiding this comment.
took me a minute to get my head around it, but this is the problematic bit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1491
Problem
The macOS nightly aborts in
AutoBatcherTest.concurrencySmokeTestwith an AddressSanitizerstack-overflowon an executor thread (for example https://github.com/hazelcast/hazelcast-cpp-client/actions/runs/34794206547). The trace repeats the same four Boost frames up to the truncation limit:auto_batcher::new_id()retried a caller that lost the race for an ID from the fresh batch by returningnew_id()from alaunch::synccontinuation and calling.unwrap(). Each lost round wrapped the caller's future in one morefuture_unwrap_shared_state. Boost completes that chain recursively on the thread that finishes the last fetch.Waiters lose repeatedly because the fetch lambda publishes the new block before Boost marks the fetch future ready. A caller that gets an ID loops straight back into the lock-free fast path and drains the rest of the batch while the executor thread is still running the coalesced continuations one by one. One waiter can lose hundreds of rounds in a row. macOS gives secondary threads a 512 KiB stack, and ASan frames are large, so the cascade overflows.
Fix
Each caller's result now lives in one
boost::promise<int64_t>:new_id()keeps the allocation-free fast path. Otherwise it creates a promise, returns its future and calls the new privatetry_get_id(p).try_get_id(p)tries the fast path, then joins or elects the single in-flight fetch (election code unchanged) and registers avoidcontinuation on it.try_get_id(p)back to the executor when it loses. Posting instead of calling inline also handles the case where the next fetch has already completed and a sync continuation would run on the current thread.Nothing waits on the continuation's own future, so no chain forms and stack depth stays constant however many rounds a waiter loses. Single-flight fetching, coalescing and error propagation work as before, and callers see the same
new_id()contract.Test
AutoBatcherTest.starvedWaiterDoesNotOverflowTheStackreproduces the starvation on every run: 128 waiters, a supplier that takes 1 ms, batch size 3. All threads coalesce onto the in-flight fetch. When it completes, the first three continuations win and their threads re-register on the next fetch ahead of the continuations still queued, so the last waiter loses every round until the others use up their quota.Verified on macOS arm64, Debug + ASan, the configuration of the failing job:
EXC_BAD_ACCESSindo_continuation)concurrencySmokeTest+ new test, 12 concurrent processesFlakeIdGeneratorApiTest.*,AutoBatcherTest.*,FlakeIdBatchTest.*against a remote controller