Repository navigation
prometheus: add test reproducing waitForCooldown deadlock (#2147) + fix (resistance to panics) - #2161
Merged
Merged
Conversation
bwplotka
added this pull request to stack #2163
October 7, 2026 13:45
bwplotka
force-pushed
the
bwplotka/repro-2147-wait-for-cooldown
branch
3 times, most recently
from
October 7, 2026 14:07
a74eb8c to
ecc1fdc
Compare
Demonstrate how a recovered panic inside histogram.Write (after flipping countAndHotIdx and before addAndResetCounts) leaves coldCounts unmerged, causing the next Write() to spin forever in waitForCooldown. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka
force-pushed
the
bwplotka/repro-2147-wait-for-cooldown
branch
from
October 7, 2026 14:35
ecc1fdc to
3a29f73
Compare
Defer addAndResetCounts immediately after waitForCooldown in histogram.Write (and merge before populating out in noObjectivesSummary.Write) so a panic inside Write cannot unlock the mutex with coldCounts unmerged and wedge subsequent Write calls in waitForCooldown. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
kakkoyun
approved these changes
Oct 8, 2026
bwplotka
removed this pull request from stack #2163
October 8, 2026 12:51
…#2162) Defer addAndResetCounts immediately after waitForCooldown in histogram.Write (and merge before populating out in noObjectivesSummary.Write) so a panic inside Write cannot unlock the mutex with coldCounts unmerged and wedge subsequent Write calls in waitForCooldown. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka
enabled auto-merge (squash)
October 8, 2026 12:51
bwplotka
changed the title
bwplotka
disabled auto-merge
October 8, 2026 12:52
bwplotka
enabled auto-merge (squash)
October 8, 2026 12:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repro of #2147
Demonstrate how a recovered panic inside histogram.Write (after flipping countAndHotIdx and before addAndResetCounts) leaves coldCounts unmerged, causing the next Write() to spin forever in waitForCooldown.
Note
This is expected to fail to reproduce the issue. #2162 can be then merged and it's clear tests were not changed, yet now passing.
Ok, but can Write(nil) or panic even happen?
In both
histogram.WriteandnoObjectivesSummary.Write, passingout == nil(Write(nil), which panics atout.Histogram = hisandout.Summary = sum) is the only thing that can trigger a recoverable panic between flippingcountAndHotIdxand mergingcoldCountsintohotCounts(all slice accesses in between are bounded bylen(h.upperBounds), and OOM on allocation is a fatalruntime.throwrather than a recoverable panic).In standard
Registry.Gather(),Write(nil)cannot happen becauseprocessMetricalways allocatesdtoMetric := &dto.Metric{}before callingmetric.Write(dtoMetric). It can only happen if a customCollectoror caller invokesWrite(nil)directly (for instance insideCollect, wheresafeCollectrecovers the panic whiledefer h.mtx.Unlock()unlocks the mutex withcoldCountsunmerged).