Skip to content

[BUG] Do not queue a curl session closed by the Retry-After cap - #4632

Open
ThomsonTan wants to merge 3 commits into
open-telemetry:mainfrom
ThomsonTan:fix/curl-retry-after-cap-requeue
Open

ThomsonTan wants to merge 3 commits into
open-telemetry:mainfrom
ThomsonTan:fix/curl-retry-after-cap-requeue

Conversation

@ThomsonTan

@ThomsonTan ThomsonTan commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Fixes #4631

Changes

When a retryable response carries a Retry-After beyond max_backoff,
PerformCurlMessage() closes the session with Cleanup(). The IO loop then
asked IsRetryable() again, still got true, and queued the closed session
at the server's retry time. doRetrySessions() stops at the first entry
that is not due, so that entry held back every later retry on the client
and kept the background thread alive. WaitBackgroundThreadExit() and
~HttpClient() could not join until the server's time, and the loop
busy-spun meanwhile.

For significant contributions please make sure you have completed the following items:

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@ThomsonTan
ThomsonTan requested a review from a team as a code owner September 23, 2026 15:50
@codecov

codecov Bot commented Sep 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.64%. Comparing base (2015ffb) to head (a3a08da).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4632      +/-   ##
==========================================
+ Coverage   86.52%   86.64%   +0.13%     
==========================================
  Files         525      525              
  Lines       20475    20475              
==========================================
+ Hits        17713    17738      +25     
+ Misses       2762     2737      -25     
Files with missing lines Coverage Δ
...lemetry/ext/http/client/curl/http_operation_curl.h 90.91% <ø> (ø)
ext/src/http/client/curl/http_client_curl.cc 93.12% <100.00%> (-0.01%) ⬇️
ext/src/http/client/curl/http_operation_curl.cc 63.15% <100.00%> (+1.99%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


started_at = std::chrono::steady_clock::now();
http_client.WaitBackgroundThreadExit();
const auto joined_in = std::chrono::steady_clock::now() - started_at;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: could we add a separate case that calls WaitBackgroundThreadExit() immediately after the capped request finishes, without sending the second request?

Here, the original bug spends the 30 seconds waiting for the second request to retry, so the later join can still return promptly. The retry assertion catches the bug, but a separate case would directly cover the shutdown scenario from #4631.

@lalitb lalitb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the quick fix!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Retry-After beyond max_backoff stalls the curl retry queue and HttpClient shutdown

2 participants