[BUG] Retry-After beyond max_backoff stalls the curl retry queue and HttpClient shutdown

[ThomsonTan]

Describe your environment

• The affected code shipped in v1.29.0, the first release with [OTLP/HTTP] Honor Retry-After header when retrying exports  #4186.
• Retry support on, which is the CMake default: OTELCPP_WITH_OTLP_RETRY_PREVIEW=ON defines ENABLE_OTLP_RETRY_PREVIEW.
• Reproduced on Windows with MSVC 14.44 (Visual Studio 2022) and libcurl 8.21.0-DEV, by compiling the curl client sources with ENABLE_OTLP_RETRY_PREVIEW defined. The code involved is not platform specific.

Steps to reproduce

1. Run an HTTP server with two endpoints:
   • POST /slow answers 429 Too Many Requests with Retry-After: 8.
   • POST /busy answers 503 the first time and 200 after that.
2. Create one opentelemetry::ext::http::client::curl::HttpClient. Give every request RetryPolicy{5, SecondsDecimal{0.2f}, SecondsDecimal{1.0f}, 1.5f}, so max_backoff (1 s) is below the server's Retry-After (8 s). The OTLP/HTTP exporters default to a 5 s max_backoff.
3. Send a request to /slow and wait for its handler. The handler gets the 429 immediately and the server sees one request, so the session was closed rather than retried, as intended.
4. Then do one of the following:
   • call WaitBackgroundThreadExit() right away, or
   • send a request to /busy and time how long its retry takes.

What is the expected behavior?

The session closed because of its Retry-After should not go into the retry queue. WaitBackgroundThreadExit() and ~HttpClient() should return promptly. The /busy retry should follow its own backoff, which takes about 0.26 s here when nothing is ahead of it in the queue.

What is the actual behavior?

• WaitBackgroundThreadExit() returned after 8000 ms and used 7.8 s of CPU while waiting.
• The /busy retry happened after 8.2 s instead of about 0.26 s.

Additional context
Add any other context about the problem here.

_{Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[mateenali66]

The 7.8 s of CPU looks separate from the Retry-After cap, and #4632 does not change it.

Once is_shutdown_ is set, wait_for is zero (http_client_curl.cc:601), so the idle branch never sets need_wait_more. doRetrySessions(true) then reports work while a retry is queued but not yet due (:637), and the loop goes straight back to curl_multi_perform without reaching curl_multi_poll.

Reproduced at 4679325c on macOS arm64 with libcurl 8.7.1, using a plain 429 on /retry/ with no Retry-After and RetryPolicy{4, 0.5, 5.0, 2.0}, joining after the first attempt. The worker thread was on CPU in the loop in 820 of 820 samples, and the same with #4632's source applied. Setting need_wait_more when doRetrySessions(true) returns true parks it in Curl_poll for 848 of 850.

Want this folded into #4632, or as its own issue?

[ThomsonTan]

The 7.8 s of CPU looks separate from the Retry-After cap, and #4632 does not change it.

Once is_shutdown_ is set, wait_for is zero (http_client_curl.cc:601), so the idle branch never sets need_wait_more. doRetrySessions(true) then reports work while a retry is queued but not yet due (:637), and the loop goes straight back to curl_multi_perform without reaching curl_multi_poll.

Reproduced at 4679325c on macOS arm64 with libcurl 8.7.1, using a plain 429 on /retry/ with no Retry-After and RetryPolicy{4, 0.5, 5.0, 2.0}, joining after the first attempt. The worker thread was on CPU in the loop in 820 of 820 samples, and the same with #4632's source applied. Setting need_wait_more when doRetrySessions(true) returns true parks it in Curl_poll for 848 of 850.

Want this folded into #4632, or as its own issue?

Opened #4636 to track this.