Skip to content

[BUG] End the Elasticsearch exporter's wait on a read or write error - #4331

Open
thc1006 wants to merge 2 commits into
open-telemetry:mainfrom
thc1006:bugfix/es-readerror-writeerror-completion-4330
Open

thc1006 wants to merge 2 commits into
open-telemetry:mainfrom
thc1006:bugfix/es-readerror-writeerror-completion-4330

Conversation

@thc1006

@thc1006 thc1006 commented Aug 2, 2026 •

Copy link
Copy Markdown
Member

Fixes #4330.

ResponseHandler::OnEvent records a completion for each session state that ends the session. ReadError and WriteError only log, so a client that reports either of them as its last event leaves the synchronous Export() waiting on its condition variable with nothing left to wake it. Two lines make those two behave like the other terminal states.

What this does not change, and one question for you

The bundled curl client dispatches nine session states and neither of these is among them, so this changes nothing for callers who use it. It matters for the HttpClient that the exporter's constructor accepts.

Worth putting in front of you rather than burying: otlp_http_client.cc handles ReadError and WriteError exactly the way the Elasticsearch exporter does today, logging at debug and carrying on. After this change the two exporters treat them differently. The SessionState comments say only "error reading response" and "error writing request", so there is no stated contract either way. If you would rather the two exporters stayed consistent, the alternative is to leave this alone and bound the wait instead, and I am happy to close this and open that one.

Evidence

Removing just the two recordCompletion() calls and rebuilding:

case without the fix
ReadErrorEndsTheWait never returns, killed at 15s
WriteErrorEndsTheWait never returns, killed at 15s
AReadErrorAfterTheHandoffEndsTheExport fails
IoErrorBeforeAResponseKeepsTheFailure fails, kSuccess where kFailure is expected
ProgressStatesDoNotDecideTheResult, both SessionDestroyed cases still pass

With the fix, the synchronous build runs 14 tests green and the async build skips the 11 that need the synchronous wait. The CTest timeout is there because a regression in this area stalls the job instead of failing an assertion.

What moved out

This pull request previously also made every terminal log line conditional on winning the completion race, and added an error line when a session is destroyed before a response arrives. That half needs a log-capturing fixture, it is a separate behaviour, and it is the reason the diff was four times this size. It is out, and I will open it separately once this one is settled. What is left is the two lines the title describes.

Verified locally: CMake in both the synchronous and async configurations with maintainer mode, Bazel, clang-format, cmake-format against the repository config, and markdownlint 0.46.0.

@thc1006
thc1006 requested a review from a team as a code owner August 2, 2026 17:50
@codecov

codecov Bot commented Aug 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.53%. Comparing base (4679325) to head (157a8dc).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #4331   +/-   ##
=======================================
  Coverage   86.53%   86.53%           
=======================================
  Files         525      525           
  Lines       20482    20482           
=======================================
  Hits        17722    17722           
  Misses       2760     2760           
Files with missing lines Coverage Δ
...orters/elasticsearch/src/es_log_record_exporter.cc 47.73% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch from 69473bc to 26ea36f Compare August 2, 2026 18:51
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
The helper tests call IsBulkResponseSuccessful() directly, so they cannot
show that the status and the body reach it. A handler that stored a fixed
status, or an Export() that never asked for one, passes all of them.

Three cases through the exporter with a fake HTTP client: an accepted bulk
response, the rejected item from open-telemetry#4295 whose shard counter still reads
"failed" : 0, and a 500 carrying a body the parser would otherwise accept.

The fake client is the same one open-telemetry#4331 adds to this file. Whichever lands
first, the other drops the duplicate when it rebases.
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
The helper tests call IsBulkResponseSuccessful() directly, so they cannot
show that the status and the body reach it. A handler that stored a fixed
status, or an Export() that never asked for one, passes all of them.

Three cases through the exporter with a fake HTTP client: an accepted bulk
response, the rejected item from open-telemetry#4295 whose shard counter still reads
"failed" : 0, and a 500 carrying a body the parser would otherwise accept.

The fake client is the same one open-telemetry#4331 adds to this file. Whichever lands
first, the other drops the duplicate when it rebases.
@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch from 26ea36f to 4dcfc60 Compare August 2, 2026 19:13
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
The helper tests call IsBulkResponseSuccessful() directly, so they cannot
show that the status and the body reach it. A handler that stored a fixed
status, or an Export() that never asked for one, passes all of them.

Three cases through the exporter with a fake HTTP client: an accepted bulk
response, the rejected item from open-telemetry#4295 whose shard counter still reads
"failed" : 0, and a 500 carrying a body the parser would otherwise accept.

The fake client is the same one open-telemetry#4331 adds to this file. Whichever lands
first, the other drops the duplicate when it rebases.
@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch from c5e8c69 to a8bcd0b Compare August 2, 2026 19:18
@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch from 2073625 to 206c68f Compare August 2, 2026 20:06
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
Reported by include-what-you-use on open-telemetry#4331, which has the same construct.
Adding it here rather than waiting for the same red run.
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 3, 2026
Reported by include-what-you-use on open-telemetry#4331, which has the same construct.
Adding it here rather than waiting for the same red run.
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 3, 2026
Reported by include-what-you-use on open-telemetry#4331, which has the same construct.
Adding it here rather than waiting for the same red run.
@thc1006

thc1006 commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

Context that saves working it out, since this one, #4297 and #4337 all sit on exporters/elasticsearch/src/es_log_record_exporter.cc and its test file.

Any two of the three conflict there, so whichever lands first the other two need a rebase. I will do those rebases, and there is no reason to take more than one of them.

Measured sizes, if it helps to pick a starting point. This one is 23 lines of production code and ends a wait that otherwise never returns. #4297 is 211 lines and stops a batch Elasticsearch rejected from reading as written. #4337 is 181 lines and the largest of the three. The test files are bigger than the production diffs in all three because each carries its own fake HTTP client; the second and third to land drop that copy when they rebase.

They fix separate things and none of them depends on the others, so the order is whatever suits you.

@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch 2 times, most recently from acc15e1 to 85ec7ba Compare August 5, 2026 03:01
@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch 2 times, most recently from d8826b4 to 9cf9661 Compare August 13, 2026 16:21
@thc1006
thc1006 marked this pull request as draft August 13, 2026 17:45
@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch 2 times, most recently from 090a25d to f1ec07c Compare August 15, 2026 19:29
@thc1006
thc1006 marked this pull request as ready for review August 16, 2026 14:04
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 24, 2026
Two CHANGELOG bullets pointed at this PR and said the same thing twice. The
first names what the exporter now decides success from, so the second goes.

The fixture comment described how this file and open-telemetry#4331 would be reconciled. That
is coordination between PRs, not something a reader of this file needs.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch from 5820fbf to d068395 Compare August 25, 2026 14:15
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 26, 2026
Two CHANGELOG bullets pointed at this PR and said the same thing twice. The
first names what the exporter now decides success from, so the second goes.

The fixture comment described how this file and open-telemetry#4331 would be reconciled. That
is coordination between PRs, not something a reader of this file needs.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
ResponseHandler::OnEvent records a completion for each session state that
ends the session, but ReadError and WriteError only log. A client that
reports either of them as its last event leaves the synchronous Export()
blocked on its condition variable with nothing left to wake it.

The bundled curl client dispatches neither state, so this does not change
behaviour for callers who use it. It matters for the HttpClient that the
exporter's constructor accepts.

The cases live in the existing synchronous fixture, which skips when the
exporter is built for async export, and a CTest timeout bounds them because
a regression here stalls the job rather than failing an assertion.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 force-pushed the bugfix/es-readerror-writeerror-completion-4330 branch from a8589d6 to f71ea10 Compare September 19, 2026 15:14
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>

# Conflicts:
#	CHANGELOG.md

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Elasticsearch exporter can still block on ReadError or WriteError

1 participant