Close querier in remote read handler to fix querier leak#7672
Open
anxkhn wants to merge 1 commit into
Open
Conversation
The remote read handler creates a storage.Querier for each sub-query in its own goroutine but never closes it, so every remote read sub-query leaked its querier and the resources it holds (inflight-query trackers, store-gateway clients, index readers) for the lifetime of the process. A client issuing many remote reads (for example Prometheus federation, Thanos sidecar remote_read, or Grafana) steadily accumulated leaked queriers. Defer querier.Close() inside the goroutine so it runs on every path, matching every other querier call site in the codebase. Add a regression test asserting that each per-query querier is closed. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Contributor
|
Good catch, seems good to me. One minor thing in the description, in tenant-federation merge queryable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
The Prometheus remote read handler (
pkg/querier/remote_read.go) handles eachsub-query in its own goroutine. Each goroutine obtains a
storage.Querierviaq.Querier(from, to)but never closes it, so every remote read sub-query leaksits querier and the resources that querier holds (inflight-query trackers,
store-gateway clients, index readers) for the lifetime of the process. A client
that issues many remote reads (for example Prometheus federation, a Thanos
sidecar's
remote_read, or Grafana) steadily accumulates leaked queriers.storage.Querierembedsstorage.LabelQuerier, whoseClose()"releases theresources of the Querier", and every other
Querier(...)call site in thecodebase pairs the acquisition with a close (for example the ingester's query
path and the tenant-federation merge queryable). The remote read handler was the
sole call site that omitted it.
This PR adds
defer querier.Close()inside the goroutine, immediately after thequerier is successfully created, so it runs on both the success path and the
Select/SeriesSetToQueryResponseerror path. It is a one-line behavioral fix(the querier is now released) with no API, flag, or config change. It also adds a
regression test that asserts the handler closes every querier it creates.
Which issue(s) this PR fixes:
No existing issue; this leak was found by code inspection. (Happy to open a
tracking issue first if maintainers prefer.)
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags (n/a - no flags)AI assistance disclosure (per the repo's GENAI_POLICY.md - keep this line in the
PR): This change was prepared with AI coding-assistant help (bug localization,
drafting, and test scaffolding). I reviewed every line, confirmed the fix against
the
storage.Querierinterface and the other querier call sites, and validatedit with the tests and linters below. I take full responsibility for the
contribution.
Commit (already on the branch, DCO-signed)
Title:
Close querier in remote read handler to fix querier leakSigned-off-by: Anas Khan 83116240+anxkhn@users.noreply.github.com
True PR diff (
origin/master...HEAD): 3 files, +42 / -0pkg/querier/remote_read.go(+1:defer querier.Close())pkg/querier/remote_read_test.go(+40:TestRemoteReadHandler_Closes_Querier+closeCountingQuerier)CHANGELOG.md(+1: BUGFIX entry)Validation run in this phase (draft)
go build -tags "netgo slicelabels" ./pkg/querier/...-> okgo test -count=1 -tags "netgo slicelabels" ./pkg/querier/ -run RemoteRead-> ok (1.627s)(full-package + -race + vet + golangci-lint v2.7.2 + goimports + misspell + gofmt
were all green in verify; re-confirmed the targeted subset here after the amend)
"loop" / task ids / "opencode" -> none (only the DCO noreply email matches
anxkhn).