Skip to content

Close querier in remote read handler to fix querier leak#7672

Open
anxkhn wants to merge 1 commit into
cortexproject:masterfrom
anxkhn:fix/remote-read-querier-close
Open

Close querier in remote read handler to fix querier leak#7672
anxkhn wants to merge 1 commit into
cortexproject:masterfrom
anxkhn:fix/remote-read-querier-close

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 5, 2026

Copy link
Copy Markdown

What this PR does:

The Prometheus remote read handler (pkg/querier/remote_read.go) handles each
sub-query in its own goroutine. Each goroutine obtains a storage.Querier via
q.Querier(from, to) but never closes it, so every remote read sub-query leaks
its querier and the resources that querier holds (inflight-query trackers,
store-gateway clients, index readers) for the lifetime of the process. A client
that issues many remote reads (for example Prometheus federation, a Thanos
sidecar's remote_read, or Grafana) steadily accumulates leaked queriers.

storage.Querier embeds storage.LabelQuerier, whose Close() "releases the
resources of the Querier", and every other Querier(...) call site in the
codebase pairs the acquisition with a close (for example the ingester's query
path and the tenant-federation merge queryable). The remote read handler was the
sole call site that omitted it.

This PR adds defer querier.Close() inside the goroutine, immediately after the
querier is successfully created, so it runs on both the success path and the
Select/SeriesSetToQueryResponse error path. It is a one-line behavioral fix
(the querier is now released) with no API, flag, or config change. It also adds a
regression test that asserts the handler closes every querier it creates.

Which issue(s) this PR fixes:

No existing issue; this leak was found by code inspection. (Happy to open a
tracking issue first if maintainers prefer.)

Checklist

  • Tests updated
  • Documentation added (n/a - no user-facing docs/flags changed)
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags (n/a - no flags)

AI assistance disclosure (per the repo's GENAI_POLICY.md - keep this line in the
PR):
This change was prepared with AI coding-assistant help (bug localization,
drafting, and test scaffolding). I reviewed every line, confirmed the fix against
the storage.Querier interface and the other querier call sites, and validated
it with the tests and linters below. I take full responsibility for the
contribution.


Commit (already on the branch, DCO-signed)

Title: Close querier in remote read handler to fix querier leak
Signed-off-by: Anas Khan 83116240+anxkhn@users.noreply.github.com
True PR diff (origin/master...HEAD): 3 files, +42 / -0

  • pkg/querier/remote_read.go (+1: defer querier.Close())
  • pkg/querier/remote_read_test.go (+40: TestRemoteReadHandler_Closes_Querier + closeCountingQuerier)
  • CHANGELOG.md (+1: BUGFIX entry)

Validation run in this phase (draft)

  • go build -tags "netgo slicelabels" ./pkg/querier/... -> ok
  • go test -count=1 -tags "netgo slicelabels" ./pkg/querier/ -run RemoteRead -> ok (1.627s)
    (full-package + -race + vet + golangci-lint v2.7.2 + goimports + misspell + gofmt
    were all green in verify; re-confirmed the targeted subset here after the amend)
  • Leak scan of the diff + commit message for local paths / workspace layout /
    "loop" / task ids / "opencode" -> none (only the DCO noreply email matches anxkhn).

The remote read handler creates a storage.Querier for each sub-query in
its own goroutine but never closes it, so every remote read sub-query
leaked its querier and the resources it holds (inflight-query trackers,
store-gateway clients, index readers) for the lifetime of the process.
A client issuing many remote reads (for example Prometheus federation,
Thanos sidecar remote_read, or Grafana) steadily accumulated leaked
queriers.

Defer querier.Close() inside the goroutine so it runs on every path,
matching every other querier call site in the codebase.

Add a regression test asserting that each per-query querier is closed.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@dosubot dosubot Bot added component/querier go Pull requests that update Go code type/bug labels Jul 5, 2026
@siddarth2810

Copy link
Copy Markdown
Contributor

Good catch, seems good to me.

One minor thing in the description, in tenant-federation merge queryable Close() returns just nil, so I'm thinking if its a weak example for "releases the resources of the Querier" ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/querier go Pull requests that update Go code size/M type/bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants