Skip to content

Observed LocalSessionManager never removes sessions that exit via WorkerQuitReason::IdleTimeout, so abandoned streamable-HTTP sessions leak their map entry permanently causing memory leak #1244

Closed

Description

@Luxas98

Summary

LocalSessionManager.sessions is only ever removed from in close_session(), which is reached solely
via an HTTP DELETE (and one setup-error path). When a session's worker instead exits on its idle
keep-alive — WorkerQuitReason::IdleTimeout — the worker's channels are dropped, but the
LocalSessionHandle entry stays in the map forever.

Any client that creates a session and does not send DELETE therefore leaks a permanent per-session
allocation. That includes clients that crash, time out, disconnect, or simply do not implement session
teardown — and, per #1048, rmcp's own OAuth discovery probe.

This is adjacent to #808 but distinct: #808 fixed pre-initialize sessions having no timeout at all.
Here the timeout works correctly and the worker does exit — the map entry is simply never cleaned up.

We did testing and observe this via https://github.com/epidemicsound/apollo-mcp-server

Affected versions

Verified present in 2.1.0 through 3.2.0 and current main — the structure is unchanged:

insert remove worker idle exit
rmcp-v2.1.0 local.rs:53 local.rs:71 (close_session only) local.rs:1035
main / rmcp-v3.2.0 local.rs:67 local.rs:85 (close_session only) local.rs:1126

There is no Drop impl and no sweeper task that removes entries. The two retain() calls in the file
operate on the completed-request cache and shadow_txs, not on sessions.

Reproduction

Any streamable-HTTP server built on LocalSessionManager — e.g. examples/servers/src/streamable_http.rs
or servers_counter_streamhttp. Then:

# create a session and abandon it, ~10/s, never sending DELETE
while :; do
  curl -s -o /dev/null -X POST http://localhost:8000/mcp \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json, text/event-stream' \
    -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"leak","version":"1"}}}'
  sleep 0.1
done

Watch RSS. A tighter assertion, without any HTTP:

// create N sessions, advance past SessionConfig::DEFAULT_KEEP_ALIVE (300s), never close_session
assert_eq!(manager.sessions.read().await.len(), 0); // currently fails: len() == N

Observed

Measured against a downstream consumer (Apollo MCP Server v1.17.0, which pins rmcp = "2.1"), driving
~10 abandoned sessions/sec for ~9 minutes, ~5,400 sessions total:

churn ON (10 sessions/s)          churn OFF
t+0     27M   <- baseline         t+1   274M
t+2    134M                       t+3   188M
t+4    239M                       t+5   129M
t+6    291M                       t+7   129M   <- floor, flat 17+ min
t+8    306M   <- plateaus

Two distinct components, both consistent with the code:

  • ~36 KB/session transient, reclaimed ~7 min after churn stops — the worker's channels, freed when
    it exits at DEFAULT_KEEP_ALIVE = 300s.
  • ~19 KB/session permanent — memory settles at a floor well above baseline and stays there. This is
    the leaked map entry.

Growth plateaus under sustained churn at roughly creation_rate x keep_alive x per_session_size, which is the steady state you would predict if the transient part is reaped and the map entry is not.

Controls, all flat: identical config with telemetry removed; the same POST rate to /mcp with a non-initialize body (rejected before a session exists); and ~9,600 requests to an unrelated
non-MCP endpoint. Only session creation drives it.

Why it matters in practice

The failure mode is self-reinforcing for long-running servers. Once the process is OOM-killed every
live session is destroyed, every client re-initializes at once, and the burst of new sessions refills
the map faster than before.

Note that tuning SessionConfig does not mitigate it — keep_alive, init_timeout and
completed_cache_ttl only affect the transient portion. The map entry leaks regardless of the timeout
value.

Suggested fix

Remove the session from LocalSessionManager.sessions when its worker terminates, not only on explicit
close_session.

Options:

  1. Have the worker signal its session id on exit and let the manager remove it — covers
    IdleTimeout, Cancelled and fatal quits uniformly.
  2. Give LocalSessionHandle (or the spawned task) ownership of a guard that removes the entry on drop.

Either way the invariant worth asserting in a test is that after keep_alive elapses with no activity,
sessions.len() returns to zero.

Related

Activity

  1. jstar0 commented on Sep 3, 2026

    @jstar0
    Contributor

    I traced the current main behavior and found two distinct compositions:

    • StreamableHttpService::spawn_session_worker waits for serve_server and then calls SessionManager::close_session for every worker exit. The existing test_keep_alive_timeout_does_not_emit_error_log regression covers this path and confirms the session is removed after an idle reap.
    • I can reproduce the map retention with the lower-level public composition of LocalSessionManager::create_session() + serve_server() directly: after the worker returns WorkerQuitReason::IdleTimeout, the transport is finished but the manager still reports the session as present because no outer task calls close_session.

    Could you confirm whether LocalSessionManager is intended to self-reap for that lower-level composition as well? If so, the fix needs a manager-owned worker-exit signal (and should guard against an old worker removing a replacement session with the same ID); if the supported contract is only the StreamableHttpService path, the issue may be limited to custom integrations.

  2. Luxas98 commented on Sep 4, 2026

    @Luxas98
    Author

    You are completely right! Sorry and thank you! We still see memory leak but it might be apollo itself and we are just missing close_session somewhere 🙇 🙇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions