Skip to content

fix(websocket): bound what a slow client can cost the process - #543

Merged
Miracle656 merged 1 commit into
Divinity-Labs:mainfrom
CollinsKRO:feat/ws-backpressure
Oct 19, 2026
Merged

Miracle656 merged 1 commit into
Divinity-Labs:mainfrom
CollinsKRO:feat/ws-backpressure

Conversation

@CollinsKRO

Copy link
Copy Markdown
Contributor

Closes #335

The bug

WebSocket#send() neither blocks nor throws when a peer is merely slow — it queues. A client on a bad mobile connection, or one whose process has been suspended, accumulated price updates in its socket's internal buffer, and nothing inspected bufferedAmount. The message was also counted as sent the moment it was queued, so a client that had stopped reading entirely still incremented ws_messages_sent_total for updates it would never see.

With the price feed pushing continuously, one stuck subscriber was enough to grow the process until it was OOM-killed — taking every other subscriber and every HTTP client with it.

The policy

Three tiers, documented in src/ws/backpressure.ts and implemented in FanOutManager.deliver:

  1. Coalesce — past WS_SLOW_CLIENT_BUFFER_BYTES (256 KiB), or inside the coalescing window, keep only the newest update and count the ones it supersedes. For a price feed a late intermediate tick is worthless; the newest tick has already superseded it.
  2. Disconnect — past WS_MAX_BUFFERED_BYTES (1 MiB), close with RFC 6455 code 1013. No amount of coalescing fixes a peer that has stopped reading entirely.
  3. Evict — congested longer than WS_MAX_CONGESTED_MS (30s), close too. Tier 1 bounds the memory such a client costs; this bounds the time. Without it a wedged client holds its slot, its bytes and a ws_clients_total series forever, and nothing says why.

Acceptance criteria

  • A client whose buffer exceeds the threshold is handled by a documented policy, not ignored — the three tiers above, in src/ws/backpressure.ts.
  • The threshold is configurable — WS_SLOW_CLIENT_BUFFER_BYTES, WS_MAX_BUFFERED_BYTES, WS_MAX_CONGESTED_MS, WS_COALESCE_INTERVAL_MS. The soft limit is clamped to half the hard one, because otherwise every client past it is disconnected on its first update over it and the coalescing tier never runs.
  • A test with a stalled socket proves memory does not grow without bound — fanoutBackpressure.test.ts drives a socket that accepts frames and never writes them, then pushes 2 000 updates through it and asserts the queue stops at the soft limit and does not climb with the next 2 000.
  • Dropped or coalesced updates are counted in a metric — ws_messages_dropped_total for superseded updates, ws_clients_dropped_total for evictions, ws_buffered_bytes for what is queued.
  • A slow client cannot affect delivery to other clients — thresholds are read per client and each send is individually guarded; asserted by a test that keeps delivering to healthy clients while a stalled one is dropped.
  • Only count a message as sent when it was actually written — ws_messages_sent_total now increments from the socket's write callback, not from send() returning. A test asserts a queued-but-never-written message counts as zero.

Two things worth calling out

The metrics were on the wrong registry. src/ws/fanout.ts registered its metrics on prom-client's global default, but /metrics renders the app's own registry from src/metrics.ts. Every websocket metric was counted somewhere Prometheus never looks. Fixed here, with a scrape assertion so it cannot regress silently.

Redis publish was on the local delivery path. onPriceUpdate published to Redis before broadcasting locally, so a stalled Redis added a network round trip to every local subscriber's delivery. Local delivery now happens first; publishing is still best-effort.

Tests

npm test — 826 passed, 1 skipped. Typecheck clean.

…ty-Labs#335)

`WebSocket#send()` neither blocks nor throws when a peer is slow — it
queues. Nothing inspected `bufferedAmount`, so a subscriber that stopped
reading accumulated price updates in its socket buffer without limit, and
`ws_messages_sent_total` counted each message the moment it was queued
rather than when it was written. One stuck subscriber could grow the
process until it was OOM-killed, taking every other subscriber and every
HTTP client with it.

`FanOutManager` now applies a per-client policy, documented in
src/ws/backpressure.ts:

- past WS_SLOW_CLIENT_BUFFER_BYTES (256 KiB) updates are coalesced to the
  newest one, since a late intermediate tick in a price feed is already
  superseded;
- past WS_MAX_BUFFERED_BYTES (1 MiB), or congested longer than
  WS_MAX_CONGESTED_MS (30s), the client is closed with RFC 6455 code 1013;
- superseded updates count in ws_messages_dropped_total and evicted
  clients in ws_clients_dropped_total, so a truncated feed is visible
  rather than silent, and ws_buffered_bytes reports total queued bytes.

Every limit is configurable and clamped so coalescing always has a window.
Thresholds are read per client, so a slow client cannot affect delivery to
any other. `ws_messages_sent_total` now increments from the socket's write
callback. The websocket metrics also move from prom-client's global default
registry onto the one /metrics actually renders.
@drips-wave

drips-wave Bot commented Oct 19, 2026

Copy link
Copy Markdown

@CollinsKRO Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Miracle656 added a commit that referenced this pull request Oct 19, 2026
`WebSocket#send()` neither blocks nor throws for a merely slow peer — it
queues, and nothing read `bufferedAmount`, so one suspended mobile client
could grow the process until it was OOM-killed, taking every other subscriber
with it.

Three tiers, and the reasoning behind each is right: coalesce past the soft
limit (an intermediate price tick that arrives late is worthless — the newest
has already superseded it, so truncate the backlog instead of draining it),
disconnect past the hard limit with close code 1013, and evict a client that
stays congested too long. The first bounds the memory a slow client costs, the
third bounds the time, and a peer that has simply stopped reading is caught by
the second.

Details I checked and was glad to find:

- The close reason is kept short because RFC 6455 caps it at 123 bytes and
  `ws` throws past that — which would have turned an eviction into an
  unhandled exception on the broadcast path.
- `ws_buffered_bytes` is aggregate-only under the `*` label. Per-client-pair
  series would be unbounded cardinality driven by caller-supplied subscription
  strings.
- The flush timer is `unref()`d, so a pending coalesce cannot hold the process
  open.
- `bufferedAmount()` is called through a guard that treats a throwing wrapper
  as zero, documented as deliberately giving a client the benefit of the doubt.
- `slowClientBufferBytes` is clamped to half `maxBufferedBytes`, so the two
  knobs cannot be configured to cancel the coalescing tier out entirely.
- `destroy()` clears the buffer sweep interval.

One fix while merging: `src/ws/backpressure.ts` had no trailing newline.

Verified on the merge result: clean merge, `tsc --noEmit` silent, 856 tests
pass with nothing red.

Closes #335

Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
@Miracle656
Miracle656 merged commit 33240aa into Divinity-Labs:main Oct 19, 2026
@Miracle656

Copy link
Copy Markdown
Collaborator

Merged. This is the best PR I've reviewed in this repo. I want to be specific about why, because the reasons are mostly things that don't show up in a diff summary.

You got the tier design right, not just a limit. The coalescing tier is the insight: an intermediate price tick that arrives late is worthless, because the newest tick has already superseded it. So the right response to a backlog is to truncate it, not drain it — and that's what keeps a healthy-but-slow mobile client connected instead of punished for its network. Then disconnect for a peer that has genuinely stopped reading, and the congestion timer because tier 1 bounds the memory a stalled client costs while nothing bounded the time. Three tiers that each answer a different failure, with the reasoning written down next to them.

Things I specifically went looking for and found already handled:

  • The 123-byte close-reason cap. RFC 6455 caps it and ws throws past it, which would have turned a backpressure eviction into an unhandled exception on the broadcast path — the one path where an exception costs you every other subscriber. Finding that comment already in the code is the single most reassuring thing in the PR.
  • ws_buffered_bytes aggregate-only under *. Per-pair series here would be unbounded cardinality driven by caller-supplied subscription strings. That exact concern is what my own open fix(ws): keep watched pairs as metric labels, bound only the unknown ones #426 is about, and you reached the same conclusion independently.
  • client.timer.unref?.() so a pending coalesce can't hold the process open.
  • bufferedAmount() behind a guard that treats a throwing wrapper as zero, with the tradeoff stated: worse to drop a healthy client than to serve a wedged one. Agreed, and worth having written down.
  • slowClientBufferBytes clamped to half maxBufferedBytes. Without it, a deployment could set the soft limit at or above the hard one and silently delete the coalescing tier — two knobs that cancel each other instead of a policy. Clamping keeps it a policy.
  • destroy() clearing the buffer sweep interval.

One fix while merging: src/ws/backpressure.ts had no trailing newline. Nothing more.

Verified on the merge result rather than your branch: clean merge, tsc --noEmit silent, 856 tests pass, nothing red.

One note for context: your PR touches pairMetricLabel in src/ws/fanout.ts, and so does my own open #426. Yours went in first and mine will rebase onto it — you hold #335, this is your work, and a maintainer's polish PR should give way rather than make you wait on it.

Minor, not worth a change: sendToClient increments ws_messages_sent_total from the send callback, so a socket wrapper that never invokes the callback would stop that counter advancing while still delivering. The real ws always calls it, so this is theoretical.

Closes #335.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

One slow websocket client can grow the server's memory without limit

2 participants