Repository navigation
fix(websocket): bound what a slow client can cost the process - #543
Conversation
…ty-Labs#335) `WebSocket#send()` neither blocks nor throws when a peer is slow — it queues. Nothing inspected `bufferedAmount`, so a subscriber that stopped reading accumulated price updates in its socket buffer without limit, and `ws_messages_sent_total` counted each message the moment it was queued rather than when it was written. One stuck subscriber could grow the process until it was OOM-killed, taking every other subscriber and every HTTP client with it. `FanOutManager` now applies a per-client policy, documented in src/ws/backpressure.ts: - past WS_SLOW_CLIENT_BUFFER_BYTES (256 KiB) updates are coalesced to the newest one, since a late intermediate tick in a price feed is already superseded; - past WS_MAX_BUFFERED_BYTES (1 MiB), or congested longer than WS_MAX_CONGESTED_MS (30s), the client is closed with RFC 6455 code 1013; - superseded updates count in ws_messages_dropped_total and evicted clients in ws_clients_dropped_total, so a truncated feed is visible rather than silent, and ws_buffered_bytes reports total queued bytes. Every limit is configurable and clamped so coalescing always has a window. Thresholds are read per client, so a slow client cannot affect delivery to any other. `ws_messages_sent_total` now increments from the socket's write callback. The websocket metrics also move from prom-client's global default registry onto the one /metrics actually renders.
|
@CollinsKRO Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
`WebSocket#send()` neither blocks nor throws for a merely slow peer — it queues, and nothing read `bufferedAmount`, so one suspended mobile client could grow the process until it was OOM-killed, taking every other subscriber with it. Three tiers, and the reasoning behind each is right: coalesce past the soft limit (an intermediate price tick that arrives late is worthless — the newest has already superseded it, so truncate the backlog instead of draining it), disconnect past the hard limit with close code 1013, and evict a client that stays congested too long. The first bounds the memory a slow client costs, the third bounds the time, and a peer that has simply stopped reading is caught by the second. Details I checked and was glad to find: - The close reason is kept short because RFC 6455 caps it at 123 bytes and `ws` throws past that — which would have turned an eviction into an unhandled exception on the broadcast path. - `ws_buffered_bytes` is aggregate-only under the `*` label. Per-client-pair series would be unbounded cardinality driven by caller-supplied subscription strings. - The flush timer is `unref()`d, so a pending coalesce cannot hold the process open. - `bufferedAmount()` is called through a guard that treats a throwing wrapper as zero, documented as deliberately giving a client the benefit of the doubt. - `slowClientBufferBytes` is clamped to half `maxBufferedBytes`, so the two knobs cannot be configured to cancel the coalescing tier out entirely. - `destroy()` clears the buffer sweep interval. One fix while merging: `src/ws/backpressure.ts` had no trailing newline. Verified on the merge result: clean merge, `tsc --noEmit` silent, 856 tests pass with nothing red. Closes #335 Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
|
Merged. This is the best PR I've reviewed in this repo. I want to be specific about why, because the reasons are mostly things that don't show up in a diff summary. You got the tier design right, not just a limit. The coalescing tier is the insight: an intermediate price tick that arrives late is worthless, because the newest tick has already superseded it. So the right response to a backlog is to truncate it, not drain it — and that's what keeps a healthy-but-slow mobile client connected instead of punished for its network. Then disconnect for a peer that has genuinely stopped reading, and the congestion timer because tier 1 bounds the memory a stalled client costs while nothing bounded the time. Three tiers that each answer a different failure, with the reasoning written down next to them. Things I specifically went looking for and found already handled:
One fix while merging: Verified on the merge result rather than your branch: clean merge, One note for context: your PR touches Minor, not worth a change: Closes #335. |
Closes #335
The bug
WebSocket#send()neither blocks nor throws when a peer is merely slow — it queues. A client on a bad mobile connection, or one whose process has been suspended, accumulated price updates in its socket's internal buffer, and nothing inspectedbufferedAmount. The message was also counted as sent the moment it was queued, so a client that had stopped reading entirely still incrementedws_messages_sent_totalfor updates it would never see.With the price feed pushing continuously, one stuck subscriber was enough to grow the process until it was OOM-killed — taking every other subscriber and every HTTP client with it.
The policy
Three tiers, documented in
src/ws/backpressure.tsand implemented inFanOutManager.deliver:WS_SLOW_CLIENT_BUFFER_BYTES(256 KiB), or inside the coalescing window, keep only the newest update and count the ones it supersedes. For a price feed a late intermediate tick is worthless; the newest tick has already superseded it.WS_MAX_BUFFERED_BYTES(1 MiB), close with RFC 6455 code1013. No amount of coalescing fixes a peer that has stopped reading entirely.WS_MAX_CONGESTED_MS(30s), close too. Tier 1 bounds the memory such a client costs; this bounds the time. Without it a wedged client holds its slot, its bytes and aws_clients_totalseries forever, and nothing says why.Acceptance criteria
src/ws/backpressure.ts.WS_SLOW_CLIENT_BUFFER_BYTES,WS_MAX_BUFFERED_BYTES,WS_MAX_CONGESTED_MS,WS_COALESCE_INTERVAL_MS. The soft limit is clamped to half the hard one, because otherwise every client past it is disconnected on its first update over it and the coalescing tier never runs.fanoutBackpressure.test.tsdrives a socket that accepts frames and never writes them, then pushes 2 000 updates through it and asserts the queue stops at the soft limit and does not climb with the next 2 000.ws_messages_dropped_totalfor superseded updates,ws_clients_dropped_totalfor evictions,ws_buffered_bytesfor what is queued.ws_messages_sent_totalnow increments from the socket's write callback, not fromsend()returning. A test asserts a queued-but-never-written message counts as zero.Two things worth calling out
The metrics were on the wrong registry.
src/ws/fanout.tsregistered its metrics on prom-client's global default, but/metricsrenders the app's own registry fromsrc/metrics.ts. Every websocket metric was counted somewhere Prometheus never looks. Fixed here, with a scrape assertion so it cannot regress silently.Redis publish was on the local delivery path.
onPriceUpdatepublished to Redis before broadcasting locally, so a stalled Redis added a network round trip to every local subscriber's delivery. Local delivery now happens first; publishing is still best-effort.Tests
npm test— 826 passed, 1 skipped. Typecheck clean.