Skip to content

ECHO: stop flagging routine SSE reconnects as monitor degradation - #1025

Draft
posthog-eu[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixmonitoring-repeated-stream-e6e60d
Draft

ECHO: stop flagging routine SSE reconnects as monitor degradation#1025
posthog-eu[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixmonitoring-repeated-stream-e6e60d

Conversation

@posthog-eu

@posthog-eu posthog-eu Bot commented Sep 1, 2026

Copy link
Copy Markdown

Problem

  • Host live monitoring "looked" unreliable: the monitor stream logged hundreds of monitor_stream_degraded events over 14 days, and the "Reconnecting" badge flickered on the host's screen.
  • Almost all of these were routine reconnects, not real outages. The monitor SSE stream auto-reconnects on a browser retry cycle (~3s) and a 5s fallback poll covers the gap, so a brief flap never reaches the host.
  • The client debounced degradation by only 3s, which is shorter than the reconnect cycle, so nearly every reconnect tripped it and emitted a degrade/reconnect pair.

Measured over the affected window:

Signal Value Meaning
Reconnect downtime p50 1s (from the 3s mark) Typical outage is a ~4s blip that self-heals
Reconnect downtime p90 ~17s Most events are short
Gap between degrades in a session p50 ~10 min Random, so no fixed connection-lifetime timeout

The server side is healthy (15s heartbeats, a 2s poll safety net, and the fallback poll), so there is no outage to fix there. The signal itself was over-sensitive.

Changes

  • Replace the 3s degrade debounce with a 10s grace window (MONITOR_DEGRADE_GRACE_MS) in useConversationMonitor.
  • Hold the stream connected through the grace window, so the "Reconnecting" badge and the monitor_stream_degraded event only reflect a stream that fails to recover across several reconnect attempts.
  • Genuine long outages (backgrounded tab, sleeping laptop) still flag.
  • Add a hook test covering the brief-flap path (no event, no badge flip) and the sustained-outage path (degrade, then a paired reconnect).

Testing

  • pnpm typecheck (tsc --noEmit): 0 errors.
  • biome check: clean.
  • vitest run: new useConversationMonitor tests pass; LiveMonitorSection tests still pass.

Agent context

  • Considered lowering the browser reconnect interval with an SSE retry: hint on the server, but rejected it: EventSource does not back off, so a shorter interval risks a reconnect storm against a genuinely down server. The client-side grace window is the targeted fix.

Created with PostHog Desktop from this inbox report.

The host live monitor's SSE stream auto-reconnects within a browser retry cycle (~3s) and a 5s fallback poll covers the gap, so a brief flap never reaches the host. The client debounced degradation by only 3s, shorter than the reconnect cycle, so nearly every routine reconnect flipped the "Reconnecting" badge and emitted a monitor_stream_degraded event.

Raise the debounce to a 10s grace window and hold the stream "connected" until it elapses. The badge and the degradation event now track genuine, host-visible outages instead of self-healing reconnects. Long real outages still flag.

Add a hook test for the brief-flap and sustained-outage paths.

Generated-By: PostHog Desktop
Task-Id: 4193467a-b415-4422-a5de-0c2b9dbc5a6b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants