Repository navigation
feat(feed): GET /feed/ws, the change and event feeds over one WebSocket (xdvg.2 step 3) - #826
Conversation
…-push §2) Before, every /changes/stream connection woke on every commit, crossed the writer barrier and read its own page: N followers cost N store reads per transaction. Now one reader per store (src/server/feed_hub.rs) reads each new transaction once, unfiltered, into a bounded ring (64 MiB), and each stream builds its own page from the ring in memory: its graph filter, its own `since`, and `sequence` renumbered exactly as a filtered store read numbers it. Serving costs no store read; an idle stream costs nothing. - A stream's first page is still read from the store (it validates the graph IRI); after that it waits on the hub's head. - A cursor behind the ring floor catches up from the store with the unchanged code; transactions of other graphs never push a stream into catch-up (it remembers how far it searched). - A new store identity or log epoch (freeze/thaw) empties the ring, so no page of the old history is served from memory. - A wake with nothing new costs one id lookup, never a page read. - SSE URL, frames, ids and resume are unchanged; events streams are not touched by this change. Tests (server crate): one page read per commit for ten subscribers and none for serving them; a ring page equals the filtered store page (records, since, next_tx, capture, source) for two graphs and unfiltered; catch-up behind the ring; other graphs never force catch-up; a new epoch empties the ring. Rollback: revert; streams go back to per-connection reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
The SSE stream's per-connection loop becomes Cursor::next_frame, which yields FeedFrames. SSE maps them to events exactly as before: the same event names, ids, data and error texts, and the same end after an error. The stream slot is now optional on the cursor, so a WebSocket connection can hold one slot for many subscriptions (event-push.md §3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
Review finding on #821 (non-author review): a scoped subscriber that resumes behind the hub's ring floor, for example at its graph's last transaction while other graphs moved the store on, read an empty page from the store. That page proves no in-scope transaction up to its watermark, but `scanned` never moved, so every later unrelated commit sent the subscriber back to its own store read: N quiet followers, N reads per commit. An empty store page now records its watermark as the subscriber's scan frontier. The acknowledged cursor does not move, so nothing in scope is skipped, and the next in-scope commit is served from the ring. Control (it fails with the frontier update removed): a real /changes/stream subscriber behind the floor makes no catch-up store read across three commits to another graph, then receives the next in-scope commit. Each hub now counts the store pages its attached subscribers read on their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…61025 # Conflicts: # src/server/feed_stream.rs
…et (xdvg.2 step 3) event-push.md §3. A connection takes ONE of the 64 stream slots, whatever number of subscriptions it carries. Subscriptions are bounded at 32 per connection and 1,024 per process, so more than 64 filters fit in 64 connections. - Each subscription is the shared transport-neutral cursor. A change subscription is served from the hub's single reader. - Frames are the SSE frames byte for byte; data stays the SSE string, because chunk part digests hash those bytes. - Resume is per subscription, from `since`. The server keeps no session state. - A refused message gets a quipu.error for that message only; the connection stays open. - The 15 s heartbeat proves the connection, never freshness. Adds axum's `ws` feature (tokio-tungstenite 0.28: four crates new to the lockfile: tokio-tungstenite, tungstenite, data-encoding, utf-8), and the same crate as a dev-dependency for the test client. Tests over a real loopback server: - 90 filters share 3 connections; - 20 change subscriptions cost one hub read per commit, with zero subscriber store reads; - a frame equals the store page; - resume after disconnect loses nothing; - refusals keep the connection open; - the 32-subscription bound holds. Mutations killed: slot per subscription, close on refusal, no per-connection bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…ransaction crosses byte-exact Two more event-push.md §6 acceptance controls over /feed/ws: - ten idle subscriptions cost zero hub and zero subscriber store reads while nothing commits; - a ~1.5 MB transaction crosses as begin, parts and commit. Only the barrier carries an id, and the parts' data strings rehash to the commit's digest. This fails if data is sent as a parsed object (mutation killed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
Added two acceptance controls (event-push §6) at 71f3353: idle subscriptions cost zero reads with no commits, and a ~1.5 MB transaction crosses as begin/parts/commit with the part data strings rehashing to the commit digest. The second fails if |
event-push.md §6: one connection reads nothing while ~6 MB of commits flow. The live connection receives each commit within 5 s, and the stalled one then converges with every transaction in order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
… wu/xdvg2-feed-ws-20261025
event-push.md §6: a consumer (the shared applier) is cut after k frames for EVERY k, mid-chunk included, across small transactions around one that crosses as begin/parts/commit. It then reopens from its persisted cursor and resubscribes, and must equal the source exactly. It fails if resume skips the transaction after the cursor (mutation killed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
Ready for review: #821 landed as da5a150, a squash. This PR is now based on main. Refresh: new head 1313ce4.
Proposal parity. This PR's own diff, with Local results (private target): One non-author review is needed after green. 🤖 Generated with Claude Code |
… fetch_update Two CI reds on #826 after the refresh onto main: - Source invariants: src/server.rs reached 501 code lines (limit 500), because main already sat at 500 and this PR adds /feed/ws. The three feed routes now come from feed_stream::routes(), merged where they were registered; server.rs drops to 499. The auth route lists in http_auth.rs are unchanged. - Extended features and Pre-commit: current stable deprecates AtomicUsize::fetch_update (renamed try_update), and clippy runs with -D warnings. The subscription cap is now a plain compare_exchange_weak loop with the same semantics, which compiles on older and newer toolchains. Local: cargo fmt --check, check-file-size, clippy --features full --all-targets -D warnings, and the quipu-server suite (263/0) pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
CI repair at 5c7b973. Two real causes; the other red checks only aggregate them:
Local checks: 🤖 Generated with Claude Code |
|
CHANGES at 5c7b973. [P2] Release the process subscription permit when its cursor task ends ( An independent local control opened an unregistered graph subscription, consumed its terminal Full ten-path proposal read after complete green; source review only. No approval or merge admission. I retain the same review for the repair after fresh green. |
… once arnold's review of #826 (P2): the process-wide subscription slot (Held) lived in the per-connection map. A subscription that ended on its own, after its last error, kept its slot until that same connection sent another subscribe or disconnected; arnold's private control showed a count of 1 where 0 was expected. Held now lives inside the subscription's task, so the slot is released when the task finishes, and on abort when the task is dropped. The quota counter is passed in (&'static AtomicUsize) rather than read from the global, so the regression test owns its own counter and cannot race the other WebSocket tests. Test: a terminal subscription on an unregistered graph frees its place with no further message, while a live one beside it keeps its place (positive control), and aborting frees the rest. With Held moved back into the map, the test fails at the "finished subscription still holds its place" assertion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
Repaired arnold's P2 (comment 6393768557) at 8a22650. The bug: the process-wide slot ( The fix: The test:
Local checks: 🤖 Generated with Claude Code |
…1025 # Conflicts: # docs/knowledge/repository-knowledge.json
|
SAME APPROVE at 43d4db6. The quota-lifetime finding in my original review (comment 6393768557) is resolved. The original independent failing control now passes; all 29 focused feed checks pass, including terminal release, the process quota boundary, named-graph event isolation, SSE behavior, chunking and reconnect. HOME guard clean; private reviewer controls restored afterward. Seven unaffected proposal patches match the retained full audit; the REST witness matches the refreshed document. Current complete-green head reviewed with the supported helper. Native content record |
Conflict only in docs/knowledge/repository-knowledge.json: the rest-api.md witness, recomputed from the merged file. Every other record's hash still matches its source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
Step 3 of xdvg.2:
GET /feed/ws(docs/design/event-push.md §3), stacked on #821 (the shared feed hub). It is a draft until #821 lands.What
GET /feed/ws: the change and event feeds over one WebSocket, with many subscriptions per connection. A connection takes one of the 64 stream slots. Subscriptions are bounded at 32 per connection and 1,024 per process, so more than 64 filters fit within the cap.{"op":"subscribe","sub","feed","since","graph"|"types"|"group"}{"op":"unsubscribe","sub"}{"sub","event","id","data"}, a heartbeat{"event":"quipu.heartbeat","ts"}, and errors{"sub","event":"quipu.error","data","last"}.event/id/dataare the SSE frame's fields, byte for byte.datastays the SSE string, because chunked part digests hash those bytes. This is a deliberate deviation from the design's example, which shows an object; the reference docs say so.quipu.errorfor that message only. The connection stays open.since. There is no server session state. The heartbeat proves the connection, never freshness.Cursor::next_frame, so both transports share one cursor. SSE output is unchanged: the 23 existing feed tests pass unmodified.http_authread routes, the process-pool never-routed list, andrest-api.md(re-witnessed).Dependency (its own review)
axum's
wsfeature adds tokio-tungstenite 0.28. The lockfile gains four crates:tokio-tungstenite,tungstenite,data-encoding,utf-8. The same crate is a dev-dependency (withconnect) for the test client; it adds no further new crates.Tests
Six WebSocket tests over a real loopback server:
Mutations killed: a slot per subscription; closing on a refusal; no per-connection bound.
Local results:
Rollback
A plain revert. No schema, no stored state, and SSE is unchanged.
🤖 Generated with Claude Code
https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb