diff --git a/doc/.vitepress/config.ts b/doc/.vitepress/config.ts index 0a11949980d..d5134cdd50c 100644 --- a/doc/.vitepress/config.ts +++ b/doc/.vitepress/config.ts @@ -118,6 +118,7 @@ export default defineConfig({ { text: "Transport", link: "/concept/transport" }, { text: "moq-lite", link: "/concept/moq-lite" }, { text: "Identity and recovery", link: "/concept/identity" }, + { text: "Sockets", link: "/concept/socket" }, { text: "hang", link: "/concept/hang" }, { text: "Archives", link: "/concept/archives" }, { text: "Audio jitter", link: "/concept/audio-jitter" }, diff --git a/doc/concept/index.md b/doc/concept/index.md index 06cc676446e..1b1884eaca3 100644 --- a/doc/concept/index.md +++ b/doc/concept/index.md @@ -38,6 +38,7 @@ watched at 100 ms by one viewer and 10 s by another. - [Transport](/concept/transport): why QUIC, and the WebTransport, WebSocket, and raw QUIC paths onto it. - [moq-lite](/concept/moq-lite): the pub/sub protocol, discovery, path patterns, subscriptions, and congestion behavior. - [Identity and recovery](/concept/identity): choosing stream instances, following restarts, and answering work on demand without mixing streams. +- [Sockets](/concept/socket): dial and accept point-to-point sessions through relays, as reliable byte pipes or as tracks of messages. - [hang](/concept/hang): the media catalog, containers, encryption, and how to extend both. - [Archives](/concept/archives): record listed tracks into object storage and serve them back from a host. - [Audio jitter](/concept/audio-jitter): how a receiver sizes its audio playout target from arrival timing. diff --git a/doc/concept/socket.md b/doc/concept/socket.md new file mode 100644 index 00000000000..d096572d04f --- /dev/null +++ b/doc/concept/socket.md @@ -0,0 +1,223 @@ +--- +title: Sockets +description: Dial and accept point-to-point sessions over MoQ, as reliable byte pipes or as tracks of messages +--- + +# Sockets + +A socket connects two endpoints through relays, the way a TCP connection +connects two hosts. Each endpoint announces one hidden broadcast, dials other +endpoints, and accepts their dials. Each connection is a session that only the +two ends can read while its id stays secret (see [Hidden names](#hidden-names)), +in one of two protocols: `tcp`, a reliable, ordered byte +pipe each way, or `track`, a track each way whose groups are messages. Use a +socket for per-client state that only the two ends need, such as a viewer's +commands and the replies to them, while whatever every client needs stays in an +ordinary broadcast. + +A socket is a convention over announcements, broadcasts, and track +subscriptions, not a wire extension. It uses no instances and no track +listings, and relays carry it unchanged. It needs moq-lite-07 or +moq-transport on every hop: an endpoint's broadcast is +[hidden](/concept/moq-lite#hidden-broadcasts), and an older moq-lite hop +never carries hidden broadcasts, so a dial across one never finds its peer. This page is the convention, enough to implement it in any +language. +[`moq_net::socket`](https://docs.rs/moq-net/latest/moq_net/socket/) implements +it in Rust. + +## Endpoints + +An endpoint with id `` announces one broadcast, once: + +```text +/.socket/ +``` + +The id is one path segment: a well-known name for a server, such as a game, or +a random id for a client. The prefix is shared by every endpoint that talks to +each other. The `.socket` segment starts with `.`, so endpoints are +[hidden](/concept/moq-lite#hidden-broadcasts) from anyone listing the prefix. + +The announcement is used only for routing: it is how a relay finds an +endpoint. Connections are never announced. An endpoint's broadcast holds only +session tracks, served on request, whatever their protocol. + +## Sessions + +A session between a dialer `A` and an acceptor `B` is named by its protocol `P`, +both ends, and a session id `S` that `A` picks: random and unguessable, at +least 128 bits, such as 32 hex digits. Both halves of the session are tracks +with the same name, `///`, one in each +endpoint's broadcast: + +| Track | In broadcast | Written by | Read by | +| --- | --- | --- | --- | +| `P/A/B/S` | `/.socket/A` | `A` | `B` | +| `P/A/B/S` | `/.socket/B` | `B` | `A` | + +The protocol is `tcp` or `track`; see [Protocols](#protocols). An endpoint +resets a request whose protocol it doesn't speak, or isn't accepting, as +unsupported. + +The dialer's id follows the protocol in both broadcasts, so an endpoint tells a +dial from an answer by the name alone: a request with another endpoint as the +dialer and itself as the acceptor is that endpoint dialing, and a request with +itself as the dialer answers one of its own dials. An endpoint resets any other +request, such as one naming another acceptor, as not found. Since `S` is random +per dial, every session is distinct, and one endpoint can dial and accept any +number of sessions at once, with the same peer or others. + +### Dial + +1. `A` picks `S` and holds its own half, `P/A/B/S` in its broadcast, + unanswered. +2. `A` subscribes to `/.socket/B` for the track `P/A/B/S`, waiting for + `B`'s endpoint to be announced if it is not yet. +3. `B` receives that subscription as a request for a track it does not have: + an incoming dial in protocol `P`, claiming to be from `A`, with session `S`. + +### Accept + +1. `B` subscribes to `/.socket/A` for the track `P/A/B/S`, and holds + `A`'s subscription unanswered. A relay can deliver the dial before `A`'s + announcement, so `B` waits for `A`'s endpoint to be announced for as long as + `A`'s subscription stays open, and gives up once it closes. +2. `A` answers that subscription only if it dialed `S` to `B` and is still + waiting, and resets any other. That makes the exchange a return-routability + check: a dial forged in `A`'s name, by anyone who can subscribe to `B`, + reaches an `A` that never dialed it, so `A` resets `B`'s subscription and `B` + resets the forger's. +3. Once `A` answers, `B` answers `A`'s subscription, and the session is open + for both. `B` writes nothing before `A` answers. + +### Reject + +`B` resets `A`'s subscription, with not found, as for a session that does not +exist. An endpoint resets a dial in a protocol it is not accepting as +unsupported, and one that accepts nothing at all answers not found, as for any +track nobody serves. A dial that gives up resets `B`'s subscription in turn. + +## Protocols + +Each side writes its own half and reads the peer's. A reader subscribes +reliably and in order: oldest group first, with no delay budget, so it waits +for a late group rather than skipping it. The protocol decides what a half +holds. + +### `tcp` + +Each half is exactly one group, sequence 0. In MoQ a group travels as one QUIC +stream on every hop, so it is reliable, strictly ordered, and flow controlled, +exactly like one direction of a TCP connection: + +- A frame is a write. The reader reads the frames in the order they were + written, with nothing skipped. +- Finishing the group is FIN: the reader reads to the end, then sees the end. +- Resetting the group, or its track, is RST: the reader fails with the error. +- A second group on a session track is a protocol error: the reader resets the + session, both halves. + +There are no track-level settings: no priority between messages, no delay +budget, no order, since the newest group is never stale. + +A session can last indefinitely: a group has no size cap, and each copy keeps +only a bounded tail of it. A writer that waits for room before each write (in +Rust, `group::Producer::ready`) runs at most a window ahead of the reader on +every hop, so a slow reader slows the writer, as TCP backpressure does, and +nothing is lost. A writer that doesn't wait sheds the front of its half, and a +reader that falls behind it fails with an eviction error instead of skipping +data. Pacing reaches across native QUIC and qmux hops; see +[unbounded groups](https://github.com/moq-dev/moq/tree/main/quest/m1/unbounded-groups) +for what remains, such as browsers, which read without pacing. + +Use `tcp` for a byte stream or a log of records that must all arrive in order, +such as a proxied TCP connection, a file transfer, or a request and its reply. + +### `track` + +Each half is a whole track, and a message is one group. A slow message never +holds up the next, and each group can carry its own priority. Messages are +ordered by group sequence. A half has no max age unless its writer sets one, so +a newer message never makes an older one stale. A reader that keeps up gets +every message, but each +hop's cache reclaims a message nobody read within its idle window (30 s by +default), or under memory pressure, except the newest, so a reader that stalls +longer can miss messages. + +The reader's subscription starts reliable and in order, like `tcp`. An +application may make it lossy for messages that go stale, such as positions or +controls, by updating the subscription with a finite delay budget: a message +that is late past the budget is skipped rather than holding up later ones. It +may also read newest first. + +Finishing a half closes the session that way. Resetting it is an error. + +Use `track` for independent messages, where a late one should not block the +next: commands, controls, acknowledgements, or telemetry. + +## Closing + +An endpoint closing, or its broadcast ending, resets every session it holds, +and stops reading them. A side that sees the peer's half reset treats the +session as closed and resets its own half, so both halves end. + +There is no multiplexing within a session: another channel is another session. +Several sessions between the same two endpoints are independent, and a +restarted endpoint starts new sessions. A channel that every client reads, such +as a game's status, stays an ordinary broadcast. + +## Hidden names + +The session id is the secret that keeps a session to its two ends: the token +grants below let any endpoint subscribe to any other, so whoever learns a +session's track name can read it, or answer it before its peer. + +The acceptor learns the name too, but cannot replay it to a third endpoint +`C`: the name binds both ends. `C` resets `P/A/B/S` as a session for another +acceptor, and `A` never dialed `P/A/C/S`, so it resets `C`'s accept of that. +The peer id of an accepted session is the endpoint that dialed it to this one. + +A session track therefore never appears where a third party can read it: + +- It is never announced. +- It is never listed: an endpoint serves its session tracks only on request, + so a moq-lite 07 track listing (`TRACKS`) of an endpoint's broadcast names + none of them, even to a reader allowed everything. +- Logs and metrics that a third party can read must not carry it. + +## Authorization + +Sockets need nothing beyond ordinary +[token grants](/bin/relay/auth#path-matching). With the token's root at the +socket prefix, each grant is a prefix: + +```jsonc +// The endpoint : publishes its own endpoint, reads every endpoint. +{ "root": "", "publish": [".socket/"], "subscribe": [".socket/**"] } +``` + +The publish grant pins the id, so no endpoint can announce as another. The +subscribe grant lets it reach any endpoint, which is what dialing and accepting +need; the session id keeps it out of other endpoints' sessions. A narrower +subscribe grant, such as a client allowed only `.socket/`, also works: +it can then dial only that server and accept only its dials. + +`moq auth sign` writes this as, for example: + +```bash +moq auth sign --key key.jwk --root "$PREFIX" --publish ".socket/$ID" --subscribe '.socket/**' > endpoint.jwt +``` + +Each client then connects to the relay at the prefix with its token, and its +endpoint's prefix is the session root. + +Signing and encryption are not part of the convention: an endpoint id is a +name the token pins, not a key, and relays are trusted with session names. A +planned mesh layer makes ids public keys. + +## Later + +Later protocols are other prefixes on the same endpoint, with no new +announcement. A multiplexed `quic` mode, with many streams per session like +QUIC, would name its sessions `quic///` and bring back several +groups or streams per session. diff --git a/doc/lib/rs/moq-net.md b/doc/lib/rs/moq-net.md index 03b7e13cec8..fe0dbd9c88c 100644 --- a/doc/lib/rs/moq-net.md +++ b/doc/lib/rs/moq-net.md @@ -20,6 +20,7 @@ What you use it for, beyond what the concept page already describes: - **Rank several CDNs.** Dial each with its own subscribe handle, `origin.clone().with_preference(origin::Preference::new(rank))` (higher wins; 0 is neutral and a negative rank demotes), all on one origin. Requests use the preferred link that has a route and fail over to the next; see [multiple CDNs](/concept/moq-lite#multiple-cdns). - **Request one instance, or follow them all.** `consumer.request(path, None)` resolves the instance serving a path now, and fails `Unroutable` only once discovery has synced with no route for it. `consumer.follow(path, None)` hands out each publisher instance serving a path from `available()`: the first once announced, then each one that takes over or returns after an end, like JS `follow(path)`. A broadcast already resolved stays on the old instance until dropped or its route goes. See [identity and recovery](/concept/identity) to choose between following and holding one instance. - **Discovery follows what you watch.** A client session asks the relay for announcements only under the prefixes its subscribe origin's readers watch, with `announced()`, `request`, or `follow`, stops asking about a prefix 2s after nothing at or beneath it is watched, and asks again on reconnect. A broadcast resolved through a discovered route keeps its path watched until dropped. `request` owns discovery of its path, so no separate watch is needed. A server session, or a client fed into an origin handle marked `with_link(origin::Link::Peer)` or `Link::Upstream`, asks for everything it may, so a relay discovers everything. An HTTP gateway (the moq-hls `Server`, the moq-rtc WHEP router) watches its whole scope while it lives, so the paths its callers name never open a request each. moq-lite-03/04 sessions, whose announcements mark no initial set, still ask for everything. +- **Dial and accept sockets.** `socket::Endpoint::new(&origin, prefix, id)` announces a hidden endpoint at `/.socket/`, used only to route dials. A session speaks one of two protocols, each with its own dial and accept. `accept_tcp()` and `accept_track()` yield an `Incoming` dial to `accept` or `reject` (dropping it rejects). `dial_tcp(peer)` and an accepted `tcp` dial yield a `socket::TcpStream`, a `group::Producer` and `group::Consumer` pair carrying one reliable, ordered byte pipe each way (finish is FIN, abort is RST, and `ready` paces the writer to the reader); `dial_track(peer, info)` and an accepted `track` dial yield a `socket::TrackStream`, this side's `track::Producer` and a `track::Subscriber` of the peer's half, subscribed reliably and in order until the application updates it, one message per group. A dial resolves once the peer accepts, and an accept checks that the claimed dialer really dialed this endpoint, waiting for its endpoint while the dial stays open. Session tracks are never announced or listed. See [Sockets](/concept/socket). - **One track per name.** Concurrent subscriptions share one request and producer. The application owns sequence numbers: the broadcast keeps nothing of an ended track, so a producer that replaces it starts at 0. A name always means the same content, so the replacement continues past its predecessor's groups with `create_group` at an explicit sequence. Readers of an aborted track resume onto the replacement when they read through an origin request, whose route always names an instance; otherwise their subscription ends with the error. - **Authored tracks default to a zero `max_age`.** `track::Info::default()` sets `max_age` to `Some(Duration::ZERO)`, the live edge only; a publisher that wants readers to reach history sets a longer bound, or `with_max_age(None)` for no publisher limit. Media tracks from `hang::container::track_info` declare `hang::container::MAX_AGE` (30 seconds). `max_age` is how long a superseded group stays fresh, never the newest group; it frees no bytes, which the cache pool's expiry reclaims. Received metadata keeps absence as no limit, and an origin's `cache_duration` can only cap it lower. - **End content by finishing tracks.** A broadcast has no finished state. When the input ends, finish each track; the broadcast stays announced and keeps serving them until you close it or drop its last producer. How long to keep it up for late readers is yours to decide: `moq_mux::Linger` holds a hang broadcast for a fixed window. diff --git a/quest/m1/socket/README.md b/quest/m1/socket/README.md index 06ca125747b..64d5b994365 100644 --- a/quest/m1/socket/README.md +++ b/quest/m1/socket/README.md @@ -2,78 +2,184 @@ ## Goal -`moq-net` gains a generic dial/accept primitive. A `Listener` at a path accepts -the `Socket`s that dialers announce. Inside a socket either side opens and -accepts named tracks the way QUIC opens and accepts streams. moq-boy, CARLA, -[byte streams](/quest/m2/stream/README.md), and the mesh -[Endpoint](/quest/m2/mesh/endpoint.md) all build on it instead of each +`moq-net` gains a generic dial/accept primitive. An `Endpoint` announces one +hidden broadcast, dials other endpoints, and accepts their dials, and each +connection is a session between the two ends: a reliable, ordered byte pipe +each way (`tcp`), or a track of messages each way (`track`). moq-boy, CARLA, [byte streams](/quest/m2/stream/README.md), and the +mesh [Endpoint](/quest/m2/mesh/endpoint.md) all build on it instead of each hand-rolling the announce, watch, and request loop. Rust, JS, and Python (through moq-ffi) expose the same surface and interoperate. Non-goals: a wire change to moq-lite (a socket is a path and track convention -over existing announce, Tracks, and subscribe streams); session caps, leases, -or idle timeouts, which stay application logic (boy's 60 s idle, CARLA's -one-driver lease and 1 s watchdog); announcement backpressure, which -[announce credit](/quest/m2/announce-credit.md) owns. +over existing announce and subscribe streams); session caps, leases, or idle +timeouts, which stay application logic (boy's 60 s idle, CARLA's one-driver +lease and 1 s watchdog); signing and encryption, which the m2 +[mesh](/quest/m2/mesh/README.md) owns by making endpoint ids public keys. ## Plan +[`doc/concept/socket.md`](/doc/concept/socket.md) is the convention JS and +Python follow, and `rs/moq-net/src/model/socket.rs` implements it. + +Decided 2026-11-04, second round (maintainer, settling the Waiting items of +[moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677)): + +- Name sessions `///`, binding a session to both + ends. An endpoint resets a dial naming another acceptor, and the dialer + serves only the name it dialed, so an endpoint `A` dialed can't replay the + session to a third endpoint posing as `A`. Done now, before the JS and Python + ports copy the scheme, rather than waiting for the m2 mesh keys. Not chosen: + keep `/` and document the replay. +- An accept waits for the claimed dialer's endpoint for as long as the dial + stays open, with no timer: a relay can deliver a dial before the dialer's + announcement, such as when two endpoints dial each other at start-up. A + forged dial claiming an endpoint nobody announces holds an accept only while + the forger holds its subscription. Not chosen: fail at once and let the + dialer redial; a bounded timer. + +Decided 2026-11-04 (maintainer, /quest-complete), on +[moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677), settling its +Waiting items and replacing the `.tcp/` layout and the one-message-per-group +pipe below; the second round above adds the acceptor to the session name: + +- Name the dialer in the session. A dial and an answer can't be confused, + the claimed dialer is named for the return-routability check, and `S` + (random per dial) makes every session unique, so one endpoint can dial and + accept any number of sessions. Not chosen: reader-named halves. +- One announcement per endpoint, used only for routing: + `/.socket/`. The session track name carries the protocol, both + ends and the session: `///`. A dial is + SUBSCRIBE `/.socket/`, track `tcp///`; the + answer is the same name in the dialer's broadcast. A requested track whose protocol the + endpoint doesn't speak, or isn't accepting, is reset as unsupported. Grants: + publish `.socket/`, subscribe `.socket/**`. The module stays `socket`. +- Two protocols ship now over the same handshake, each with its own dial and + accept, and an incoming dial lands in its protocol's queue: + - `tcp///`: a session half is a single group, not a track of + messages. A group is one QUIC stream on every hop, so it's reliable, + strictly ordered and flow controlled, like a TCP direction. Frames are + writes, finishing the group is FIN, and resetting it is RST. There are no + track-level settings (budget, max-delay, order), since the newest group is + never stale. A second group on a session track is a protocol error that + resets the session. Rust: `Endpoint::dial_tcp` / `accept_tcp`, yielding + `socket::TcpStream { producer: group::Producer, consumer: group::Consumer }` + plus `peer()` and `session()`. Not chosen: `track::Ordered` with one message + per group. + - `track///`: the same dial, accept, reject and return-routability + check, but the app gets the whole session tracks, with groups as messages + and its own priority, budget and order, and no single-group rule. The + reader starts reliable and ascending (no delay budget) until the app + updates it. A half has no max age unless the app sets one, so a newer + message never makes an older one stale. Rust: `Endpoint::dial_track` / + `accept_track`, yielding + `socket::TrackStream { producer: track::Producer, consumer: track::Subscriber }`. +- Long-lived `tcp` sessions rely on unbounded groups (since + [moq-dev/moq#7411](https://github.com/moq-dev/moq/pull/7411) a group has no + size cap, and each copy keeps a ~32 MiB tail). With one reader per copy, + [fastest-reader pacing](/quest/m1/unbounded-groups/fastest-reader.md) acts + as TCP backpressure: a writer awaiting `group::Producer::ready` runs at most + 16 MiB past its reader on each hop, and a reader that falls off the tail + fails with `Evicted` rather than skipping data. Where pacing doesn't reach + yet, that `Evicted` is the failure mode, never silent loss: browsers and JS + read unpaced ([JS fastest-reader pacing](/quest/m2/js-fastest-reader.md)), + and a session past its paused-credit cap (about two paused group streams at + the client defaults) reads and sheds instead of pausing. That remaining + pacing work stays in the [unbounded groups](/quest/m1/unbounded-groups/README.md) + epic, which this depends on for lossless `tcp` in those cases. + +Implementation choices in +[moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677): + +- A `tcp` session runs a small task on the endpoint origin's driver. It copies + the peer's group into the group the app reads, a frame at a time and paced by + that reader, so it can end that group itself: on a second group from the + peer, on the endpoint closing, or when the peer resets. It also finishes or + aborts this side's track the way the app ended its group, and resets the + session when the app drops its reader early. Once group 0 arrives, it + watches for a second group through the subscriber's largest position rather + than `poll_recv_group`: polling `poll_recv_group` on every wake while + reading the group it handed out stalled the group's delivery (see the PR). +- `TrackStream::consumer` is the handshake's own `track::Subscriber`, not a bare + `track::Consumer`: it is already subscribed reliably and in order, so no + message falls between the accept and the app's first read, and the app + overrides it with `update` or reads in sequence through `ordered()`. +- An endpoint takes dials in a protocol from its first accept in it, like a TCP + listener. Until its first accept in any protocol it has no request handler, + so a dial to it is reset with not found at once. A dial reserves its own + half, which the peer's answer finds without a handler, so a dial-only + endpoint never needs polling. A dial waits for the peer's endpoint to be + announced (drop it to give up), and an accept waits for the claimed + dialer's endpoint until the dialer drops its subscription. +- Accepted dials queue while the app accepts, each holding the dialer's + subscription open. A queued dial whose dialer gave up leaves its queue on the + endpoint's next poll, which each new dial triggers, so a queue holds only + open dials, even in a protocol the app stopped accepting. +- Session track names are the secret, so they never appear in a lite-07 track + listing or anywhere else a third party reads. moq-net lists only tracks a + producer creates or a handler's `Lister` answers, never one served on + request, so an endpoint serves its session tracks only on request, and the + dialer's own half is reserved unlisted + (`broadcast::Producer::reserve_unlisted_track`, crate-private). + +Decided 2026-11-03 (maintainer, /quest-grill in /quest-complete), still +standing where the 2026-11-04 decisions above don't replace it: + +- Endpoint sockets, one session per connection. Each endpoint announces one + hidden broadcast once, and connections are never announced. `` is a + well-known name for a server (a game) or a random id for a client. Why: the + old flow had the listener answer `L/.accept/` under the dialer's + instance, which broke instance uniqueness (lite-07 treats routes sharing an + instance as replicas), let two listeners accept one dial, announced every + connection publicly, and needed lite-07 instances and track listings on + every hop. +- Dial: `A` picks a random, unguessable session id `S` (128 bits) and + subscribes to `B`'s endpoint; `B` surfaces the requested track as an + incoming dial claiming to be from `A`. Accept: `B` subscribes to `A`'s + endpoint, and `A` answers only a session it dialed and resets any other, a + return-routability check that fails a dial forged in `A`'s name. `B` sends + nothing until `A` answers. Reject: `B` resets `A`'s subscription. +- No multiplexing: another channel is another session, a pair of endpoints + holds several, and a restart is a new session. +- Grants are prefix only, rooted at the prefix. A relay test signs real + `moq-auth` tokens. +- The handshake uses no instances or listings; lite-07 instances only affect + relay route choice. Tests cross moq-transport hops. Corrected 2026-11-04 in + [moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677): it needs + moq-lite-07 or moq-transport on every hop, since a moq-lite hop before 07 + has no hidden opt-in and never carries an endpoint's `.socket/` + broadcast; [older hops](/quest/m1/socket/old-hop-dial.md) make such a dial + fail instead of waiting. +- Fixed channels become sessions or messages on one: boy's `command` and + per-viewer latency, CARLA's `control`, ack, and e-stop. Shared state such as + boy's `status` stays an ordinary broadcast. +- The stream types are plain structs with public `producer` (this side's half) + and `consumer` (the peer's half) fields, plus read-only `peer()` and + `session()`, and no read, write, or message methods of their own: apps use + the group and track APIs directly, so JS, Python, and the bindings mirror a + minimal surface. + Decided 2026-11-02 (maintainer, Kixel planning): the whole epic moves from m2 to m1, so the next cut publishes sockets in `@moq/net` and moq.pro's relays carry them. Why: Kixel's T3 Code prototype ([moq-dev/kixel.ai T3 epic](https://github.com/moq-dev/kixel.ai/blob/main/quest/m0/t3/README.md)) sends T3's commands over sockets, one group per message, in place of byte -streams. `@moq/net` publishes from `release`, and accept relies on lite-07's -Tracks stream, which only `main` has, so a backport was declined and the cut -is the route. Byte streams, fastest-reader pacing, and the mesh libraries -keep their own milestones. +streams. `@moq/net` publishes from `release`. Byte streams, fastest-reader +pacing, and the mesh libraries keep their own milestones. Decided 2026-11-01 with the maintainer (planning interview from moq.pro's TCP -replacement arc): - -- Name and home: `Listener`, `dial`, and `Socket` in a `socket` module of - `moq-net` (`moq_net::socket`), the `socket` export of `@moq/net`, and - moq-ffi beside origin. Not a separate crate: it uses only the origin, - broadcast, and track APIs every binding already wraps. `Session` is taken by - the transport connection. -- Layout is owned by the listener and hidden. A listener at `L` keeps `L` for - whatever it shares with everyone (boy's game, CARLA's cameras), so fanout - is unchanged. A dialer announces `L/.dial/` and the listener answers on - `L/.accept/`. Discovery is a one-segment watch of `L/.dial/*`, which is - cheap for relays, unlike a globstar over the whole origin. Auth enforces the - id with a mid-path wildcard grant (`mesh/*/*/.dial/`), which - `moq-pattern` supports. The dot keeps sessions out of ordinary discovery - ([hidden broadcasts](/doc/concept/moq-lite.md#hidden-broadcasts)), so boy's game list - never sees viewers. -- Identity is a dialer-chosen id plus a pinned epoch. The id is the dialer's - own: an endpoint ID in the mesh, a random id in the demos. Each announce - epoch is one socket: a new epoch ends the old socket and starts a new one, - and the listener's request is pinned to the epoch (`origin::Consumer::request` - with an owned `Epoch`, as CARLA already does), so a restarted dialer never - inherits state. A socket ends when either side's broadcast ends or its epoch - is replaced. -- Streams are tracks. A uni stream is a track the opener publishes in its own - socket broadcast. A bi stream is the same track name in both. The peer - learns of new tracks from the Lite07 Tracks stream (`TRACKS_ADD`, `REMOVE`, - `FINISH`), which pushes them like QUIC's accept. `requested_track` was - declined because it is pull-shaped: only the side being subscribed to can - accept. Track names opened by the two sides must never collide (for example - a role prefix, like QUIC's stream-id low bit); the Rust quest settles the - scheme. -- Accept is unbounded and lazy. Announced dialers queue without limit, and a - dialer's broadcast is requested only when the application accepts it, so an - application that stops accepting holds no subscriptions. Backpressure on the - announcements themselves (paced by the fastest consumer, made hard by merging - overlapping prefixes) is a separate design, settled as wire-level - [announce credit](/quest/m2/announce-credit.md). +replacement arc), still standing: + +- Home: a `socket` module of `moq-net` (`moq_net::socket`), the `socket` + export of `@moq/net`, and moq-ffi beside origin. Not a separate crate: it + uses only the origin, broadcast, and track APIs every binding already wraps. - The convention is specified on a doc.moq.dev concept page, - `doc/concept/socket.md`, written with the Rust implementation. It is not a - section of the moq-lite draft or a new draft: it is an application + `doc/concept/socket.md`, not in the moq-lite draft: it is an application convention over moq-lite. -- The demos move per-client echoes onto the socket: boy's per-viewer latency - and CARLA's driver ack and sequence each go on that client's `.accept` - track. State every viewer needs (the game, who is driving, who holds the - e-stop) stays in the shared broadcast at `L`. + +Later, not built now: a multiplexed `quic` protocol, `quic///` on +the same endpoint with no new announcement, which brings back several groups +or streams per session; and signed or encrypted sessions with ids as public +keys, in the m2 [mesh](/quest/m2/mesh/README.md). This epic owns removing the hand-rolled loops: after the ports, no announce-watch-request pairing remains in `rs/moq-boy`, `js/moq-boy`, or @@ -81,14 +187,13 @@ announce-watch-request pairing remains in `rs/moq-boy`, `js/moq-boy`, or ## Required -- [Rust sockets](/quest/m1/socket/rust.md) - `moq_net::socket` with `Listener`, `dial`, and `Socket`, plus the concept page - [JS sockets](/quest/m1/socket/js.md) - the same surface in `@moq/net`, interoperating with Rust - [A socket dial over an older hop fails instead of waiting](/quest/m1/socket/old-hop-dial.md) - a dial through a pre-lite-07 hop fails `Unsupported` - [Python sockets](/quest/m1/socket/py.md) - moq-ffi and `py/moq-rs` bindings, interoperating with Rust -- [moq-boy on sockets](/quest/m1/socket/boy.md) - the emulator accepts viewers through a listener and echoes latency per socket -- [CARLA on sockets](/quest/m1/socket/carla.md) - trucks accept drivers through a listener and ack them per socket +- [moq-boy on sockets](/quest/m1/socket/boy.md) - viewers dial the emulator's endpoint and send commands and receive latency on their session +- [CARLA on sockets](/quest/m1/socket/carla.md) - drivers dial the truck's endpoint and send control and receive acks on their session ## Related -- [Byte streams](/quest/m2/stream/README.md) - TCP over sockets, one bi track per connection -- [Endpoint API](/quest/m2/mesh/endpoint.md) - the mesh's `connect`/`accept`, a socket plus signing +- [Byte streams](/quest/m2/stream/README.md) - TCP over sockets, one session per connection +- [Endpoint API](/quest/m2/mesh/endpoint.md) - the mesh's `connect`/`accept`, a socket endpoint plus signing diff --git a/quest/m1/socket/boy.md b/quest/m1/socket/boy.md index c00ea1d2909..b7c484b96ce 100644 --- a/quest/m1/socket/boy.md +++ b/quest/m1/socket/boy.md @@ -2,26 +2,35 @@ ## Goal -The moq-boy emulator accepts viewers through a `Listener` at its game path, -and each viewer dials it, so neither `rs/moq-boy` nor `js/moq-boy` watches -announcements or requests broadcasts by hand. Each viewer's latency comes back -on its own socket instead of the shared `status` map, and crowd play, pausing -when nobody watches, and the 60 s idle stop behave as before. +The moq-boy emulator accepts viewers on a socket endpoint named for its game, +and each viewer dials it from its own endpoint, so neither `rs/moq-boy` nor +`js/moq-boy` watches announcements or requests broadcasts by hand. Each +viewer's commands and latency travel on its own session instead of a viewer +broadcast and the shared `status` map, and crowd play, pausing when nobody +watches, and the 60 s idle stop behave as before. ## Plan -See the [epic](/quest/m1/socket/README.md) for the layout and the reply split. +See the [epic](/quest/m1/socket/README.md) for the convention. -- Viewers dial `boy/game/` with their random id. Their `command` track - moves from `{viewerPrefix}/{name}/{viewerId}` to the socket. Drop the +- The emulator runs an endpoint named for its game, and each viewer an + endpoint with its random id that dials it. Settle and record the prefix + (such as `boy`, so endpoints sit at `boy/.socket/`). +- The viewer's `command` track at `{viewerPrefix}/{name}/{viewerId}` becomes + the viewer's messages on a `track` session, one command per group: commands + are independent messages, and a stale one shouldn't hold up the next, which + a `tcp` session's single group would. Drop the separate viewer-prefix configuration in `rs/moq-boy/src/main.rs`. -- Per-viewer latency moves from `status`'s `latency` map to a track on that - viewer's `.accept` broadcast. Game state stays in `status`. -- Requests become pinned to the viewer's epoch (boy requests unpinned today), - so a reload is a new socket. -- The demo relay's anonymous publish grant (`demo/relay/prod.toml`, and the - `moq auth serve` line it documents) covers `demo/boy/game/*/.dial/*` - instead of boy's use of `demo/viewer/**`. Whichever of this and +- Per-viewer latency moves from `status`'s `latency` map to the emulator's + messages on that viewer's session. Game state stays in the ordinary + `status` broadcast. +- A reload is a new session; the emulator counts a viewer as gone when its + session ends. +- The demo relay's anonymous grants (`demo/relay/prod.toml`, and the + `moq auth serve` line it documents) let an anonymous viewer publish its own + endpoint and subscribe to the game endpoints, instead of boy's use of + `demo/viewer/**`. Anonymous viewers share one grant, so record how a viewer's + endpoint id is kept from being claimed by another. Whichever of this and [CARLA on sockets](/quest/m1/socket/carla.md) lands second removes `demo/viewer/**`. - Update `doc/bin/demo.md` and the boy README or CHANGELOG where they describe @@ -29,5 +38,4 @@ See the [epic](/quest/m1/socket/README.md) for the layout and the reply split. ## Required -- [Rust sockets](/quest/m1/socket/rust.md) - the emulator's listener - [JS sockets](/quest/m1/socket/js.md) - the viewer's dial diff --git a/quest/m1/socket/carla.md b/quest/m1/socket/carla.md index 8a8a84d97df..085c577d653 100644 --- a/quest/m1/socket/carla.md +++ b/quest/m1/socket/carla.md @@ -2,32 +2,36 @@ ## Goal -Each CARLA truck accepts drivers through a `Listener` at `carla/`, and -the web driver dials it, so `demo/carla/sim` and `demo/carla/web` no longer -watch `viewer/carla//` or request driver broadcasts by hand. Each -driver's ack and sequence echo come back on its own socket. The one-driver -lease, takeover, e-stop, the 1 s watchdog, and handback behave as before. +Each CARLA truck runs a socket endpoint named for the truck and accepts +drivers on it, and the web driver dials it from its own endpoint, so +`demo/carla/sim` and `demo/carla/web` no longer watch `viewer/carla//` +or request driver broadcasts by hand. Each driver's control, ack, sequence +echo, and e-stop travel on its own sessions. The one-driver lease, takeover, +e-stop, the 1 s watchdog, and handback behave as before. ## Plan -See the [epic](/quest/m1/socket/README.md) for the layout and the reply split. +See the [epic](/quest/m1/socket/README.md) for the convention. -- The worker (`publish.py`) uses the Python `Listener`; `paths.ts` and - `truck.ts` dial `carla/` with the per-lease driver id. A new lease - is a new epoch and so a new socket, which matches today's lease binding. -- The `control` track moves onto the socket. The driver's ack, sequence, and - round-trip echo move from the shared `status` to the driver's `.accept` - broadcast. `driver` and `estop_by` stay in `status`, because every viewer - shows them. -- Decide in the PR whether the `safety-` e-stop broadcast becomes its own - socket or a second track on the driver's socket, and record why. -- The anonymous publish grant covers `demo/carla/*/.dial/*`. Whichever of - this and [moq-boy on sockets](/quest/m1/socket/boy.md) lands second removes +- The worker (`publish.py`) runs a Python endpoint per truck; `paths.ts` and + `truck.ts` dial it from an endpoint with the per-lease driver id. A new lease + is a new session, which matches today's lease binding. +- The `control` track becomes the driver's messages on a `track` session, one + control per group, so a late control never blocks a newer one; the driver + may read the truck's replies with a finite delay budget. The + driver's ack, sequence, and round-trip echo move from the shared `status` to + the truck's messages on that session. `driver` and `estop_by` stay in the + ordinary `status` broadcast, because every viewer shows them. +- The `safety-` e-stop broadcast becomes messages on the driver's session + or a second session of its own; decide in the PR and record why, such as + keeping the e-stop off the control session's queue. +- The anonymous grants let a driver publish its own endpoint and subscribe to + the truck endpoints. Whichever of this and + [moq-boy on sockets](/quest/m1/socket/boy.md) lands second removes `demo/viewer/**` from `demo/relay/prod.toml` and its documented - `moq auth serve` line. The worker's token keeps publish on `carla/**` - and swaps its subscribe scope from `viewer/carla/**` to `carla/*/.dial/*`, - because publish and subscribe are granted separately; say so in - `demo/carla/README.md`. + `moq auth serve` line. The worker's token keeps publish on `carla/**`, which + covers its endpoints, and swaps its subscribe scope from `viewer/carla/**` + to the driver endpoints; say so in `demo/carla/README.md`. - Update `demo/carla/README.md` (paths, lease) and `doc/bin/demo.md`. - Decided 2026-11-03 (maintainer, follow-up planning after the 2026-11-03 /quest-afk and /quest-complete runs): [moq-dev/moq#7678](https://github.com/moq-dev/moq/pull/7678) keys carla's Python driver lease on a local run counter, because moq-ffi's @@ -38,5 +42,5 @@ See the [epic](/quest/m1/socket/README.md) for the layout and the reply split. ## Required -- [Python sockets](/quest/m1/socket/py.md) - the truck's listener +- [Python sockets](/quest/m1/socket/py.md) - the truck's endpoint - [JS sockets](/quest/m1/socket/js.md) - the driver's dial diff --git a/quest/m1/socket/js.md b/quest/m1/socket/js.md index 5d7e930e55c..e9a70cc0596 100644 --- a/quest/m1/socket/js.md +++ b/quest/m1/socket/js.md @@ -2,17 +2,26 @@ ## Goal -`@moq/net` exports `socket` with the same `Listener`, `dial`, and `Socket` -surface as Rust, and a Rust dialer and a JS listener (and the reverse) open -and accept uni and bi streams through a relay. +`@moq/net` exports `socket` with the same `Endpoint`, `TcpStream`, and +`TrackStream` surface as Rust, and a Rust endpoint and a JS endpoint dial and +accept sessions in both protocols with each other through a relay, carrying +data both ways. ## Plan -Follow `doc/concept/socket.md` and mirror the -[Rust surface](/quest/m1/socket/rust.md), including its track-naming scheme. -Add the Rust/JS pair to `just test interop`, and document the export in -`doc/lib/js/net.md`. - -## Required - -- [Rust sockets](/quest/m1/socket/rust.md) - the reference surface and concept page +Follow `doc/concept/socket.md` and mirror the Rust surface +(`rs/moq-net/src/model/socket.rs`): an `Endpoint` announcing +`/.socket/`; `dialTcp(peer)` / `acceptTcp()` and +`dialTrack(peer)` / `acceptTrack()` in JS's casing, each landing in its +protocol's queue; an incoming dial to accept or reject; a `TcpStream` with a +group `producer` and `consumer` (one group each way: frames are writes, finish +is FIN, abort is RST, and a second group resets the session); and a +`TrackStream` with a track `producer` and a reliably subscribed `consumer`. +Both carry `peer` and `session`, and no message methods of their own. Session +tracks are `///` in both endpoints, served only on +request so no listing names them; an unknown or unaccepted protocol is reset +as unsupported. JS reads without fastest-reader pacing until +[JS fastest-reader pacing](/quest/m2/js-fastest-reader.md), so a JS `tcp` +reader that falls a group tail behind fails with an eviction error; say so in +the docs. Add the Rust/JS pairs to `just test interop`, and document the +export in `doc/lib/js/net.md`. diff --git a/quest/m1/socket/old-hop-dial.md b/quest/m1/socket/old-hop-dial.md index df58d3d1866..646a5884758 100644 --- a/quest/m1/socket/old-hop-dial.md +++ b/quest/m1/socket/old-hop-dial.md @@ -2,9 +2,12 @@ ## Goal -A socket dial through a hop older than lite-07 fails with `Unsupported` -instead of waiting forever. Today the older hop re-mints the answer's -instance, so the dialer never sees an answer it recognizes. +A socket dial or accept through a hop older than moq-lite-07 fails with +`Unsupported` instead of waiting forever. An endpoint's `.socket/` +broadcast is hidden, and a moq-lite hop before 07 has no hidden opt-in on its +announce requests, so it never carries the broadcast: a dial across one never +finds its peer's endpoint, and an acceptor reached by a dialer behind one waits +for the dialer's endpoint for as long as the dial stays open. ## Plan @@ -15,20 +18,26 @@ Decided 2026-11-03 (maintainer, follow-up planning after the 2026-11-03 /quest-a - m2 From [moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677), which -lands Rust sockets and documents the hang (accept already fails -`Unsupported` over an older hop). - -- One option from #7677: the dialer also watches `path/.accept/` and - fails `Unsupported` on an answer whose instance `is_minted()`. Weigh the - cost of a second announce interest per pending dial against alternatives, - and record the choice. +landed Rust sockets on one hidden endpoint broadcast. Its first plan here, an +answer instance check, belonged to the earlier `.accept/` design, which +used instances; the endpoint design uses none. + +Facts from #7677's head, over the shared relay harness +(`rs/moq-net/tests/integration/socket.rs`): a moq-lite-06 client's request for +`room/.socket/bob` fails `Unroutable` where `room/plain/bob` resolves, and a +`tcp` session times out for lite-06 to lite-06, lite-07 to lite-06, and +lite-03 to lite-03. With a lite-06 dialer and a lite-07 acceptor, the acceptor +hears the dial, and its accept then waits until the dialer gives up. + +- Find where an endpoint can learn that a hop on the path can't carry hidden + broadcasts, such as the session's negotiated version, or a relay answering + a hidden request it can't route with a distinct error, and fail the dial or + accept `Unsupported` there. Weigh one that sees only the first hop against + one that covers relay-to-relay hops, and record the choice. - Python inherits it through moq-ffi. If [JS sockets](/quest/m1/socket/js.md) - has landed, its dial gets the same check. -- Test through a pre-lite-07 hop: the dial fails promptly with + has landed, its dial and accept get the same check. +- Test through a pre-lite-07 hop: the dial and the accept fail promptly with `Unsupported`. -Public API: none (an existing error). Wire: none. - -## Required - -- [Rust sockets](/quest/m1/socket/rust.md) - the dial this bounds +Public API: none (an existing error). Wire: none, unless a relay's answer +needs one. diff --git a/quest/m1/socket/py.md b/quest/m1/socket/py.md index 7bd6904159d..efc7e42eb6c 100644 --- a/quest/m1/socket/py.md +++ b/quest/m1/socket/py.md @@ -2,17 +2,17 @@ ## Goal -moq-ffi exposes `Listener`, `dial`, and `Socket` beside origin, and -`py/moq-rs` wraps them, so a Python listener accepts a Rust dialer's socket -and the reverse, with uni and bi streams each way. +moq-ffi exposes the socket `Endpoint`, `TcpStream`, and `TrackStream` beside +origin, and `py/moq-rs` wraps them, so a Python endpoint accepts a Rust +endpoint's dial and the reverse, in both protocols, carrying data both ways. ## Plan -Bind the [Rust surface](/quest/m1/socket/rust.md) through moq-ffi the way -origin's request and announce APIs are bound today (`rs/moq-ffi/src/origin.rs`, +Bind the Rust surface (`moq_net::socket`): `Endpoint::new`, `dial_tcp`, +`accept_tcp`, `dial_track`, `accept_track`, an incoming dial's `accept` and +`reject`, and each stream's `producer`, `consumer`, `peer`, and `session` +(group handles for `TcpStream`, track handles for `TrackStream`), through +moq-ffi the way origin's request +and announce APIs are bound today (`rs/moq-ffi/src/origin.rs`, `py/moq-rs/moq/origin.py`). Add a Python/Rust interop test, and document the API wherever `py/moq-rs` documents origin. - -## Required - -- [Rust sockets](/quest/m1/socket/rust.md) - the surface this binds diff --git a/quest/m1/socket/rust.md b/quest/m1/socket/rust.md deleted file mode 100644 index dc382147ec4..00000000000 --- a/quest/m1/socket/rust.md +++ /dev/null @@ -1,39 +0,0 @@ -# [M] Rust sockets - -## Goal - -`moq_net::socket` lets a Rust application bind a `Listener` at a path and -`accept()` the sockets dialed to it, or `dial(origin, path, id)` one itself. -Either side of a `Socket` can `open_uni`, `open_bi`, `accept_uni`, and -`accept_bi` tracks, and learns when the socket ends. `doc/concept/socket.md` -specifies the convention, so other languages can implement it from the page. - -## Plan - -The [epic](/quest/m1/socket/README.md) records the decisions: the -`L/.dial/` and `L/.accept/` layout, a dialer-chosen id with a pinned -epoch, tracks as streams discovered through the Tracks stream, and lazy, -unbounded accept. - -- `Listener` watches `L/.dial/*` and queues each announced `(id, epoch)`. - `accept()` requests the dialer's broadcast pinned to its epoch and publishes - `L/.accept/`. A new epoch for a queued or live id replaces it. -- `dial` publishes `L/.dial/` and requests `L/.accept/`; it resolves - once the listener answers. Settle and record here whether a dial times out - or waits indefinitely for a listener. -- `Socket` holds the local broadcast producer and the remote consumer, and - exposes both for applications that use fixed track names (boy, CARLA) - rather than open/accept. -- A bi stream returns a track producer and consumer pair; a uni stream - returns one end. Settle and record the naming scheme that keeps the two - sides' track names from colliding. Each opened track takes a priority from - the caller. -- The socket ends with a typed error when either broadcast ends, its epoch - is replaced, or the session drops. -- Write `doc/concept/socket.md` (paths, epochs, tracks as streams, ending) - and link it from the concept index and `doc/lib/rs`. -- Tests on controlled time with no sleeps: dial and accept through a relay; - a bi and a uni stream each way; a dialer re-announcing with a new epoch - ends the old socket; an unaccepted dialer holds no subscription; a listener - restart ends its sockets; a dialer for another id cannot be accepted as - this one. diff --git a/quest/m2/announce-credit.md b/quest/m2/announce-credit.md index 180afc0e7bb..5a9b46aeca8 100644 --- a/quest/m2/announce-credit.md +++ b/quest/m2/announce-credit.md @@ -7,7 +7,7 @@ the wire: ANNOUNCE_REQUEST carries an initial route credit, the subscriber raises it later on its side of the stream, and the publisher never holds more live advertisements on the stream than the credit. A peer that floods distinct paths is held to the credit instead of closing the session at the 100k -announce cap, so a flood of `.dial/*` sockets cannot drop a cluster link or the +announce cap, so a flood of distinct paths cannot drop a cluster link or the other announcements it carries. Rust and JS implement it on the next unpublished lite version after lite-07. @@ -30,8 +30,8 @@ Decided 2026-11-01 with the maintainer (the follow-up to the budget. - Declined: per-route refusal (frees the subscriber but not the publisher, and the flood still crosses the wire once), and aggregating to a covering - prefix at the limit (loses the per-path inventory a socket `Listener` - needs). + prefix at the limit (loses the per-path inventory a reader watching many + broadcasts needs). - The publisher withholds routes beyond the credit, coalesced to the latest state per path as it already does, and sends them as an ANNOUNCE_END or a credit raise frees room. Which withheld route goes first is the diff --git a/quest/m2/mesh/README.md b/quest/m2/mesh/README.md index 993a27673cb..54f91ed6266 100644 --- a/quest/m2/mesh/README.md +++ b/quest/m2/mesh/README.md @@ -52,9 +52,10 @@ Decided 2026-10-25 with the maintainer (planning interview), still in force: reaching an ID is subscribing there. There is no separate directory. - The token is the binding: enrollment proves the device key, then the coordinator issues a grant to publish `mesh//**` and subscribe - `mesh/**`, plus publish on `mesh/*/*/.dial/` so the node can dial - another's [socket](/quest/m1/socket/README.md) under its own id (amended - 2026-11-01). Relays need no new check. Membership is the coordinator's + `mesh/**`, plus publish on its own [socket](/quest/m1/socket/README.md) + endpoint, `.socket/`, so it can dial and accept under its own id + (amended 2026-11-01, 2026-11-03 when sockets became endpoints, and + 2026-11-04 when the segment became `.socket`). Relays need no new check. Membership is the coordinator's token only; a Tailnet-Lock analog was declined. - Protection is per broadcast, with levels that nest: identity (the token only), authentication (per-frame Ed25519 signatures, the default), and diff --git a/quest/m2/mesh/endpoint.md b/quest/m2/mesh/endpoint.md index fbb6adbbc37..97073c5085e 100644 --- a/quest/m2/mesh/endpoint.md +++ b/quest/m2/mesh/endpoint.md @@ -15,17 +15,21 @@ Decided 2026-10-25 (maintainer): iroh-shaped, not drop-in compatible; tickets, `EndpointAddr`, and `Router` are out unless porting proves they are needed. - A service is the iroh ALPN analog: a name under `mesh//`. -- Decided 2026-11-01 (maintainer, socket planning interview): an `Endpoint` - wraps [sockets](/quest/m1/socket/rust.md). A service is a `Listener` at - `mesh//`, `connect` dials it with the caller's own endpoint ID, - and `accept` reads the remote endpoint ID from the `.dial/` segment, - which the grant pins. Extend `moq auth`'s mesh grant with publish on - `mesh/*/*/.dial/`, and update `doc/bin/relay/auth.md`. +- Decided 2026-11-01 (maintainer, socket planning interview), and updated + 2026-11-03 when sockets became endpoints and 2026-11-04 when the segment + became `.socket` with the protocol in the session name: an `Endpoint` wraps + a [socket](/doc/concept/socket.md) endpoint. Settle the layout here: such as + one socket endpoint per service at `mesh//.socket/`, or one per + node with the service named in the session. `connect` dials with the + caller's own endpoint ID, and `accept` reads the remote ID from the dial, + which the socket's answer check proves and the grant pins (publish + `.socket/` only). Update `moq auth`'s mesh grant and + `doc/bin/relay/auth.md` to match. - Streams are [byte streams](/quest/m2/stream/rust.md) at the protection level the service declares: identity reads `moq-net` directly, and authentication, the mesh default, reads through `moq_crypto::sign` against the - `PublicKey` parsed from the path it subscribes to: the listener's - `mesh//` for a dialer, and the `.dial/` segment for a listener. + `PublicKey` parsed from the peer's endpoint id, which names the endpoint + broadcast it subscribes to. - The secret key comes from `moq_crypto::sign::SecretKey::load_or_generate`. - A mesh daemon (Kixel's, outside this repository, decided 2026-11-02) is built on this API, so a process either embeds an `Endpoint` or talks to a @@ -34,5 +38,4 @@ Decided 2026-10-25 (maintainer): iroh-shaped, not drop-in compatible; tickets, ## Required -- [Rust sockets](/quest/m1/socket/rust.md) - the dial and accept this wraps - [Rust byte streams](/quest/m2/stream/rust.md) - the stream this API returns diff --git a/quest/m2/stream/README.md b/quest/m2/stream/README.md index e4862999940..62c725bcd44 100644 --- a/quest/m2/stream/README.md +++ b/quest/m2/stream/README.md @@ -17,9 +17,9 @@ fastest-reader pacing gives. Decided 2026-10-25 with the maintainer (mesh planning interview): - TCP-level proxying over groups, never IP over datagrams. -- A forwarded service is one [socket](/quest/m1/socket/README.md), and each - TCP connection is one bi track on it (amended 2026-11-01, below), one group - each way; each TCP `read()` becomes one frame, because a frame's length must +- A forwarded service is one [socket](/quest/m1/socket/README.md) endpoint, + and each TCP connection is one session on it (amended 2026-11-03, below), + one group each way; each TCP `read()` becomes one frame, because a frame's length must be known before it is written and a read has one. It coalesces whatever is buffered with no added timer (the local TCP socket's own Nagle or `TCP_NODELAY` still applies). @@ -39,13 +39,17 @@ Decided 2026-10-25 with the maintainer (mesh planning interview): identity, signing, and service names belong to the [mesh](/quest/m2/mesh/README.md), which builds on this. -Amended 2026-11-01 with the maintainer (socket planning interview): connection -setup is the [socket](/quest/m1/socket/README.md) epic's `Listener` and `dial`, -which settles the path layout (`L/.dial/`, `L/.accept/`). The earlier -one track per direction with a group per connection is replaced by one bi -track per connection. Each connection then gets its own priority, pacing, and -eviction, and a long-lived connection never sits in an old group that newer -ones crowd out of a track's window. +Amended 2026-11-01 with the maintainer (socket planning interview), +2026-11-03 when sockets became endpoints, and 2026-11-04 when the socket's +`tcp` protocol became one group each way: connection setup is the +[socket](/quest/m1/socket/README.md) epic's `Endpoint`, `dial_tcp`, and +`accept_tcp`, which settle the layout (`/.socket/`, a +`tcp///` track each way). Each TCP connection is one `tcp` +session, whose halves are a single group each way for the connection's life: +frames are writes, finishing is FIN, resetting is RST, and the writer awaits +`group::Producer::ready` for backpressure. Each session gets its own priority, +pacing, and eviction, and a long-lived connection never sits in an old group +that newer ones crowd out of a track's window. This epic owns the end-to-end proof: an unmodified TCP client and server (SSH and an HTTP bulk transfer) talk through a two-relay chain with loss injected on @@ -54,7 +58,6 @@ the window. ## Required -- [Rust sockets](/quest/m1/socket/rust.md) - the dial/accept primitive and tracks as streams - [Fastest-reader pacing](/quest/m1/unbounded-groups/fastest-reader.md) - relays push back instead of evicting a reader that keeps up - [Rust byte streams](/quest/m2/stream/rust.md) - a duplex `AsyncRead`/`AsyncWrite` stream, a listener and a dialer, and `moq` CLI forwarding - [JS byte streams](/quest/m2/stream/js.md) - the same surface in `@moq/net` diff --git a/quest/m2/stream/rust.md b/quest/m2/stream/rust.md index 1a7b7404a4a..7f001cb5dfe 100644 --- a/quest/m2/stream/rust.md +++ b/quest/m2/stream/rust.md @@ -2,22 +2,23 @@ ## Goal -A Rust application opens a duplex byte stream on a -[socket](/quest/m1/socket/rust.md) and gets an `AsyncRead + AsyncWrite` that -carries TCP semantics over one bi track: ordered, -lossless, FIN as half-close, reset with a code. A socket's listener accepts the streams -its dialers open. `moq` gains TCP forwarding on both ends so an +A Rust application opens a duplex byte stream as a +[socket](/doc/concept/socket.md) session and gets an `AsyncRead + AsyncWrite` +that carries TCP semantics over the session's two tracks: ordered, lossless, +FIN as half-close, reset with a code. A socket endpoint accepts the streams +other endpoints dial. `moq` gains TCP forwarding on both ends so an unmodified client and server talk through a relay. ## Plan -See the [epic](/quest/m2/stream/README.md) for the mapping: a socket per -forwarded service, a bi track per connection with one group each way, a frame -per TCP read, no fill handler. +See the [epic](/quest/m2/stream/README.md) for the mapping: a socket endpoint +per forwarded service, a session per connection with one group each way, a +frame per TCP read, no fill handler. -- Build on `moq_net::socket`: the path layout and connection naming are the - socket's (`open_bi` and `accept_bi`), so this quest adds only the - `AsyncRead`/`AsyncWrite` adapter and the TCP mapping. Place the adapter +- Build on `moq_net::socket`: the path layout, connection naming, FIN and RST + are the socket's (`Endpoint::dial_tcp` and `accept_tcp`, each yielding a + `TcpStream` of one group each way), so this quest adds only the + `AsyncRead`/`AsyncWrite` adapter over a `TcpStream` and the TCP mapping. Place the adapter beside the socket module unless it pulls in a dependency moq-net avoids; report the choice. - Each stream carries a subscription priority the caller sets; nothing is @@ -32,5 +33,4 @@ per TCP read, no fill handler. ## Required -- [Rust sockets](/quest/m1/socket/rust.md) - the socket each stream is a bi track on - [Fastest-reader pacing](/quest/m1/unbounded-groups/fastest-reader.md) - lossless delivery to a reader that keeps up diff --git a/rs/moq-net/CHANGELOG.md b/rs/moq-net/CHANGELOG.md index 456e1513be5..678445b1b56 100644 --- a/rs/moq-net/CHANGELOG.md +++ b/rs/moq-net/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `socket`: `Endpoint`, which announces a hidden `/.socket/` broadcast, dials other endpoints, and accepts their dials in two protocols: `tcp` sessions (`dial_tcp`, `accept_tcp`, `TcpStream`), one group each way as a reliable, ordered byte pipe, and `track` sessions (`dial_track`, `accept_track`, `TrackStream`), a track each way with one message per group; see `doc/concept/socket.md`. - `transport::SETUP_RECEIVE_WINDOW`, the connection credit a server offers before it admits a session: enough for the SETUP and one in-band AUTH. ## [0.3.10](https://github.com/moq-dev/moq/compare/moq-net-v0.3.9...moq-net-v0.3.10) - 2026-10-05 diff --git a/rs/moq-net/src/model/broadcast.rs b/rs/moq-net/src/model/broadcast.rs index a6345ce9c13..ed7306044f3 100644 --- a/rs/moq-net/src/model/broadcast.rs +++ b/rs/moq-net/src/model/broadcast.rs @@ -460,6 +460,15 @@ impl Producer { })) } + /// [`Self::reserve_track`] for a track no listing ever names, so only a reader who + /// already knows the name finds it, as a socket session's track. + pub(super) fn reserve_unlisted_track(&self, name: impl Into>) -> Result { + let mut state = self.state.lock(); + let request = track::Request::new(self.info.clone(), name.into()).with_stats(self.stats.clone()); + state.insert_track(request.weak())?; + Ok(request) + } + /// Create a track with a unique name using the given suffix. /// /// Uses [`Self::unique_name`]; minted names are never reused, even after closure. @@ -534,6 +543,11 @@ impl Producer { self.alive.close(); } + /// `Ready` once the broadcast ended, by [`Self::close`] or the last producer dropping. + pub(super) fn poll_closed(&self, waiter: &kio::Waiter) -> Poll<()> { + self.alive.token.poll_closed(waiter) + } + /// Whether any [`Consumer`] handle exists, which is not itself demand for a track. pub(super) fn is_held(&self) -> bool { self.alive.token.wanted() @@ -859,6 +873,10 @@ impl Requests { // Cache the served track so concurrent lookups coalesce onto it. If a live track already // holds the name (a publish raced the request), `insert` keeps it rather than shadowing it. let _ = state.cache_track(pending.weak()); + // Claimed after the lock is released, since claiming wakes the track's subscribers + // and an inline executor can re-enter the broadcast from that wake. A close landing + // in between rejects the request, as it would one still queued. + drop(state); // Attribute the served track to this broadcast's ingress scope (no-op untagged). Poll::Ready(Ok(pending.claim().with_stats(self.stats.clone()))) } diff --git a/rs/moq-net/src/model/mod.rs b/rs/moq-net/src/model/mod.rs index a4b29958b6f..3f46801d4d9 100644 --- a/rs/moq-net/src/model/mod.rs +++ b/rs/moq-net/src/model/mod.rs @@ -4,6 +4,7 @@ pub mod cache; pub mod datagram; pub mod frame; pub mod group; +pub mod socket; pub mod track; // The origin + announce subsystem shares one implementation (a route table). diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index af40f4365d3..26eca71a138 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -2087,6 +2087,12 @@ pub struct Producer { } impl Producer { + /// A handle that queues work on this origin's [`Driver`] without keeping the origin + /// alive: work queued once the driver is gone is dropped. + pub(super) fn tasks(&self) -> crate::util::TasksWeak { + self.tasks.downgrade() + } + /// Build a producer from a [`Config`] (identity + cache pool) with no scoped /// prefix and no pre-existing broadcasts, paired with the [`Driver`] that runs /// the origin's lifecycle work. diff --git a/rs/moq-net/src/model/socket.rs b/rs/moq-net/src/model/socket.rs new file mode 100644 index 00000000000..801d1d0d42a --- /dev/null +++ b/rs/moq-net/src/model/socket.rs @@ -0,0 +1,1349 @@ +//! Point-to-point sessions over an origin, the way TCP connects two hosts. +//! +//! An [`Endpoint`] announces one hidden broadcast, `/.socket/`, used only to +//! route dials to it. From it, an endpoint dials other endpoints and accepts their dials. +//! Each connection is a session: two tracks, one in each endpoint's broadcast, both named +//! `///`. Sessions are never announced or listed. +//! +//! A session speaks one of two protocols: +//! +//! - `tcp`: one reliable, ordered byte pipe each way, as TCP gives. Each half is a single +//! group. Dial with [`Endpoint::dial_tcp`], accept with [`Endpoint::accept_tcp`], and +//! read and write a [`TcpStream`]. +//! - `track`: each half is a whole track, so messages are groups, with the track's own +//! priority, delay budget and order. Dial with [`Endpoint::dial_track`], accept with +//! [`Endpoint::accept_track`], and read and write a [`TrackStream`]. +//! +//! A dialer `A` picks a random session id `S` and subscribes to `/.socket/` for +//! the track `/A/B/S`. Endpoint `B` surfaces that request as an [`Incoming`] dial +//! from `A` in its protocol's queue. To accept it, `B` subscribes to `/.socket/` +//! for the same track name, and `A` answers only a session it dialed, so the exchange +//! checks that `A` really asked `B`: a dial forged in `A`'s name is reset. Naming both ends +//! binds a session to them, so an endpoint `A` dialed cannot replay the session to a third +//! endpoint `C`: `C` resets a name for another acceptor, and `A` never dialed +//! `/A/C/S`. Each side then writes its own half and reads the peer's. +//! +//! See `doc/concept/socket.md` for the convention other languages follow. + +use std::collections::VecDeque; +use std::marker::PhantomData; +use std::sync::Arc; +use std::task::{Poll, ready}; + +use crate::{Error, Path}; + +use super::WeakEntry; +use super::origin::{self, Route}; +use super::track::{self, Delay, Order, Subscription}; +use super::{frame, group}; + +/// The segment every endpoint's broadcast sits under. +const SOCKET: &str = ".socket"; + +/// Refuse an id that is not one plain path segment: empty, or holding a `/` or a `\`. +fn check_id(id: &str) -> Result<(), Error> { + if id.is_empty() || id.contains(['/', '\\']) { + return Err(crate::InvalidPattern::InvalidSegment(id.to_string()).into()); + } + Ok(()) +} + +/// `/.socket/`, the broadcast of the endpoint `id`. +fn endpoint_path(prefix: &Path, id: &str) -> Result { + check_id(id)?; + Ok(prefix.join(&Path::from_static(SOCKET))?.join(&Path::new(id)?)?) +} + +/// A fresh session id: 128 random bits as hex, so nobody can guess a session's tracks. +fn session_id() -> Arc { + use rand::RngExt; + format!("{:032x}", rand::rng().random::()).into() +} + +/// The track name of a session: `///`, the same in +/// both endpoints. +fn track_name(protocol: &str, dialer: &str, acceptor: &str, session: &str) -> String { + format!("{protocol}/{dialer}/{acceptor}/{session}") +} + +/// A session's track name, split. +#[derive(Debug, PartialEq, Eq)] +struct Name<'a> { + protocol: &'a str, + dialer: &'a str, + acceptor: &'a str, + session: &'a str, +} + +/// Split a session's track name, or `None` for a name outside the scheme. +fn parse(name: &str) -> Option> { + let mut segments = name.split('/'); + let [protocol, dialer, acceptor, session] = std::array::from_fn(|_| segments.next().unwrap_or_default()); + if segments.next().is_some() { + return None; + } + for segment in [protocol, dialer, acceptor, session] { + check_id(segment).ok()?; + } + Some(Name { + protocol, + dialer, + acceptor, + session, + }) +} + +/// A `track` session half's default: every message stays until read, rather than only +/// the newest as for an authored track. +fn messages() -> track::Info { + track::Info::default().with_max_age(None) +} + +/// What a session's reader asks for: every group, oldest first, however late it is. +fn reliable() -> Subscription { + Subscription::default() + .with_order(Order::Ascending) + .with_delay(Delay::default().with_max(None)) +} + +mod sealed { + use super::*; + + /// The protocols an endpoint speaks. + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + pub enum Protocol { + Tcp, + Track, + } + + /// What an endpoint's handles share. + pub struct Shared { + pub(super) id: Arc, + pub(super) prefix: Path, + /// Where the peers' endpoints are requested. + pub(super) origin: origin::Consumer, + /// Where the `tcp` sessions' pumps run: the origin's driver. + pub(super) tasks: crate::util::TasksWeak, + /// This endpoint's broadcast. + pub(super) publication: origin::Publication, + pub(super) state: kio::Lock, + } + + /// A session's two halves, once the handshake is done. + pub struct Parts { + pub shared: Arc, + /// This side's half, served to the peer. + pub producer: track::Producer, + /// The peer's half, subscribed reliably and in order. + pub subscriber: track::Subscriber, + pub peer: Arc, + pub session: Arc, + } + + /// What a protocol does with a session's halves. + pub trait Open: Sized { + /// The protocol's segment, the first of a session's track name. + const PROTOCOL: Protocol; + + /// Make the stream an application reads and writes. + fn open(parts: Parts) -> Self; + } +} + +use sealed::{Parts, Protocol, Shared}; + +impl Protocol { + fn as_str(self) -> &'static str { + match self { + Self::Tcp => "tcp", + Self::Track => "track", + } + } + + fn parse(protocol: &str) -> Option { + match protocol { + "tcp" => Some(Self::Tcp), + "track" => Some(Self::Track), + _ => None, + } + } +} + +/// A session's stream: a [`TcpStream`] or a [`TrackStream`]. Names the protocol a dial, +/// an incoming dial, or an accept is for. +pub trait Stream: sealed::Open {} + +/// A `tcp` session: one reliable, ordered byte pipe each way, as TCP gives. +/// +/// Each half is the session track's only group, which is one QUIC stream on every hop, +/// so it is reliable, ordered and flow controlled like a TCP direction. A frame is a +/// write. Finishing [`Self::producer`] is FIN, and aborting or dropping it is RST; the +/// peer reads the same from [`Self::consumer`]. Reading ends once the peer finished, and +/// fails with its reset. A second group on a peer's half is a protocol error that resets +/// the session. +/// +/// Await [`group::Producer::poll_ready`] (or `ready`) before each write: it waits until the peer reads, so +/// the writer runs at most a window ahead of the reader and nothing is lost. A writer +/// that never waits sheds the front of its half, and a reader that falls behind it fails +/// with [`Error::Evicted`]. +/// +/// Dropping [`Self::consumer`] before the peer finished resets the session, since nobody +/// reads what the peer writes. The endpoint closing resets it too. +pub struct TcpStream { + /// This side's half: the frames written here reach the peer in order. + pub producer: group::Producer, + /// The peer's half: its frames, in order. + pub consumer: group::Consumer, + peer: Arc, + session: Arc, +} + +impl TcpStream { + /// The peer endpoint's id. + pub fn peer(&self) -> &str { + &self.peer + } + + /// The session's id, which the dialer chose. + pub fn session(&self) -> &str { + &self.session + } +} + +impl Stream for TcpStream {} + +impl sealed::Open for TcpStream { + const PROTOCOL: Protocol = Protocol::Tcp; + + fn open(parts: Parts) -> Self { + let Parts { + shared, + producer: track, + subscriber, + peer, + session, + } = parts; + let (producer, ours) = match track.create_group(group::Info { sequence: 0 }) { + Ok(group) => { + let demand = group.demand(); + (group, Some(demand)) + } + // The track already ended, such as by the endpoint closing: hand out a half + // that reports it. + Err(err) => { + let group = + group::Producer::new(group::Info { sequence: 0 }, track::Info::default(), Default::default()); + let _ = group.handle().abort(err); + (group, None) + } + }; + // The application reads a copy of the peer's half that the pump fills, so the + // session can end it: the peer's own copy is the session's, not this side's. + let local = group::Producer::new(group::Info { sequence: 0 }, track::Info::default(), Default::default()); + let consumer = local.consume(); + let pump = Pump { + shared: shared.clone(), + track: Some(track), + ours, + subscriber: Some(subscriber), + received: false, + remote: None, + local_handle: local.handle(), + local: Some(local), + pending: None, + }; + shared.tasks.push(kio::Pending::new(pump)); + Self { + producer, + consumer, + peer, + session, + } + } +} + +/// A `track` session: each half is a whole track, and each message a group. +/// +/// [`Self::consumer`] starts subscribed reliably and in order: oldest first, with no +/// delay budget, so a late group is waited for rather than skipped. Each hop's cache +/// still reclaims a group nobody read within its idle window (30 s by default), or under +/// memory pressure, except the newest, so a reader that stalls longer can miss groups. +/// Update its subscription, such as with a finite +/// [`Delay`] or [`Order::Descending`], to make it lossy or newest first, and read it in +/// group order with [`track::Subscriber::ordered`]. Groups are written with +/// [`track::Producer::append_group`], each with its own priority on the wire. +/// +/// Finishing [`Self::producer`] closes the session for the peer, and aborting it is an +/// error. The endpoint closing aborts it too. When the peer's half ends with an error, +/// abort this side's half as well, and drop the stream, so both halves end. +pub struct TrackStream { + /// This side's half: the session's track in this endpoint's broadcast. + pub producer: track::Producer, + /// The peer's half: the session's track in the peer's broadcast, subscribed reliably + /// and in order until updated. + pub consumer: track::Subscriber, + peer: Arc, + session: Arc, +} + +impl TrackStream { + /// The peer endpoint's id. + pub fn peer(&self) -> &str { + &self.peer + } + + /// The session's id, which the dialer chose. + pub fn session(&self) -> &str { + &self.session + } +} + +impl Stream for TrackStream {} + +impl sealed::Open for TrackStream { + const PROTOCOL: Protocol = Protocol::Track; + + fn open(parts: Parts) -> Self { + Self { + producer: parts.producer, + consumer: parts.subscriber, + peer: parts.peer, + session: parts.session, + } + } +} + +/// Dials from peers in one protocol, waiting to be accepted. +#[derive(Default)] +struct Queue { + /// Whether the protocol is accepted: from its first accept on. + listening: bool, + dials: VecDeque, + /// The accepts waiting for a dial. + waiting: kio::WaiterList, +} + +/// An endpoint's handler for the requests to its broadcast: the dials peers send. +/// +/// A poller takes it out to poll it with no lock held, since taking a request wakes the +/// dialer's subscription, and an inline executor can re-enter the endpoint from that wake. +enum Handler { + /// Not made yet, until the first accept, so the endpoint answers a request for a + /// session it does not hold with [`Error::NotFound`] at once. + None, + /// Waiting for a poller. + Idle(super::broadcast::Requests), + /// Taken by a poller, which puts it back and wakes the pollers that found it taken, + /// so each polls it, and is woken by it, in turn. + Taken(kio::WaiterList), +} + +/// What the endpoint's pollers share. +struct State { + /// Where the dials peers send arrive. + handler: Handler, + tcp: Queue, + track: Queue, + /// This side's half of every session handed out, aborted when the endpoint closes. + streams: Vec, + /// Why the endpoint stopped, once it did. + end: Option, +} + +impl State { + fn queue(&mut self, protocol: Protocol) -> &mut Queue { + match protocol { + Protocol::Tcp => &mut self.tcp, + Protocol::Track => &mut self.track, + } + } + + /// Keep `producer`'s track, so the endpoint closing ends it. + fn hold(&mut self, producer: &track::Producer) { + self.streams.retain(|stream| !stream.is_closed()); + self.streams.push(producer.weak()); + } +} + +impl Shared { + /// Queue every dial on hand and take the next in `protocol`, registering `waiter` for + /// more, and fail with the endpoint's end once it stopped. + /// + /// A dial of this endpoint's reserves its own track, so a request reaching the handler + /// with this endpoint as the dialer names a session it never dialed, or one it gave up, + /// such as an accept of a dial forged or replayed in this endpoint's name: it is reset + /// with [`Error::NotFound`], which fails that accept. So is a request naming another + /// acceptor, such as a session replayed from the endpoint it was dialed to. A request in + /// a protocol nobody accepts is reset with [`Error::Unsupported`]. + /// + /// The handler is polled with no lock held (see [`Handler`]), and resets and wakes go + /// out after the lock is released. + fn poll_dial(&self, protocol: Protocol, waiter: &kio::Waiter) -> Poll> { + let mut requests = { + let mut state = self.state.lock(); + state.queue(protocol).listening = true; + if state.end.is_some() { + None + } else if let Handler::Taken(waiting) = &mut state.handler { + waiting.register(waiter); + None + } else { + match std::mem::replace(&mut state.handler, Handler::Taken(kio::WaiterList::new())) { + Handler::Idle(requests) => Some(requests), + Handler::None => Some(self.publication.requests()), + Handler::Taken(_) => unreachable!("checked above"), + } + } + }; + let mut taken = Vec::new(); + let mut ended = None; + if let Some(requests) = &mut requests { + loop { + match requests.poll_requested_track(waiter) { + Poll::Pending => break, + Poll::Ready(Ok(request)) => taken.push(request), + Poll::Ready(Err(err)) => { + ended = Some(err); + break; + } + } + } + } + + let mut resets = Vec::new(); + let mut wakes = Vec::new(); + let mut retired = None; + let mut state = self.state.lock(); + if let Some(err) = ended { + state.end.get_or_insert(err); + } + // Put the handler back, unless the endpoint ended meanwhile: then it is dropped + // after the lock is released. + if let Some(handler) = requests.take() { + let next = match state.end { + None => Handler::Idle(handler), + Some(_) => { + retired = Some(handler); + Handler::None + } + }; + if let Handler::Taken(mut waiting) = std::mem::replace(&mut state.handler, next) { + wakes.push(waiting.take()); + } + } + // Dials whose dialer gave up leave both queues on every poll, which each new dial + // triggers, so a queue holds no more than the dials still open, even in a protocol + // the application stopped accepting. + for protocol in [Protocol::Tcp, Protocol::Track] { + let queue = state.queue(protocol); + if queue.dials.iter().any(|request| !request.demand().is_wanted()) { + let (open, abandoned): (VecDeque<_>, VecDeque<_>) = std::mem::take(&mut queue.dials) + .into_iter() + .partition(|request| request.demand().is_wanted()); + queue.dials = open; + resets.extend(abandoned.into_iter().map(|request| (request, Error::Cancel))); + } + } + for request in taken { + // `None` for a name naming no dial to this endpoint: outside the scheme, this + // endpoint's own, or for another acceptor. + let target = match parse(request.name()) { + Some(name) if name.acceptor == &*self.id && name.dialer != &*self.id => { + Some(Protocol::parse(name.protocol)) + } + _ => None, + }; + match target { + // Taken while the endpoint closed: reset like the dials it queued. + _ if state.end.is_some() => resets.push((request, Error::Cancel)), + Some(Some(protocol)) if state.queue(protocol).listening => { + let queue = state.queue(protocol); + queue.dials.push_back(request); + wakes.push(queue.waiting.take()); + } + Some(_) => resets.push((request, Error::Unsupported)), + None => resets.push((request, Error::NotFound)), + } + } + let next = match &state.end { + Some(err) => Poll::Ready(Err(err.clone())), + None => { + let queue = state.queue(protocol); + match queue.dials.pop_front() { + Some(request) => Poll::Ready(Ok(request)), + None => { + queue.waiting.register(waiter); + Poll::Pending + } + } + } + }; + drop(state); + if let Some(retired) = retired { + // Closed first, as `close` may not have yet: dropping the last handler rejects + // the dials still queued with `Dropped`, rather than the close's `Unroutable`. + self.publication.close(); + drop(retired); + } + for mut wake in wakes { + wake.wake(); + } + for (request, err) in resets { + // Not the name: a session's name is its secret. + tracing::debug!(%err, "reset a request for no session"); + request.reject(err); + } + next + } + + /// Poll for the endpoint's end, registering `waiter` for it. + fn poll_end(&self, waiter: &kio::Waiter) -> Result<(), Error> { + if let Some(err) = &self.state.lock().end { + return Err(err.clone()); + } + match self.publication.poll_closed(waiter) { + Poll::Ready(()) => Err(Error::Closed), + Poll::Pending => Ok(()), + } + } + + fn close(&self) { + let mut guard = self.state.lock(); + let state = &mut *guard; + state.end.get_or_insert(Error::Closed); + let mut wakes = Vec::new(); + // A poller holding the handler drops it once it sees the end. + let requests = match std::mem::replace(&mut state.handler, Handler::None) { + Handler::Idle(requests) => Some(requests), + Handler::Taken(mut waiting) => { + wakes.push(waiting.take()); + None + } + Handler::None => None, + }; + let mut queued = Vec::new(); + for queue in [&mut state.tcp, &mut state.track] { + queued.extend(queue.dials.drain(..)); + wakes.push(queue.waiting.take()); + } + let streams = std::mem::take(&mut state.streams); + drop(guard); + + self.publication.close(); + // Reset outside the lock. + for stream in streams { + stream.abort(Error::Cancel); + } + for request in queued { + request.reject(Error::Cancel); + } + for mut wake in wakes { + wake.wake(); + } + drop(requests); + } +} + +/// A socket endpoint: announces `/.socket/`, dials other endpoints, and +/// accepts their dials. +/// +/// It accepts dials in a protocol from that protocol's first accept on, like a TCP +/// listener: before that, a dial in it is reset with [`Error::Unsupported`], and after it, +/// dials queue until accepted. Until the endpoint's first accept in any protocol, a dial +/// to it is reset with [`Error::NotFound`], as for any track nobody serves. +/// +/// Dropping it, or [`Self::close`], retracts the broadcast, resets the dials waiting to be +/// accepted, and resets every session it handed out: a [`TcpStream`] entirely, and this +/// side's half of a [`TrackStream`]. The peer's half of a [`TrackStream`] is read through +/// its [`TrackStream::consumer`], which the application owns: drop the streams too, which +/// releases the subscriptions. +/// +/// `tcp` sessions run a task on the origin's driver, which must keep running. +pub struct Endpoint { + shared: Arc, +} + +impl Endpoint { + /// Announce the endpoint `id` under `prefix` on `origin`, which also reaches the peers. + /// + /// `id` names the endpoint: a well-known name for a server, or a random one for a + /// client. It must be one path segment. Fails like + /// [`origin::Producer::create_broadcast`] when `origin` may not publish + /// `/.socket/`. + pub fn new( + origin: &origin::Producer, + prefix: impl TryInto>, + id: &str, + ) -> Result { + let prefix = crate::path::convert(prefix)?; + let publication = origin.create_broadcast(endpoint_path(&prefix, id)?)?; + publication.announce(Route::default())?; + Ok(Self { + shared: Arc::new(Shared { + id: id.into(), + prefix, + origin: origin.consume(), + tasks: origin.tasks(), + publication, + state: kio::Lock::new(State { + handler: Handler::None, + tcp: Queue::default(), + track: Queue::default(), + streams: Vec::new(), + end: None, + }), + }), + }) + } + + /// This endpoint's id. + pub fn id(&self) -> &str { + &self.shared.id + } + + /// The prefix the endpoint's broadcast sits under, and where it finds its peers. + pub fn prefix(&self) -> &Path { + &self.shared.prefix + } + + /// Dial the endpoint `peer` in a new `tcp` session, resolving once the peer accepts it. + /// + /// Waits for as long as the peer takes to accept: drop the returned future to give up, + /// which resets the session. That includes waiting for the peer's endpoint to be + /// announced. Fails with the peer's reset when it rejects the dial, or does not accept + /// `tcp` sessions, and with [`Error::Unsupported`] for a dial to this endpoint itself. + pub fn dial_tcp(&self, peer: &str) -> Result>, Error> { + self.dial(peer, None) + } + + /// Dial the endpoint `peer` in a new `track` session; see [`Self::dial_tcp`]. + /// + /// `info` describes this side's half; `None` keeps every message until read, with no + /// max age. + pub fn dial_track( + &self, + peer: &str, + info: impl Into>, + ) -> Result>, Error> { + self.dial(peer, Some(info.into().unwrap_or_else(messages))) + } + + fn dial(&self, peer: &str, info: Option) -> Result>, Error> { + let shared = &self.shared; + let path = endpoint_path(&shared.prefix, peer)?; + if peer == &*shared.id { + return Err(Error::Unsupported); + } + if let Some(err) = &shared.state.lock().end { + return Err(err.clone()); + } + let session = session_id(); + let name = track_name(S::PROTOCOL.as_str(), &shared.id, peer, &session); + // This side's half, waiting for the peer to ask for it, which only the peer can: + // nobody else knows the session. + let answer = shared.publication.reserve_unlisted_track(name.clone())?; + // Followed rather than requested, so a peer not announced yet is waited for. + let resolving = Box::new(shared.origin.follow(path, None)?); + Ok(kio::Pending::new(Dialing { + shared: shared.clone(), + peer: peer.into(), + session, + name, + info, + demand: answer.demand(), + answer: Some(Answer::Reserved(answer)), + step: Step::Resolving(resolving), + stream: PhantomData, + })) + } + + fn poll_accept(&self, waiter: &kio::Waiter) -> Poll, Error>> { + let shared = &self.shared; + let request = ready!(shared.poll_dial(S::PROTOCOL, waiter))?; + let name = parse(request.name()).expect("sorted as a dial"); + Poll::Ready(Ok(Incoming { + shared: shared.clone(), + peer: name.dialer.into(), + session: name.session.into(), + request: Some(request), + stream: PhantomData, + })) + } + + /// Poll for the next `tcp` dial from a peer; fails with [`Error::Closed`] once the + /// endpoint closed. + /// + /// The first call starts accepting `tcp` dials. They queue until taken, or until their + /// dialer gives up, so keep accepting, and accept or reject each one. + pub fn poll_accept_tcp(&self, waiter: &kio::Waiter) -> Poll, Error>> { + self.poll_accept(waiter) + } + + /// Wait for the next `tcp` dial from a peer; see [`Self::poll_accept_tcp`]. + #[cfg(feature = "async")] + pub async fn accept_tcp(&self) -> Result, Error> { + kio::wait(|waiter| self.poll_accept_tcp(waiter)).await + } + + /// Poll for the next `track` dial from a peer; see [`Self::poll_accept_tcp`]. + pub fn poll_accept_track(&self, waiter: &kio::Waiter) -> Poll, Error>> { + self.poll_accept(waiter) + } + + /// Wait for the next `track` dial from a peer; see [`Self::poll_accept_tcp`]. + #[cfg(feature = "async")] + pub async fn accept_track(&self) -> Result, Error> { + kio::wait(|waiter| self.poll_accept_track(waiter)).await + } + + /// Close the endpoint; see [`Endpoint`]. + pub fn close(&self) { + self.shared.close(); + } +} + +impl Drop for Endpoint { + fn drop(&mut self) { + self.shared.close(); + } +} + +/// A dial from a peer, from [`Endpoint::accept_tcp`] or [`Endpoint::accept_track`]: accept +/// or reject it. +/// +/// Dropping it rejects it with [`Error::Cancel`]. +pub struct Incoming { + shared: Arc, + peer: Arc, + session: Arc, + /// The dialer's request for this side's half; taken by an accept or a reject. + request: Option, + stream: PhantomData S>, +} + +impl Incoming { + /// The id the dialer claims. An accept checks it: the dial connects only if that + /// endpoint really dialed this session to this endpoint. + pub fn peer(&self) -> &str { + &self.peer + } + + /// The session's id, which the dialer chose. + pub fn session(&self) -> &str { + &self.session + } + + /// Reject the dial, resetting the dialer's request with [`Error::NotFound`], as for a + /// session that does not exist. + pub fn reject(mut self) { + if let Some(request) = self.request.take() { + request.reject(Error::NotFound); + } + } + + fn accept_with(mut self, info: Option) -> kio::Pending> { + let shared = self.shared.clone(); + let (peer, session) = (self.peer.clone(), self.session.clone()); + let request = self.request.take().expect("taken only by accept, reject or drop"); + // Followed rather than requested: the dialer announced its endpoint before it + // dialed, but a relay can deliver the dial first. A dial claiming an endpoint + // nobody announces waits for as long as its dialer holds it open. + let step = match endpoint_path(&shared.prefix, &peer).and_then(|path| shared.origin.follow(path, None)) { + Ok(following) => Step::Resolving(Box::new(following)), + Err(err) => Step::Failed(err), + }; + kio::Pending::new(Accepting { + shared: shared.clone(), + name: track_name(S::PROTOCOL.as_str(), &peer, &shared.id, &session), + peer, + session, + info, + demand: request.demand(), + request: Some(request), + step, + stream: PhantomData, + }) + } +} + +impl Incoming { + /// Accept the dial, resolving once the dialer answers, which proves it dialed. + /// + /// Nothing is written before the dialer answers. + pub fn accept(self) -> kio::Pending> { + self.accept_with(None) + } +} + +impl Incoming { + /// Accept the dial, resolving once the dialer answers, which proves it dialed. + /// + /// `info` describes this side's half, as for [`Endpoint::dial_track`]. Nothing is + /// written before the dialer answers. + pub fn accept(self, info: impl Into>) -> kio::Pending> { + self.accept_with(Some(info.into().unwrap_or_else(messages))) + } +} + +impl Drop for Incoming { + fn drop(&mut self) { + if let Some(request) = self.request.take() { + request.reject(Error::Cancel); + } + } +} + +/// How far a dial or an accept got in reading the peer's half. +enum Step { + /// Waiting for the peer's endpoint broadcast to be announced. + Resolving(Box), + /// Subscribed to the peer's half, waiting for the peer to serve it. + Subscribing(track::Subscribing), + /// The peer serves its half. + Subscribed(Box), + /// It failed before it started. + Failed(Error), + /// Handed out, or failed. + Done, +} + +impl Step { + /// Advance until the peer serves its half, subscribing to `track` in its endpoint. + fn poll(&mut self, waiter: &kio::Waiter, track: &str) -> Poll> { + loop { + match self { + Self::Resolving(following) => { + let broadcast = ready!(following.poll_available(waiter))?; + let subscribing = broadcast.track(track)?.subscribe(reliable()).into_inner(); + *self = Self::Subscribing(subscribing); + } + Self::Subscribing(subscribing) => { + let subscriber = ready!(subscribing.poll_ok(waiter))?; + *self = Self::Subscribed(Box::new(subscriber)); + } + Self::Subscribed(_) => return Poll::Ready(Ok(())), + Self::Failed(err) => return Poll::Ready(Err(err.clone())), + Self::Done => return Poll::Ready(Err(Error::Closed)), + } + } + } + + fn take_subscriber(&mut self) -> track::Subscriber { + match std::mem::replace(self, Self::Done) { + Self::Subscribed(subscriber) => *subscriber, + _ => unreachable!("taken once subscribed"), + } + } +} + +/// Hand out a session's stream once its handshake is done, unless the endpoint stopped, +/// in which case this side's half is aborted, outside the lock. +fn open( + shared: &Arc, + producer: track::Producer, + step: &mut Step, + peer: &Arc, + session: &Arc, +) -> Result { + let mut state = shared.state.lock(); + if let Some(err) = state.end.clone() { + drop(state); + let _ = producer.abort(err.clone()); + return Err(err); + } + state.hold(&producer); + drop(state); + Ok(S::open(Parts { + shared: shared.clone(), + producer, + subscriber: step.take_subscriber(), + peer: peer.clone(), + session: session.clone(), + })) +} + +/// This side's half of a dial. +enum Answer { + /// Reserved, until the peer asks for it. + Reserved(track::Request), + /// Served, once the peer asked. + Served(track::Producer), +} + +impl Answer { + /// Reset the half for the peer with `err`. + fn reset(self, err: Error) { + match self { + Self::Reserved(request) => request.reject(err), + Self::Served(producer) => { + let _ = producer.abort(err); + } + } + } +} + +/// A dial, from [`Endpoint::dial_tcp`] or [`Endpoint::dial_track`]: resolves once the peer +/// accepts. +/// +/// Dropping it gives up, resetting the session. +pub struct Dialing { + shared: Arc, + peer: Arc, + session: Arc, + /// The session's track name. + name: String, + info: Option, + /// Whether the peer asked for this side's half, which it does on accept. + demand: track::Demand, + answer: Option, + step: Step, + stream: PhantomData S>, +} + +impl Dialing { + /// The session's id. + pub fn session(&self) -> &str { + &self.session + } + + /// Poll for the peer to accept; see [`Endpoint::dial_tcp`]. + pub fn poll_ok(&mut self, waiter: &kio::Waiter) -> Poll> { + let result = self.poll_inner(waiter); + if let Poll::Ready(Err(err)) = &result { + if let Some(answer) = self.answer.take() { + answer.reset(err.clone()); + } + self.step = Step::Done; + } + result + } + + fn poll_inner(&mut self, waiter: &kio::Waiter) -> Poll> { + self.shared.poll_end(waiter)?; + + // The peer asks for this side's half when it accepts: serve it, which proves to the + // peer that this endpoint dialed. + if let Some(Answer::Reserved(_)) = &self.answer { + // Polled until it parks, so the next change wakes this. + while let Poll::Ready(changed) = self.demand.poll_wanted_changed(waiter) { + // Ends once the reservation does: the endpoint closed. + changed?; + } + if self.demand.wanted().is_some() + && let Some(Answer::Reserved(request)) = self.answer.take() + { + self.answer = Some(Answer::Served(request.accept(self.info.take()))); + } + } + + ready!(self.step.poll(waiter, &self.name))?; + // The peer serves its half only after asking for this side's, so a conforming peer + // asked already; wait for its request regardless. + let Some(Answer::Served(_)) = &self.answer else { + return Poll::Pending; + }; + let Some(Answer::Served(producer)) = self.answer.take() else { + unreachable!("checked above"); + }; + Poll::Ready(open(&self.shared, producer, &mut self.step, &self.peer, &self.session)) + } +} + +impl kio::Task for Dialing { + type Output = Result; + + fn poll(&mut self, waiter: &kio::Waiter) -> Poll { + self.poll_ok(waiter) + } +} + +impl Drop for Dialing { + fn drop(&mut self) { + // Given up: reset the session for the peer. + if let Some(answer) = self.answer.take() { + answer.reset(Error::Cancel); + } + } +} + +/// An accept, from an [`Incoming`] dial: resolves once the dialer answers. +pub struct Accepting { + shared: Arc, + peer: Arc, + session: Arc, + /// The session's track name. + name: String, + info: Option, + /// The dialer's request for this side's half, served once the dialer answers. + request: Option, + /// Whether the dialer still subscribes to this side's half: the dial stays open while + /// it does. + demand: track::Demand, + step: Step, + stream: PhantomData S>, +} + +impl Accepting { + /// Poll for the dialer to answer, which proves it dialed. + /// + /// Waits for the dialer's endpoint to be announced for as long as the dial stays open, + /// and fails with [`Error::Cancel`] once the dialer gives up. Fails with the dialer's + /// reset when it never dialed this session, such as a dial forged in its name, and + /// resets the dialer's request with the same error. + pub fn poll_ok(&mut self, waiter: &kio::Waiter) -> Poll> { + let result = self.poll_inner(waiter); + if let Poll::Ready(Err(err)) = &result { + self.step = Step::Done; + if let Some(request) = self.request.take() { + request.reject(err.clone()); + } + } + result + } + + fn poll_inner(&mut self, waiter: &kio::Waiter) -> Poll> { + self.shared.poll_end(waiter)?; + if self.demand.poll_state(waiter) != track::DemandState::Active { + return Poll::Ready(Err(Error::Cancel)); + } + ready!(self.step.poll(waiter, &self.name))?; + + // Accepted outside the lock, since it wakes the dialer. + let request = self.request.take().ok_or(Error::Closed)?; + let producer = request.accept(self.info.take()); + Poll::Ready(open(&self.shared, producer, &mut self.step, &self.peer, &self.session)) + } +} + +impl kio::Task for Accepting { + type Output = Result; + + fn poll(&mut self, waiter: &kio::Waiter) -> Poll { + self.poll_ok(waiter) + } +} + +impl Drop for Accepting { + fn drop(&mut self) { + // Given up: reset the dialer's request. + if let Some(request) = self.request.take() { + request.reject(Error::Cancel); + } + } +} + +/// A `tcp` session's task on the origin's driver. +/// +/// It copies the peer's half into the group the application reads, a frame at a time and +/// paced by that reader, so backpressure reaches the peer. It ends this side's half the +/// way the application ended its group, and resets the session on a second group from the +/// peer, the application dropping its reader, or the endpoint closing. +struct Pump { + shared: Arc, + /// This side's half, held so the application can write only its one group. + track: Option, + /// This side's group, watched for how the application ends it without keeping it open. + ours: Option, + /// The peer's half, watched for its one group, and for a second. + subscriber: Option, + /// Whether the peer's group arrived. + received: bool, + /// The peer's group, from its arrival until it ends. + remote: Option, + /// The application's copy of the peer's group, until it ends. + local: Option, + local_handle: group::Handle, + /// A frame read from the peer, waiting for the application's reader to make room. + pending: Option, +} + +impl Pump { + /// Reset everything still open with `err`. + fn reset(&mut self, err: Error) { + if let Some(local) = self.local.take() { + let _ = local.abort(err.clone()); + } + if let Some(track) = self.track.take() { + let _ = track.abort(err); + } + self.ours = None; + self.subscriber = None; + self.remote = None; + self.pending = None; + } + + fn poll_session(&mut self, waiter: &kio::Waiter) -> Result<(), Error> { + self.shared.poll_end(waiter)?; + + // This side's half ends the way the application ended its group. + if let Some(ours) = &self.ours + && let Poll::Ready(outcome) = ours.poll_closed(waiter) + { + self.ours = None; + match outcome { + Ok(()) => { + if let Some(track) = &self.track { + let _ = track.finish(); + } + } + Err(err) => { + if let Some(track) = self.track.take() { + let _ = track.abort(err); + } + } + } + } + + // The peer's half holds one group: a second is a protocol error, even after the + // first finished, for as long as either half is open. + if !self.received + && let Some(subscriber) = &mut self.subscriber + { + match subscriber.poll_recv_group(waiter) { + Poll::Pending => {} + Poll::Ready(Ok(Some(group))) => { + if group.sequence != 0 { + tracing::debug!(sequence = group.sequence, "a second group on a tcp session"); + return Err(Error::ProtocolViolation); + } + self.received = true; + self.remote = Some(group); + } + Poll::Ready(Ok(None)) => self.subscriber = None, + // Ended short of its group: the peer's reset. + Poll::Ready(Err(err)) => return Err(err), + } + } + if self.received + && let Some(subscriber) = &mut self.subscriber + { + // Not `poll_recv_group`: polled on every wake while reading the group it handed + // out, it stalled that group's delivery. + while let Poll::Ready(largest) = subscriber.poll_largest(waiter) { + match largest { + Some(largest) if largest.group > 0 => { + tracing::debug!(sequence = largest.group, "a second group on a tcp session"); + return Err(Error::ProtocolViolation); + } + Some(_) => {} + // The track ended, finished or reset: `poll_largest` reports both + // alike, so ask which, and pass on the peer's reset. + None => match subscriber.poll_finished(waiter) { + Poll::Ready(Ok(_)) => { + self.subscriber = None; + break; + } + Poll::Ready(Err(err)) => return Err(err), + Poll::Pending => break, + }, + } + } + } + + if self.local.is_none() { + return Ok(()); + } + // Nobody reads what the peer writes. + if self.local_handle.poll_unwanted(waiter).is_ready() { + return Err(Error::Cancel); + } + + let Some(remote) = &mut self.remote else { + // Finished without a group: an empty half. + if !self.received + && self.subscriber.is_none() + && let Some(mut local) = self.local.take() + { + let _ = local.finish(); + } + return Ok(()); + }; + let local = self.local.as_mut().expect("checked above"); + loop { + if let Some(frame) = &self.pending { + let info = frame::Info { + size: frame.payload.len() as u64, + timestamp: None, + }; + if local.poll_ready(waiter, &info).is_pending() { + return Ok(()); + } + let frame = self.pending.take().expect("checked above"); + local.write_frame(None, frame.payload)?; + } + match remote.poll_read_frame(waiter) { + Poll::Pending => return Ok(()), + Poll::Ready(Ok(Some(frame))) => self.pending = Some(frame), + Poll::Ready(Ok(None)) => { + local.finish()?; + self.local = None; + self.remote = None; + return Ok(()); + } + Poll::Ready(Err(err)) => return Err(err), + } + } + } +} + +impl kio::Task for Pump { + type Output = (); + + fn poll(&mut self, waiter: &kio::Waiter) -> Poll<()> { + if let Err(err) = self.poll_session(waiter) { + self.reset(err); + } + // Done once both halves ended; dropping the track then leaves its end in place. + if self.ours.is_none() && self.local.is_none() { + return Poll::Ready(()); + } + Poll::Pending + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn track_names_round_trip() { + let name = track_name("tcp", "alice", "bob", "0123abcd"); + assert_eq!(name, "tcp/alice/bob/0123abcd"); + assert_eq!( + parse(&name), + Some(Name { + protocol: "tcp", + dialer: "alice", + acceptor: "bob", + session: "0123abcd" + }) + ); + } + + #[test] + fn foreign_track_names_are_skipped() { + for name in [ + "command", + "alice/s", + "tcp/alice/s", + "tcp/alice/bob/", + "tcp//bob/s", + "tcp/alice//s", + "/alice/bob/s", + "tcp/alice/bob/s/0", + "tcp/a\\b/bob/s", + "tcp/alice/bob/s\\", + ] { + assert_eq!(parse(name), None, "{name}"); + } + } + + #[test] + fn an_endpoint_path_holds_one_segment() { + let prefix = Path::from_static("room"); + assert_eq!(endpoint_path(&prefix, "alice").unwrap().as_str(), "room/.socket/alice"); + assert!(endpoint_path(&prefix, "a/b").is_err()); + assert!(endpoint_path(&prefix, "").is_err()); + assert!(endpoint_path(&prefix, "a\\b").is_err()); + } + + /// Taking a dial wakes the dialer's subscription, and an inline executor can re-enter + /// the endpoint from that wake, such as to accept again or to close it, so the endpoint + /// polls its handler with no lock held. + #[test] + fn taking_a_dial_wakes_outside_the_lock() { + use std::sync::atomic::{AtomicBool, Ordering}; + use std::task::Wake; + + struct Close { + shared: Arc, + woken: AtomicBool, + } + impl Wake for Close { + fn wake(self: Arc) { + // The handler is out being polled, so this waits for it. + let accept = self.shared.poll_dial(Protocol::Track, &kio::Waiter::noop()); + assert!(accept.is_pending()); + self.shared.close(); + self.woken.store(true, Ordering::SeqCst); + } + } + + let (done, finished) = std::sync::mpsc::channel(); + let thread = std::thread::spawn(move || { + let (origin, _driver) = origin::Producer::new(origin::Config::new(crate::Hop::new(1).unwrap())); + let endpoint = Endpoint::new(&origin, "room", "bob").unwrap(); + let noop = kio::Waiter::noop(); + // Listening, so the dial queues for the handler. + assert!(endpoint.poll_accept_tcp(&noop).is_pending()); + + let broadcast = endpoint.shared.publication.broadcast().consume(); + let dial = broadcast + .track("tcp/alice/bob/0123456789abcdef0123456789abcdef") + .unwrap() + .subscribe(None) + .into_inner(); + let close = Arc::new(Close { + shared: endpoint.shared.clone(), + woken: AtomicBool::new(false), + }); + let waiter = kio::Waiter::new(std::task::Waker::from(close.clone())); + assert!(dial.poll_ok(&waiter).is_pending()); + + // The wake closes the endpoint while this takes the dial, which it resets. + assert!(matches!( + endpoint.poll_accept_tcp(&noop), + Poll::Ready(Err(Error::Closed)) + )); + assert!(close.woken.load(Ordering::SeqCst)); + assert!(matches!(dial.poll_ok(&noop), Poll::Ready(Err(_)))); + done.send(()).unwrap(); + }); + let timeout = std::time::Duration::from_secs(10); + if let Err(std::sync::mpsc::RecvTimeoutError::Timeout) = finished.recv_timeout(timeout) { + panic!("deadlocked re-entering the endpoint"); + } + thread.join().unwrap(); + } + + /// A dial queued in a protocol the application stopped accepting leaves the queue once + /// its dialer gives up, so dialing and giving up again never piles up requests. + #[test] + fn abandoned_dials_leave_the_queue() { + let (origin, _driver) = origin::Producer::new(origin::Config::new(crate::Hop::new(1).unwrap())); + let endpoint = Endpoint::new(&origin, "room", "bob").unwrap(); + let noop = kio::Waiter::noop(); + // Accepting both protocols, then only `tcp`. + assert!(endpoint.poll_accept_track(&noop).is_pending()); + assert!(endpoint.poll_accept_tcp(&noop).is_pending()); + + let broadcast = endpoint.shared.publication.broadcast().consume(); + let queued = || endpoint.shared.state.lock().track.dials.len(); + let dial = |session: u32| { + let dial = broadcast + .track(&format!("track/alice/bob/{session:032x}")) + .unwrap() + .subscribe(None) + .into_inner(); + assert!(dial.poll_ok(&noop).is_pending()); + dial + }; + for session in 0..3 { + let dial = dial(session); + assert!(endpoint.poll_accept_tcp(&noop).is_pending()); + assert_eq!(queued(), 1, "dial {session}"); + drop(dial); + } + let _open = dial(3); + let _also = dial(4); + assert!(endpoint.poll_accept_tcp(&noop).is_pending()); + assert_eq!(queued(), 2); + } + + #[test] + fn session_ids_are_128_bits() { + let (a, b) = (session_id(), session_id()); + assert_eq!(a.len(), 32); + assert_ne!(a, b); + } +} diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index e198ac52c3e..4aa66e060fe 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -4282,6 +4282,21 @@ impl TrackWeak { true } + /// Abort the track with `err`, as [`Producer::abort`] does, unless it already closed. + /// + /// For an owner that hands a track's [`Producer`] out and must end it on its own + /// shutdown without keeping it alive. Leaves the listing alone: such a track is one + /// served on request, which is never listed. + pub(super) fn abort(&self, err: Error) { + let Some(producer) = self.state.produce() else { + return; + }; + let Ok(state) = producer.write() else { + return; + }; + commit_abort(state, err).abort(); + } + /// Whether its producer declared the track's end, which is how its listing ends too. pub(super) fn is_finished(&self) -> bool { self.state.read().end.is_some() diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index 21a710fc538..d521ca74ed8 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -62,6 +62,7 @@ mod session_panic; mod settled_end_survives_abort; mod setup_negotiation; mod slow_joiner; +mod socket; mod source_errors; mod stream_credit; mod subscribe_priority; diff --git a/rs/moq-net/tests/integration/socket.rs b/rs/moq-net/tests/integration/socket.rs new file mode 100644 index 00000000000..24ee7d76e00 --- /dev/null +++ b/rs/moq-net/tests/integration/socket.rs @@ -0,0 +1,851 @@ +#![cfg(feature = "async")] + +//! Sockets through a relay: endpoints on their own origins, each with a session to a +//! relay, dialing and accepting `tcp` and `track` sessions over moq-lite-07 and +//! moq-transport. + +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Duration; + +use bytes::Bytes; +use moq_net::broadcast::TracksEvent; +use moq_net::socket::{Endpoint, TcpStream, TrackStream}; +use moq_net::{Error, Hop, Version, frame, group, origin, track}; + +use crate::support::harness::{MockConnectOptions, MockPair, connect_mock, spawn}; + +const TIMEOUT: Duration = Duration::from_secs(10); + +/// Long enough for every hop to settle, in simulated time. +const SETTLE: Duration = Duration::from_secs(1); + +const LITE: &str = "moq-lite-07-wip"; +const IETF: &str = "moq-transport-18"; + +fn origin(hop: u64) -> origin::Producer { + let (origin, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); + spawn(driver); + origin +} + +/// A relay and the clients connected to it. +struct Fixture { + relay: origin::Producer, + next: u64, + sessions: Vec, +} + +impl Fixture { + fn new() -> Self { + Self { + relay: origin(1), + next: 2, + sessions: Vec::new(), + } + } + + /// A client publishing and subscribing through one origin, connected to the relay over + /// `version`. + async fn client(&mut self, version: &str) -> origin::Producer { + let client = origin(self.next); + self.next += 1; + let mut options = MockConnectOptions::new(version.parse::().unwrap()); + options.client_publish = Some(client.consume()); + options.client_subscribe = Some(client.clone()); + options.server_publish = Some(self.relay.consume()); + options.server_subscribe = Some(self.relay.clone()); + // Discovery follows the endpoint's own requests, as it does for an application. + options.client_discovers_all = false; + self.sessions.push(connect_mock(options).await); + client + } + + /// An endpoint `id` on its own client of the relay. + async fn endpoint(&mut self, version: &str, id: &str) -> Endpoint { + let client = self.client(version).await; + Endpoint::new(&client, "room", id).unwrap() + } +} + +const MB: usize = 1 << 20; + +/// Dial `acceptor` from `dialer` in a `tcp` session while `acceptor` accepts, returning +/// both ends. +async fn connect_tcp(dialer: &Endpoint, acceptor: &Endpoint) -> (TcpStream, TcpStream) { + let dial = dialer.dial_tcp(acceptor.id()).unwrap(); + let accept = async { + let incoming = acceptor.accept_tcp().await?; + assert_eq!(incoming.peer(), dialer.id()); + incoming.accept().await + }; + let (dialed, accepted) = kio::clock::timeout(TIMEOUT, async { futures::join!(dial, accept) }) + .await + .expect("connect timeout"); + (dialed.expect("dial"), accepted.expect("accept")) +} + +/// Dial `acceptor` from `dialer` in a `track` session; see [`connect_tcp`]. +async fn connect_track(dialer: &Endpoint, acceptor: &Endpoint) -> (TrackStream, TrackStream) { + let dial = dialer.dial_track(acceptor.id(), None).unwrap(); + let accept = async { + let incoming = acceptor.accept_track().await?; + assert_eq!(incoming.peer(), dialer.id()); + incoming.accept(None).await + }; + let (dialed, accepted) = kio::clock::timeout(TIMEOUT, async { futures::join!(dial, accept) }) + .await + .expect("connect timeout"); + (dialed.expect("dial"), accepted.expect("accept")) +} + +/// Read the next write from a `tcp` session, or `None` once the peer finished. +async fn read(reader: &mut group::Consumer) -> Result>, Error> { + kio::clock::timeout(TIMEOUT, reader.read_frame()) + .await + .expect("read timeout") + .map(|frame| frame.map(|frame| frame.payload.to_vec())) +} + +/// Write `payload` as one message of a `track` session: one group. +fn send(track: &track::Producer, payload: &'static [u8]) { + let mut group = track.append_group().unwrap(); + group.write_frame(None, payload).unwrap(); + group.finish().unwrap(); +} + +/// Read the next message from a `track` session, or `None` once the peer finished. +async fn recv(reader: &mut track::Ordered) -> Result>, Error> { + kio::clock::timeout(TIMEOUT, async { + let Some(mut group) = reader.next_group().await? else { + return Ok(None); + }; + let frame = group.read_frame().await?.expect("a frame"); + Ok(Some(frame.payload.to_vec())) + }) + .await + .expect("read timeout") +} + +/// Writes flow both ways, in order, over every pairing of moq-lite-07 and moq-transport, +/// and finishing a half is FIN: the handshake uses neither instances nor track listings. +#[kio::test] +async fn tcp_sessions_carry_writes_both_ways() { + for (dialer, acceptor) in [(LITE, LITE), (LITE, IETF), (IETF, LITE), (IETF, IETF)] { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(dialer, "alice").await; + let bob = fixture.endpoint(acceptor, "bob").await; + let (mut dialed, mut accepted) = connect_tcp(&alice, &bob).await; + assert_eq!(dialed.peer(), "bob"); + assert_eq!(accepted.peer(), "alice"); + assert_eq!(dialed.session(), accepted.session()); + + for write in [b"one", b"two", b"six"] { + dialed.producer.write_frame(None, &write[..]).unwrap(); + } + for write in [b"one", b"two", b"six"] { + assert_eq!( + read(&mut accepted.consumer).await.unwrap().unwrap(), + write, + "{dialer} to {acceptor}" + ); + } + accepted.producer.write_frame(None, &b"pong"[..]).unwrap(); + assert_eq!(read(&mut dialed.consumer).await.unwrap().unwrap(), b"pong"); + + // FIN, each way. + dialed.producer.finish().unwrap(); + assert_eq!( + read(&mut accepted.consumer).await.unwrap(), + None, + "{dialer} to {acceptor}" + ); + accepted.producer.finish().unwrap(); + assert_eq!( + read(&mut dialed.consumer).await.unwrap(), + None, + "{dialer} to {acceptor}" + ); + } +} + +/// A long session is one group of many writes, read in order. +#[kio::test] +async fn a_long_tcp_session_stays_in_order() { + const WRITES: u32 = 2000; + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let (mut dialed, mut accepted) = connect_tcp(&alice, &bob).await; + + let write = async { + for i in 0..WRITES { + let payload = Bytes::copy_from_slice(&i.to_be_bytes()); + dialed + .producer + .ready(&frame::Info { + size: 4, + timestamp: None, + }) + .await; + dialed.producer.write_frame(None, payload).unwrap(); + } + dialed.producer.finish().unwrap(); + }; + let read_all = async { + for i in 0..WRITES { + assert_eq!(read(&mut accepted.consumer).await.unwrap().unwrap(), i.to_be_bytes()); + } + assert_eq!(read(&mut accepted.consumer).await.unwrap(), None); + }; + futures::join!(write, read_all); +} + +/// Aborting a half is RST: the peer's read fails instead of ending. +#[kio::test] +async fn a_reset_tcp_half_fails_the_read() { + for version in [LITE, IETF] { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(version, "alice").await; + let bob = fixture.endpoint(version, "bob").await; + let (dialed, mut accepted) = connect_tcp(&alice, &bob).await; + let mut producer = dialed.producer; + producer.write_frame(None, &b"partial"[..]).unwrap(); + assert_eq!(read(&mut accepted.consumer).await.unwrap().unwrap(), b"partial"); + producer.abort(Error::Cancel).unwrap(); + assert!( + read(&mut accepted.consumer).await.is_err(), + "{version}: a reset read as FIN" + ); + } +} + +/// A slow reader paces the writer through every hop, and a session longer than a group's +/// live tail loses nothing: the writer waits instead of running ahead. +#[kio::test] +async fn a_slow_tcp_reader_paces_the_writer() { + // Well past the 32 MiB a group copy holds, and the window every hop may buffer. + const WRITES: usize = 192; + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let (mut dialed, mut accepted) = connect_tcp(&alice, &bob).await; + + let written = Arc::new(AtomicUsize::new(0)); + let write = { + let written = written.clone(); + async move { + for i in 0..WRITES { + let payload = Bytes::from(vec![i as u8; MB]); + let info = frame::Info { + size: MB as u64, + timestamp: None, + }; + dialed.producer.ready(&info).await; + dialed.producer.write_frame(None, payload).unwrap(); + written.fetch_add(1, Ordering::SeqCst); + } + dialed.producer.finish().unwrap(); + } + }; + let read_all = async { + // Not reading for a while: the writer stalls well short of the end. + kio::clock::sleep(Duration::from_secs(5)).await; + let ahead = written.load(Ordering::SeqCst); + assert!(ahead < WRITES, "the writer ran {ahead} writes ahead of an idle reader"); + for i in 0..WRITES { + let frame = read(&mut accepted.consumer).await.unwrap().expect("a write"); + assert_eq!(frame.len(), MB); + assert!(frame.iter().all(|byte| *byte == i as u8), "write {i} out of order"); + // Slowly. + kio::clock::sleep(Duration::from_millis(10)).await; + } + assert_eq!(read(&mut accepted.consumer).await.unwrap(), None); + }; + kio::clock::timeout(Duration::from_secs(120), async { futures::join!(write, read_all) }) + .await + .expect("session timeout"); +} + +/// A `tcp` session carol dialed by hand, as a nonconforming implementation would. +struct ByHand { + /// Bob's end. + accepted: TcpStream, + /// Carol's half, which she writes groups to as she likes. + half: track::Producer, + _dialed: track::Subscriber, + _endpoint: origin::Publication, + _carol: origin::Producer, +} + +/// Dial bob by hand from carol in a `tcp` session. +async fn dial_by_hand(fixture: &mut Fixture, bob: &Endpoint) -> ByHand { + let carol = fixture.client(LITE).await; + let endpoint = carol.create_broadcast("room/.socket/carol").unwrap(); + endpoint.announce(Default::default()).unwrap(); + let mut requests = endpoint.requests(); + let name = "tcp/carol/bob/0123456789abcdef0123456789abcdef"; + let broadcast = kio::clock::timeout(TIMEOUT, carol.consume().request("room/.socket/bob", None)) + .await + .expect("request timeout") + .expect("bob's endpoint"); + let dial = broadcast.track(name).unwrap().subscribe(None); + + let accept = async { + let incoming = bob.accept_tcp().await.unwrap(); + incoming.accept().await + }; + let answer = async { + let request = kio::wait(|waiter| requests.poll_requested_track(waiter)).await.unwrap(); + assert_eq!(request.name(), name); + request.accept(None) + }; + let (accepted, half, dialed) = kio::clock::timeout(TIMEOUT, async { futures::join!(accept, answer, dial) }) + .await + .expect("connect timeout"); + ByHand { + accepted: accepted.expect("accept"), + half, + _dialed: dialed.expect("bob's half"), + _endpoint: endpoint, + _carol: carol, + } +} + +/// Write `sequence` as a finished group of one write. +fn write_group(half: &track::Producer, sequence: u64) { + let mut group = half.create_group(group::Info { sequence }).unwrap(); + group.write_frame(None, &b"data"[..]).unwrap(); + group.finish().unwrap(); +} + +/// A peer that writes a second group in a `tcp` session breaks the protocol: the session +/// is reset, both halves. +#[kio::test] +async fn a_second_tcp_group_resets_the_session() { + let mut fixture = Fixture::new(); + let bob = fixture.endpoint(LITE, "bob").await; + // Bound whole, so carol's endpoint and read stay open to the end. + let mut session = dial_by_hand(&mut fixture, &bob).await; + let (accepted, half) = (&mut session.accepted, &session.half); + + write_group(half, 0); + write_group(half, 1); + // Bob reads at most the first group's write, then the reset. + let mut reads = 0; + loop { + match read(&mut accepted.consumer).await { + Ok(Some(_)) => reads += 1, + Ok(None) => panic!("a second group read as FIN"), + Err(_) => break, + } + assert!(reads <= 1, "read past the first group"); + } + // And bob's own half was reset with it. + kio::clock::sleep(SETTLE).await; + assert!(accepted.producer.write_frame(None, &b"late"[..]).is_err()); +} + +/// A second group after the first one finished, and was read to its end, still resets +/// the half that remains open. +#[kio::test] +async fn a_second_tcp_group_after_fin_resets_the_session() { + let mut fixture = Fixture::new(); + let bob = fixture.endpoint(LITE, "bob").await; + // Bound whole, so carol's endpoint and read stay open to the end. + let mut session = dial_by_hand(&mut fixture, &bob).await; + let (accepted, half) = (&mut session.accepted, &session.half); + + write_group(half, 0); + assert_eq!(read(&mut accepted.consumer).await.unwrap().unwrap(), b"data"); + assert_eq!(read(&mut accepted.consumer).await.unwrap(), None); + accepted.producer.write_frame(None, &b"still open"[..]).unwrap(); + + write_group(half, 1); + kio::clock::sleep(SETTLE).await; + assert!(accepted.producer.write_frame(None, &b"late"[..]).is_err()); +} + +/// The peer resetting its half after its group finished, and was read to its end, resets +/// the half that remains open, as a TCP RST after FIN does. Finishing it instead leaves +/// that half open, as a half-close does. +#[kio::test] +async fn a_peer_reset_after_fin_resets_the_session() { + for reset in [true, false] { + let mut fixture = Fixture::new(); + let bob = fixture.endpoint(LITE, "bob").await; + // Bound whole, so carol's endpoint and read stay open to the end. + let session = dial_by_hand(&mut fixture, &bob).await; + let ByHand { mut accepted, half, .. } = session; + + write_group(&half, 0); + assert_eq!(read(&mut accepted.consumer).await.unwrap().unwrap(), b"data"); + assert_eq!(read(&mut accepted.consumer).await.unwrap(), None); + accepted.producer.write_frame(None, &b"still open"[..]).unwrap(); + + if reset { + half.abort(Error::Cancel).unwrap(); + } else { + half.finish().unwrap(); + } + kio::clock::sleep(SETTLE).await; + let late = accepted.producer.write_frame(None, &b"late"[..]); + assert_eq!(late.is_err(), reset, "reset: {reset}"); + } +} + +/// Messages flow both ways, in order, over every pairing of moq-lite-07 and +/// moq-transport. +#[kio::test] +async fn track_sessions_carry_messages_both_ways() { + for (dialer, acceptor) in [(LITE, LITE), (LITE, IETF), (IETF, LITE), (IETF, IETF)] { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(dialer, "alice").await; + let bob = fixture.endpoint(acceptor, "bob").await; + let (dialed, accepted) = connect_track(&alice, &bob).await; + assert_eq!(dialed.peer(), "bob"); + assert_eq!(dialed.session(), accepted.session()); + let mut reader = accepted.consumer.ordered(); + let mut replies = dialed.consumer.ordered(); + + for message in [b"one", b"two", b"six"] { + send(&dialed.producer, message); + } + for message in [b"one", b"two", b"six"] { + assert_eq!( + recv(&mut reader).await.unwrap().unwrap(), + message, + "{dialer} to {acceptor}" + ); + } + send(&accepted.producer, b"pong"); + assert_eq!(recv(&mut replies).await.unwrap().unwrap(), b"pong"); + + // Finishing a half closes the pipe that way. + dialed.producer.finish().unwrap(); + assert_eq!(recv(&mut reader).await.unwrap(), None); + } +} + +/// Messages are read in sequence even when they are written, and so arrive, out of order. +#[kio::test] +async fn track_messages_are_read_in_sequence() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let (dialed, accepted) = connect_track(&alice, &bob).await; + let mut reader = accepted.consumer.ordered(); + + let write = |sequence: u64, payload: &'static [u8]| { + let mut group = dialed.producer.create_group(group::Info { sequence }).unwrap(); + group.write_frame(None, payload).unwrap(); + group.finish().unwrap(); + }; + write(1, b"second"); + // Let the later message arrive first. + assert!(kio::clock::timeout(SETTLE, reader.next_group()).await.is_err()); + write(0, b"first"); + assert_eq!(recv(&mut reader).await.unwrap().unwrap(), b"first"); + assert_eq!(recv(&mut reader).await.unwrap().unwrap(), b"second"); +} + +/// With a finite delay budget, a message that never arrives stops blocking the ones after +/// it once the budget runs out. +#[kio::test] +async fn a_lost_track_message_does_not_block_later_ones() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let (dialed, accepted) = connect_track(&alice, &bob).await; + let lossy = accepted + .consumer + .subscription() + .with_delay(track::Delay::default().with_max(Duration::from_millis(500))); + accepted.consumer.control().update(lossy).unwrap(); + let mut reader = accepted.consumer.ordered(); + + let write = |sequence: u64, payload: &'static [u8]| { + let mut group = dialed.producer.create_group(group::Info { sequence }).unwrap(); + group.write_frame(None, payload).unwrap(); + group.finish().unwrap(); + }; + // Message 0 is never written: message 1 waits out the budget, then is read. + write(1, b"one"); + assert_eq!(recv(&mut reader).await.unwrap().unwrap(), b"one"); + write(2, b"two"); + assert_eq!(recv(&mut reader).await.unwrap().unwrap(), b"two"); +} + +/// A pair of endpoints holds several sessions at once in either protocol, each its own +/// pipe. +#[kio::test] +async fn several_sessions_share_a_pair() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let (mut first, mut first_accepted) = connect_tcp(&alice, &bob).await; + let (mut second, mut second_accepted) = connect_tcp(&alice, &bob).await; + // Either side may dial, in either protocol. + let (third, third_accepted) = connect_track(&bob, &alice).await; + assert_ne!(first.session(), second.session()); + + second.producer.write_frame(None, &b"second"[..]).unwrap(); + first.producer.write_frame(None, &b"first"[..]).unwrap(); + send(&third_accepted.producer, b"third"); + assert_eq!(read(&mut first_accepted.consumer).await.unwrap().unwrap(), b"first"); + assert_eq!(read(&mut second_accepted.consumer).await.unwrap().unwrap(), b"second"); + assert_eq!(recv(&mut third.consumer.ordered()).await.unwrap().unwrap(), b"third"); + first.producer.finish().unwrap(); + second.producer.finish().unwrap(); +} + +/// Rejecting a dial resets it. +#[kio::test] +async fn a_rejected_dial_fails() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let dial = alice.dial_tcp("bob").unwrap(); + let reject = async { + bob.accept_tcp().await.unwrap().reject(); + }; + let (dialed, ()) = kio::clock::timeout(TIMEOUT, async { futures::join!(dial, reject) }) + .await + .expect("reject timeout"); + assert!(dialed.is_err(), "a rejected dial connected"); +} + +/// A dial in a protocol the peer does not accept is reset, while the peer still accepts +/// the protocol it listens on. +#[kio::test] +async fn a_protocol_nobody_accepts_is_reset() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let dial = alice.dial_track("bob", None).unwrap(); + let accept = async { kio::clock::timeout(SETTLE, bob.accept_tcp()).await }; + let (dialed, accepted) = kio::clock::timeout(TIMEOUT, async { futures::join!(dial, accept) }) + .await + .expect("dial timeout"); + assert!(dialed.is_err(), "a track dial reached a tcp listener"); + assert!(accepted.is_err(), "a track dial surfaced as a tcp dial"); + + // An unknown protocol is reset too. + let mallory = fixture.client(LITE).await; + let broadcast = kio::clock::timeout(TIMEOUT, mallory.consume().request("room/.socket/bob", None)) + .await + .expect("request timeout") + .expect("bob's endpoint"); + let unknown = broadcast + .track("quic/mallory/bob/0123456789abcdef0123456789abcdef") + .unwrap() + .subscribe(None); + let unknown = async { kio::clock::timeout(TIMEOUT, unknown).await }; + let accept = async { kio::clock::timeout(SETTLE, bob.accept_tcp()).await }; + let (unknown, accepted) = futures::join!(unknown, accept); + assert!(unknown.expect("unknown timeout").is_err()); + assert!(accepted.is_err(), "an unknown protocol surfaced as a dial"); +} + +/// A dial waits for its peer's endpoint to be announced, and a dial to itself fails. +#[kio::test] +async fn a_dial_waits_for_its_peer() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + assert!(matches!(alice.dial_tcp("alice"), Err(Error::Unsupported))); + let mut dial = alice.dial_tcp("bob").unwrap(); + assert!( + kio::clock::timeout(SETTLE, &mut dial).await.is_err(), + "a dial to nobody resolved" + ); + + let bob = fixture.endpoint(LITE, "bob").await; + let accept = async { bob.accept_tcp().await?.accept().await }; + let (dialed, accepted) = kio::clock::timeout(TIMEOUT, async { futures::join!(dial, accept) }) + .await + .expect("connect timeout"); + let (mut dialed, accepted) = (dialed.unwrap(), accepted.unwrap()); + assert_eq!(dialed.session(), accepted.session()); + dialed.producer.finish().unwrap(); +} + +/// An accept waits for the dialer's endpoint while the dial stays open, since a relay can +/// deliver a dial before the dialer's announcement: carol dials bob before she announces +/// her endpoint, and bob's accept connects once she does. +#[kio::test] +async fn an_accept_waits_for_its_dialer() { + let mut fixture = Fixture::new(); + let bob = fixture.endpoint(LITE, "bob").await; + let carol = fixture.client(LITE).await; + let name = "tcp/carol/bob/0123456789abcdef0123456789abcdef"; + let broadcast = kio::clock::timeout(TIMEOUT, carol.consume().request("room/.socket/bob", None)) + .await + .expect("request timeout") + .expect("bob's endpoint"); + let dial = broadcast.track(name).unwrap().subscribe(None); + + let incoming = kio::clock::timeout(TIMEOUT, bob.accept_tcp()) + .await + .expect("accept timeout") + .unwrap(); + assert_eq!(incoming.peer(), "carol"); + let mut accept = incoming.accept(); + assert!( + kio::clock::timeout(SETTLE, &mut accept).await.is_err(), + "bob's accept ended before carol announced" + ); + + let endpoint = carol.create_broadcast("room/.socket/carol").unwrap(); + let mut requests = endpoint.requests(); + endpoint.announce(Default::default()).unwrap(); + let answer = async { + let request = kio::wait(|waiter| requests.poll_requested_track(waiter)).await.unwrap(); + assert_eq!(request.name(), name); + request.accept(None) + }; + let (accepted, _half, dialed) = kio::clock::timeout(TIMEOUT, async { futures::join!(accept, answer, dial) }) + .await + .expect("connect timeout"); + assert_eq!(accepted.expect("accept").peer(), "carol"); + dialed.expect("bob's half"); +} + +/// An accept of a dial claiming an endpoint nobody announces waits only while the dial +/// stays open: the forger dropping it fails the accept. +#[kio::test] +async fn an_accept_ends_with_its_dial() { + let mut fixture = Fixture::new(); + let bob = fixture.endpoint(LITE, "bob").await; + let mallory = fixture.client(LITE).await; + let broadcast = kio::clock::timeout(TIMEOUT, mallory.consume().request("room/.socket/bob", None)) + .await + .expect("request timeout") + .expect("bob's endpoint"); + let forged = broadcast + .track("tcp/carol/bob/0123456789abcdef0123456789abcdef") + .unwrap() + .subscribe(None); + let incoming = kio::clock::timeout(TIMEOUT, bob.accept_tcp()) + .await + .expect("accept timeout") + .unwrap(); + assert_eq!(incoming.peer(), "carol"); + let mut accept = incoming.accept(); + assert!( + kio::clock::timeout(SETTLE, &mut accept).await.is_err(), + "bob's accept ended while the dial was open" + ); + + drop(forged); + let accepted = kio::clock::timeout(TIMEOUT, accept).await.expect("accept timeout"); + assert!(accepted.is_err(), "bob connected to nobody"); +} + +/// A session is bound to both its ends: an endpoint alice dialed knows the session's id, +/// but replaying it to carol connects neither under bob's name nor under carol's. +#[kio::test] +async fn a_replayed_session_never_connects() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let carol = fixture.endpoint(LITE, "carol").await; + let (mut dialed, mut accepted) = connect_tcp(&alice, &bob).await; + let session = accepted.session().to_string(); + + // Replayed from bob's side, which knows the session. + let mallory = fixture.client(LITE).await; + let broadcast = kio::clock::timeout(TIMEOUT, mallory.consume().request("room/.socket/carol", None)) + .await + .expect("request timeout") + .expect("carol's endpoint"); + + // Under the name alice dialed: carol is not its acceptor, so she resets it unheard. + let replayed = broadcast + .track(&format!("tcp/alice/bob/{session}")) + .unwrap() + .subscribe(None); + // Carol accepts first, so the replay reaches her listening. + let accept = async { kio::clock::timeout(SETTLE, carol.accept_tcp()).await }; + let (accepted_by_carol, replayed) = futures::join!(accept, async { kio::clock::timeout(TIMEOUT, replayed).await }); + assert!( + replayed.expect("replay timeout").is_err(), + "carol served a replayed session" + ); + assert!(accepted_by_carol.is_err(), "a session for bob surfaced at carol"); + + // Renamed for carol: alice never dialed it, so she resets carol's accept. + let renamed = broadcast + .track(&format!("tcp/alice/carol/{session}")) + .unwrap() + .subscribe(None); + let incoming = kio::clock::timeout(TIMEOUT, carol.accept_tcp()) + .await + .expect("accept timeout") + .unwrap(); + assert_eq!(incoming.peer(), "alice"); + let (accepted_by_carol, renamed) = + kio::clock::timeout(TIMEOUT, async { futures::join!(incoming.accept(), renamed) }) + .await + .expect("connect timeout"); + assert!(accepted_by_carol.is_err(), "carol connected to a replayed session"); + assert!(renamed.is_err(), "the replayer read carol's half"); + + // The real session is untouched. + dialed.producer.write_frame(None, &b"still here"[..]).unwrap(); + assert_eq!(read(&mut accepted.consumer).await.unwrap().unwrap(), b"still here"); +} + +/// A third party asking bob for a session in alice's name surfaces at bob as a dial from +/// alice, but alice never dialed it, so she resets bob's accept, and the forger reads +/// nothing. The same holds in either protocol. +#[kio::test] +async fn a_forged_dial_never_connects() { + for protocol in ["tcp", "track"] { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let mallory = fixture.client(LITE).await; + + let broadcast = kio::clock::timeout(TIMEOUT, mallory.consume().request("room/.socket/bob", None)) + .await + .expect("request timeout") + .expect("bob's endpoint"); + let forged = broadcast + .track(&format!("{protocol}/alice/bob/0123456789abcdef0123456789abcdef")) + .unwrap() + .subscribe(None); + + let (incoming_peer, accepted) = match protocol { + "tcp" => { + let incoming = kio::clock::timeout(TIMEOUT, bob.accept_tcp()) + .await + .expect("accept timeout") + .unwrap(); + let peer = incoming.peer().to_string(); + let accepted = kio::clock::timeout(TIMEOUT, incoming.accept()).await; + (peer, accepted.expect("accept timeout").map(drop)) + } + _ => { + let incoming = kio::clock::timeout(TIMEOUT, bob.accept_track()) + .await + .expect("accept timeout") + .unwrap(); + let peer = incoming.peer().to_string(); + let accepted = kio::clock::timeout(TIMEOUT, incoming.accept(None)).await; + (peer, accepted.expect("accept timeout").map(drop)) + } + }; + assert_eq!(incoming_peer, "alice"); + assert!(accepted.is_err(), "bob connected to a forged {protocol} dial"); + let forged = kio::clock::timeout(TIMEOUT, forged).await.expect("forged timeout"); + assert!(forged.is_err(), "the forger read bob's {protocol} half"); + + // Alice never heard of it as a dial either. + assert!(kio::clock::timeout(SETTLE, alice.accept_tcp()).await.is_err()); + assert!(kio::clock::timeout(SETTLE, alice.accept_track()).await.is_err()); + } +} + +/// A dial dropped before the peer accepts resets the accept. +#[kio::test] +async fn a_dropped_dial_resets_the_accept() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let mut dial = alice.dial_tcp("bob").unwrap(); + // Polled until bob hears of it, then dropped. + let incoming = kio::clock::timeout(TIMEOUT, async { + match futures::future::select(std::pin::pin!(bob.accept_tcp()), &mut dial).await { + futures::future::Either::Left((incoming, _)) => incoming, + futures::future::Either::Right(_) => panic!("the dial resolved unaccepted"), + } + }) + .await + .expect("accept timeout") + .unwrap(); + drop(dial); + let accepted = kio::clock::timeout(TIMEOUT, incoming.accept()) + .await + .expect("accept timeout"); + assert!(accepted.is_err(), "bob connected to a dropped dial"); +} + +/// Closing an endpoint resets its sessions for the peer, and fails its pending work. +#[kio::test] +async fn closing_an_endpoint_ends_its_sessions() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let (mut tcp_dialed, mut tcp_accepted) = connect_tcp(&alice, &bob).await; + let (track_dialed, track_accepted) = connect_track(&alice, &bob).await; + + bob.close(); + assert!( + read(&mut tcp_dialed.consumer).await.is_err(), + "bob's tcp half outlived his endpoint" + ); + // Both halves of bob's own tcp session end. + assert!(read(&mut tcp_accepted.consumer).await.is_err()); + assert!(tcp_accepted.producer.write_frame(None, &b"late"[..]).is_err()); + assert!( + recv(&mut track_dialed.consumer.ordered()).await.is_err(), + "bob's track half outlived his endpoint" + ); + assert!(track_accepted.producer.append_group().is_err()); + assert!(matches!(bob.accept_tcp().await, Err(Error::Closed))); + assert!(matches!(bob.accept_track().await, Err(Error::Closed))); + assert!(matches!(bob.dial_tcp("alice"), Err(Error::Closed))); + // Alice saw bob's half reset, so she reset hers: the session is gone both ways. + assert!(tcp_dialed.producer.finish().is_err()); +} + +/// Session tracks are the secret that keeps third parties out, so no listing names them: +/// a reader allowed everything lists each endpoint's broadcast while a session runs and +/// finds no track. +#[kio::test] +async fn session_tracks_are_never_listed() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let bob = fixture.endpoint(LITE, "bob").await; + let (mut dialed, mut accepted) = connect_tcp(&alice, &bob).await; + let (track_dialed, track_accepted) = connect_track(&alice, &bob).await; + dialed.producer.write_frame(None, &b"secret"[..]).unwrap(); + assert_eq!(read(&mut accepted.consumer).await.unwrap().unwrap(), b"secret"); + send(&track_dialed.producer, b"secret"); + assert_eq!( + recv(&mut track_accepted.consumer.ordered()).await.unwrap().unwrap(), + b"secret" + ); + + let eve = fixture.client(LITE).await; + for id in ["alice", "bob"] { + let broadcast = kio::clock::timeout(TIMEOUT, eve.consume().request(format!("room/.socket/{id}"), None)) + .await + .expect("request timeout") + .expect("the endpoint"); + let mut tracks = broadcast.tracks().unwrap(); + let event = kio::clock::timeout(TIMEOUT, tracks.next()) + .await + .expect("listing timeout") + .unwrap(); + assert!(matches!(event, Some(TracksEvent::Synced)), "{id} listed {event:?}"); + assert!( + kio::clock::timeout(SETTLE, tracks.next()).await.is_err(), + "{id} listed a track" + ); + } + dialed.producer.finish().unwrap(); +} + +/// An endpoint takes dials only from its first accept on, like a TCP listener: before it, +/// a dial is reset rather than left waiting. +#[kio::test] +async fn an_endpoint_not_accepting_resets_dials() { + let mut fixture = Fixture::new(); + let alice = fixture.endpoint(LITE, "alice").await; + let _bob = fixture.endpoint(LITE, "bob").await; + let dialed = kio::clock::timeout(TIMEOUT, alice.dial_tcp("bob").unwrap()) + .await + .expect("dial timeout"); + assert!(dialed.is_err(), "a dial to an endpoint not accepting connected"); +} diff --git a/rs/moq-relay/tests/integration/main.rs b/rs/moq-relay/tests/integration/main.rs index 479ab6c2005..e3e02dbf499 100644 --- a/rs/moq-relay/tests/integration/main.rs +++ b/rs/moq-relay/tests/integration/main.rs @@ -34,6 +34,8 @@ mod runtime_workers; mod session_revalidate; mod shutdown_signal; mod smoke; +#[cfg(feature = "test-support")] +mod socket_auth; mod stack; mod tenant_matrix; mod upstream_links; diff --git a/rs/moq-relay/tests/integration/socket_auth.rs b/rs/moq-relay/tests/integration/socket_auth.rs new file mode 100644 index 00000000000..6b394d201c8 --- /dev/null +++ b/rs/moq-relay/tests/integration/socket_auth.rs @@ -0,0 +1,289 @@ +//! Sockets through a relay that admits with `moq-auth` tokens, verified by the reference +//! auth server, granting only the prefixes `doc/concept/socket.md` lists. +//! +//! Every token is rooted at the socket prefix `room`. An endpoint's token publishes its own +//! endpoint, `.socket/`, and subscribes to every endpoint, `.socket/**`, so the +//! session track names are what keep third parties out: a forged dial never connects, and no +//! listing gives a session's name away. + +use std::time::Duration; + +use moq_net::broadcast::TracksEvent; +use moq_net::socket::{Endpoint, TcpStream, TrackStream}; +use moq_net::transport::memory; +use moq_relay::{Accepted, Connection}; + +const TIMEOUT: Duration = Duration::from_secs(10); + +/// Long enough for a listing to have reported a track had it listed one. The relay links +/// are in memory, so this is many round trips. +const SETTLE: Duration = Duration::from_millis(500); + +/// The socket prefix, and every token's root. +const ROOT: &str = "room"; + +const LITE: &str = "moq-lite-07-wip"; +const IETF: &str = "moq-transport-18"; + +/// A key, the reference auth server verifying tokens it signs, and a relay asking it. +struct Fixture { + key: moq_auth::Key, + cluster: moq_relay::cluster::Cluster, + auth: moq_relay::auth::Auth, + _keys: tempfile::TempDir, + _server: tokio::task::JoinHandle<()>, +} + +impl Fixture { + async fn new() -> Self { + let keys = tempfile::tempdir().expect("tempdir"); + let key = moq_auth::Key::generate(moq_auth::Algorithm::HS256, None).expect("generate key"); + let key_path = keys.path().join("key.jwk"); + key.to_file(&key_path).expect("write key"); + + let mut policy = moq_auth::serve::Policy::default(); + policy.keys = Some(moq_auth::serve::Keys::File(key_path)); + let server = moq_auth::serve::Server::new(policy).expect("auth server"); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind auth"); + let url = format!("http://{}/", listener.local_addr().unwrap()).parse().unwrap(); + let server = tokio::spawn(async move { + let _ = server.serve(listener).await; + }); + + let mut config = moq_relay::auth::Config::default(); + config.url = Some(url); + let auth = config + .init("test", &moq_tokio::tls::Connect::default(), false) + .expect("auth init"); + Self { + key, + cluster: crate::spawn_cluster(Default::default()), + auth, + _keys: keys, + _server: server, + } + } + + /// Sign a token rooted at [`ROOT`] with these grants. + fn token(&self, publish: &str, subscribe: &str) -> String { + let claims = moq_auth::Claims::default() + .with_root(ROOT) + .with_publish([publish.parse().expect("publish pattern")]) + .with_subscribe([subscribe.parse().expect("subscribe pattern")]); + self.key.sign(&claims).expect("sign") + } + + /// A client at [`ROOT`] presenting `token`, publishing and subscribing through one + /// origin, over `version`. + async fn client(&self, token: &str, version: &str) -> (moq_net::origin::Producer, moq_net::Session) { + let origin = moq_tokio::origin::spawn(); + let url = format!("moql://relay.example.com/{ROOT}?jwt={token}").parse().unwrap(); + let client = moq_net::Client::new() + .with_publisher(origin.consume()) + .with_subscriber(origin.clone()) + .with_url(url); + let version = version.parse().expect("version"); + let session = self.connect(client, version).await.expect("connect"); + (origin, session) + } + + /// Dial the relay over a fresh in-memory link, as `client` speaking `version`. + /// + /// On Tokio rather than kio's simulator, which `crate::memory` uses, since the + /// reference auth server is a real HTTP server. + async fn connect( + &self, + client: moq_net::Client, + version: moq_net::Version, + ) -> Result { + let mut config = memory::Config::default(); + config.protocol = Some(version.alpn().to_string()); + let (client_link, relay_link) = memory::pair(config); + + let (cluster, auth) = (self.cluster.clone(), self.auth.clone()); + tokio::spawn(async move { + let server = moq_net::Server::new().with_versions(version.into()); + let Ok(handshake) = server.accept_request(moq_net::time::Instant::now(), relay_link).await else { + return; + }; + let request = Accepted::new(handshake, moq_auth::Transport::Quic); + Connection::new(request, cluster, auth).run().await; + }); + + let (session, driver) = client + .with_versions(version.into()) + .connect(moq_net::time::Instant::now(), client_link) + .await?; + tokio::spawn(moq_net::time::run(driver)); + Ok(session) + } + + /// The endpoint `id`'s client, holding the grants `doc/concept/socket.md` lists: publish + /// its own endpoint, subscribe to every endpoint. + async fn endpoint(&self, id: &str, version: &str) -> (Endpoint, moq_net::origin::Producer, moq_net::Session) { + let (origin, session) = self + .client(&self.token(&format!(".socket/{id}"), ".socket/**"), version) + .await; + // The session is rooted at the prefix, so the endpoint's prefix is the root. + let endpoint = Endpoint::new(&origin, "", id).expect("endpoint"); + (endpoint, origin, session) + } +} + +/// Dial `acceptor` from `dialer` in a `tcp` session while `acceptor` accepts, returning +/// both ends. +async fn connect_tcp(dialer: &Endpoint, acceptor: &Endpoint) -> (TcpStream, TcpStream) { + let dial = dialer.dial_tcp(acceptor.id()).expect("dial"); + let accept = async { + let incoming = acceptor.accept_tcp().await?; + assert_eq!(incoming.peer(), dialer.id()); + incoming.accept().await + }; + let (dialed, accepted) = kio::clock::timeout(TIMEOUT, async { futures::join!(dial, accept) }) + .await + .expect("connect timeout"); + (dialed.expect("dial"), accepted.expect("accept")) +} + +/// Dial `acceptor` from `dialer` in a `track` session; see [`connect_tcp`]. +async fn connect_track(dialer: &Endpoint, acceptor: &Endpoint) -> (TrackStream, TrackStream) { + let dial = dialer.dial_track(acceptor.id(), None).expect("dial"); + let accept = async { + let incoming = acceptor.accept_track().await?; + assert_eq!(incoming.peer(), dialer.id()); + incoming.accept(None).await + }; + let (dialed, accepted) = kio::clock::timeout(TIMEOUT, async { futures::join!(dial, accept) }) + .await + .expect("connect timeout"); + (dialed.expect("dial"), accepted.expect("accept")) +} + +/// Read the next write from a `tcp` session. +async fn read(reader: &mut moq_net::group::Consumer) -> Vec { + kio::clock::timeout(TIMEOUT, reader.read_frame()) + .await + .expect("read timeout") + .expect("frame") + .expect("a frame") + .payload + .to_vec() +} + +/// Write `payload` as one message of a `track` session: one group. +fn send(track: &moq_net::track::Producer, payload: &'static [u8]) { + let mut group = track.append_group().expect("group"); + group.write_frame(None, payload).expect("frame"); + group.finish().expect("finish"); +} + +/// Read the next message of a `track` session. +async fn recv(reader: &mut moq_net::track::Ordered) -> Vec { + kio::clock::timeout(TIMEOUT, async { + let mut group = reader.next_group().await.expect("group").expect("a group"); + let frame = group.read_frame().await.expect("frame").expect("a frame"); + frame.payload.to_vec() + }) + .await + .expect("read timeout") +} + +/// The prefix grants admit a dial and its accept in either protocol, and the session +/// carries data, over moq-lite-07 and across a moq-transport hop. +#[tokio::test] +async fn prefix_grants_admit_a_session() { + for (dialer, acceptor) in [(LITE, LITE), (LITE, IETF), (IETF, LITE)] { + let fixture = Fixture::new().await; + let (alice, _alice_origin, _alice_session) = fixture.endpoint("alice", dialer).await; + let (bob, _bob_origin, _bob_session) = fixture.endpoint("bob", acceptor).await; + let (mut dialed, mut accepted) = connect_tcp(&alice, &bob).await; + dialed.producer.write_frame(None, &b"ping"[..]).expect("write"); + assert_eq!(read(&mut accepted.consumer).await, b"ping", "{dialer} to {acceptor}"); + accepted.producer.write_frame(None, &b"pong"[..]).expect("write"); + assert_eq!(read(&mut dialed.consumer).await, b"pong", "{dialer} to {acceptor}"); + dialed.producer.finish().expect("finish"); + accepted.producer.finish().expect("finish"); + + let (dialed, accepted) = connect_track(&alice, &bob).await; + send(&dialed.producer, b"ping"); + assert_eq!( + recv(&mut accepted.consumer.ordered()).await, + b"ping", + "{dialer} to {acceptor}" + ); + } +} + +/// Mallory may read every endpoint, so she can ask bob for a session in alice's name. Bob +/// sees a dial from alice, but alice never dialed it and resets his accept, so it never +/// connects and mallory reads nothing. +#[tokio::test] +async fn a_forged_dial_never_connects() { + let fixture = Fixture::new().await; + let (alice, _alice_origin, _alice_session) = fixture.endpoint("alice", LITE).await; + let (bob, _bob_origin, _bob_session) = fixture.endpoint("bob", LITE).await; + let (_mallory, mallory_origin, _mallory_session) = fixture.endpoint("mallory", LITE).await; + // Alice keeps accepting, as a server would, which is how she resets the forged accept. + let alice_accepts = async { + let _ = alice.accept_tcp().await; + panic!("alice heard of a dial"); + }; + + let forge = async { + let broadcast = mallory_origin + .consume() + .request(".socket/bob", None) + .await + .expect("bob's endpoint"); + let forged = broadcast + .track("tcp/alice/bob/0123456789abcdef0123456789abcdef") + .expect("track") + .subscribe(None); + let incoming = bob.accept_tcp().await.expect("accept"); + assert_eq!(incoming.peer(), "alice"); + let (accepted, forged) = futures::join!(incoming.accept(), forged); + assert!(accepted.is_err(), "bob connected to a forged dial"); + assert!(forged.is_err(), "mallory read bob's half"); + }; + kio::clock::timeout(TIMEOUT, async { + futures::select_biased! { + () = futures::FutureExt::fuse(Box::pin(forge)) => {} + () = futures::FutureExt::fuse(Box::pin(alice_accepts)) => {} + } + }) + .await + .expect("forge timeout"); +} + +/// A reader allowed every endpoint lists both ends of a running session and finds no +/// track, so it cannot learn a session's name to read or forge it. +#[tokio::test] +async fn a_reader_of_everything_lists_no_session() { + let fixture = Fixture::new().await; + let (alice, _alice_origin, _alice_session) = fixture.endpoint("alice", LITE).await; + let (bob, _bob_origin, _bob_session) = fixture.endpoint("bob", LITE).await; + let (mut dialed, mut accepted) = connect_tcp(&alice, &bob).await; + dialed.producer.write_frame(None, &b"secret"[..]).expect("write"); + assert_eq!(read(&mut accepted.consumer).await, b"secret"); + let (track_dialed, track_accepted) = connect_track(&alice, &bob).await; + send(&track_dialed.producer, b"secret"); + assert_eq!(recv(&mut track_accepted.consumer.ordered()).await, b"secret"); + + let (eve_origin, _eve_session) = fixture.client(&fixture.token(".socket/eve", "**"), LITE).await; + for id in ["alice", "bob"] { + let broadcast = kio::clock::timeout(TIMEOUT, eve_origin.consume().request(format!(".socket/{id}"), None)) + .await + .expect("request timeout") + .expect("the endpoint"); + let mut tracks = broadcast.tracks().expect("tracks"); + let event = kio::clock::timeout(TIMEOUT, tracks.next()) + .await + .expect("listing timeout") + .expect("listing"); + assert!(matches!(event, Some(TracksEvent::Synced)), "{id} listed {event:?}"); + assert!( + kio::clock::timeout(SETTLE, tracks.next()).await.is_err(), + "{id} listed a track" + ); + } +}