diff --git a/demo/carla/README.md b/demo/carla/README.md index b43465d6701..55ad677d74d 100644 --- a/demo/carla/README.md +++ b/demo/carla/README.md @@ -3,9 +3,10 @@ Drive simulated trucks from a browser over MoQ. Trucks share one [CARLA](https://carla.org) world and drive themselves until someone takes control. Each truck publishes its cameras and a status track; a web player -watches them on a map and publishes a steering **control** track back. Control -follows the [moq-boy](../boy) pattern: the viewer publishes, the truck -subscribes, and the truck echoes what it accepted on `status`. +watches them on a map, and a driver steers one by dialing the truck's +[socket](../../doc/concept/socket.md) endpoint: its controls and the truck's +acknowledgements travel on a session only the two of them read, while what +every viewer shows stays on `status`. ```text ├── web/ # the player: the element, with map, video, keyboard and steering-wheel input (@moq/demo-carla) @@ -44,7 +45,7 @@ The player alone runs against any relay with `just carla web `. Append `?preview` to the local URL for a fixture-driven layout review that opens no connection; it is removed from production builds. -## Broadcasts +## Broadcasts and sessions ```text carla/{truck}/{camera}.hang <- one media broadcast per camera (hang catalog) @@ -52,16 +53,21 @@ carla/{truck}/{camera}.hang <- one media broadcast per camera (hang catalo hd 720p H.264, for the focused view status JSON Snapshot telemetry -viewer/carla/{truck}/{driverId} <- one broadcast per driver - control JSON Snapshot commands +carla/.socket/{truck} <- the truck's socket endpoint, which drivers dial +carla/.socket/driver-{id} <- one endpoint per driver lease ``` -The worker consumes `viewer/carla/{truck}/` for each truck, fanning in every -driver's `control` track. Cameras are sibling broadcasts, so switching views -never creates a second claim on the same wheel, and the driver scope is -disjoint from every media path. +Each truck accepts `track` sessions on its endpoint. A driver dials from an +endpoint of its own, one session per lease, and sends one control per message; +the truck answers each control it accepted from the lease holder on the same +session. Cameras are sibling broadcasts, so switching views never creates a +second claim on the same wheel. -### Control (player to truck) +Sockets need moq-lite-07 or moq-transport on every hop, so the worker and the +player offer only `moq-lite-07-wip` and `moq-transport-22`; a relay must +accept one of them. + +### Control (player to truck, on the driver's session) ```json { "type": "drive", "sequence": 42, "steering": -1.0..1.0, @@ -72,13 +78,27 @@ disjoint from every media path. { "type": "respawn", "sequence": 45, "timestamps": [...] } ``` -### Status (truck to player) +The truck reads them oldest first, skipping one that falls a heartbeat (250 ms) +behind the newest, so a late control never holds up a newer one, while a burst, +such as a reset right before a drive, still arrives whole. + +### Acknowledgement (truck to player, on the driver's session) + +```json +{ "sequence": 44, "media_pipeline": [ { "label": "received→decoded", "ms": 12 } ] } +``` + +Sent for each `drive`, `estop`, and `reset_estop` the truck accepted from the +lease holder, so the player measures its round trip and its own media pipeline. +A bystander's e-stop, a command the gate refused, or an accepted `respawn` +(confirmed from `status` instead) gets none. + +### Status (truck to every viewer) ```json { "accepted": { "steering": .., "throttle": .., "brake": .., "gear": ".." }, "autonomy": "autonomous|manual", "estopped": true, - "driver": "ab12cd", "estop_by": "ef34gh", "sequence": 44, - "media_pipeline": { "ab12cd": [ { "label": "received→decoded", "ms": 12 } ] }, + "driver": "driver-ab12cd", "estop_by": "driver-ef34gh", "map": "Town10HD_Opt", "pose": { "x": 12.35, "y": -78.9, "yaw": 180.1, "speed": 9.88 }, "measured": { "steering": .., "throttle": .., "brake": .., "gear": ".." }, @@ -104,13 +124,14 @@ collision. The worker's gate (`sim/moq_carla_sim/drive.py`) owns every truck's controls: - **One driver at a time.** The first driver to send a command holds the - lease until its broadcast ends or its 1 s watchdog expires; the player - heartbeats every 250 ms. The lease is bound to the driver's path and the - publisher instance serving it, so a driver that reconnects at the same - path is a new claimant rather than a continuation. -- **E-stop latches** until an explicit reset, from any viewer: the player - latches and resets it through a short-lived safety broadcast without taking - the lease. A latched truck nobody drives is also cleared when it respawns + lease until its session ends or its 1 s watchdog expires; the player + heartbeats every 250 ms. The lease is bound to the driver's session, so a + driver that redials, even from the same endpoint id, is a new claimant + rather than a continuation. +- **E-stop latches** until an explicit reset, from any viewer. The lease + holder sends it on its session; any other viewer dials a short-lived session + of its own, so a bystander's stop neither takes the lease nor waits behind + the holder's controls. A latched truck nobody drives is also cleared when it respawns as wedged, below. - **Gear changes are interlocked**: the gate keeps the current gear unless the command releases throttle and applies some brake. @@ -128,10 +149,15 @@ The worker's gate (`sim/moq_carla_sim/drive.py`) owns every truck's controls: e-stop latched, that has not moved 2 m in 5 minutes of a running world returns to a free spawn point under autonomy, its e-stop cleared. -The lease is cooperative. Anyone who can publish under `viewer/carla/` can -announce a driver path, so it stops two honest drivers fighting over a truck, -not a deliberate takeover. Anything beyond a simulator needs per-driver tokens -scoped to the driver's own path. +The lease is cooperative. Anonymous drivers share one grant, publish +`carla/.socket/driver-*`, so anyone can announce an endpoint under the id that +`status` names as the holder. That never continues the holder's lease, since a +lease is one session, but it can disrupt it: the relay may route the truck's +accept of the holder's next session to the impostor. Truck names must not look +like driver ids, or the same grant would let anyone impersonate a truck; the +worker refuses one. So the lease stops two honest drivers fighting over a +truck, not a deliberate disruption. Anything beyond a simulator needs +per-driver tokens scoped to the driver's own endpoint. ## The simulator @@ -193,8 +219,9 @@ checkout. just carla venv just carla service-install -# The units read this. MOQ_URL carries a token allowed to publish `carla/**` -# and subscribe `viewer/carla/**`; keeping it here keeps it off the command +# The units read this. MOQ_URL carries a token allowed to publish `carla/**`, +# which covers the trucks' endpoints, and subscribe `carla/.socket/driver-*`, +# the endpoints drivers dial from; keeping it here keeps it off the command # line. CARLA_IMAGE is the tag the moq-dev/carla build printed, since the worker # refuses a stock simulator. cat >~/.config/moq/carla.env <<'EOF' diff --git a/demo/carla/sim/moq_carla_sim/drive.py b/demo/carla/sim/moq_carla_sim/drive.py index fde62c302a9..48c530f4793 100644 --- a/demo/carla/sim/moq_carla_sim/drive.py +++ b/demo/carla/sim/moq_carla_sim/drive.py @@ -94,15 +94,16 @@ class Command: `drive`, `estop`, `reset_estop`, and `respawn` arrive on the wire; `driver_left` and `watchdog_expired` are synthesized locally. - `driver_id` is the driver's path segment, which `status` echoes. `instance` - names the run that sent it, one per publisher instance serving the path: the - same id under another instance is another run of the driver, so it is a - different driver for the lease. + `driver_id` is the driver's socket endpoint id, which `status` echoes. + `session` is the socket session that carried it, one per lease the player + takes: the same id on another session is a different driver for the lease. + The session id is the secret that keeps a session to its two ends, so it + never leaves this process. """ kind: CommandKind driver_id: str = "" - instance: str | None = None + session: str | None = None sequence: int = 0 state: DriveState | None = None timestamps: tuple[TimestampEntry, ...] = () @@ -114,19 +115,19 @@ class Drive: The first driver to send a command owns the truck until it disconnects or its command lease expires. Other drivers cannot silently steal control. - The lease binds to the driver's broadcast path and the run serving it, so a - driver whose broadcast ends or restarts loses it. - It is still cooperative, not enforced: the holder's id is echoed on the - public `status` track, so anyone allowed to publish there can announce at - that path under another instance, which ends the holder's broadcast and frees - the lease for the newcomer. This stops two honest drivers colliding; it - does not stop a deliberate takeover. + The lease binds to the driver's socket session, so a driver whose session + ends loses it, and another session claiming the same driver id is another + driver. It is still cooperative, not enforced: anonymous drivers share one + grant, so anyone can announce an endpoint under the holder's id, which the + public `status` track echoes. Their own sessions never continue the + holder's lease, but they can disrupt it: a relay may route the truck's + accept of the holder's next session to the impostor instead. This stops two + honest drivers colliding; it does not stop a deliberate disruption. """ def __init__(self) -> None: self._driver: str | None = None - self._instance: str | None = None - self._sequence: int | None = None + self._session: str | None = None self._state = DriveState.stopped() self._takeover_pedals: tuple[float, float] | None = None self._estopped = False @@ -136,10 +137,6 @@ def __init__(self) -> None: def driver(self) -> str | None: return self._driver - @property - def sequence(self) -> int | None: - return self._sequence - @property def state(self) -> DriveState: return self._state @@ -153,8 +150,8 @@ def estop_by(self) -> str | None: return self._estop_by def holds(self, command: Command) -> bool: - """Whether `command` comes from the run of the driver holding the lease.""" - return self._driver is not None and (self._driver, self._instance) == (command.driver_id, command.instance) + """Whether `command` comes from the session holding the lease.""" + return self._driver is not None and (self._driver, self._session) == (command.driver_id, command.session) def accepted(self) -> Accepted: return Accepted(state=self._state, estopped=self._estopped) @@ -166,8 +163,7 @@ def apply(self, command: Command) -> bool: return False claiming = self._driver is None self._driver = command.driver_id - self._instance = command.instance - self._sequence = command.sequence + self._session = command.session state = (command.state or DriveState.stopped()).sanitized() pedals = (state.throttle, state.brake) if claiming: @@ -188,11 +184,7 @@ def apply(self, command: Command) -> bool: # latches and brings the truck to a stop, but it must never transfer # the driving lease to its sender. Otherwise a bystander could # e-stop, then reset_estop, and drive off with a lease they never - # legitimately claimed. Only refresh the acknowledgement sequence - # when the sender already holds the lease, so a bystander's e-stop - # cannot masquerade as the authoritative command either. - if self.holds(command): - self._sequence = command.sequence + # legitimately claimed. self._state = DriveState.stopped() self._estopped = True if not was_estopped: @@ -202,9 +194,7 @@ def apply(self, command: Command) -> bool: if not self._estopped: return False # Reset clears the global latch without claiming or transferring the - # lease. Only the holder's sequence is authoritative for that lease. - if self.holds(command): - self._sequence = command.sequence + # lease. self._state = DriveState.stopped() self._estopped = False self._estop_by = None @@ -228,7 +218,6 @@ def _can_shift(state: DriveState) -> bool: def _clear_driver(self) -> None: self._driver = None - self._instance = None - self._sequence = None + self._session = None self._state = DriveState.stopped() self._takeover_pedals = None diff --git a/demo/carla/sim/moq_carla_sim/main.py b/demo/carla/sim/moq_carla_sim/main.py index 88585fb09d0..2993e64e914 100644 --- a/demo/carla/sim/moq_carla_sim/main.py +++ b/demo/carla/sim/moq_carla_sim/main.py @@ -37,6 +37,15 @@ # A local development relay, which needs no token. DEFAULT_URL = "http://localhost:4443" +# Drivers dial each truck's socket endpoint, which only these versions carry: +# an endpoint is a hidden broadcast, and moq-lite before 07 hides it from every +# peer. The web player offers the same list. +VERSIONS = ("moq-lite-07-wip", "moq-transport-22") + +# The id form anonymous drivers may publish endpoints under. A truck named like +# one could be impersonated by any of them. +DRIVER_ID_PREFIX = "driver-" + def default_trucks() -> list[str]: trucks = json.loads(TRUCKS_FILE.read_text()) @@ -80,6 +89,9 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: args = parser.parse_args(argv) if not args.trucks: args.trucks = default_trucks() + for truck in args.trucks: + if truck.startswith(DRIVER_ID_PREFIX): + parser.error(f"truck {truck!r} is named like a driver, whose endpoints anyone may publish") return args @@ -110,11 +122,6 @@ def broadcast_path(truck: str, camera: str) -> str: return f"carla/{truck}/{camera}.hang" -def driver_prefix(truck: str) -> str: - """Where a truck's drivers publish, each at `{prefix}/{driverId}`.""" - return f"viewer/carla/{truck}" - - # The one task allowed to end the run without failing it. SHUTDOWN = "shutdown" @@ -238,7 +245,7 @@ async def run(args: argparse.Namespace) -> None: # shares the connection. events = asyncio.create_task(bridge.run(), name="bridge") try: - async with moq.connect(args.url) as client: + async with moq.connect(args.url, versions=VERSIONS) as client: try: for truck in world.trucks: route = client.route(broadcast_path(truck.name, args.camera), hidden=True) @@ -248,16 +255,17 @@ async def run(args: argparse.Namespace) -> None: session = TruckSession(truck, route, video, args.location, map_name, world.clock) sessions.append(session) log.info( - "publishing %s, consuming %s/*", + "publishing %s, accepting drivers at %s", broadcast_path(truck.name, args.camera), - driver_prefix(truck.name), + session.drivers.path, ) + relay = client.session + if relay is None: + raise RuntimeError("relay session is not connected") + origin = relay.publish() tasks = [ - asyncio.create_task( - session.run(client, driver_prefix(truck.name)), - name=f"truck:{truck.name}", - ) + asyncio.create_task(session.run(origin), name=f"truck:{truck.name}") for session, truck in zip(sessions, world.trucks) ] diff --git a/demo/carla/sim/moq_carla_sim/publish.py b/demo/carla/sim/moq_carla_sim/publish.py index 34f334ce683..a7f36b14e44 100644 --- a/demo/carla/sim/moq_carla_sim/publish.py +++ b/demo/carla/sim/moq_carla_sim/publish.py @@ -1,19 +1,22 @@ """MoQ wiring: video and status out, driver control in. -The broadcast layout is the one `demo/carla/README.md` documents and the web -player speaks: +The layout is the one `demo/carla/README.md` documents and the web player +speaks: carla/{truck}/{camera}.hang video (sd, hd) + status - viewer/carla/{truck}/{driverId} control + carla/.socket/{truck} the truck's socket endpoint + carla/.socket/driver-{id} each driver's socket endpoint """ from __future__ import annotations import asyncio -import itertools +import contextlib +import json import logging import math from dataclasses import dataclass +from datetime import timedelta from typing import Any import moq @@ -98,8 +101,6 @@ class Status: estop_by: str | None autonomous: bool driver: str | None - sequence: int | None - media_pipeline: dict[str, list[dict[str, Any]]] location: str | None map_name: str | None telemetry: Telemetry | None @@ -128,7 +129,6 @@ def status_payload(status: Status) -> dict[str, Any]: "capabilities": ["respawn"], "autonomy": "autonomous" if status.autonomous else "manual", "estopped": status.estopped, - "media_pipeline": status.media_pipeline, } # Omitted rather than null when absent; the player reads a missing field # as unset. @@ -136,8 +136,6 @@ def status_payload(status: Status) -> dict[str, Any]: payload["driver"] = status.driver if status.estop_by is not None: payload["estop_by"] = status.estop_by - if status.sequence is not None: - payload["sequence"] = status.sequence if status.location is not None: payload["location"] = status.location if status.map_name is not None: @@ -215,7 +213,7 @@ async def unused(self) -> None: # The player's pipeline stages. Anything else a driver sends is dropped, so an -# anonymous lease holder cannot grow the status the truck echoes. +# anonymous lease holder cannot grow the reply the truck echoes. TIMESTAMP_LABELS = ("received", "decoded", "rendered") @@ -235,8 +233,8 @@ def _parse_timestamps(raw: Any) -> tuple[TimestampEntry, ...]: return tuple(entries.values()) -def parse_command(driver_id: str, message: Any, instance: str | None = None) -> Command | None: - """Turn one `control` track message from a driver run into a Command. +def parse_command(driver_id: str, message: Any, session: str | None = None) -> Command | None: + """Turn one control message from a driver's session into a Command. Returns None for anything unrecognized. A driver is an untrusted publisher, so a malformed message is dropped here rather than reaching the gate: the @@ -262,7 +260,7 @@ def parse_command(driver_id: str, message: Any, instance: str | None = None) -> return Command( kind="drive", driver_id=driver_id, - instance=instance, + session=session, sequence=sequence, state=DriveState( steering=_number(message.get("steering")), @@ -274,7 +272,7 @@ def parse_command(driver_id: str, message: Any, instance: str | None = None) -> ) if kind in ("estop", "reset_estop", "respawn"): - return Command(kind=kind, driver_id=driver_id, instance=instance, sequence=sequence, timestamps=timestamps) + return Command(kind=kind, driver_id=driver_id, session=session, sequence=sequence, timestamps=timestamps) return None @@ -295,114 +293,124 @@ def _number_or_none(value: Any) -> float | None: return number if math.isfinite(number) else None -async def watch_drivers( - client: moq.Client, - prefix: str, - commands: asyncio.Queue[Command], -) -> None: - """Discover driver broadcasts at `{prefix}/{driverId}` and fan their commands in. +# Every endpoint the trucks and their drivers talk through sits under this +# prefix, as `{prefix}/.socket/{id}`. +SOCKET_PREFIX = "carla" - Runs until cancelled. Each driver path the listing reports present gets its own - task, so one slow or stuck driver cannot hold up the others. That task follows - the path across the driver's runs, one publisher instance each; a run that - another takes over, or whose path goes absent, ends, and with it its lease. - """ - drivers: dict[str, asyncio.Task[None]] = {} +# How far a control may fall behind the newest before it is skipped: one player +# heartbeat, by which time a newer control has superseded it. Within it, a burst +# arrives whole and in order, so a reset never loses to the drive right after it. +CONTROL_DELAY = timedelta(milliseconds=250) - try: - async with client.announced(prefix, filter="*") as announced: - async for event in announced: - # Both variants carry `announce`; uniffi's stubs type only the enum. - path = event.announce.prefix # pyright: ignore[reportAttributeAccessIssue] - driver_id = path.removeprefix(prefix + "/") - # The filter admits one segment below the prefix; anything else - # is a route that only overlaps it, not a driver's broadcast. - if driver_id == path or not driver_id or "/" in driver_id: - continue - if isinstance(event, moq.AnnounceEventAbsent): - task = drivers.pop(driver_id, None) - if task is not None: - task.cancel() - continue - drivers[driver_id] = asyncio.create_task( - _follow_driver(driver_id, client, path, commands), - name=f"driver:{driver_id}", - ) - finally: - for task in drivers.values(): - task.cancel() - - -# Names each driver run for the lease. The bindings hand out a run's broadcast -# without its route instance, and a process-wide count keeps a run's late -# `driver_left` from releasing a later run of the same driver. -_runs = itertools.count(1) - - -async def _follow_driver( - driver_id: str, - client: moq.Client, - path: str, - commands: asyncio.Queue[Command], -) -> None: - """Pump each run of one driver in turn, ending a run once another takes over or it goes offline.""" - run: asyncio.Task[None] | None = None - try: - async with client.follow(path) as following: - while True: - try: - broadcast = await following.available() - # A refusal leaves the follow waiting for another run; only shutdown ends it. - except moq.Error as err: # pyright: ignore[reportGeneralTypeIssues] - if moq.is_shutdown(err): - raise - log.info("driver %s refused: %s", driver_id, err) - continue - if run is not None: - run.cancel() - run = None - if broadcast is None: - continue - instance = str(next(_runs)) - run = asyncio.create_task( - _pump_driver(driver_id, instance, broadcast, commands), - name=f"driver:{driver_id}:{instance}", - ) - # uniffi's stubs declare the error a plain class; at runtime it is an Exception. - except moq.Error as err: # pyright: ignore[reportGeneralTypeIssues] - log.info("driver %s follow ended: %s", driver_id, err) - finally: - if run is not None: - run.cancel() - - -async def _pump_driver( - driver_id: str, - instance: str, - broadcast: moq.BroadcastConsumer, - commands: asyncio.Queue[Command], -) -> None: - """Feed one driver run's `control` track into the command queue. - - Always emits `driver_left` on the way out, however it leaves, so the gate - releases a lease even when the track ends abnormally. + +class Drivers: + """A truck's socket endpoint: accepts each driver's session, fans its controls + into the command queue, and answers the driver on the same session. + + Each driver dials from an endpoint of its own, one `track` session per lease, + whose messages are one control per group. Replies go back the same way, so a + driver's acknowledgements never reach anyone else. """ - log.info("driver %s connected (run %s)", driver_id, instance) + + def __init__(self, truck: str, commands: asyncio.Queue[Command]) -> None: + self._truck = truck + self._commands = commands + # Keyed like the lease, by driver and session, since a dialer picks its own session id. + self._replies: dict[tuple[str, str], moq.TrackProducer] = {} + + @property + def path(self) -> str: + """Where the truck's endpoint is announced.""" + return f"{SOCKET_PREFIX}/.socket/{self._truck}" + + def reply(self, command: Command, message: dict[str, Any]) -> None: + """Send `message` on the session that carried `command`, if it is still open. + + A session that has just ended is not an error: its pump is already on + its way to releasing the lease. + """ + if command.session is None: + return + producer = self._replies.get((command.driver_id, command.session)) + if producer is None: + return + try: + producer.write_frame(json.dumps(message, separators=(",", ":")).encode()) + # uniffi's stubs declare the error a plain class; at runtime it is an Exception. + except moq.Error as err: # pyright: ignore[reportGeneralTypeIssues] + log.info("%s: reply failed: %s", self._truck, err) + + async def run(self, origin: moq.OriginProducer) -> None: + """Accept drivers until cancelled, each session in a task of its own, + so one slow or stuck driver cannot hold up the others.""" + sessions: set[asyncio.Task[None]] = set() + try: + async with moq.socket.Endpoint(origin, SOCKET_PREFIX, self._truck) as endpoint: + while True: + incoming = await endpoint.accept_track() + task = asyncio.create_task(self._serve(incoming), name=f"driver:{incoming.peer}") + sessions.add(task) + task.add_done_callback(sessions.discard) + finally: + for task in sessions: + task.cancel() + await asyncio.gather(*sessions, return_exceptions=True) + + async def _serve(self, incoming: moq.socket.IncomingTrack) -> None: + """Feed one driver session's controls into the command queue. + + Emits `driver_left` once the session was open, however it ends, so the + gate releases a lease even when the session fails. Cancellation is the + exception: that is the truck shutting down, whose gate no longer reads + the queue, so waiting to enqueue there would hang the shutdown. + """ + driver = incoming.peer + try: + stream = await incoming.accept() + # A dial forged in another endpoint's name fails here, as does one whose + # dialer gave up first; neither ever reached the gate. + except moq.Error as err: # pyright: ignore[reportGeneralTypeIssues] + log.info("driver %s not accepted: %s", driver, err) + return + + session = stream.session + key = (driver, session) + if key in self._replies: + # It would share the open session's lease and replies; refuse it. + log.info("driver %s dialed a session id it already has open", driver) + stream.producer.finish() + return + stream.consumer.update(moq.Subscription(delay=moq.Delay(max=CONTROL_DELAY), order=moq.Order.ASCENDING)) + self._replies[key] = stream.producer + log.info("driver %s connected", driver) + try: + while (frame := await stream.consumer.read_frame()) is not None: + command = parse_command(driver, _decode(frame.payload), session) + if command is not None: + await self._commands.put(command) + except moq.Error as err: # pyright: ignore[reportGeneralTypeIssues] + log.info("driver %s session ended: %s", driver, err) + finally: + del self._replies[key] + # Closes this side too; a session the peer reset has nothing left to close. + with contextlib.suppress(moq.Error): # pyright: ignore[reportGeneralTypeIssues] + stream.producer.finish() + log.info("driver %s disconnected", driver) + await self._commands.put(Command(kind="driver_left", driver_id=driver, session=session)) + + +def _decode(payload: bytes) -> Any: + """One control message, or None for bytes that are not JSON, which parse_command drops.""" try: - control = moq.json.SnapshotConsumer(broadcast, "control") - async for message in control: - command = parse_command(driver_id, message.value, instance) - if command is not None: - await commands.put(command) - # uniffi's stubs declare the error a plain class; at runtime it is an Exception. - except moq.Error as err: # pyright: ignore[reportGeneralTypeIssues] - log.info("driver %s track ended: %s", driver_id, err) - finally: - log.info("driver %s disconnected (run %s)", driver_id, instance) - await commands.put(Command(kind="driver_left", driver_id=driver_id, instance=instance)) + return json.loads(payload) + except ValueError: + return None __all__ = [ + "CONTROL_DELAY", + "SOCKET_PREFIX", + "Drivers", "StatusPublisher", "Collision", "Motion", @@ -411,5 +419,4 @@ async def _pump_driver( "Telemetry", "parse_command", "status_payload", - "watch_drivers", ] diff --git a/demo/carla/sim/moq_carla_sim/runtime.py b/demo/carla/sim/moq_carla_sim/runtime.py index 51aceae2e6b..af553f0a2c6 100644 --- a/demo/carla/sim/moq_carla_sim/runtime.py +++ b/demo/carla/sim/moq_carla_sim/runtime.py @@ -1,8 +1,8 @@ """The per-truck control loop. -One `TruckSession` per truck: it owns the safety gate, the MoQ broadcast, and -the truck's bridge stream, whose compressed video the bridge publishes into -that broadcast. +One `TruckSession` per truck: it owns the safety gate, the MoQ broadcast, the +socket endpoint its drivers dial, and the truck's bridge stream, whose +compressed video the bridge publishes into that broadcast. """ from __future__ import annotations @@ -15,7 +15,7 @@ import moq from .drive import Accepted, Command, Drive, TimestampEntry -from .publish import Pose, Status, StatusPublisher, Telemetry, status_payload, watch_drivers +from .publish import Drivers, Pose, Status, StatusPublisher, Telemetry, status_payload from .world import drive_hold, status_hold if TYPE_CHECKING: @@ -112,7 +112,7 @@ def __init__( # memory-growth path under one that floods; a full queue instead blocks # that driver's own task. self._commands: asyncio.Queue[Command] = asyncio.Queue(maxsize=COMMAND_QUEUE_DEPTH) - self._media_pipeline: dict[str, list[dict[str, Any]]] = {} + self._drivers = Drivers(truck.name, self._commands) self._deadline: float | None = None self._stuck = StuckDetector() # The route starts hidden; the player discovers trucks by announcement, @@ -130,19 +130,21 @@ def stop_video(self) -> None: def commands(self) -> asyncio.Queue[Command]: return self._commands - async def run(self, client: moq.Client, drivers: str) -> None: + @property + def drivers(self) -> Drivers: + return self._drivers + + async def run(self, origin: moq.OriginProducer) -> None: """Consume driver commands and actuate until cancelled. - Driver discovery runs in a task group rather than a bare task, so a + The driver endpoint runs in a task group rather than a bare task, so a failure there fails the session. A detached task's exception is never retrieved: the truck would keep publishing video and status while silently accepting no driver ever again, which reads as healthy. """ self._publish() async with asyncio.TaskGroup() as group: - watcher = group.create_task( - watch_drivers(client, drivers, self._commands), name=f"drivers:{self._truck.name}" - ) + watcher = group.create_task(self._drivers.run(origin), name=f"drivers:{self._truck.name}") # Video demand is what renders the world for this truck, and the # bridge follows the same demand to pick its camera. demand = group.create_task( @@ -175,35 +177,31 @@ async def _loop(self) -> None: self._publish() continue accepted = self._drive.apply(command) - changed = accepted self._hold_clock() + if not accepted: + continue - if command.kind in ("drive", "estop", "reset_estop"): - if accepted and self._drive.holds(command): - # Only the lease holder's accepted traffic refreshes the - # watchdog and its pipeline timing. A bystander's e-stop - # still latches, but must not extend the holder's lease, and - # an e-stopped holder's refused drive frames must not keep - # the lease alive forever. - self._media_pipeline[command.driver_id] = latency_breakdown(command.timestamps) - self._deadline = asyncio.get_running_loop().time() + COMMAND_TIMEOUT - elif command.kind == "driver_left" and accepted: - # Only the holder's run has a pipeline entry, so another run of - # the same driver id leaving must not drop it. - self._media_pipeline.pop(command.driver_id, None) + # Only the lease holder's accepted traffic refreshes the watchdog + # and is acknowledged. A bystander's e-stop still latches, but must + # not extend the holder's lease, and an e-stopped holder's refused + # drive frames must not keep the lease alive forever. + acknowledged = command.kind in ("drive", "estop", "reset_estop") and self._drive.holds(command) + if acknowledged: + self._deadline = asyncio.get_running_loop().time() + COMMAND_TIMEOUT + elif command.kind == "driver_left": self._deadline = None - elif command.kind == "watchdog_expired" and accepted: + elif command.kind == "watchdog_expired": self._deadline = None - self._media_pipeline.clear() log.warning("%s: control watchdog expired, returning to autonomy", self._truck.name) - if not changed: - continue - # Actuate before publishing, so the world and the status echo can # never disagree about what was accepted. self._actuate() self._publish() + if acknowledged: + self._drivers.reply( + command, {"sequence": command.sequence, "media_pipeline": latency_breakdown(command.timestamps)} + ) async def _report_video(self) -> None: """Log the published frame rate per rendition. @@ -283,7 +281,6 @@ def _respawn(self) -> None: self._truck.respawn() self._drive = Drive() self._deadline = None - self._media_pipeline.clear() self._stuck.reset() self._hold_clock() @@ -344,8 +341,6 @@ def _publish(self, telemetry: Telemetry | None = None) -> None: estop_by=self._drive.estop_by, autonomous=self._truck.autonomous, driver=self._drive.driver, - sequence=self._drive.sequence, - media_pipeline=self._media_pipeline, location=self._location, map_name=self._map_name, telemetry=self._truck.telemetry() if telemetry is None else telemetry, diff --git a/demo/carla/sim/systemd/install-user-service.sh b/demo/carla/sim/systemd/install-user-service.sh index 129ea0f4f3c..9ea99676e51 100755 --- a/demo/carla/sim/systemd/install-user-service.sh +++ b/demo/carla/sim/systemd/install-user-service.sh @@ -44,8 +44,8 @@ Before starting, write the relay settings to: CARLA_IMAGE=localhost/moq-carla: The token is a demo-project JWT scoped to publish carla/** and subscribe -viewer/carla/**. For local development, MOQ_URL=http://localhost:4443 needs -no token. +carla/.socket/driver-*, the endpoints drivers dial from. For local +development, MOQ_URL=http://localhost:4443 needs no token. Then run: just carla service-enable diff --git a/demo/carla/sim/systemd/install-user-service.test.sh b/demo/carla/sim/systemd/install-user-service.test.sh index fcd3de472dd..b539a046740 100755 --- a/demo/carla/sim/systemd/install-user-service.test.sh +++ b/demo/carla/sim/systemd/install-user-service.test.sh @@ -45,7 +45,7 @@ CARLA_SYSTEMD_USER_DIR="$tmp/units" \ grep -Fxq ' MOQ_URL=https://cdn.moq.dev/demo?jwt=' "$tmp/install.out" grep -Fxq ' CARLA_IMAGE=localhost/moq-carla:' "$tmp/install.out" grep -Fq 'scoped to publish carla/** and subscribe' "$tmp/install.out" -grep -Fq 'viewer/carla/**. For local development, MOQ_URL=http://localhost:4443 needs' "$tmp/install.out" +grep -Fq 'carla/.socket/driver-*, the endpoints drivers dial from. For local' "$tmp/install.out" test "$(stat -c %a "$tmp/units")" = 700 test "$(stat -c %a "$tmp/config")" = 700 test "$(stat -c %a "$tmp/units/carla-sim.service")" = 600 diff --git a/demo/carla/sim/tests/conftest.py b/demo/carla/sim/tests/conftest.py index 1aa31b9689e..1c9d18fa0b0 100644 --- a/demo/carla/sim/tests/conftest.py +++ b/demo/carla/sim/tests/conftest.py @@ -10,21 +10,6 @@ class MoqError(Exception): pass -class AnnounceEvent: - """One announcement event; the subclass names which kind, as in moq's.""" - - def __init__(self, announce: SimpleNamespace) -> None: - self.announce = announce - - -class AnnounceEventPresent(AnnounceEvent): - pass - - -class AnnounceEventAbsent(AnnounceEvent): - pass - - class SnapshotProducer: """Records every published value; demand comes from the wrapped track.""" @@ -39,30 +24,18 @@ def update(self, value: object, timestamp: object) -> None: self.values.append(value) -class SnapshotConsumer: - """Reads the named fake track of a broadcast, wrapping its parsed values in timed records, - like the runtime consumer.""" - - def __init__(self, broadcast: object, name: str, *, priority: int = 0) -> None: - self._track = broadcast.tracks[name] - - async def __aiter__(self): - async for value in self._track: - yield SimpleNamespace(value=value, at=None) - - moq = ModuleType("moq") moq.Error = MoqError -# The stub has no shutdown errors: every `MoqError` it raises is a failure. -moq.is_shutdown = lambda err: False -moq.AnnounceEventPresent = AnnounceEventPresent -moq.AnnounceEventAbsent = AnnounceEventAbsent +moq.Subscription = lambda **kwargs: SimpleNamespace(**kwargs) +moq.Delay = lambda **kwargs: SimpleNamespace(**kwargs) +moq.Order = SimpleNamespace(ASCENDING="ascending") +# Tests that run a truck's endpoint script it in place of this. +moq.socket = ModuleType("moq.socket") moq.Timestamp = lambda value, scale: SimpleNamespace(value=value, scale=scale) moq.TrackInfo = lambda **kwargs: SimpleNamespace(**kwargs) moq_json = ModuleType("moq.json") moq_json.SnapshotProducer = SnapshotProducer -moq_json.SnapshotConsumer = SnapshotConsumer moq.json = moq_json moq_media = ModuleType("moq.media") diff --git a/demo/carla/sim/tests/test_drive.py b/demo/carla/sim/tests/test_drive.py index d639f61417d..c54d61cb4e0 100644 --- a/demo/carla/sim/tests/test_drive.py +++ b/demo/carla/sim/tests/test_drive.py @@ -35,25 +35,21 @@ def test_first_driver_keeps_control() -> None: assert state.state.throttle == 0.6 -def test_the_lease_belongs_to_one_run_of_a_driver() -> None: - """A driver id announced under another instance is another driver: it cannot - steer while the first run holds the lease, nor release it.""" +def test_the_lease_belongs_to_one_session_of_a_driver() -> None: + """A driver id on another session is another driver: it cannot steer while + the first session holds the lease, nor release it.""" state = Drive() - first = Command( - kind="drive", driver_id="abc", instance="run-1", sequence=1, state=DriveState(throttle=0.5, brake=0.0) - ) - second = Command( - kind="drive", driver_id="abc", instance="run-2", sequence=1, state=DriveState(throttle=1.0, brake=0.0) - ) - assert state.apply(Command(kind="drive", driver_id="abc", instance="run-1")) + first = Command(kind="drive", driver_id="abc", session="s1", sequence=1, state=DriveState(throttle=0.5, brake=0.0)) + second = Command(kind="drive", driver_id="abc", session="s2", sequence=1, state=DriveState(throttle=1.0, brake=0.0)) + assert state.apply(Command(kind="drive", driver_id="abc", session="s1")) assert state.apply(first) assert not state.apply(second) assert state.state.throttle == 0.5 - assert not state.apply(Command(kind="driver_left", driver_id="abc", instance="run-2")) + assert not state.apply(Command(kind="driver_left", driver_id="abc", session="s2")) assert state.driver == "abc" - # The first run leaving frees the lease for the second. - assert state.apply(Command(kind="driver_left", driver_id="abc", instance="run-1")) + # The first session leaving frees the lease for the second. + assert state.apply(Command(kind="driver_left", driver_id="abc", session="s1")) assert state.driver is None assert state.apply(second) assert state.holds(second) @@ -77,7 +73,6 @@ def test_watchdog_stops_and_releases_driver() -> None: assert state.apply(drive("driver", 1, 1.0)) assert state.apply(Command(kind="watchdog_expired")) assert state.driver is None - assert state.sequence is None assert state.state == DriveState.stopped() @@ -104,40 +99,24 @@ def test_estop_from_a_bystander_latches_without_transferring_the_lease() -> None assert state.estopped assert state.estop_by == "second" - # ...but the lease stays with "first", and the bystander's sequence is not - # recorded as the authoritative command. + # ...but the lease stays with "first". assert state.driver == "first" - assert state.sequence == 1 - # The holder heartbeat is acknowledged without replacing the actor that - # originally latched the stop. + # The holder's own e-stop does not replace the actor that originally + # latched the stop. assert state.apply(estop("first", 3)) assert state.estop_by == "second" - assert state.sequence == 3 + assert state.holds(estop("first", 3)) - # Anyone may clear the global latch, but doing so neither transfers the - # lease nor records the bystander's sequence as authoritative. + # Anyone may clear the global latch, but doing so does not transfer the + # lease. assert state.apply(reset("second", 4)) assert not state.estopped assert state.estop_by is None assert state.driver == "first" - assert state.sequence == 3 assert not state.apply(drive("second", 5, 1.0)) -def test_estop_from_holder_refreshes_the_acknowledgement_sequence() -> None: - state = Drive() - assert state.apply(drive("holder", 5, 0.3)) - assert state.sequence == 5 - - # The holder's own e-stop is acknowledged (sequence advances) and it keeps - # the lease. - assert state.apply(estop("holder", 6)) - assert state.estopped - assert state.sequence == 6 - assert state.driver == "holder" - - def test_axes_are_clamped_into_range() -> None: state = Drive() assert state.apply(Command(kind="drive", driver_id="driver", sequence=1)) diff --git a/demo/carla/sim/tests/test_main.py b/demo/carla/sim/tests/test_main.py index d999629431e..8ec5a54bffe 100644 --- a/demo/carla/sim/tests/test_main.py +++ b/demo/carla/sim/tests/test_main.py @@ -6,7 +6,8 @@ import pytest from moq_carla_sim import main as main_module -from moq_carla_sim.main import broadcast_path, default_trucks, driver_prefix, parse_args +from moq_carla_sim.main import broadcast_path, default_trucks, parse_args +from moq_carla_sim.publish import Drivers from moq_carla_sim.watchdog import Heartbeat @@ -37,7 +38,13 @@ def test_relay_url_defaults_to_the_local_relay(monkeypatch: pytest.MonkeyPatch) def test_paths_match_the_player() -> None: assert broadcast_path("truck1", "front") == "carla/truck1/front.hang" - assert driver_prefix("truck1") == "viewer/carla/truck1" + assert Drivers("truck1", asyncio.Queue()).path == "carla/.socket/truck1" + + +def test_a_truck_named_like_a_driver_is_refused() -> None: + """Anonymous drivers may publish `driver-*` endpoints, so such a truck could be impersonated.""" + with pytest.raises(SystemExit): + parse_args(["--url", "http://localhost:4443", "--truck", "driver-1"]) def test_log_level_configures_python_and_rust(monkeypatch: pytest.MonkeyPatch) -> None: @@ -188,7 +195,7 @@ def destroy(self) -> None: class FakeConnect: async def __aenter__(self): events.append("session.open") - return SimpleNamespace() + return SimpleNamespace(session=SimpleNamespace(publish=SimpleNamespace)) async def __aexit__(self, *_exc) -> None: events.append("session.close") @@ -209,7 +216,13 @@ async def fake_connect_bridge(_path: str) -> FakeBridge: monkeypatch.setattr(main_module.World, "connect", staticmethod(lambda *_args: FakeWorld())) monkeypatch.setattr(main_module, "connect_bridge", fake_connect_bridge) - monkeypatch.setattr(main_module.moq, "connect", lambda _url: FakeConnect(), raising=False) + + def fake_connect(_url: str, *, versions: tuple[str, ...]) -> FakeConnect: + # Drivers dial the trucks' endpoints, which only these versions carry. + assert versions == ("moq-lite-07-wip", "moq-transport-22") + return FakeConnect() + + monkeypatch.setattr(main_module.moq, "connect", fake_connect, raising=False) monkeypatch.setattr(main_module, "supervise", fake_supervise) asyncio.run(main_module.run(parse_args(["--url", "http://localhost:4443"]))) diff --git a/demo/carla/sim/tests/test_publish.py b/demo/carla/sim/tests/test_publish.py index c4736d765ab..9b41c3d3b74 100644 --- a/demo/carla/sim/tests/test_publish.py +++ b/demo/carla/sim/tests/test_publish.py @@ -1,4 +1,5 @@ import asyncio +import json from types import SimpleNamespace import moq @@ -25,8 +26,6 @@ def status(**overrides) -> Status: estop_by=None, autonomous=False, driver=None, - sequence=None, - media_pipeline={}, location=None, map_name=None, telemetry=None, @@ -89,12 +88,15 @@ def test_respawn_command_is_carla_only_control_input() -> None: def test_status_names_the_driver_that_latched_estop() -> None: - payload = status_payload(status(estopped=True, estop_by="bystander", driver="holder", sequence=4)) + payload = status_payload(status(estopped=True, estop_by="bystander", driver="holder")) assert payload["driver"] == "holder" assert payload["estop_by"] == "bystander" assert payload["mode"] == "simulated" assert payload["capabilities"] == ["respawn"] + # Acknowledgements are the holder's alone, on its own session. + assert "sequence" not in payload + assert "media_pipeline" not in payload def test_status_carries_the_map_and_a_rounded_pose() -> None: @@ -176,113 +178,98 @@ def closed(_payload: object, _timestamp: object) -> None: publisher.publish({}) -class Scripted: - """A scripted async source. `send` returns once the reader has handled the - item and come back for the next one.""" +class Session: + """One accepted driver session: controls the driver sends, replies the truck writes.""" - def __init__(self) -> None: - self._items: asyncio.Queue = asyncio.Queue() - self._idle = asyncio.Event() + def __init__(self, id: str) -> None: + self.session = id + self.consumer = self + self.producer = self + self.subscription = None + self.replies: list[dict] = [] + self.finished = False + self._frames: asyncio.Queue = asyncio.Queue() - async def next(self): - if self._items.empty(): - self._idle.set() - item = await self._items.get() - self._idle.clear() - return item + def send(self, message) -> None: + payload = message if isinstance(message, bytes) else json.dumps(message).encode() + self._frames.put_nowait(SimpleNamespace(payload=payload)) - async def send(self, item) -> None: - self._idle.clear() - self._items.put_nowait(item) - await self._idle.wait() + def end(self, error: BaseException | None = None) -> None: + self._frames.put_nowait(error) + def update(self, subscription) -> None: + self.subscription = subscription -class Announcements(Scripted): - """The scripted listing of driver paths.""" + async def read_frame(self): + frame = await self._frames.get() + if isinstance(frame, BaseException): + raise frame + return frame - async def __aenter__(self): - return self + def write_frame(self, payload: bytes) -> None: + if self.finished: + raise moq.Error("session closed") + self.replies.append(json.loads(payload)) - async def __aexit__(self, *_exc) -> None: - pass - - def __aiter__(self): - return self + def finish(self) -> None: + self.finished = True - async def __anext__(self): - return await self.next() - async def present(self, path: str) -> None: - await self.send(moq.AnnounceEventPresent(SimpleNamespace(prefix=path))) +class Incoming: + """A dial claiming `peer`; accepting it fails the return-routability check when forged.""" - async def absent(self, path: str) -> None: - await self.send(moq.AnnounceEventAbsent(SimpleNamespace(prefix=path))) + def __init__(self, peer: str, session: Session | None) -> None: + self.peer = peer + self._session = session + async def accept(self) -> Session: + if self._session is None: + raise moq.Error("not dialed") + return self._session -class Following(Scripted): - """A scripted follow of one path: each `run` hands out another instance, `offline` a `None`, - and `refuse` an error.""" - - async def __aenter__(self): - return self - - async def __aexit__(self, *_exc) -> None: - pass - async def available(self): - item = await self.next() - if isinstance(item, BaseException): - raise item - return item - - async def refuse(self) -> None: - """Fail the next `available()`, as a handler refusing the path does.""" - await self.send(moq.Error("refused")) - - async def run(self, sequence: int) -> None: - """Hand out an instance that serves one drive command numbered `sequence`, then stays open.""" - - async def control(): - yield {"type": "drive", "sequence": sequence, "throttle": 0.5} - await asyncio.Event().wait() - - await self.send(SimpleNamespace(tracks={"control": control()})) - - async def offline(self) -> None: - await self.send(None) +class Endpoint: + """A truck's endpoint, handing out scripted dials.""" + def __init__(self) -> None: + self.dials: asyncio.Queue = asyncio.Queue() + self.opened: list[tuple[str, str]] = [] + self.closed = False -class Drivers: - """A relay holding driver broadcasts.""" + def __call__(self, origin, prefix: str, id: str) -> "Endpoint": + self.opened.append((prefix, id)) + return self - def __init__(self) -> None: - self.announcements = Announcements() - self.followed: list[str] = [] - self._followings: dict[str, Following] = {} + async def __aenter__(self) -> "Endpoint": + return self - def announced(self, prefix: str, *, filter: str): - assert (prefix, filter) == ("viewer/carla/truck1", "*") - return self.announcements + async def __aexit__(self, *_exc) -> None: + self.closed = True - def following(self, path: str) -> Following: - return self._followings.setdefault(path, Following()) + async def accept_track(self) -> Incoming: + return await self.dials.get() - def follow(self, path: str) -> Following: - self.followed.append(path) - return self.following(path) + def dial(self, peer: str, id: str | None) -> Session | None: + session = None if id is None else Session(id) + self.dials.put_nowait(Incoming(peer, session)) + return session -async def watching(scenario) -> None: - """Run `scenario(relay, commands, gate)` against a live `watch_drivers`.""" - relay = Drivers() +async def serving(scenario, monkeypatch: pytest.MonkeyPatch) -> None: + """Run `scenario(endpoint, drivers, commands, gate)` against a live `Drivers`.""" + endpoint = Endpoint() + monkeypatch.setattr(moq.socket, "Endpoint", endpoint, raising=False) commands: asyncio.Queue = asyncio.Queue() - watcher = asyncio.create_task(publish.watch_drivers(relay, "viewer/carla/truck1", commands)) + drivers = publish.Drivers("truck1", commands) + task = asyncio.create_task(drivers.run(SimpleNamespace())) try: - await scenario(relay, commands, Drive()) + await scenario(endpoint, drivers, commands, Drive()) finally: - watcher.cancel() + task.cancel() with pytest.raises(asyncio.CancelledError): - await watcher + await task + assert endpoint.opened == [("carla", "truck1")] + assert endpoint.closed async def next_command(commands: asyncio.Queue, gate: Drive): @@ -291,92 +278,164 @@ async def next_command(commands: asyncio.Queue, gate: Drive): return command -DRIVER = "viewer/carla/truck1/abc" +DRIVE = {"type": "drive", "sequence": 1, "throttle": 0.5} -def test_a_driver_whose_broadcast_goes_absent_loses_the_lease() -> None: - async def scenario(relay: Drivers, commands: asyncio.Queue, gate: Drive) -> None: - await relay.announcements.present(DRIVER) - await relay.following(DRIVER).run(1) +def test_a_driver_whose_session_ends_loses_the_lease(monkeypatch: pytest.MonkeyPatch) -> None: + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + session = endpoint.dial("driver-abc", "s1") + session.send(DRIVE) command = await next_command(commands, gate) - assert (command.kind, command.driver_id) == ("drive", "abc") - assert gate.driver == "abc" + assert (command.kind, command.driver_id, command.session) == ("drive", "driver-abc", "s1") + assert gate.driver == "driver-abc" - await relay.announcements.absent(DRIVER) + session.end() left = await next_command(commands, gate) - assert (left.kind, left.instance) == ("driver_left", command.instance) + assert (left.kind, left.session) == ("driver_left", "s1") assert gate.driver is None + assert session.finished - asyncio.run(watching(scenario)) + asyncio.run(serving(scenario, monkeypatch)) -def test_a_driver_whose_run_goes_offline_loses_the_lease_until_it_returns() -> None: - async def scenario(relay: Drivers, commands: asyncio.Queue, gate: Drive) -> None: - await relay.announcements.present(DRIVER) - await relay.following(DRIVER).run(1) - first = await next_command(commands, gate) +def test_a_failed_session_still_releases_its_lease(monkeypatch: pytest.MonkeyPatch) -> None: + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + session = endpoint.dial("driver-abc", "s1") + session.send(DRIVE) + await next_command(commands, gate) - await relay.following(DRIVER).offline() + session.end(moq.Error("reset")) left = await next_command(commands, gate) - assert (left.kind, left.instance) == ("driver_left", first.instance) + assert (left.kind, left.session) == ("driver_left", "s1") assert gate.driver is None - await relay.following(DRIVER).run(2) - second = await next_command(commands, gate) - assert second.kind == "drive" and second.instance != first.instance - assert gate.driver == "abc" + asyncio.run(serving(scenario, monkeypatch)) + + +def test_shutdown_never_waits_on_a_full_command_queue(monkeypatch: pytest.MonkeyPatch) -> None: + """Shutdown stops the gate first, so a pump cancelled then must not wait to report its departure.""" + + async def scenario() -> None: + endpoint = Endpoint() + monkeypatch.setattr(moq.socket, "Endpoint", endpoint, raising=False) + commands: asyncio.Queue = asyncio.Queue(maxsize=1) + drivers = publish.Drivers("truck1", commands) + task = asyncio.create_task(drivers.run(SimpleNamespace())) + session = endpoint.dial("driver-abc", "s1") + session.send(DRIVE) + session.send(DRIVE) + # The queue is full and the second control is waiting for room. + for _ in range(10): + await asyncio.sleep(0) + assert commands.full() - asyncio.run(watching(scenario)) + task.cancel() + await asyncio.wait_for(asyncio.gather(task, return_exceptions=True), timeout=1) + assert session.finished + asyncio.run(scenario()) -def test_a_refused_driver_is_still_followed() -> None: - """A refusal leaves the follow waiting, so the driver's next run still takes the lease.""" - async def scenario(relay: Drivers, commands: asyncio.Queue, gate: Drive) -> None: - await relay.announcements.present(DRIVER) - await relay.following(DRIVER).refuse() - await relay.following(DRIVER).run(1) +def test_a_new_session_of_the_same_driver_is_a_new_driver(monkeypatch: pytest.MonkeyPatch) -> None: + """A lease is one session: an endpoint posing under the holder's id never continues it.""" + + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + first = endpoint.dial("driver-abc", "s1") + first.send(DRIVE) + await next_command(commands, gate) + + second = endpoint.dial("driver-abc", "s2") + second.send({**DRIVE, "throttle": 1.0}) command = await next_command(commands, gate) - assert (command.kind, command.driver_id) == ("drive", "abc") - assert relay.followed == [DRIVER] + assert command.session == "s2" + assert not gate.holds(command) - asyncio.run(watching(scenario)) + second.end() + assert (await next_command(commands, gate)).kind == "driver_left" + assert gate.driver == "driver-abc" + assert gate.holds(Command(kind="drive", driver_id="driver-abc", session="s1")) + asyncio.run(serving(scenario, monkeypatch)) -def test_a_driver_that_restarts_under_a_new_instance_is_a_new_driver() -> None: - """The old run's lease ends with it, even though the path and id are the same.""" - async def scenario(relay: Drivers, commands: asyncio.Queue, gate: Drive) -> None: - await relay.announcements.present(DRIVER) - await relay.following(DRIVER).run(1) - first = await next_command(commands, gate) +def test_a_second_session_under_an_open_driver_and_session_id_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + first = endpoint.dial("driver-abc", "s1") + first.send(DRIVE) + await next_command(commands, gate) - await relay.following(DRIVER).run(2) - received = [await next_command(commands, gate) for _ in range(2)] - assert {(command.kind, command.instance == first.instance) for command in received} == { - ("driver_left", True), - ("drive", False), - } - # The path stays present across the restart, so one follow sees both runs. - assert relay.followed == [DRIVER] + copy = endpoint.dial("driver-abc", "s1") + copy.send({**DRIVE, "throttle": 1.0}) + first.send({**DRIVE, "sequence": 2}) + command = await next_command(commands, gate) + assert command.sequence == 2 + assert copy.finished + assert commands.empty() + drivers.reply(command, {"sequence": 2}) + assert first.replies == [{"sequence": 2}] - # Whichever order those two arrived in, the new run holds the lease - # by its next command, and the old run can no longer release it. - second = next(command.instance for command in received if command.kind == "drive") - assert not gate.holds(Command(kind="drive", driver_id="abc", instance=first.instance)) - gate.apply(Command(kind="drive", driver_id="abc", instance=second, sequence=3, state=DriveState())) - assert gate.holds(Command(kind="drive", driver_id="abc", instance=second)) - assert not gate.apply(Command(kind="driver_left", driver_id="abc", instance=first.instance)) - assert gate.driver == "abc" + asyncio.run(serving(scenario, monkeypatch)) - asyncio.run(watching(scenario)) +def test_a_forged_dial_never_reaches_the_gate(monkeypatch: pytest.MonkeyPatch) -> None: + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + endpoint.dial("driver-forged", None) + session = endpoint.dial("driver-abc", "s1") + session.send(DRIVE) + command = await next_command(commands, gate) + assert command.driver_id == "driver-abc" + assert commands.empty() -def test_only_direct_children_of_the_prefix_are_drivers() -> None: - async def scenario(relay: Drivers, commands: asyncio.Queue, gate: Drive) -> None: - await relay.announcements.present("viewer/carla") - await relay.announcements.present("viewer/carla/truck1/abc/deeper") - await relay.announcements.present("viewer/carla/truck10") - assert relay.followed == [] + asyncio.run(serving(scenario, monkeypatch)) + + +def test_a_malformed_control_is_dropped_and_the_session_keeps_going(monkeypatch: pytest.MonkeyPatch) -> None: + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + session = endpoint.dial("driver-abc", "s1") + session.send(b"\xff not json") + session.send(["not", "a", "command"]) + session.send(DRIVE) + command = await next_command(commands, gate) + assert command.kind == "drive" + + asyncio.run(serving(scenario, monkeypatch)) + + +def test_a_late_control_never_blocks_a_newer_one(monkeypatch: pytest.MonkeyPatch) -> None: + """The truck reads oldest first, skipping a control one heartbeat behind the newest.""" + + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + session = endpoint.dial("driver-abc", "s1") + session.send(DRIVE) + await next_command(commands, gate) + assert session.subscription == moq.Subscription( + delay=moq.Delay(max=publish.CONTROL_DELAY), order=moq.Order.ASCENDING + ) - asyncio.run(watching(scenario)) + asyncio.run(serving(scenario, monkeypatch)) + + +def test_replies_reach_only_their_own_open_session(monkeypatch: pytest.MonkeyPatch) -> None: + async def scenario(endpoint: Endpoint, drivers, commands: asyncio.Queue, gate: Drive) -> None: + first = endpoint.dial("driver-abc", "s1") + second = endpoint.dial("driver-def", "s2") + first.send(DRIVE) + second.send(DRIVE) + await next_command(commands, gate) + await next_command(commands, gate) + + drivers.reply(Command(kind="drive", driver_id="driver-abc", session="s1"), {"sequence": 1}) + assert first.replies == [{"sequence": 1}] + assert second.replies == [] + # A session id is the dialer's pick, so only the pair names a session. + drivers.reply(Command(kind="drive", driver_id="driver-def", session="s1"), {"sequence": 9}) + assert first.replies == [{"sequence": 1}] + assert second.replies == [] + + first.end() + await next_command(commands, gate) + # The pump releasing the lease may race a reply to the gone session. + drivers.reply(Command(kind="drive", driver_id="driver-abc", session="s1"), {"sequence": 2}) + assert first.replies == [{"sequence": 1}] + + asyncio.run(serving(scenario, monkeypatch)) diff --git a/demo/carla/sim/tests/test_runtime.py b/demo/carla/sim/tests/test_runtime.py index cfa4b76d015..8912de2a135 100644 --- a/demo/carla/sim/tests/test_runtime.py +++ b/demo/carla/sim/tests/test_runtime.py @@ -25,6 +25,16 @@ def release(self, reason: str) -> None: self.held.discard(reason) +class Replies: + """Records what the truck answers on each driver session.""" + + def __init__(self) -> None: + self.sent: list[tuple[str, dict]] = [] + + def reply(self, command: Command, message: dict) -> None: + self.sent.append((command.session, message)) + + class DemandPublisher: def __init__(self) -> None: self.use = asyncio.Event() @@ -69,7 +79,7 @@ async def scenario() -> None: session._drive = Drive() session._commands = commands session._deadline = None - session._media_pipeline = {} + session._drivers = Replies() actuated = [] published = asyncio.Event() session._truck.actuate = lambda accepted, *, has_driver: actuated.append((accepted, has_driver)) @@ -92,6 +102,51 @@ async def scenario() -> None: asyncio.run(scenario()) +def test_only_the_holders_accepted_commands_are_acknowledged_on_its_session() -> None: + """The ack and the pipeline echo are the holder's alone, so they ride its session.""" + + async def scenario() -> None: + stages = (TimestampEntry(label="received", ts=10.0), TimestampEntry(label="decoded", ts=14.0)) + commands = asyncio.Queue() + for command in ( + Command(kind="drive", driver_id="holder", session="s1", sequence=1, state=DriveState(), timestamps=stages), + # Another session of the same id is another driver, refused while the lease is held. + Command(kind="drive", driver_id="holder", session="s2", sequence=2, state=DriveState()), + # A bystander's e-stop latches, but is not the lease's to acknowledge. + Command(kind="estop", driver_id="bystander", session="s3", sequence=3), + Command(kind="estop", driver_id="holder", session="s1", sequence=4), + ): + commands.put_nowait(command) + + session = runtime.TruckSession.__new__(runtime.TruckSession) + session._clock = FakeClock() + session._truck = SimpleNamespace(name="truck1", autonomous=False) + session._drive = Drive() + session._commands = commands + session._deadline = None + session._drivers = Replies() + session._actuate = lambda: None + session._publish = lambda: None + + task = asyncio.create_task(session._loop()) + try: + for _ in range(10): + if commands.empty(): + break + await asyncio.sleep(0) + await asyncio.sleep(0) + assert session._drivers.sent == [ + ("s1", {"sequence": 1, "media_pipeline": [{"label": "received→decoded", "ms": 4}]}), + ("s1", {"sequence": 4, "media_pipeline": []}), + ] + finally: + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + asyncio.run(scenario()) + + def test_only_unestopped_lease_holder_can_respawn_truck() -> None: async def scenario() -> None: commands = asyncio.Queue() @@ -111,7 +166,7 @@ async def scenario() -> None: assert session._drive.apply(Command(kind="estop", driver_id="bystander", sequence=2)) session._commands = commands session._deadline = None - session._media_pipeline = {"holder": []} + session._drivers = Replies() session._actuate = lambda: None session._publish = lambda: published.append(True) @@ -123,7 +178,6 @@ async def scenario() -> None: await asyncio.sleep(0) assert respawns == ["truck1"] assert session._drive.driver is None - assert session._media_pipeline == {} assert published == [True, True] finally: task.cancel() @@ -133,7 +187,7 @@ async def scenario() -> None: asyncio.run(scenario()) -def test_video_demand_renders_the_world_for_the_truck(monkeypatch: pytest.MonkeyPatch) -> None: +def test_video_demand_renders_the_world_for_the_truck() -> None: """The stream's demand is what turns the simulator's renderer on for a truck.""" async def wait_forever(*_args: object) -> None: @@ -157,9 +211,9 @@ async def watch_demand(on_demand) -> None: session._status = DemandPublisher() session._publish = lambda: None session._loop = wait_forever - monkeypatch.setattr(runtime, "watch_drivers", wait_forever) + session._drivers = SimpleNamespace(run=wait_forever) - run = asyncio.create_task(session.run(SimpleNamespace(), "viewer/carla/truck1")) + run = asyncio.create_task(session.run(SimpleNamespace())) await watching.wait() run.cancel() await asyncio.gather(run, return_exceptions=True) @@ -168,7 +222,7 @@ async def watch_demand(on_demand) -> None: asyncio.run(scenario()) -def test_session_exits_when_its_video_demand_fails(monkeypatch: pytest.MonkeyPatch) -> None: +def test_session_exits_when_its_video_demand_fails() -> None: """A world that cannot start rendering for a viewer fails the session.""" async def wait_forever(*_args: object) -> None: @@ -186,10 +240,10 @@ async def scenario() -> None: session._status = DemandPublisher() session._publish = lambda: None session._loop = wait_forever - monkeypatch.setattr(runtime, "watch_drivers", wait_forever) + session._drivers = SimpleNamespace(run=wait_forever) with pytest.raises(ExceptionGroup) as failure: - await session.run(SimpleNamespace(), "viewer/carla/truck1") + await session.run(SimpleNamespace()) assert len(failure.value.exceptions) == 1 assert str(failure.value.exceptions[0]) == "CARLA rejected settings" @@ -237,7 +291,7 @@ def telemetry_session(pose, *, autonomous: bool = True) -> runtime.TruckSession: ) session._drive = Drive() session._deadline = None - session._media_pipeline = {} + session._drivers = Replies() session._stuck = StuckDetector() return session @@ -563,7 +617,7 @@ async def scenario() -> None: session._drive = Drive() session._commands = commands session._deadline = None - session._media_pipeline = {} + session._drivers = Replies() session._stuck = StuckDetector() session._actuate = lambda: None session._publish = lambda: None @@ -668,7 +722,7 @@ async def scenario() -> None: session._drive = Drive() session._commands = asyncio.Queue() session._deadline = None - session._media_pipeline = {} + session._drivers = Replies() session._actuate = lambda: None session._publish = lambda: published.append(True) diff --git a/demo/carla/web/src/element.ts b/demo/carla/web/src/element.ts index 79288b029e3..52f1b3348a0 100644 --- a/demo/carla/web/src/element.ts +++ b/demo/carla/web/src/element.ts @@ -8,6 +8,7 @@ */ import * as Moq from "@moq/net"; +import { VERSIONS } from "./paths.ts"; import { mount, PREVIEW } from "./player.ts"; /** The attributes the element observes. */ @@ -30,13 +31,22 @@ export default class MoqCarla extends HTMLElement { /** The relay connection, open while the element is in a document. */ readonly connection: Moq.Connection; + // Owned rather than borrowed from the connection, since a driver's socket + // endpoint publishes into it. + readonly #origin = new Moq.Origin.Producer(); + // Set when the element is connected to the DOM. readonly #connected = new Moq.Signals.Signal(false); #dispose?: () => void; constructor() { super(); - this.connection = new Moq.Connection({ enabled: PREVIEW ? false : this.#connected }); + this.connection = new Moq.Connection({ + enabled: PREVIEW ? false : this.#connected, + versions: VERSIONS, + publish: this.#origin.consume(), + consume: this.#origin, + }); cleanup.register(this, this.connection); } @@ -46,7 +56,7 @@ export default class MoqCarla extends HTMLElement { // Render into a Shadow DOM, reusing the existing one on reconnect. const shadow = this.shadowRoot ?? this.attachShadow({ mode: "open" }); - this.#dispose = mount(shadow, this.connection); + this.#dispose = mount(shadow, this.connection, this.#origin); } /** Called by the browser when the element leaves its document. */ diff --git a/demo/carla/web/src/paths.test.ts b/demo/carla/web/src/paths.test.ts index 7d8c87c8b36..2974f6de172 100644 --- a/demo/carla/web/src/paths.test.ts +++ b/demo/carla/web/src/paths.test.ts @@ -13,7 +13,7 @@ describe("parseCamera", () => { "carla/truck1/front", "carla/truck1/rear/extra.hang", "other/truck1/front.hang", - "viewer/carla/truck1/abc", + "carla/.socket/truck1", ]) { expect(parseCamera(path)).toBeUndefined(); } diff --git a/demo/carla/web/src/paths.ts b/demo/carla/web/src/paths.ts index 9c9394d80e7..b71eea0fc19 100644 --- a/demo/carla/web/src/paths.ts +++ b/demo/carla/web/src/paths.ts @@ -1,9 +1,10 @@ /** - * The broadcast paths shared with the simulator. + * The paths and connection settings shared with the simulator. * - * Trucks publish one media broadcast per camera at `carla/{truck}/{camera}.hang`. - * Drivers publish their `control` track at `viewer/carla/{truck}/{driver}`, a - * prefix the demo relay lets anonymous viewers publish under. + * Trucks publish one media broadcast per camera at `carla/{truck}/{camera}.hang`, + * and accept drivers on a socket endpoint named for the truck. A driver dials it + * from an endpoint of its own, `driver-{id}`, a form the demo relay lets + * anonymous viewers publish. * * @module */ @@ -27,7 +28,16 @@ export function parseCamera(path: string): Camera | undefined { return { truck, id: `${truck}/${file.slice(0, -".hang".length)}` }; } -/** Where one driver of a truck publishes its control broadcast. */ -export function driverPath(truck: string, driver: string): Moq.Path.Valid { - return Moq.Path.from(`viewer/carla/${truck}/${driver}`); +/** The prefix every truck and driver socket endpoint sits under, as `carla/.socket/{id}`. */ +export const SOCKET = Moq.Path.from("carla"); + +/** + * The versions offered to the relay, most preferred first: the ones that carry socket + * endpoints, which are hidden broadcasts that moq-lite before 07 never carries. + */ +export const VERSIONS = ["moq-lite-07-wip", "moq-transport-22"] as const; + +/** A fresh driver endpoint id. Public, since the truck's status names its driver. */ +export function driverId(): string { + return `driver-${crypto.randomUUID().replaceAll("-", "").slice(0, 12)}`; } diff --git a/demo/carla/web/src/player.ts b/demo/carla/web/src/player.ts index 010dfdec7da..564d6a0a765 100644 --- a/demo/carla/web/src/player.ts +++ b/demo/carla/web/src/player.ts @@ -4,8 +4,8 @@ * Connects to the relay, discovers camera broadcasts by announcement (like * moq-boy), renders each on a map with a video preview, and lets you take * control of one vehicle with the keyboard or a steering wheel. The - * {@link Truck} backend publishes a `control` track and subscribes to the - * vehicle's `status` track for round-trip confirmation. + * {@link Truck} backend dials the vehicle's socket endpoint to drive it and + * subscribes to its `status` track. * * @module */ @@ -53,12 +53,13 @@ import { export const PREVIEW = import.meta.env.DEV && new URLSearchParams(location.search).has("preview"); /** - * Render the player into `root` and run it on `connection`. + * Render the player into `root` and run it on `connection`, which serves and + * fills `origin`. * * Everything it starts, window listeners and the input loop included, stops * when the returned function is called. */ -export function mount(root: ShadowRoot, connection: Moq.Connection): () => void { +export function mount(root: ShadowRoot, connection: Moq.Connection, origin: Moq.Origin.Producer): () => void { root.innerHTML = `${template}`; preloadAerials(); @@ -126,7 +127,14 @@ export function mount(root: ShadowRoot, connection: Moq.Connection): () => void // A restarted simulator keeps its path present, and the watch broadcast // follows it, so only an absence removes the tile. if (entry.kind !== "absent" && !tiles.has(entry.prefix)) { - const truck = new Truck({ broadcastId: entry.prefix, connection, driving, focused, hovered }); + const truck = new Truck({ + broadcastId: entry.prefix, + connection, + origin, + driving, + focused, + hovered, + }); tiles.set(entry.prefix, new TruckTile(truck)); vehicles.set(tiles.size); refreshFocusLayout(); @@ -1092,8 +1100,7 @@ export function mount(root: ShadowRoot, connection: Moq.Connection): () => void ? "Manual" : "Unknown"; q(".mode").className = `mode${s?.estopped ? " estopped" : ""}`; - const me = effect.get(this.truck.driverId); - const mine = me ? s?.media_pipeline[me] : undefined; + const mine = effect.get(this.truck.pipeline); const roundTrip = effect.get(this.truck.roundTripMs); q(".rtt").textContent = roundTrip == null ? "Not measured" : `${Math.round(roundTrip)} ms`; q(".lat").textContent = mine?.length @@ -1232,7 +1239,7 @@ export function mount(root: ShadowRoot, connection: Moq.Connection): () => void const { populatePreview } = await import("./preview.ts"); if (abort.signal.aborted) return; const cleanup = populatePreview((path) => { - const truck = new Truck({ broadcastId: path, connection, driving, focused, hovered }); + const truck = new Truck({ broadcastId: path, connection, origin, driving, focused, hovered }); const tile = new TruckTile(truck); tiles.set(path, tile); vehicles.set(tiles.size); diff --git a/demo/carla/web/src/preview.ts b/demo/carla/web/src/preview.ts index c342a52a85e..7d7de0980d6 100644 --- a/demo/carla/web/src/preview.ts +++ b/demo/carla/web/src/preview.ts @@ -30,7 +30,6 @@ export function populatePreview(add: (path: Moq.Path.Valid) => Truck) { estopped: false, map: "Town10HD_Opt", pose, - media_pipeline: {}, }, }); cleanup.push(() => { diff --git a/demo/carla/web/src/schemas.test.ts b/demo/carla/web/src/schemas.test.ts index 8e446a590d0..8b7f0181037 100644 --- a/demo/carla/web/src/schemas.test.ts +++ b/demo/carla/web/src/schemas.test.ts @@ -4,7 +4,6 @@ import { TruckStatusSchema } from "./schemas.ts"; const status = { accepted: { steering: 0, throttle: 0, brake: 1, gear: "neutral" }, estopped: false, - media_pipeline: {}, }; describe("TruckStatusSchema", () => { diff --git a/demo/carla/web/src/schemas.ts b/demo/carla/web/src/schemas.ts index 408f481f8c6..f65eabf0c4c 100644 --- a/demo/carla/web/src/schemas.ts +++ b/demo/carla/web/src/schemas.ts @@ -66,6 +66,14 @@ const LatencyEntrySchema = z.object({ label: z.string(), ms: z.number(), }); +export type LatencyEntry = z.infer; + +/** The truck's answer on a driver's session to each command it accepted from the lease holder. */ +export const ReplySchema = z.object({ + sequence: z.number(), + media_pipeline: z.array(LatencyEntrySchema), +}); +export type Reply = z.infer; /** Per-update telemetry the truck publishes on the `status` track. */ export const TruckStatusSchema = z.object({ @@ -76,8 +84,6 @@ export const TruckStatusSchema = z.object({ estopped: z.boolean(), estop_by: z.optional(z.string()), driver: z.optional(z.string()), - sequence: z.optional(z.number()), - media_pipeline: z.record(z.string(), z.array(LatencyEntrySchema)), location: z.optional(z.string()), // Absent until the simulator's first physics tick reports where the vehicle is. map: z.optional(z.string()), diff --git a/demo/carla/web/src/truck.test.ts b/demo/carla/web/src/truck.test.ts index 84e89a503e4..de263f6927a 100644 --- a/demo/carla/web/src/truck.test.ts +++ b/demo/carla/web/src/truck.test.ts @@ -2,12 +2,14 @@ import { describe, expect, test } from "bun:test"; import * as Json from "@moq/json"; import * as Moq from "@moq/net"; import { applyKeys, REST } from "./input.ts"; +import { SOCKET } from "./paths.ts"; import type { TruckStatus } from "./schemas.ts"; -import { SafetyCommandBroadcast, shiftGear, Truck } from "./truck.ts"; +import { shiftGear, Truck } from "./truck.ts"; describe("Truck video demand", () => { test("only hovered or focused cameras subscribe, and focus suppresses other previews", async () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const hovered = new Moq.Signals.Signal(undefined); @@ -16,6 +18,7 @@ describe("Truck video demand", () => { new Truck({ broadcastId: Moq.Path.from(id), connection, + origin, driving, focused, hovered, @@ -45,6 +48,7 @@ describe("Truck video demand", () => { } finally { for (const truck of trucks) truck.close(); connection.close(); + origin.close(); } }); }); @@ -55,33 +59,34 @@ describe("Truck status", () => { accepted: { steering: 0.25, throttle, brake: 0, gear: "forward" }, autonomy: "manual", estopped: false, - media_pipeline: {}, }; } - function statusOrigin() { - const origin = new Moq.Broadcast.Producer(); - const track = origin.createTrack("status", { compress: true }); + function statusBroadcast() { + const broadcast = new Moq.Broadcast.Producer(); + const track = broadcast.createTrack("status", { compress: true }); const producer = new Json.Snapshot.Producer({ track }); - return { origin, producer }; + return { broadcast, producer }; } test("clears valid telemetry when the status track ends", async () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); - const { origin, producer } = statusOrigin(); + const { broadcast, producer } = statusBroadcast(); try { const available = truck.statusState.changed(); (truck.broadcast.out.active as Moq.Signals.Signal).set( - origin.consume(), + broadcast.consume(), ); producer.update({ value: status(0.5) }); expect(await available).toBe("available"); @@ -93,23 +98,26 @@ describe("Truck status", () => { expect(truck.status.peek()).toBeUndefined(); } finally { producer.finish(); - origin.close(); + broadcast.close(); truck.close(); connection.close(); + origin.close(); } }); test("a status from before the truck saw an e-stop does not unlatch it", async () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); - const { origin, producer } = statusOrigin(); + const { broadcast, producer } = statusBroadcast(); const next = async (update: TruckStatus) => { const changed = truck.status.changed(); producer.update({ value: update }); @@ -118,7 +126,7 @@ describe("Truck status", () => { try { (truck.broadcast.out.active as Moq.Signals.Signal).set( - origin.consume(), + broadcast.consume(), ); await next(status(0.5)); truck.emergencyStop(); @@ -131,27 +139,30 @@ describe("Truck status", () => { expect(truck.estopped.peek()).toBe(false); } finally { producer.finish(); - origin.close(); + broadcast.close(); truck.close(); connection.close(); + origin.close(); } }); test("marks status unavailable when an active broadcast disappears", async () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); - const { origin, producer } = statusOrigin(); + const { broadcast, producer } = statusBroadcast(); const active = truck.broadcast.out.active as Moq.Signals.Signal; try { - active.set(origin.consume()); + active.set(broadcast.consume()); producer.update({ value: status(0.5) }); while (truck.statusState.peek() !== "available") await truck.statusState.changed(); @@ -161,35 +172,38 @@ describe("Truck status", () => { expect(truck.status.peek()).toBeUndefined(); } finally { producer.finish(); - origin.close(); + broadcast.close(); truck.close(); connection.close(); + origin.close(); } }); test("a canceled status consumer cannot overwrite its replacement", async () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); - const first = statusOrigin(); - const second = statusOrigin(); + const first = statusBroadcast(); + const second = statusBroadcast(); const active = truck.broadcast.out.active as Moq.Signals.Signal; try { - active.set(first.origin.consume()); + active.set(first.broadcast.consume()); first.producer.update({ value: status(0.25) }); while (truck.statusState.peek() !== "available") await truck.statusState.changed(); const transitions: string[] = []; const unsubscribe = truck.statusState.subscribe((state) => transitions.push(state)); try { - active.set(second.origin.consume()); + active.set(second.broadcast.consume()); second.producer.update({ value: status(0.75) }); while (truck.status.peek()?.accepted.throttle !== 0.75) await truck.status.changed(); await Promise.resolve(); @@ -203,10 +217,11 @@ describe("Truck status", () => { } finally { first.producer.finish(); second.producer.finish(); - first.origin.close(); - second.origin.close(); + first.broadcast.close(); + second.broadcast.close(); truck.close(); connection.close(); + origin.close(); } }); }); @@ -214,15 +229,17 @@ describe("Truck status", () => { describe("Truck.setDrive", () => { function truck() { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const t = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); - return { t, connection }; + return { t, connection, origin }; } // Guards setDrive's contract: local state advances however small the step, @@ -230,7 +247,7 @@ describe("Truck.setDrive", () => { // in input.test.ts's stepDrive suite; this one calls setDrive directly and so // cannot catch a gate added at the call site. test("accumulates sub-epsilon frames instead of discarding them", () => { - const { t, connection } = truck(); + const { t, connection, origin } = truck(); try { const dt = 1 / 120; for (let i = 0; i < 120; i++) { @@ -242,27 +259,31 @@ describe("Truck.setDrive", () => { } finally { t.close(); connection.close(); + origin.close(); } }); test("a single sub-epsilon frame still moves the local state", () => { - const { t, connection } = truck(); + const { t, connection, origin } = truck(); try { t.setDrive(applyKeys(t.drive.peek(), new Set(["d"]), 1 / 60)); expect(t.drive.peek().steering).toBeGreaterThan(0); } finally { t.close(); connection.close(); + origin.close(); } }); test("claims with measured steering and gear but zero throttle and full brake", async () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal("truck1/front"); const t = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); @@ -272,7 +293,6 @@ describe("Truck.setDrive", () => { accepted: { ...REST }, autonomy: "autonomous", estopped: false, - media_pipeline: {}, measured, }); expect(t.beginTakeover()).toBe(true); @@ -283,17 +303,19 @@ describe("Truck.setDrive", () => { } finally { t.close(); connection.close(); + origin.close(); } }); test("refuses a takeover before the truck reports status", () => { - const { t, connection } = truck(); + const { t, connection, origin } = truck(); try { expect(t.beginTakeover()).toBe(false); expect(t.drive.peek()).toEqual(REST); } finally { t.close(); connection.close(); + origin.close(); } }); }); @@ -301,11 +323,13 @@ describe("Truck.setDrive", () => { describe("Truck safety controls", () => { test("a viewer e-stop does not claim the driving session", () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); @@ -318,37 +342,19 @@ describe("Truck safety controls", () => { } finally { truck.close(); connection.close(); - } - }); - - test("one safety snapshot reaches every current truck consumer", async () => { - const safety = new SafetyCommandBroadcast({ - cmd: { type: "estop" }, - sequence: 7, - timestamps: [], - }); - const front = new Json.Snapshot.Consumer>({ - track: safety.broadcast.track("control"), - }); - const rear = new Json.Snapshot.Consumer>({ - track: safety.broadcast.track("control"), - }); - - try { - expect((await front.next())?.value).toEqual({ type: "estop", sequence: 7, timestamps: [] }); - expect((await rear.next())?.value).toEqual({ type: "estop", sequence: 7, timestamps: [] }); - } finally { - safety.close(); + origin.close(); } }); test("a focused viewer resets only after the truck confirms the latch", () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); @@ -363,7 +369,6 @@ describe("Truck safety controls", () => { autonomy: "unknown", estopped: true, estop_by: "bystander", - media_pipeline: {}, }); truck.resetEstop(); expect(driving.peek()).toBeUndefined(); @@ -371,16 +376,19 @@ describe("Truck safety controls", () => { } finally { truck.close(); connection.close(); + origin.close(); } }); test("the tile safety action exits a confirmed e-stop without claiming control", () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal(undefined); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); @@ -392,7 +400,6 @@ describe("Truck safety controls", () => { accepted: { ...REST }, autonomy: "manual", estopped: true, - media_pipeline: {}, }); truck.toggleEmergencyStop(); @@ -402,6 +409,169 @@ describe("Truck safety controls", () => { } finally { truck.close(); connection.close(); + origin.close(); + } + }); +}); + +describe("Truck sessions", () => { + const decoder = new TextDecoder(); + const encoder = new TextEncoder(); + + /** A truck whose socket endpoint accepts on the same origin as the player. */ + function setup() { + const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); + const driving = new Moq.Signals.Signal(undefined); + const focused = new Moq.Signals.Signal(undefined); + const truck = new Truck({ + broadcastId: Moq.Path.from("carla/truck1/front.hang"), + connection, + origin, + driving, + focused, + }); + const endpoint = new Moq.socket.Endpoint(origin, SOCKET, "truck1"); + const close = () => { + truck.close(); + endpoint.close(); + connection.close(); + origin.close(); + }; + return { truck, driving, endpoint, close }; + } + + async function read(stream: Moq.socket.TrackStream, reader = stream.consumer.ordered()) { + const frame = await reader.readFrame(); + return frame && JSON.parse(decoder.decode(frame.payload)); + } + + test("a driver's commands and the truck's acknowledgements travel on its own session", async () => { + const { truck, driving, endpoint, close } = setup(); + try { + const incoming = endpoint.acceptTrack(); + driving.set(truck.id); + const dial = await incoming; + expect(dial.peer).toBe(truck.driverId.peek() as string); + expect(dial.peer).toMatch(/^driver-/); + const stream = await dial.accept(); + + // Taking control heartbeats at once, so the truck hears the claim. + const command = await read(stream); + expect(command.type).toBe("drive"); + + const measured = truck.roundTripMs.changed(); + const pipeline = [{ label: "received→decoded", ms: 4 }]; + const reply = { sequence: command.sequence, media_pipeline: pipeline }; + stream.producer.writeFrame({ payload: encoder.encode(JSON.stringify(reply)) }); + expect(await measured).toBeGreaterThanOrEqual(0); + expect(truck.pipeline.peek()).toEqual(pipeline); + } finally { + close(); + } + }); + + test("the truck ending the session ends the lease", async () => { + const { truck, driving, endpoint, close } = setup(); + try { + const incoming = endpoint.acceptTrack(); + driving.set(truck.id); + const stream = await (await incoming).accept(); + await read(stream); + + const released = driving.changed(); + stream.producer.finish(); + expect(await released).toBeUndefined(); + // The lease's endpoint goes with it. + while (truck.driverId.peek() !== undefined) await truck.driverId.changed(); + } finally { + close(); + } + }); + + test("releasing control ends the session cleanly", async () => { + const { truck, driving, endpoint, close } = setup(); + try { + const incoming = endpoint.acceptTrack(); + driving.set(truck.id); + const stream = await (await incoming).accept(); + const reader = stream.consumer.ordered(); + await read(stream, reader); + + driving.set(undefined); + // Anything the heartbeat sent first, then the clean end. + while ((await read(stream, reader)) !== undefined); + } finally { + close(); + } + }); + + test("a bystander's e-stop dials a session of its own without taking the lease", async () => { + const { truck, driving, endpoint, close } = setup(); + try { + const incoming = endpoint.acceptTrack(); + truck.emergencyStop(); + const dial = await incoming; + expect(dial.peer).toMatch(/^driver-/); + const stream = await dial.accept(); + const reader = stream.consumer.ordered(); + expect((await read(stream, reader)).type).toBe("estop"); + expect(driving.peek()).toBeUndefined(); + expect(truck.driverId.peek()).toBeUndefined(); + + // The truck's status echoing the latch is what lets the session go. + const broadcast = new Moq.Broadcast.Producer(); + const producer = new Json.Snapshot.Producer({ + track: broadcast.createTrack("status", { compress: true }), + }); + try { + (truck.broadcast.out.active as Moq.Signals.Signal).set( + broadcast.consume(), + ); + producer.update({ + value: { accepted: { ...REST }, autonomy: "manual", estopped: true, estop_by: dial.peer }, + }); + expect(await read(stream, reader)).toBeUndefined(); + } finally { + producer.finish(); + broadcast.close(); + } + } finally { + close(); + } + }); +}); + +describe("Truck safety sessions", () => { + test("an e-stop whose session ends unconfirmed stops showing as latched", async () => { + const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); + const driving = new Moq.Signals.Signal(undefined); + const focused = new Moq.Signals.Signal(undefined); + const truck = new Truck({ + broadcastId: Moq.Path.from("carla/truck1/front.hang"), + connection, + origin, + driving, + focused, + }); + const endpoint = new Moq.socket.Endpoint(origin, SOCKET, "truck1"); + try { + const incoming = endpoint.acceptTrack(); + truck.emergencyStop(); + expect(truck.estopped.peek()).toBe(true); + const stream = await (await incoming).accept(); + await stream.consumer.ordered().readFrame(); + + // The truck goes away before its status ever echoed the latch. + const cleared = truck.estopped.changed(); + stream.producer.finish(); + expect(await cleared).toBe(false); + } finally { + truck.close(); + endpoint.close(); + connection.close(); + origin.close(); } }); }); @@ -409,11 +579,13 @@ describe("Truck safety controls", () => { describe("Truck respawn", () => { test("requires the truck's current lease and clears old input", async () => { const connection = new Moq.Connection({ enabled: false }); + const origin = new Moq.Origin.Producer(); const driving = new Moq.Signals.Signal(undefined); const focused = new Moq.Signals.Signal("truck1/front"); const truck = new Truck({ broadcastId: Moq.Path.from("carla/truck1/front.hang"), connection, + origin, driving, focused, }); @@ -429,7 +601,6 @@ describe("Truck respawn", () => { autonomy: "manual", estopped: false, driver: "someone-else", - media_pipeline: {}, } satisfies TruckStatus; truck.status.set(status); expect(truck.respawn()).toBe(false); @@ -452,6 +623,7 @@ describe("Truck respawn", () => { } finally { truck.close(); connection.close(); + origin.close(); } }); }); diff --git a/demo/carla/web/src/truck.ts b/demo/carla/web/src/truck.ts index cbba17bf2ce..ad283e18466 100644 --- a/demo/carla/web/src/truck.ts +++ b/demo/carla/web/src/truck.ts @@ -1,10 +1,11 @@ /** * The non-UI truck steering session backend. * - * Shaped like moq-boy's `Game`: a video pipeline pulled from `@moq/watch`, a - * `control` track we publish, and a `status` track we subscribe to for - * round-trip confirmation. The UI layer reads signals from this class to render - * video, the control panel, and telemetry. + * A video pipeline pulled from `@moq/watch`, the truck's `status` track that + * every viewer reads, and, while driving, a socket session to the truck that + * carries this driver's commands one way and the truck's acknowledgements the + * other. The UI layer reads signals from this class to render video, the + * control panel, and telemetry. * * @module */ @@ -13,9 +14,9 @@ import * as Json from "@moq/json"; import * as Moq from "@moq/net"; import * as Watch from "@moq/watch"; import { changed, REST, takeoverState } from "./input.ts"; -import { driverPath, parseCamera } from "./paths.ts"; -import type { DriveState, Gear, TruckStatus } from "./schemas.ts"; -import { TruckStatusSchema } from "./schemas.ts"; +import { driverId, parseCamera, SOCKET } from "./paths.ts"; +import type { DriveState, Gear, LatencyEntry, Reply, TruckStatus } from "./schemas.ts"; +import { ReplySchema, TruckStatusSchema } from "./schemas.ts"; /** Configuration for creating a Truck session. */ export interface TruckConfig { @@ -23,6 +24,8 @@ export interface TruckConfig { broadcastId: Moq.Path.Valid; /** MoQ connection to the relay. */ connection: Moq.Connection; + /** The origin the connection serves and fills, which driver endpoints publish into. */ + origin: Moq.Origin.Producer; /** Shared signal tracking which truck is currently being driven. */ driving: Moq.Signals.Signal; /** Shared signal tracking which truck is currently focused. */ @@ -38,28 +41,21 @@ interface Command { timestamps: { label: string; ts: number }[]; } -/** A retained safety snapshot that fans out to every truck consumer. */ -export class SafetyCommandBroadcast { - readonly broadcast: Moq.Broadcast.Producer; - readonly #producer: Json.Snapshot.Producer>; - - constructor(command: Command, broadcast = new Moq.Broadcast.Producer()) { - this.broadcast = broadcast; - const track = this.broadcast.createTrack("control", { compress: true }); - this.#producer = new Json.Snapshot.Producer>({ track }); - this.#producer.update({ - value: { ...command.cmd, sequence: command.sequence, timestamps: command.timestamps }, - }); - } +/** A driver session: pending until the truck accepts, `null` once the dial or the session failed. */ +type Session = Moq.Signals.Signal; - close() { - this.#producer.finish(); - this.broadcast.close(); - } -} - -const CONTROL_HEARTBEAT = Nano.fromMilli(Milli(250)); +const CONTROL_HEARTBEAT = Milli(250); const MAX_PENDING_ACKS = 64; +const encoder = new TextEncoder(); +const decoder = new TextDecoder(); + +/** Send one command as its own message, so a late one never holds up the next. */ +function write(stream: Moq.socket.TrackStream, command: Command) { + // A session that already ended has nothing to send to; its reply reader reports the end. + if (stream.producer.closed.peek() !== undefined) return; + const message = { ...command.cmd, sequence: command.sequence, timestamps: command.timestamps }; + stream.producer.writeFrame({ payload: encoder.encode(JSON.stringify(message)) }); +} /** * A single truck streaming + steering session (the non-UI backend). @@ -81,6 +77,8 @@ export class Truck { readonly estopped = new Moq.Signals.Signal(false); readonly respawning = new Moq.Signals.Signal(false); readonly roundTripMs = new Moq.Signals.Signal(undefined); + /** This driver's media pipeline stages, as the truck last measured them. */ + readonly pipeline = new Moq.Signals.Signal(undefined); // The driver's locally-requested drive state (what we are sending). readonly drive = new Moq.Signals.Signal({ ...REST }); @@ -105,20 +103,23 @@ export class Truck { // takes it as a signal input rather than a property set afterwards. readonly #canvas = new Moq.Signals.Signal(undefined); + readonly #origin: Moq.Origin.Producer; + constructor(config: TruckConfig) { - const { broadcastId, connection, driving, focused, hovered } = config; + const { broadcastId, connection, origin, driving, focused, hovered } = config; const camera = parseCamera(broadcastId); if (!camera) throw new Error(`not a camera broadcast: ${broadcastId}`); this.broadcastId = broadcastId; this.id = camera.id; this.name = camera.truck; this.#driving = driving; + this.#origin = origin; // Active when this truck is the one being driven. this.#signals.run(this.#runActive.bind(this)); this.#signals.run(this.#runControlLease.bind(this)); this.#signals.run(this.#runRespawn.bind(this)); - this.#signals.interval(() => this.#heartbeat(), CONTROL_HEARTBEAT); + this.#signals.interval(() => this.#heartbeat(), Nano.fromMilli(CONTROL_HEARTBEAT)); // Video pipeline. this.broadcast = new Watch.Broadcast({ @@ -167,9 +168,9 @@ export class Truck { // Subscribe to the truck's status (telemetry) track. this.#signals.run(this.#runStatus.bind(this)); - // Publish control while this truck is being driven. - this.#signals.run(this.#runControl.bind(this, connection)); - this.#signals.run(this.#runSafetyControl.bind(this, connection)); + // Dial the truck while driving it, or to deliver a bystander's e-stop. + this.#signals.run(this.#runControl.bind(this)); + this.#signals.run(this.#runSafetyControl.bind(this)); } /** Attach the canvas the UI created so the renderer can draw into it. */ @@ -206,10 +207,10 @@ export class Truck { this.#send({ type: "drive", ...state }); } - /** Request an emergency stop. */ /** - * Latch e-stop on this truck even without holding the driving lease. A viewer - * sends it on a one-shot safety broadcast, leaving the driving session alone. + * Latch e-stop on this truck even without holding the driving lease. The + * holder sends it on its driving session; a viewer dials a short safety + * session of its own, so it neither takes the lease nor waits for one. */ emergencyStop() { this.respawning.set(false); @@ -330,8 +331,8 @@ export class Truck { #runControlLease(effect: Moq.Signals.Effect) { if (!effect.get(this.active)) { - // Releasing control closes the driving broadcast, which may not have - // delivered this player's e-stop yet: carry it on the safety broadcast, + // Releasing control closes the driving session, which may not have + // delivered this player's e-stop yet: carry it on a safety session, // which stays up until a status echoes the latch. if (this.#estopUnconfirmed) this.#sendSafety({ type: "estop" }); this.respawning.set(false); @@ -398,73 +399,132 @@ export class Truck { if (status.estopped) this.#estopUnconfirmed = false; if (!this.#estopUnconfirmed) this.estopped.set(status.estopped); this.#acknowledgeSafety(status); - this.#acknowledge(status); this.status.set(status); this.statusState.set("available"); } }); } - #runControl(connection: Moq.Connection, effect: Moq.Signals.Effect) { - const conn = effect.get(connection.origin); - if (!conn) return; - + #runControl(effect: Moq.Signals.Effect) { if (!effect.get(this.active)) return; - const driverId = crypto.randomUUID().replaceAll("-", "").slice(0, 12); - this.driverId.set(driverId); - - const driverBroadcastRoute = conn.route(driverPath(this.name, driverId), { hidden: true }); - const driverBroadcast = driverBroadcastRoute.broadcast; + // A fresh endpoint and session per lease: the truck binds its single + // driver to the session, so a new lease is a new claimant. + const id = driverId(); + this.driverId.set(id); effect.cleanup(() => { - driverBroadcast.close(); this.driverId.set(undefined); this.roundTripMs.set(undefined); + this.pipeline.set(undefined); this.#sentAt.clear(); }); - const track = driverBroadcast.createTrack("control", { compress: true }); - const producer = new Json.Snapshot.Producer>({ track }); - effect.cleanup(() => producer.finish()); - effect.run(this.#runControlTrack.bind(this, track, producer)); - // A fresh instance per lease, which each publication mints: the truck binds its single - // driver to this path and instance. - driverBroadcastRoute.announce(); + const session = this.#dial(effect, id); + effect.run((nested) => { + const stream = nested.get(session); + // The truck refused the dial or ended the session, which ended the lease. + if (stream === null) return this.#release(); + if (!stream) return; + nested.spawn(async () => { + if (await this.#readReplies(nested, stream)) this.#release(); + }); + nested.run((inner) => { + const command = inner.get(this.#command); + if (command) write(stream, command); + }); + }); } - #runSafetyControl(connection: Moq.Connection, effect: Moq.Signals.Effect) { - const conn = effect.get(connection.origin); + #runSafetyControl(effect: Moq.Signals.Effect) { const command = effect.get(this.#safetyCommand); - if (!conn || !command) return; - - const driverId = `safety-${crypto.randomUUID().replaceAll("-", "").slice(0, 12)}`; - const broadcastRoute = conn.route(driverPath(this.name, driverId), { hidden: true }); - const broadcast = broadcastRoute.broadcast; - const safety = new SafetyCommandBroadcast(command, broadcast); - broadcastRoute.announce(); - effect.cleanup(() => safety.close()); + if (!command) return; + + const session = this.#dial(effect, driverId()); + effect.run((nested) => { + const stream = nested.get(session); + if (stream === null) return this.#undelivered(command); + if (!stream) return; + write(stream, command); + // The truck answers a bystander with nothing but the session's end. + nested.spawn(async () => { + if (await this.#readReplies(nested, stream)) this.#undelivered(command); + }); + }); + } + + /** + * Drop a safety command whose session failed before the status confirmed it. + * + * The local latch then follows the truck's status again, rather than showing a + * stop that never arrived: the viewer sees it and can press again. + */ + #undelivered(command: Command) { + if (this.#safetyCommand.peek() !== command) return; + console.warn(`${String(command.cmd.type)} for ${this.name} was not delivered`); + this.#safetyCommand.set(undefined); + this.#estopUnconfirmed = false; + this.estopped.set(this.status.peek()?.estopped ?? false); } - #runControlTrack( - track: Moq.Track.Producer, - producer: Json.Snapshot.Producer>, - effect: Moq.Signals.Effect, - ) { - if (effect.get(track.closed) !== undefined) return; + /** Dial the truck from a new endpoint `id`, both closed with `effect`. */ + #dial(effect: Moq.Signals.Effect, id: string): Session { + const session: Session = new Moq.Signals.Signal(undefined); + const endpoint = new Moq.socket.Endpoint(this.#origin, SOCKET, id); + effect.cleanup(() => { + // Finishing first ends the session cleanly rather than resetting it. + session.peek()?.producer.finish(); + endpoint.close(); + }); + effect.spawn(async () => { + try { + session.set(await endpoint.dialTrack(this.name, undefined, { signal: effect.abort })); + } catch (err) { + if (effect.abort.aborted) return; + console.warn(`dialing ${this.name} failed:`, err); + session.set(null); + } + }); + return session; + } - const command = effect.get(this.#command); - if (!command) return; + /** Read the truck's acknowledgements, returning true once the session ends unless `effect` closed it. */ + async #readReplies(effect: Moq.Signals.Effect, stream: Moq.socket.TrackStream): Promise { + // An acknowledgement a heartbeat late is superseded by the next one. + stream.consumer.update({ order: "ascending", delay: { max: CONTROL_HEARTBEAT } }); + const replies = stream.consumer.ordered(); + try { + for (;;) { + const frame = await effect.race(replies.readFrame()); + if (effect.abort.aborted) return false; + if (!frame) break; + let reply: Reply; + try { + reply = ReplySchema.parse(JSON.parse(decoder.decode(frame.payload))); + } catch (err) { + console.warn("Invalid reply JSON:", err); + continue; + } + this.#acknowledge(reply); + } + } catch (err) { + if (effect.abort.aborted) return false; + console.warn(`session with ${this.name} failed:`, err); + } + return true; + } - producer.update({ value: { ...command.cmd, sequence: command.sequence, timestamps: command.timestamps } }); + /** Give up driving this truck, if this player still is. */ + #release() { + if (this.#driving.peek() === this.id) this.#driving.set(undefined); } - #acknowledge(status: TruckStatus) { - if (status.driver !== this.driverId.peek() || status.sequence == null) return; - const sentAt = this.#sentAt.get(status.sequence); + #acknowledge(reply: Reply) { + this.pipeline.set(reply.media_pipeline); + const sentAt = this.#sentAt.get(reply.sequence); if (sentAt == null) return; this.roundTripMs.set(performance.now() - sentAt); for (const sequence of this.#sentAt.keys()) { - if (sequence <= status.sequence) this.#sentAt.delete(sequence); + if (sequence <= reply.sequence) this.#sentAt.delete(sequence); } } diff --git a/demo/relay/prod.toml b/demo/relay/prod.toml index 7ec7b44c0b0..fb27ffc869d 100644 --- a/demo/relay/prod.toml +++ b/demo/relay/prod.toml @@ -45,13 +45,15 @@ key = "key.pem" # # moq auth serve --listen 127.0.0.1:4440 --key root.jwk \ # --public-subscribe 'anon/**' --public-subscribe 'demo/**' \ -# --public-publish 'anon/**' --public-publish 'demo/viewer/**' \ +# --public-publish 'anon/**' \ # --public-publish 'demo/boy/.socket/viewer-*' \ +# --public-publish 'demo/carla/.socket/driver-*' \ # --mtls-publish '**' --mtls-subscribe '**' --mtls-link peer # -# Anonymous MoQ Boy viewers share one grant, so it covers only `viewer-*` endpoint ids: -# a game's endpoint, `demo/boy/.socket/`, stays the emulator's, which signs in -# with a token. Each viewer's id is 128 random bits, so no viewer can guess another's. +# Anonymous MoQ Boy viewers and CARLA drivers get one shared grant each, covering only +# `viewer-*` and `driver-*` endpoint ids: a game's or truck's endpoint stays its +# publisher's, which signs in with a token. Each viewer's id is 128 random bits, so no +# viewer can guess another's. # # For multiple keys with rotation, give it `--key-dir /path/to/keys/` instead: # keys are named {kid}.jwk and resolved by the kid in the JWT header. diff --git a/doc/bin/demo.md b/doc/bin/demo.md index e9665306ec8..f03666ee29e 100644 --- a/doc/bin/demo.md +++ b/doc/bin/demo.md @@ -66,7 +66,7 @@ just carla # relay + trucks + player What it demonstrates: - **GPU to wire.** Camera images never leave the GPU: `moq-video` encodes them on the device that rendered them, and a camera renders only while someone subscribes. -- **Control with a lease.** A driver publishes a JSON `control` track under `viewer/carla/{truck}/`; the truck accepts one driver at a time, bound to the instance it announced with, and returns to autonomy when the lease lapses. +- **Control with a lease.** A driver dials the truck's [socket](/concept/socket) endpoint and sends JSON controls on its own session, which carries the truck's acknowledgements back; the truck accepts one driver at a time, bound to that session, and returns to autonomy when the lease lapses. - **Measured feedback.** Each truck's `status` track carries what the gate accepted and what the vehicle did, so the player draws requested against measured steering. Code: [`demo/carla`](https://github.com/moq-dev/moq/tree/main/demo/carla). diff --git a/packaging/moq-relay/relay.toml b/packaging/moq-relay/relay.toml index 801d3b409e5..1a26eb48d79 100644 --- a/packaging/moq-relay/relay.toml +++ b/packaging/moq-relay/relay.toml @@ -33,7 +33,7 @@ key = "/var/lib/moq-relay/key.pem" # # moq auth serve --listen 127.0.0.1:4440 --key /var/lib/moq-relay/root.jwk \ # --public-subscribe 'anon/**' --public-subscribe 'demo/**' \ -# --public-publish 'anon/**' --public-publish 'demo/viewer/**' +# --public-publish 'anon/**' # # Generate the key with `moq auth generate`. See https://moq.dev/docs/auth. [auth] diff --git a/quest/m1/README.md b/quest/m1/README.md index 5133c9d9ede..96293c1a688 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -43,7 +43,6 @@ the same reason, gates the cut through the - [The overcommitted receive-window cells run again](/quest/m1/receive-windows-evicted.md) - `benchmark_receive_windows` measures its 256-stream and 4 MiB-frame cells again, reporting delivered frames and evictions instead of stalling - [Datagram tracks](/quest/m1/datagram-tracks/README.md) - a track is datagrams or streams; datagram tracks have their own request and sequence, no ranges, no cache, and map to object ID 0 on moq-transport - [Stream credit by priority](/quest/m1/stream-credit/README.md) - every stream opens through one priority entry point, and a higher-priority group takes over or resets a lower one when credit runs out -- [Sockets](/quest/m1/socket/README.md) - a generic dial/accept `Socket` in moq-net, Rust, JS, and Python, with tracks as streams; moq-boy and CARLA move onto it - [Audio jitter target](/quest/m1/audio-jitter-target/README.md) - the playout target is default in both languages; time-stretched browser playback without steady-state underruns and a native trace replay remain - [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS container consumer drops groups below its read cursor like Rust - [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - a buffered `` asks for the oldest group first, so a voice reply's opening arrives first, and audio's head hold shrinks to the floor diff --git a/quest/m1/socket/README.md b/quest/m1/socket/README.md deleted file mode 100644 index e55cc90e90f..00000000000 --- a/quest/m1/socket/README.md +++ /dev/null @@ -1,196 +0,0 @@ -# Sockets over MoQ - -## Goal - -`moq-net` gains a generic dial/accept primitive. An `Endpoint` announces one -hidden broadcast, dials other endpoints, and accepts their dials, and each -connection is a session between the two ends: a reliable, ordered byte pipe -each way (`tcp`), or a track of messages each way (`track`). moq-boy, CARLA, [byte streams](/quest/m3/stream/README.md), and the -mesh [Endpoint](/quest/m3/mesh/endpoint.md) all build on it instead of each -hand-rolling the announce, watch, and request loop. Rust, JS, and Python -(through moq-ffi) expose the same surface and interoperate. - -Non-goals: a wire change to moq-lite (a socket is a path and track convention -over existing announce and subscribe streams); session caps, leases, or idle -timeouts, which stay application logic (boy's 60 s idle, CARLA's one-driver -lease and 1 s watchdog); signing and encryption, which the m3 -[mesh](/quest/m3/mesh/README.md) owns by making endpoint ids public keys. - -## Plan - -[`doc/concept/socket.md`](/doc/concept/socket.md) is the convention JS and -Python follow, and `rs/moq-net/src/model/socket.rs` implements it. - -Decided 2026-11-04, second round (maintainer, settling the Waiting items of -[moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677)): - -- Name sessions `///`, binding a session to both - ends. An endpoint resets a dial naming another acceptor, and the dialer - serves only the name it dialed, so an endpoint `A` dialed can't replay the - session to a third endpoint posing as `A`. Done now, before the JS and Python - ports copy the scheme, rather than waiting for the m2 mesh keys. Not chosen: - keep `/` and document the replay. -- An accept waits for the claimed dialer's endpoint for as long as the dial - stays open, with no timer: a relay can deliver a dial before the dialer's - announcement, such as when two endpoints dial each other at start-up. A - forged dial claiming an endpoint nobody announces holds an accept only while - the forger holds its subscription. Not chosen: fail at once and let the - dialer redial; a bounded timer. - -Decided 2026-11-04 (maintainer, /quest-complete), on -[moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677), settling its -Waiting items and replacing the `.tcp/` layout and the one-message-per-group -pipe below; the second round above adds the acceptor to the session name: - -- Name the dialer in the session. A dial and an answer can't be confused, - the claimed dialer is named for the return-routability check, and `S` - (random per dial) makes every session unique, so one endpoint can dial and - accept any number of sessions. Not chosen: reader-named halves. -- One announcement per endpoint, used only for routing: - `/.socket/`. The session track name carries the protocol, both - ends and the session: `///`. A dial is - SUBSCRIBE `/.socket/`, track `tcp///`; the - answer is the same name in the dialer's broadcast. A requested track whose protocol the - endpoint doesn't speak, or isn't accepting, is reset as unsupported. Grants: - publish `.socket/`, subscribe `.socket/**`. The module stays `socket`. -- Two protocols ship now over the same handshake, each with its own dial and - accept, and an incoming dial lands in its protocol's queue: - - `tcp///`: a session half is a single group, not a track of - messages. A group is one QUIC stream on every hop, so it's reliable, - strictly ordered and flow controlled, like a TCP direction. Frames are - writes, finishing the group is FIN, and resetting it is RST. There are no - track-level settings (budget, max-delay, order), since the newest group is - never stale. A second group on a session track is a protocol error that - resets the session. Rust: `Endpoint::dial_tcp` / `accept_tcp`, yielding - `socket::TcpStream { producer: group::Producer, consumer: group::Consumer }` - plus `peer()` and `session()`. Not chosen: `track::Ordered` with one message - per group. - - `track///`: the same dial, accept, reject and return-routability - check, but the app gets the whole session tracks, with groups as messages - and its own priority, budget and order, and no single-group rule. The - reader starts reliable and ascending (no delay budget) until the app - updates it. A half has no max age unless the app sets one, so a newer - message never makes an older one stale. Rust: `Endpoint::dial_track` / - `accept_track`, yielding - `socket::TrackStream { producer: track::Producer, consumer: track::Subscriber }`. -- Long-lived `tcp` sessions rely on unbounded groups (since - [moq-dev/moq#7411](https://github.com/moq-dev/moq/pull/7411) a group has no - size cap, and each copy keeps a ~32 MiB tail). With one reader per copy, - [fastest-reader pacing](https://github.com/moq-dev/moq/pull/7664) acts - as TCP backpressure: a writer awaiting `group::Producer::ready` runs at most - 16 MiB past its reader on each hop, and a reader that falls off the tail - fails with `Evicted` rather than skipping data. Where pacing doesn't reach - yet, that `Evicted` is the failure mode, never silent loss: browsers and JS - read unpaced ([JS fastest-reader pacing](/quest/m4/js-fastest-reader.md)), - and a session past its paused-credit cap (about two paused group streams at - the client defaults) reads and sheds instead of pausing. That remaining - pacing work stays in the [unbounded groups](/quest/m2/unbounded-groups/README.md) - epic, which this depends on for lossless `tcp` in those cases. - -Implementation choices in -[moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677): - -- A `tcp` session runs a small task on the endpoint origin's driver. It copies - the peer's group into the group the app reads, a frame at a time and paced by - that reader, so it can end that group itself: on a second group from the - peer, on the endpoint closing, or when the peer resets. It also finishes or - aborts this side's track the way the app ended its group, and resets the - session when the app drops its reader early. Once group 0 arrives, it - watches for a second group through the subscriber's largest position rather - than `poll_recv_group`: polling `poll_recv_group` on every wake while - reading the group it handed out stalled the group's delivery (see the PR). -- `TrackStream::consumer` is the handshake's own `track::Subscriber`, not a bare - `track::Consumer`: it is already subscribed reliably and in order, so no - message falls between the accept and the app's first read, and the app - overrides it with `update` or reads in sequence through `ordered()`. -- An endpoint takes dials in a protocol from its first accept in it, like a TCP - listener. Until its first accept in any protocol it has no request handler, - so a dial to it is reset with not found at once. A dial reserves its own - half, which the peer's answer finds without a handler, so a dial-only - endpoint never needs polling. A dial waits for the peer's endpoint to be - announced (drop it to give up), and an accept waits for the claimed - dialer's endpoint until the dialer drops its subscription. -- Accepted dials queue while the app accepts, each holding the dialer's - subscription open. A queued dial whose dialer gave up leaves its queue on the - endpoint's next poll, which each new dial triggers, so a queue holds only - open dials, even in a protocol the app stopped accepting. -- Session track names are the secret, so they never appear in a lite-07 track - listing or anywhere else a third party reads. moq-net lists only tracks a - producer creates or a handler's `Lister` answers, never one served on - request, so an endpoint serves its session tracks only on request, and the - dialer's own half is reserved unlisted - (`broadcast::Producer::reserve_unlisted_track`, crate-private). - -Decided 2026-11-03 (maintainer, /quest-grill in /quest-complete), still -standing where the 2026-11-04 decisions above don't replace it: - -- Endpoint sockets, one session per connection. Each endpoint announces one - hidden broadcast once, and connections are never announced. `` is a - well-known name for a server (a game) or a random id for a client. Why: the - old flow had the listener answer `L/.accept/` under the dialer's - instance, which broke instance uniqueness (lite-07 treats routes sharing an - instance as replicas), let two listeners accept one dial, announced every - connection publicly, and needed lite-07 instances and track listings on - every hop. -- Dial: `A` picks a random, unguessable session id `S` (128 bits) and - subscribes to `B`'s endpoint; `B` surfaces the requested track as an - incoming dial claiming to be from `A`. Accept: `B` subscribes to `A`'s - endpoint, and `A` answers only a session it dialed and resets any other, a - return-routability check that fails a dial forged in `A`'s name. `B` sends - nothing until `A` answers. Reject: `B` resets `A`'s subscription. -- No multiplexing: another channel is another session, a pair of endpoints - holds several, and a restart is a new session. -- Grants are prefix only, rooted at the prefix. A relay test signs real - `moq-auth` tokens. -- The handshake uses no instances or listings; lite-07 instances only affect - relay route choice. Tests cross moq-transport hops. Corrected 2026-11-04 in - [moq-dev/moq#7677](https://github.com/moq-dev/moq/pull/7677): it needs - moq-lite-07 or moq-transport on every hop, since a moq-lite hop before 07 - has no hidden opt-in and never carries an endpoint's `.socket/` - broadcast. Relays' cluster links carry hidden broadcasts on every version, - so an endpoint's own client session is the hop that decides: while one is - moq-lite before 07, its dials and accepts fail `Unsupported` instead of - waiting ([moq-dev/moq#7888](https://github.com/moq-dev/moq/pull/7888)). -- Fixed channels become sessions or messages on one: boy's `command` and - per-viewer latency, CARLA's `control`, ack, and e-stop. Shared state such as - boy's `status` stays an ordinary broadcast. -- The stream types are plain structs with public `producer` (this side's half) - and `consumer` (the peer's half) fields, plus read-only `peer()` and - `session()`, and no read, write, or message methods of their own: apps use - the group and track APIs directly, so JS, Python, and the bindings mirror a - minimal surface. - -Decided 2026-11-02 (maintainer, Kixel planning): the whole epic moves from -m2 to m1, so the next cut publishes sockets in `@moq/net` and moq.pro's -relays carry them. Why: Kixel's T3 Code prototype (its T3 epic, in the -private kixel.ai repo) sends T3's commands over sockets, one group per message, -in place of byte streams. `@moq/net` publishes from `release`. Byte streams, -fastest-reader pacing, and the mesh libraries keep their own milestones. - -Decided 2026-11-01 with the maintainer (planning interview from moq.pro's TCP -replacement arc), still standing: - -- Home: a `socket` module of `moq-net` (`moq_net::socket`), the `socket` - export of `@moq/net`, and moq-ffi beside origin. Not a separate crate: it - uses only the origin, broadcast, and track APIs every binding already wraps. -- The convention is specified on a doc.moq.dev concept page, - `doc/concept/socket.md`, not in the moq-lite draft: it is an application - convention over moq-lite. - -Later, not built now: a multiplexed `quic` protocol, `quic///` on -the same endpoint with no new announcement, which brings back several groups -or streams per session; and signed or encrypted sessions with ids as public -keys, in the m3 [mesh](/quest/m3/mesh/README.md). - -This epic owns removing the hand-rolled loops: after the ports, no -announce-watch-request pairing remains in `rs/moq-boy`, `js/moq-boy`, or -`demo/carla`. - -## Required - -- [CARLA on sockets](/quest/m1/socket/carla.md) - drivers dial the truck's endpoint and send control and receive acks on their session - -## Related - -- [Byte streams](/quest/m3/stream/README.md) - TCP over sockets, one session per connection -- [Endpoint API](/quest/m3/mesh/endpoint.md) - the mesh's `connect`/`accept`, a socket endpoint plus signing diff --git a/quest/m1/socket/carla.md b/quest/m1/socket/carla.md deleted file mode 100644 index 170c8678751..00000000000 --- a/quest/m1/socket/carla.md +++ /dev/null @@ -1,42 +0,0 @@ -# [M] CARLA on sockets - -## Goal - -Each CARLA truck runs a socket endpoint named for the truck and accepts -drivers on it, and the web driver dials it from its own endpoint, so -`demo/carla/sim` and `demo/carla/web` no longer watch `viewer/carla//` -or request driver broadcasts by hand. Each driver's control, ack, sequence -echo, and e-stop travel on its own sessions. The one-driver lease, takeover, -e-stop, the 1 s watchdog, and handback behave as before. - -## Plan - -See the [epic](/quest/m1/socket/README.md) for the convention. - -- The worker (`publish.py`) runs a Python endpoint per truck; `paths.ts` and - `truck.ts` dial it from an endpoint with the per-lease driver id. A new lease - is a new session, which matches today's lease binding. -- The `control` track becomes the driver's messages on a `track` session, one - control per group, so a late control never blocks a newer one; the driver - may read the truck's replies with a finite delay budget. The - driver's ack, sequence, and round-trip echo move from the shared `status` to - the truck's messages on that session. `driver` and `estop_by` stay in the - ordinary `status` broadcast, because every viewer shows them. -- The `safety-` e-stop broadcast becomes messages on the driver's session - or a second session of its own; decide in the PR and record why, such as - keeping the e-stop off the control session's queue. -- The anonymous grants let a driver publish its own endpoint and subscribe to - the truck endpoints. moq-boy moved to sockets already, so this removes - `demo/viewer/**` from `demo/relay/prod.toml` and its documented - `moq auth serve` line. Anonymous sessions share one grant, so follow boy's - scheme: random driver ids under a fixed id prefix that the anonymous grant - covers and no truck id takes, as `demo/boy/.socket/viewer-*` does for boy. The worker's token keeps publish on `carla/**`, which - covers its endpoints, and swaps its subscribe scope from `viewer/carla/**` - to the driver endpoints; say so in `demo/carla/README.md`. -- Update `demo/carla/README.md` (paths, lease) and `doc/bin/demo.md`. -- Decided 2026-11-03 (maintainer, follow-up planning after the 2026-11-03 /quest-afk and /quest-complete runs): [moq-dev/moq#7678](https://github.com/moq-dev/moq/pull/7678) keys carla's - Python driver lease on a local run counter, because moq-ffi's - `MoqBroadcastConsumer` exposes no instance. If the move to sockets doesn't - make that counter moot, add a `MoqBroadcastConsumer::instance()` accessor - (and its Python wrapper) and key the lease on the real instance. Options - shown: m0; ✅ m1, in this quest; m2. diff --git a/quest/m3/mesh/README.md b/quest/m3/mesh/README.md index 199dabeba4f..46b56ce1831 100644 --- a/quest/m3/mesh/README.md +++ b/quest/m3/mesh/README.md @@ -52,7 +52,7 @@ Decided 2026-10-25 with the maintainer (planning interview), still in force: reaching an ID is subscribing there. There is no separate directory. - The token is the binding: enrollment proves the device key, then the coordinator issues a grant to publish `mesh//**` and subscribe - `mesh/**`, plus publish on its own [socket](/quest/m1/socket/README.md) + `mesh/**`, plus publish on its own [socket](/doc/concept/socket.md) endpoint, `.socket/`, so it can dial and accept under its own id (amended 2026-11-01, 2026-11-03 when sockets became endpoints, and 2026-11-04 when the segment became `.socket`). Relays need no new check. Membership is the coordinator's diff --git a/quest/m3/stream/README.md b/quest/m3/stream/README.md index f10a6ab4b1a..5dd0332469d 100644 --- a/quest/m3/stream/README.md +++ b/quest/m3/stream/README.md @@ -17,7 +17,7 @@ fastest-reader pacing gives. Decided 2026-10-25 with the maintainer (mesh planning interview): - TCP-level proxying over groups, never IP over datagrams. -- A forwarded service is one [socket](/quest/m1/socket/README.md) endpoint, +- A forwarded service is one [socket](/doc/concept/socket.md) endpoint, and each TCP connection is one session on it (amended 2026-11-03, below), one group each way; each TCP `read()` becomes one frame, because a frame's length must be known before it is written and a read has one. It coalesces whatever is @@ -42,7 +42,7 @@ Decided 2026-10-25 with the maintainer (mesh planning interview): Amended 2026-11-01 with the maintainer (socket planning interview), 2026-11-03 when sockets became endpoints, and 2026-11-04 when the socket's `tcp` protocol became one group each way: connection setup is the -[socket](/quest/m1/socket/README.md) epic's `Endpoint`, `dial_tcp`, and +[socket](/doc/concept/socket.md) `Endpoint`, `dial_tcp`, and `accept_tcp`, which settle the layout (`/.socket/`, a `tcp///` track each way). Each TCP connection is one `tcp` session, whose halves are a single group each way for the connection's life: diff --git a/quest/m5/announce-credit.md b/quest/m5/announce-credit.md index 00f8e6ffe8d..15ee60d2919 100644 --- a/quest/m5/announce-credit.md +++ b/quest/m5/announce-credit.md @@ -14,7 +14,7 @@ unpublished lite version after lite-07. ## Plan Decided 2026-11-01 with the maintainer (the follow-up to the -[sockets](/quest/m1/socket/README.md) interview): +sockets interview): - Backpressure is fixed at the wire, not by a local cap. The draft already tells implementations to limit matching broadcasts but gives the subscriber diff --git a/quest/m5/socket-old-hop-signal.md b/quest/m5/socket-old-hop-signal.md index af62fd91140..ac61be45479 100644 --- a/quest/m5/socket-old-hop-signal.md +++ b/quest/m5/socket-old-hop-signal.md @@ -94,5 +94,4 @@ Wire: a new per-path marker on the current moq-lite wip. ## Related -- [Sockets](/quest/m1/socket/README.md) - the dial and accept this reports on, and #7888's first-hop check - [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the route layer a marker may ride, and how interest crosses cluster links