Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions .claude/skills/verify-reconnect/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,34 +25,33 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re

```sh
quest gate just rs tla moq-net/verify/reconnect/Handover.tla
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartHandoff.cfg
```

The first checks every rule over four sessions: about 1.5M distinct states, under a minute. The second and third check the same rules under `Restart::Wait` and `Restart::Handoff` over three sessions: about 330k states each. All pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.
The first checks every rule over four sessions: about 1.5M distinct states, under a minute. The second checks the same rules under `Restart::Handoff` over three sessions: about 330k states. All pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.

## Open findings

None.

## Reading a failure

Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the policy with the helpers in `reconnect/tests.rs` (`next`, `handshake`, `on_fallback`, `probe`, `connected`, `failed`, `probed`, a server's `drain().send(..)`), one call per step, on the simulated clock. A path through `Shared` needs `moq-tokio`'s connection tests instead.
Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the policy with the helpers in `reconnect/tests.rs` (`next`, `handshake`, `on_fallback`, `probe`, `connected`, `failed`, `probed`, a server's `goaway(..)`), one call per step, on the simulated clock. A path through `Shared` needs `moq-tokio`'s connection tests instead.

## Coverage

Sync commit: `8f8eec388694cd74555b9a8adfbf63d0bc07da92`. Run `git log 8f8eec3886.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `27b46ef35bdf284ba61158521723768436cd8460`. Run `git log 27b46ef35b.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
| `rs/moq-net/src/reconnect.rs` `Reconnect::poll` (queue, predecessor, phases), `follow` | `Pop`, `Retire`, `IdleClose`, `FollowPred`, `Dial`, `Abandon`, `BackoffEnds`, `LiveEnds`, `ProbeAsk`, `ProbeAbandon` |
| `rs/moq-net/src/reconnect.rs` `established`, set by `end` and by a GOAWAY in `poll`, and `walk_failed` and `closed` failing fast until it is set | `established`, `EndGoaway`, `LiveEnds`, `Failed`, `Abandon`, `EndClosed` |
| `rs/moq-net/src/reconnect.rs` `connected`, `fallback`, `failed`, `probed` | `Connected`, `Failed`, `ProbeFails`, `ProbeGetsThrough` |
| `rs/moq-net/src/reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` |
| `rs/moq-net/src/reconnect.rs` `Restart::Wait`, `Restart::Handoff` | `LiveEnds` and `WaitOut` with `Restart = "wait"` (`RestartWait.cfg`) and `"handoff"` (`RestartHandoff.cfg`) |
| `rs/moq-net/src/reconnect.rs` `Restart::Handoff` | `LiveEnds` and `WaitOut` with `Restart = "handoff"` (`RestartHandoff.cfg`) |
| `rs/moq-net/src/reconnect/redirect.rs` `Redirect::target` | `Refused` and `Endpoints` |
| `rs/moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `retired`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` |

## Not covered

One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait` and `Restart::Handoff`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing.
One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Handoff`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing.
10 changes: 7 additions & 3 deletions cpp/obs/src/moq-settings.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -204,8 +204,9 @@ const std::vector<Field> &Fields()
1, 60000, 100, "Delay before the first reconnect attempt; it grows from here."));
f.push_back(Number(BACKOFF_MAX, "Reconnect delay cap (ms)", Millis(d.backoff.max, BACKOFF_MAX), 1,
600000, 1000, "Ceiling on the growing reconnect delay."));
f.push_back(Number(BACKOFF_TIMEOUT, "Give up after (ms)", Millis(d.backoff.timeout, BACKOFF_TIMEOUT), 0,
3600000, 1000,
// The library's default never gives up, which this field spells as 0.
f.push_back(Number(BACKOFF_TIMEOUT, "Give up after (ms)",
d.backoff.timeout ? Millis(d.backoff.timeout, BACKOFF_TIMEOUT) : 0, 0, 3600000, 1000,
"Total time to keep reconnecting after a drop before the stream fails. 0 retries "
"forever. A first connect that reaches no server fails the stream at once."));

Expand Down Expand Up @@ -340,7 +341,10 @@ std::string Configure(obs_data_t *settings, moq::ClientConfig &config)

config.backoff.initial = Milliseconds(settings, BACKOFF_INITIAL, error);
config.backoff.max = Milliseconds(settings, BACKOFF_MAX, error);
config.backoff.timeout = Milliseconds(settings, BACKOFF_TIMEOUT, error);
// 0 here means never give up, which the library spells as no timeout: its zero gives up
// at the first failure.
if (const auto timeout = Milliseconds(settings, BACKOFF_TIMEOUT, error); timeout.count() != 0)
config.backoff.timeout = timeout;

// Automatic leaves the tri-states and the congestion family unset, which keeps the
// transport free to pick.
Expand Down
3 changes: 2 additions & 1 deletion cpp/obs/test/moq-settings-test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,8 @@ int main()
CHECK(settings.default_ints["quic_idle_timeout_ms"] == Ms(*library.quic.idle_timeout));
CHECK(!library.quic.keep_alive);
CHECK(settings.default_ints["quic_keep_alive_ms"] == 0);
CHECK(settings.default_ints["backoff_timeout_ms"] == Ms(*library.backoff.timeout));
CHECK(!library.backoff.timeout);
CHECK(settings.default_ints["backoff_timeout_ms"] == 0);
CHECK(settings.default_bools["websocket_enabled"] == *library.websocket.enabled);
CHECK(settings.default_strings["quic_gso"].empty());

Expand Down
8 changes: 5 additions & 3 deletions dart/moq/lib/src/aliases.dart
Original file line number Diff line number Diff line change
Expand Up @@ -275,9 +275,11 @@ typedef ConnectionPhaseClosed = ClosedMoqConnectionPhase;
typedef ConnectionState = MoqConnectionState;

/// Retry pacing for the automatic reconnect: initial delay, multiplier, ceiling, and give-up window.
/// Until a session has stayed up past 1s or been drained by a GOAWAY, a failed
/// dial or a closed session, such as a refused credential, ends the connection
/// with its error rather than retrying.
/// A null field keeps its default, zero means zero (retry at once, or give up
/// at the first failure), and `maxUnlimited` lifts the ceiling. Until a session
/// has stayed up past 1s or been drained by a GOAWAY, a failed dial or a closed
/// session, such as a refused credential, ends the connection with its error
/// rather than retrying.
typedef Backoff = MoqBackoff;

/// Whether a protocol code is from the session or stream registry.
Expand Down
14 changes: 14 additions & 0 deletions dart/moq_ffi/lib/src/moq.dart
Original file line number Diff line number Diff line change
Expand Up @@ -3554,11 +3554,13 @@ class MoqBackoff {
final Duration? initial;
final int? multiplier;
final Duration? max;
final bool maxUnlimited;
final Duration? timeout;
MoqBackoff({
this.initial = null,
this.multiplier = null,
this.max = null,
this.maxUnlimited = false,
this.timeout = null,
});
}
Expand All @@ -3585,6 +3587,11 @@ class FfiConverterMoqBackoff {
);
final max = max_lifted.value;
new_offset += max_lifted.bytesRead;
final maxUnlimited_lifted = FfiConverterBool.read(
Uint8List.view(buf.buffer, new_offset),
);
final maxUnlimited = maxUnlimited_lifted.value;
new_offset += maxUnlimited_lifted.bytesRead;
final timeout_lifted = FfiConverterOptionalDuration.read(
Uint8List.view(buf.buffer, new_offset),
);
Expand All @@ -3595,6 +3602,7 @@ class FfiConverterMoqBackoff {
initial: initial,
multiplier: multiplier,
max: max,
maxUnlimited: maxUnlimited,
timeout: timeout,
),
new_offset - buf.offsetInBytes,
Expand All @@ -3606,6 +3614,7 @@ class FfiConverterMoqBackoff {
FfiConverterOptionalDuration.allocationSize(value.initial) +
FfiConverterOptionalUInt32.allocationSize(value.multiplier) +
FfiConverterOptionalDuration.allocationSize(value.max) +
FfiConverterBool.allocationSize(value.maxUnlimited) +
FfiConverterOptionalDuration.allocationSize(value.timeout) +
0;
final buf = Uint8List(total_length);
Expand All @@ -3627,6 +3636,10 @@ class FfiConverterMoqBackoff {
value.max,
Uint8List.view(buf.buffer, new_offset),
);
new_offset += FfiConverterBool.write(
value.maxUnlimited,
Uint8List.view(buf.buffer, new_offset),
);
new_offset += FfiConverterOptionalDuration.write(
value.timeout,
Uint8List.view(buf.buffer, new_offset),
Expand All @@ -3638,6 +3651,7 @@ class FfiConverterMoqBackoff {
return FfiConverterOptionalDuration.allocationSize(value.initial) +
FfiConverterOptionalUInt32.allocationSize(value.multiplier) +
FfiConverterOptionalDuration.allocationSize(value.max) +
FfiConverterBool.allocationSize(value.maxUnlimited) +
FfiConverterOptionalDuration.allocationSize(value.timeout) +
0;
}
Expand Down
2 changes: 1 addition & 1 deletion doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,7 @@ services.moq-relay = {
```

Each dialed session binds its own UDP socket on an ephemeral port of
`connect.bind`'s address. Once `connect.backoff.timeout` (10s when unset or `0`,
`connect.bind`'s address. Once `connect.backoff.timeout` (10s when unset or `none`,
which would otherwise retry forever) passes without a session, the redial leaves from a
fresh port, so a network path that drops one UDP flow can't hold a link down
while other ports to the same peer would work. A fixed `connect.bind` port opts
Expand Down
2 changes: 1 addition & 1 deletion doc/lib/c/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ inside. The header is the reference: each function carries a doc comment.
- **Errors.** Negative return codes are named by `MOQ_ERROR_*`, and `moq_error()` gives the reason on the calling thread. Auth rejections have their own codes so you don't retry them. For the peer's protocol code use `moq_error_protocol()`; do not parse `moq_error()`. A panic in a binding background task reports `MOQ_ERROR_PANIC` through its terminal callback; unrelated objects keep running. A session closed because its credential expired has kind `MOQ_PROTOCOL_KIND_EXPIRED`, and a request refused for an expired token returns `MOQ_ERROR_EXPIRED`; refresh the token before retrying. Malformed credentials report `MOQ_PROTOCOL_KIND_MALFORMED_AUTH_TOKEN`; correct the token before reconnecting with it. A catalog that needs a newer moq-hang draft than this build reads returns `MOQ_ERROR_UNSUPPORTED_CATALOG`; upgrade the library to read it. A request-cap closure has kind `MOQ_PROTOCOL_KIND_TOO_MANY_REQUESTS`; reduce the workload or adjust the relay's [session limits](/bin/relay/config#session-and-peer) before reconnecting.
- **Tokens.** Set `moq_client_config.tokens` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. `moq_session_tokens_set()` replaces them before they expire: the next dial presents them, and a live AUTH session at once. A live session without AUTH (lite before 07, or moq-transport without the extension) keeps the old credential until it redials, and `moq_session_tokens_answered()` reports `MOQ_ERROR_UNSUPPORTED` for it. Otherwise that call reports the peer's verdict. Follow what the peer granted with `moq_session_grants()`. For a per-request token, `moq_session_credential()` presents it on the live session, or the next one while offline, and returns a credential handle at once: name it with `moq_route_set_auth()` or `moq_claim_set_auth()`, and setting a new one renews the announcement in place, without redialing. Only the session that minted it carries it, a dialed one across its reconnects; another session announcing the same route uses its own scope. `moq_credential_answered()` reports the peer's verdict: moq-transport accepts at once, since only the request carries it, moq-lite needs AUTH, where the grant joins the session's, and is `MOQ_ERROR_UNSUPPORTED` without it. `moq_credential_free()` withdraws the token once no route or claim names it, answered or not, and ends a wait still pending.
- **Threading.** Call any function from any thread. Callbacks run on moq-c's runtime thread; the callback and its `user_data` must be safe to use there. Synchronize access to data shared with other threads, and dispatch UI work to your UI thread. Raw publish calls block until the codec takes the frame, which paces a publisher.
- **Zeroed config is the default.** A zeroed `moq_client_config` means the default for every field, so new fields don't break callers. A field whose default isn't zero has a `has_*` flag: `backoff_timeout_us = 0` means "retry forever" only with `has_backoff_timeout = true`.
- **Zeroed config is the default.** A zeroed `moq_client_config` means the default for every field, so new fields don't break callers. A field whose default isn't zero has a `has_*` flag: `backoff_timeout_us = 0` gives up at the first failure only with `has_backoff_timeout = true`, and without it the session never gives up.
- **Subscription defaults.** A zeroed `moq_subscription`, a NULL pointer, and `moq_subscription_defaults()` all mean moq-net's defaults, whose zero `delay` starts at the newest group and skips a superseded one at once. Set `delay.max_unlimited` for no subscriber limit, and `delay.min_unstated` to leave the target latency unstated.
- **Compressed tracks.** `moq_track_info.compression` set to `MOQ_COMPRESSION_DEFLATE` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames.
- **fMP4 import.** `MOQ_CONTAINER_FORMAT_FMP4` splits each fragment into one frame per sample.
Expand Down
2 changes: 1 addition & 1 deletion doc/lib/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ The wrapped bindings share one feature set, so the language pages only show
how it looks in that language:

- **Connect** to a relay with TLS options and a JWT in the URL, or **serve** sessions yourself and accept or reject each request by path.
- **Reconnect** automatically with backoff when the transport drops. Until a session has stayed up past 1s or been drained by a GOAWAY it fails fast instead: a relay that never answered with a session, or closed it, as one refusing a credential does, ends the connection with its error, for the app to retry.
- **Reconnect** automatically with backoff when the transport drops. Until a session has stayed up past 1s or been drained by a GOAWAY it fails fast instead: a relay that never answered with a session, or closed it, as one refusing a credential does, ends the connection with its error, for the app to retry. The backoff's `max` and `timeout` read the same everywhere: unset takes the default (5s, and never give up), zero means zero (retry at once, or give up at the first failure), and a separate off spelling lifts the bound (`none` in moq-tokio's flags and config, `null` in JS, `max_unlimited` or a null `timeout` in the bindings).
- **Discover** broadcasts by prefix, wait for a specific one, or request one by path. Advertise an exact path (a created broadcast is invisible until announced), or claim a prefix and serve requests beneath it on demand.
- **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate. The audio decoder reports the playout delay to buffer ahead of the playhead, fixed or estimated from arrivals, so read each frame as soon as the last is handled rather than pacing reads to the speaker.
- **Raw stream tracks** of arbitrary bytes, with per-subscriber priority and max delay, and separate live datagram tracks without groups or caching.
Expand Down
Loading