diff --git a/.claude/skills/verify-reconnect/SKILL.md b/.claude/skills/verify-reconnect/SKILL.md index 3807179c5cf..5ac64091797 100644 --- a/.claude/skills/verify-reconnect/SKILL.md +++ b/.claude/skills/verify-reconnect/SKILL.md @@ -25,11 +25,10 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re ```sh quest gate just rs tla moq-net/verify/reconnect/Handover.tla -quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartHandoff.cfg ``` -The first checks every rule over four sessions: about 1.5M distinct states, under a minute. The second and third check the same rules under `Restart::Wait` and `Restart::Handoff` over three sessions: about 330k states each. All pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. +The first checks every rule over four sessions: about 1.5M distinct states, under a minute. The second checks the same rules under `Restart::Handoff` over three sessions: about 330k states. All pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. ## Open findings @@ -37,11 +36,11 @@ None. ## Reading a failure -Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the policy with the helpers in `reconnect/tests.rs` (`next`, `handshake`, `on_fallback`, `probe`, `connected`, `failed`, `probed`, a server's `drain().send(..)`), one call per step, on the simulated clock. A path through `Shared` needs `moq-tokio`'s connection tests instead. +Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the policy with the helpers in `reconnect/tests.rs` (`next`, `handshake`, `on_fallback`, `probe`, `connected`, `failed`, `probed`, a server's `goaway(..)`), one call per step, on the simulated clock. A path through `Shared` needs `moq-tokio`'s connection tests instead. ## Coverage -Sync commit: `8f8eec388694cd74555b9a8adfbf63d0bc07da92`. Run `git log 8f8eec3886.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `27b46ef35bdf284ba61158521723768436cd8460`. Run `git log 27b46ef35b.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | @@ -49,10 +48,10 @@ Sync commit: `8f8eec388694cd74555b9a8adfbf63d0bc07da92`. Run `git log 8f8eec3886 | `rs/moq-net/src/reconnect.rs` `established`, set by `end` and by a GOAWAY in `poll`, and `walk_failed` and `closed` failing fast until it is set | `established`, `EndGoaway`, `LiveEnds`, `Failed`, `Abandon`, `EndClosed` | | `rs/moq-net/src/reconnect.rs` `connected`, `fallback`, `failed`, `probed` | `Connected`, `Failed`, `ProbeFails`, `ProbeGetsThrough` | | `rs/moq-net/src/reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` | -| `rs/moq-net/src/reconnect.rs` `Restart::Wait`, `Restart::Handoff` | `LiveEnds` and `WaitOut` with `Restart = "wait"` (`RestartWait.cfg`) and `"handoff"` (`RestartHandoff.cfg`) | +| `rs/moq-net/src/reconnect.rs` `Restart::Handoff` | `LiveEnds` and `WaitOut` with `Restart = "handoff"` (`RestartHandoff.cfg`) | | `rs/moq-net/src/reconnect/redirect.rs` `Redirect::target` | `Refused` and `Endpoints` | | `rs/moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `retired`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` | ## Not covered -One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait` and `Restart::Handoff`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing. +One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Handoff`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing. diff --git a/cpp/obs/src/moq-settings.cpp b/cpp/obs/src/moq-settings.cpp index f8c56abb9e8..855b4eb17a6 100644 --- a/cpp/obs/src/moq-settings.cpp +++ b/cpp/obs/src/moq-settings.cpp @@ -204,8 +204,9 @@ const std::vector &Fields() 1, 60000, 100, "Delay before the first reconnect attempt; it grows from here.")); f.push_back(Number(BACKOFF_MAX, "Reconnect delay cap (ms)", Millis(d.backoff.max, BACKOFF_MAX), 1, 600000, 1000, "Ceiling on the growing reconnect delay.")); - f.push_back(Number(BACKOFF_TIMEOUT, "Give up after (ms)", Millis(d.backoff.timeout, BACKOFF_TIMEOUT), 0, - 3600000, 1000, + // The library's default never gives up, which this field spells as 0. + f.push_back(Number(BACKOFF_TIMEOUT, "Give up after (ms)", + d.backoff.timeout ? Millis(d.backoff.timeout, BACKOFF_TIMEOUT) : 0, 0, 3600000, 1000, "Total time to keep reconnecting after a drop before the stream fails. 0 retries " "forever. A first connect that reaches no server fails the stream at once.")); @@ -340,7 +341,10 @@ std::string Configure(obs_data_t *settings, moq::ClientConfig &config) config.backoff.initial = Milliseconds(settings, BACKOFF_INITIAL, error); config.backoff.max = Milliseconds(settings, BACKOFF_MAX, error); - config.backoff.timeout = Milliseconds(settings, BACKOFF_TIMEOUT, error); + // 0 here means never give up, which the library spells as no timeout: its zero gives up + // at the first failure. + if (const auto timeout = Milliseconds(settings, BACKOFF_TIMEOUT, error); timeout.count() != 0) + config.backoff.timeout = timeout; // Automatic leaves the tri-states and the congestion family unset, which keeps the // transport free to pick. diff --git a/cpp/obs/test/moq-settings-test.cpp b/cpp/obs/test/moq-settings-test.cpp index 47e564c71ed..1cd273357ea 100644 --- a/cpp/obs/test/moq-settings-test.cpp +++ b/cpp/obs/test/moq-settings-test.cpp @@ -158,7 +158,8 @@ int main() CHECK(settings.default_ints["quic_idle_timeout_ms"] == Ms(*library.quic.idle_timeout)); CHECK(!library.quic.keep_alive); CHECK(settings.default_ints["quic_keep_alive_ms"] == 0); - CHECK(settings.default_ints["backoff_timeout_ms"] == Ms(*library.backoff.timeout)); + CHECK(!library.backoff.timeout); + CHECK(settings.default_ints["backoff_timeout_ms"] == 0); CHECK(settings.default_bools["websocket_enabled"] == *library.websocket.enabled); CHECK(settings.default_strings["quic_gso"].empty()); diff --git a/dart/moq/lib/src/aliases.dart b/dart/moq/lib/src/aliases.dart index 163033153b6..d74555e4385 100644 --- a/dart/moq/lib/src/aliases.dart +++ b/dart/moq/lib/src/aliases.dart @@ -275,9 +275,11 @@ typedef ConnectionPhaseClosed = ClosedMoqConnectionPhase; typedef ConnectionState = MoqConnectionState; /// Retry pacing for the automatic reconnect: initial delay, multiplier, ceiling, and give-up window. -/// Until a session has stayed up past 1s or been drained by a GOAWAY, a failed -/// dial or a closed session, such as a refused credential, ends the connection -/// with its error rather than retrying. +/// A null field keeps its default, zero means zero (retry at once, or give up +/// at the first failure), and `maxUnlimited` lifts the ceiling. Until a session +/// has stayed up past 1s or been drained by a GOAWAY, a failed dial or a closed +/// session, such as a refused credential, ends the connection with its error +/// rather than retrying. typedef Backoff = MoqBackoff; /// Whether a protocol code is from the session or stream registry. diff --git a/dart/moq_ffi/lib/src/moq.dart b/dart/moq_ffi/lib/src/moq.dart index 5667110f389..6b9238c61b4 100644 --- a/dart/moq_ffi/lib/src/moq.dart +++ b/dart/moq_ffi/lib/src/moq.dart @@ -3554,11 +3554,13 @@ class MoqBackoff { final Duration? initial; final int? multiplier; final Duration? max; + final bool maxUnlimited; final Duration? timeout; MoqBackoff({ this.initial = null, this.multiplier = null, this.max = null, + this.maxUnlimited = false, this.timeout = null, }); } @@ -3585,6 +3587,11 @@ class FfiConverterMoqBackoff { ); final max = max_lifted.value; new_offset += max_lifted.bytesRead; + final maxUnlimited_lifted = FfiConverterBool.read( + Uint8List.view(buf.buffer, new_offset), + ); + final maxUnlimited = maxUnlimited_lifted.value; + new_offset += maxUnlimited_lifted.bytesRead; final timeout_lifted = FfiConverterOptionalDuration.read( Uint8List.view(buf.buffer, new_offset), ); @@ -3595,6 +3602,7 @@ class FfiConverterMoqBackoff { initial: initial, multiplier: multiplier, max: max, + maxUnlimited: maxUnlimited, timeout: timeout, ), new_offset - buf.offsetInBytes, @@ -3606,6 +3614,7 @@ class FfiConverterMoqBackoff { FfiConverterOptionalDuration.allocationSize(value.initial) + FfiConverterOptionalUInt32.allocationSize(value.multiplier) + FfiConverterOptionalDuration.allocationSize(value.max) + + FfiConverterBool.allocationSize(value.maxUnlimited) + FfiConverterOptionalDuration.allocationSize(value.timeout) + 0; final buf = Uint8List(total_length); @@ -3627,6 +3636,10 @@ class FfiConverterMoqBackoff { value.max, Uint8List.view(buf.buffer, new_offset), ); + new_offset += FfiConverterBool.write( + value.maxUnlimited, + Uint8List.view(buf.buffer, new_offset), + ); new_offset += FfiConverterOptionalDuration.write( value.timeout, Uint8List.view(buf.buffer, new_offset), @@ -3638,6 +3651,7 @@ class FfiConverterMoqBackoff { return FfiConverterOptionalDuration.allocationSize(value.initial) + FfiConverterOptionalUInt32.allocationSize(value.multiplier) + FfiConverterOptionalDuration.allocationSize(value.max) + + FfiConverterBool.allocationSize(value.maxUnlimited) + FfiConverterOptionalDuration.allocationSize(value.timeout) + 0; } diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index d4d34a75e13..0478a5eb2f2 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -363,7 +363,7 @@ services.moq-relay = { ``` Each dialed session binds its own UDP socket on an ephemeral port of -`connect.bind`'s address. Once `connect.backoff.timeout` (10s when unset or `0`, +`connect.bind`'s address. Once `connect.backoff.timeout` (10s when unset or `none`, which would otherwise retry forever) passes without a session, the redial leaves from a fresh port, so a network path that drops one UDP flow can't hold a link down while other ports to the same peer would work. A fixed `connect.bind` port opts diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index 96b721cb197..0625844c8ac 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -119,7 +119,7 @@ inside. The header is the reference: each function carries a doc comment. - **Errors.** Negative return codes are named by `MOQ_ERROR_*`, and `moq_error()` gives the reason on the calling thread. Auth rejections have their own codes so you don't retry them. For the peer's protocol code use `moq_error_protocol()`; do not parse `moq_error()`. A panic in a binding background task reports `MOQ_ERROR_PANIC` through its terminal callback; unrelated objects keep running. A session closed because its credential expired has kind `MOQ_PROTOCOL_KIND_EXPIRED`, and a request refused for an expired token returns `MOQ_ERROR_EXPIRED`; refresh the token before retrying. Malformed credentials report `MOQ_PROTOCOL_KIND_MALFORMED_AUTH_TOKEN`; correct the token before reconnecting with it. A catalog that needs a newer moq-hang draft than this build reads returns `MOQ_ERROR_UNSUPPORTED_CATALOG`; upgrade the library to read it. A request-cap closure has kind `MOQ_PROTOCOL_KIND_TOO_MANY_REQUESTS`; reduce the workload or adjust the relay's [session limits](/bin/relay/config#session-and-peer) before reconnecting. - **Tokens.** Set `moq_client_config.tokens` instead of putting the credential in the URL. The first is the connection credential, sent in SETUP and, while an offered version can't carry it there (lite before 07), as the URL's `?jwt=`; each other token widens the grant over AUTH. A `?jwt=` already in the URL stays the connection credential, which is how to reach a relay older than this release. `moq_session_tokens_set()` replaces them before they expire: the next dial presents them, and a live AUTH session at once. A live session without AUTH (lite before 07, or moq-transport without the extension) keeps the old credential until it redials, and `moq_session_tokens_answered()` reports `MOQ_ERROR_UNSUPPORTED` for it. Otherwise that call reports the peer's verdict. Follow what the peer granted with `moq_session_grants()`. For a per-request token, `moq_session_credential()` presents it on the live session, or the next one while offline, and returns a credential handle at once: name it with `moq_route_set_auth()` or `moq_claim_set_auth()`, and setting a new one renews the announcement in place, without redialing. Only the session that minted it carries it, a dialed one across its reconnects; another session announcing the same route uses its own scope. `moq_credential_answered()` reports the peer's verdict: moq-transport accepts at once, since only the request carries it, moq-lite needs AUTH, where the grant joins the session's, and is `MOQ_ERROR_UNSUPPORTED` without it. `moq_credential_free()` withdraws the token once no route or claim names it, answered or not, and ends a wait still pending. - **Threading.** Call any function from any thread. Callbacks run on moq-c's runtime thread; the callback and its `user_data` must be safe to use there. Synchronize access to data shared with other threads, and dispatch UI work to your UI thread. Raw publish calls block until the codec takes the frame, which paces a publisher. -- **Zeroed config is the default.** A zeroed `moq_client_config` means the default for every field, so new fields don't break callers. A field whose default isn't zero has a `has_*` flag: `backoff_timeout_us = 0` means "retry forever" only with `has_backoff_timeout = true`. +- **Zeroed config is the default.** A zeroed `moq_client_config` means the default for every field, so new fields don't break callers. A field whose default isn't zero has a `has_*` flag: `backoff_timeout_us = 0` gives up at the first failure only with `has_backoff_timeout = true`, and without it the session never gives up. - **Subscription defaults.** A zeroed `moq_subscription`, a NULL pointer, and `moq_subscription_defaults()` all mean moq-net's defaults, whose zero `delay` starts at the newest group and skips a superseded one at once. Set `delay.max_unlimited` for no subscriber limit, and `delay.min_unstated` to leave the target latency unstated. - **Compressed tracks.** `moq_track_info.compression` set to `MOQ_COMPRESSION_DEFLATE` on a raw track sends each group DEFLATE-compressed to opt-in `moq-lite-07-wip` peers; other peers, and every reader here, get the original frames. - **fMP4 import.** `MOQ_CONTAINER_FORMAT_FMP4` splits each fragment into one frame per sample. diff --git a/doc/lib/index.md b/doc/lib/index.md index 2e9d79ad55a..96a7f245a28 100644 --- a/doc/lib/index.md +++ b/doc/lib/index.md @@ -66,7 +66,7 @@ The wrapped bindings share one feature set, so the language pages only show how it looks in that language: - **Connect** to a relay with TLS options and a JWT in the URL, or **serve** sessions yourself and accept or reject each request by path. -- **Reconnect** automatically with backoff when the transport drops. Until a session has stayed up past 1s or been drained by a GOAWAY it fails fast instead: a relay that never answered with a session, or closed it, as one refusing a credential does, ends the connection with its error, for the app to retry. +- **Reconnect** automatically with backoff when the transport drops. Until a session has stayed up past 1s or been drained by a GOAWAY it fails fast instead: a relay that never answered with a session, or closed it, as one refusing a credential does, ends the connection with its error, for the app to retry. The backoff's `max` and `timeout` read the same everywhere: unset takes the default (5s, and never give up), zero means zero (retry at once, or give up at the first failure), and a separate off spelling lifts the bound (`none` in moq-tokio's flags and config, `null` in JS, `max_unlimited` or a null `timeout` in the bindings). - **Discover** broadcasts by prefix, wait for a specific one, or request one by path. Advertise an exact path (a created broadcast is invisible until announced), or claim a prefix and serve requests beneath it on demand. - **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate. The audio decoder reports the playout delay to buffer ahead of the playhead, fixed or estimated from arrivals, so read each frame as soon as the last is handled rather than pacing reads to the speaker. - **Raw stream tracks** of arbitrary bytes, with per-subscriber priority and max delay, and separate live datagram tracks without groups or caching. diff --git a/go/wrapper/backoff_internal_test.go b/go/wrapper/backoff_internal_test.go index e4122d81c4e..8ba8cfcb0f0 100644 --- a/go/wrapper/backoff_internal_test.go +++ b/go/wrapper/backoff_internal_test.go @@ -10,34 +10,32 @@ import ( func ptr[T any](v T) *T { return &v } -// The conversion is where Go's zero value meets the native encoding, and the two -// disagree: zero means "unset" to a Go caller but "no delay" and "retry forever" -// to the reconnect loop. Passing it through as zero turns the most natural -// literal a caller writes, Backoff{}, into an unthrottled dial loop. -func TestBackoffFfiLeavesUnsetFieldsToTheNativeDefaults(t *testing.T) { +// Each bound reads nil as the native default, MaxUnlimited as no cap, and zero as zero, +// as moq-ffi's MoqBackoff does. +func TestBackoffFfiPassesEachBoundThrough(t *testing.T) { cases := []struct { name string in Backoff want ffi.MoqBackoff }{ { - name: "zero value is the native default, never an unpaced loop", + name: "zero value is the native default", in: Backoff{}, want: ffi.MoqBackoff{}, }, { - name: "a partial override keeps the defaults for everything else", - in: Backoff{Max: time.Second}, - want: ffi.MoqBackoff{Max: ptr(time.Second)}, + name: "MaxUnlimited lifts the cap", + in: Backoff{MaxUnlimited: true}, + want: ffi.MoqBackoff{MaxUnlimited: true}, }, { - name: "RetryForever is the only way to reach the native zero timeout", - in: Backoff{Timeout: RetryForever}, - want: ffi.MoqBackoff{Timeout: ptr(time.Duration(0))}, + name: "zero is zero", + in: Backoff{Initial: ptr(time.Duration(0)), Max: ptr(time.Duration(0)), Timeout: ptr(time.Duration(0))}, + want: ffi.MoqBackoff{Initial: ptr(time.Duration(0)), Max: ptr(time.Duration(0)), Timeout: ptr(time.Duration(0))}, }, { name: "explicit values pass through", - in: Backoff{Initial: 500 * time.Millisecond, Multiplier: 3, Max: 10 * time.Second, Timeout: time.Minute}, + in: Backoff{Initial: ptr(500 * time.Millisecond), Multiplier: 3, Max: ptr(10 * time.Second), Timeout: ptr(time.Minute)}, want: ffi.MoqBackoff{ Initial: ptr(500 * time.Millisecond), Multiplier: ptr[uint32](3), @@ -62,7 +60,7 @@ func TestBackoffFfiLeavesUnsetFieldsToTheNativeDefaults(t *testing.T) { // The generated converter panics on a negative duration, so it is refused first. func TestBackoffFfiRejectsNegatives(t *testing.T) { - for _, in := range []Backoff{{Initial: -time.Second}, {Max: -time.Hour}, {Timeout: -2 * time.Second}} { + for _, in := range []Backoff{{Initial: ptr(-time.Second)}, {Max: ptr(-time.Hour)}, {Timeout: ptr(-2 * time.Second)}} { if _, err := in.ffi(); err == nil { t.Errorf("Backoff%+v.ffi() succeeded, want an error", in) } diff --git a/go/wrapper/client.go b/go/wrapper/client.go index 4af0e2b820c..dd41fea4e38 100644 --- a/go/wrapper/client.go +++ b/go/wrapper/client.go @@ -209,56 +209,35 @@ func (c ClientConfig) ffi() (ffi.MoqClientConfig, error) { // closed session, such as a refused credential, ends the connection with its // error rather than retrying. // -// Every field is optional: the zero value means the native default (1s, x2, 5s, -// and no give-up), so a partial Backoff overrides only what it sets. Pass -// RetryForever as Timeout to keep retrying indefinitely. +// A nil duration means the native default (1s, 5s, and no give-up), so a partial +// Backoff overrides only what it sets, and zero means zero: Initial or Max of 0 +// retries at once, and a Timeout of 0 gives up at the first failure. A zero +// Multiplier keeps the default of x2. MaxUnlimited lifts the cap instead of Max. type Backoff struct { - Initial time.Duration // delay before the first retry - Multiplier uint32 // applied to the delay after each failure - Max time.Duration // ceiling on the delay - Timeout time.Duration // give up after this long + Initial *time.Duration // delay before the first retry + Multiplier uint32 // applied to the delay after each failure + Max *time.Duration // ceiling on the delay + MaxUnlimited bool // no ceiling on the delay; exclusive with Max + Timeout *time.Duration // give up after this long } -// RetryForever, passed as Backoff.Timeout, keeps a reconnecting session retrying -// indefinitely instead of giving up. -const RetryForever time.Duration = -1 - -// ffi leaves the zero fields unset, which is load-bearing rather than cosmetic: -// the native side reads a zero timeout as "retry forever" and a zero delay as no -// pacing at all, so passing Go's zero value straight through would turn -// Backoff{} into an unthrottled dial loop. +// ffi refuses a negative duration, which the generated converter would panic on. func (b Backoff) ffi() (ffi.MoqBackoff, error) { - var out ffi.MoqBackoff - var err error - if out.Initial, err = backoffDuration("backoff initial", b.Initial); err != nil { - return out, err - } - if out.Max, err = backoffDuration("backoff max", b.Max); err != nil { - return out, err - } + out := ffi.MoqBackoff{Initial: b.Initial, Max: b.Max, MaxUnlimited: b.MaxUnlimited, Timeout: b.Timeout} if b.Multiplier != 0 { out.Multiplier = &b.Multiplier } - // Zero is the native encoding of "forever" and also Go's zero value, so the - // two are spelled apart here: Backoff{} keeps the default and forever is - // explicit at the call site. - if b.Timeout == RetryForever { - forever := time.Duration(0) - out.Timeout = &forever - } else if out.Timeout, err = backoffDuration("backoff timeout", b.Timeout); err != nil { - return out, err + for _, d := range []struct { + name string + value *time.Duration + }{{"backoff initial", b.Initial}, {"backoff max", b.Max}, {"backoff timeout", b.Timeout}} { + if err := bridge.NonNegative(d.value, d.name); err != nil { + return out, err + } } return out, nil } -// backoffDuration leaves a zero duration unset and refuses a negative one. -func backoffDuration(name string, d time.Duration) (*time.Duration, error) { - if d == 0 { - return nil, nil - } - return &d, bridge.NonNegative(&d, name) -} - // Client is a connected MoQ client with automatic origin wiring. When no origin // is configured, both sides share one origin, so a broadcast announced here is // also discoverable here. diff --git a/go/wrapper/reconnect_test.go b/go/wrapper/reconnect_test.go index ae7300869cd..691d091a194 100644 --- a/go/wrapper/reconnect_test.go +++ b/go/wrapper/reconnect_test.go @@ -82,9 +82,8 @@ func dialWorker(t *testing.T, ctx context.Context, url string) *moq.Client { client, err := moq.Dial(ctx, url, moq.ClientConfig{ TLS: moq.ClientTLS{Insecure: true}, Backoff: moq.Backoff{ - Initial: 50 * time.Millisecond, - Max: time.Second, - Timeout: moq.RetryForever, + Initial: ptr(50 * time.Millisecond), + Max: ptr(time.Second), }, }) if err != nil { @@ -360,7 +359,7 @@ func TestFirstDialState(t *testing.T) { Timeout: ptr(100 * time.Millisecond), TLS: moq.ClientTLS{Insecure: true}, WebSocket: moq.WebSocketConfig{Enabled: ptr(false)}, - Backoff: moq.Backoff{Initial: 50 * time.Millisecond, Max: 50 * time.Millisecond, Timeout: moq.RetryForever}, + Backoff: moq.Backoff{Initial: ptr(50 * time.Millisecond), Max: ptr(50 * time.Millisecond)}, }) if err != nil { t.Fatal(err) diff --git a/js/net/api/index.api.md b/js/net/api/index.api.md index 0a60027e0ec..a2c0525e19c 100644 --- a/js/net/api/index.api.md +++ b/js/net/api/index.api.md @@ -166,11 +166,11 @@ export namespace Connection { websocket?: Connection.WebSocketProps; webtransport?: Connection.WebTransportProps; } - export interface Drain { + export interface Goaway { readonly timeout?: Clock.Duration; readonly uri: string; } - export interface Goaway { + export interface Handover { handover?: Clock.Duration; redirect?: Connection.Redirect; } @@ -182,7 +182,7 @@ export namespace Connection { consume?: Origin.Producer; delay?: Connection.Backoff; enabled?: boolean | Signals.Signal; - goaway?: Connection.Goaway; + goaway?: Connection.Handover; publish?: Origin.Consumer; share?: boolean; signal?: never; @@ -201,7 +201,7 @@ export namespace Connection { readonly auth: Auth.Handle; close(): Promise; closed: Promise; - readonly draining: Signals.GetPromise; + readonly draining: Signals.GetPromise; readonly peerImplementation: Promise; readonly probe: Signals.Getter; stats(): Promise; diff --git a/js/net/src/connection/goaway.ts b/js/net/src/connection/goaway.ts index b08f0fc27f5..7bd0b406ec4 100644 --- a/js/net/src/connection/goaway.ts +++ b/js/net/src/connection/goaway.ts @@ -1,6 +1,6 @@ /** * GOAWAY handling for the reconnect loop: the peer's drain signal and the policy for the - * redirect it may name. Mirrors `Goaway` and `Redirect` in `moq_tokio::connection`. + * redirect it may name. Mirrors `Handover` and `Redirect` in `moq_net::reconnect`. * * @module */ @@ -11,7 +11,7 @@ import { type Endpoint, isLocal, parse, sameHost } from "./endpoint.ts"; import type { Transport } from "./transport.ts"; /** A peer's GOAWAY as a session received it. */ -export interface Drain { +export interface Goaway { /** Where to reconnect, including any credentials it needs. Empty means the same endpoint. */ readonly uri: string; /** When the peer force-closes the session. Undefined when the wire carried none. */ @@ -32,8 +32,8 @@ export interface Drain { */ export type Redirect = "follow" | "same-host" | "ignore"; -/** How a reconnecting connection reacts to a peer's GOAWAY. */ -export interface GoawayProps { +/** How a reconnecting connection hands over to a new session after a peer's GOAWAY. */ +export interface Handover { /** What to do with the URI the peer names (default: `same-host`). */ redirect?: Redirect; diff --git a/js/net/src/connection/pool.ts b/js/net/src/connection/pool.ts index c3ae5bea28c..c5f66c166c7 100644 --- a/js/net/src/connection/pool.ts +++ b/js/net/src/connection/pool.ts @@ -28,7 +28,7 @@ import { type WebTransportProps as WebTransportPropsType, } from "./connect.ts"; import { DISCOVERY_REMOVED } from "./forward.ts"; -import type { Drain as DrainType, GoawayProps, Redirect as RedirectType } from "./goaway.ts"; +import type { Goaway as GoawayType, Handover as HandoverType, Redirect as RedirectType } from "./goaway.ts"; import { pumpAnnounced, Reload, type ReloadDelay, type ReloadStatus } from "./reload.ts"; import type { Session as SessionType } from "./session.ts"; import type { Probe as ProbeType, Stats as StatsType } from "./stats.ts"; @@ -91,7 +91,7 @@ export interface ConnectionProps { * migrates on GOAWAY, dialing the replacement while the old session finishes its groups. * This only tunes where it may go and for how long the old session serves. */ - goaway?: GoawayProps; + goaway?: HandoverType; /** A Connection owns the abort signal for each connection attempt. */ signal?: never; @@ -417,12 +417,12 @@ export namespace Connection { export type AcceptProps = AcceptPropsType; /** Backoff settings for a private reconnect loop. @reexport */ export type Backoff = ReloadDelay; - /** How a connection reacts to the relay's GOAWAY. @reexport */ - export type Goaway = GoawayProps; + /** How a connection hands over to a new session after the relay's GOAWAY. @reexport */ + export type Handover = HandoverType; /** What to do with the URI a relay names in its GOAWAY. @reexport */ export type Redirect = RedirectType; - /** A relay's GOAWAY as a one-shot session received it. @reexport */ - export type Drain = DrainType; + /** A relay's GOAWAY as a session received it. @reexport */ + export type Goaway = GoawayType; /** Current state of a {@link Connection}. @reexport */ export type Status = ReloadStatus; /** The current connection's PROBE estimates. @reexport */ diff --git a/js/net/src/connection/reload.ts b/js/net/src/connection/reload.ts index 58921292135..c040a34842a 100644 --- a/js/net/src/connection/reload.ts +++ b/js/net/src/connection/reload.ts @@ -21,7 +21,7 @@ import { } from "./connect.ts"; import { parse } from "./endpoint.ts"; import { DISCOVERY_REMOVED } from "./forward.ts"; -import { DEFAULT_HANDOVER, type Drain, dialed, type GoawayProps, handover, pinnedTransport, target } from "./goaway.ts"; +import { DEFAULT_HANDOVER, dialed, type Goaway, type Handover, handover, pinnedTransport, target } from "./goaway.ts"; import type { Session } from "./session.ts"; import { estimates, type Probe } from "./stats.ts"; import { tick } from "./ticker.ts"; @@ -123,7 +123,7 @@ export type ReloadProps = Omit< delay?: ReloadDelay; /** How to react to the peer's GOAWAY; every field falls back to its default. */ - goaway?: GoawayProps; + goaway?: Handover; }; /** @@ -307,7 +307,7 @@ export class Reload { delay: ReloadDelay; /** How to react to the peer's GOAWAY (not reactive). */ - goaway: GoawayProps; + goaway: Handover; /** * The URL an accepted GOAWAY redirect assigned, or undefined while the loop dials @@ -790,7 +790,7 @@ export class Reload { * to go next and leave the old session serving until it drains. Returns false when the * redirect is refused, which ends the sequence rather than redialing. */ - #migrate(connection: Session, dialing: URL, drain: Drain): boolean { + #migrate(connection: Session, dialing: URL, drain: Goaway): boolean { // The pin is a WebTransport option. A WebSocket that won the race never used it. const pinned = pinnedTransport(connection.transport, this.#pinned()); diff --git a/js/net/src/connection/session.ts b/js/net/src/connection/session.ts index b7fada73658..ed24ed999b3 100644 --- a/js/net/src/connection/session.ts +++ b/js/net/src/connection/session.ts @@ -3,7 +3,7 @@ import type * as announce from "../announce.ts"; import type { RouteOptions, Routes } from "../announced.ts"; import type * as Auth from "../auth.ts"; import type * as Path from "../path.ts"; -import type { Drain } from "./goaway.ts"; +import type { Goaway } from "./goaway.ts"; import type { Probe, Stats } from "./stats.ts"; import type { Transport } from "./transport.ts"; @@ -89,7 +89,7 @@ export interface Session { * The URI is peer-controlled: apply a redirect policy you trust before dialing it or presenting * tokens to it, as the reconnecting `Connection` does with `goaway.redirect`. */ - readonly draining: GetPromise; + readonly draining: GetPromise; /** * Resolves when the session closes: `null` for a clean close, an `Error.Session` when the diff --git a/js/net/src/goaway-requests.test.ts b/js/net/src/goaway-requests.test.ts index e68f74f3cd1..3668b13c801 100644 --- a/js/net/src/goaway-requests.test.ts +++ b/js/net/src/goaway-requests.test.ts @@ -4,7 +4,7 @@ import { Manual } from "@moq/clock/test"; import { Once } from "@moq/signals"; import { accept } from "./connection/accept.ts"; import { connect } from "./connection/connect.ts"; -import type { Drain } from "./connection/goaway.ts"; +import type { Goaway as Received } from "./connection/goaway.ts"; import { GoAway } from "./ietf/goaway.ts"; import type { Session } from "./ietf/session.ts"; import { Subscriber } from "./ietf/subscriber.ts"; @@ -247,7 +247,7 @@ test("ietf: requests open on the draining session until the replacement wins", a test("announcing and subscribing after GOAWAY still open streams", async () => { for (const version of [IetfVersion.DRAFT_18, IetfVersion.DRAFT_22]) { - const goaway = new Once(); + const goaway = new Once(); goaway.set({ uri: "" }); let opened = false; const session: Session = { diff --git a/js/net/src/ietf/connection.ts b/js/net/src/ietf/connection.ts index b478adf752d..a5808b7adf9 100644 --- a/js/net/src/ietf/connection.ts +++ b/js/net/src/ietf/connection.ts @@ -3,7 +3,7 @@ import { type GetPromise, type Getter, Once, race, Signal } from "@moq/signals"; import * as announce from "../announced.ts"; import type * as Auth from "../auth.ts"; import { AuthSession } from "../auth_session.ts"; -import type { Drain } from "../connection/goaway.ts"; +import type { Goaway } from "../connection/goaway.ts"; import type { Session } from "../connection/session.ts"; import { detect, type Probe, type Stats, transportStats } from "../connection/stats.ts"; import { type Transport, transportOf } from "../connection/transport.ts"; @@ -96,7 +96,7 @@ export class Connection implements Session { #cluster?: Cluster.Hops; // The peer's GOAWAY, read off its SETUP stream. - #goaway = new Once(); + #goaway = new Once(); /** The implementation the peer advertised; see {@link Session.peerImplementation}. */ readonly peerImplementation: Promise; @@ -610,7 +610,7 @@ export class Connection implements Session { } /** Settles with the peer's GOAWAY; see {@link Session.draining}. */ - get draining(): GetPromise { + get draining(): GetPromise { return this.#goaway; } diff --git a/js/net/src/ietf/goaway.ts b/js/net/src/ietf/goaway.ts index 787aedece22..5f2a25796f3 100644 --- a/js/net/src/ietf/goaway.ts +++ b/js/net/src/ietf/goaway.ts @@ -1,5 +1,5 @@ import { Duration } from "@moq/clock"; -import type { Drain } from "../connection/goaway.ts"; +import type { Goaway } from "../connection/goaway.ts"; import { ProtocolViolation } from "../error.ts"; import type { Reader, Writer } from "../stream.ts"; import { toNumber } from "../util/bigint.ts"; @@ -45,7 +45,7 @@ export class GoAway { } /** The drain signal this message carries. A zero timeout is the wire saying "none". */ - drain(): Drain { + drain(): Goaway { return { uri: this.newSessionUri, timeout: this.timeout > 0n ? Duration.millis(toNumber(this.timeout)) : undefined, diff --git a/js/net/src/ietf/subscriber.ts b/js/net/src/ietf/subscriber.ts index 2b898d488b8..878ee98f48d 100644 --- a/js/net/src/ietf/subscriber.ts +++ b/js/net/src/ietf/subscriber.ts @@ -4,7 +4,7 @@ import * as announce from "../announced.ts"; import type { Grant } from "../auth.ts"; import type { AuthSession } from "../auth_session.ts"; import * as broadcast from "../broadcast.ts"; -import type { Drain } from "../connection/goaway.ts"; +import type { Goaway } from "../connection/goaway.ts"; import { BroadcastCache } from "../consume.ts"; import type * as datagram from "../datagram.ts"; import * as DatagramStream from "../datagram_stream.ts"; @@ -217,7 +217,7 @@ export class Subscriber { #activeCount: boolean; // Settles when the peer sends GOAWAY, ranking this session's routes last. - #goaway?: GetPromise; + #goaway?: GetPromise; // Our grant: a subscription it stops covering is cancelled. Undefined until // the peer answers, which allows everything. #grant?: Getter; @@ -253,7 +253,7 @@ export class Subscriber { /** Whether both SETUPs declared MoQ Active Count. */ activeCount?: boolean; /** Settles when the peer sends GOAWAY. */ - goaway?: GetPromise; + goaway?: GetPromise; /** The union of our tokens' grants, which bounds what we subscribe to. */ grant?: Getter; auth?: AuthSession; diff --git a/js/net/src/lite/connection.ts b/js/net/src/lite/connection.ts index bd8fb35f268..567c614678b 100644 --- a/js/net/src/lite/connection.ts +++ b/js/net/src/lite/connection.ts @@ -3,7 +3,7 @@ import { Effect, type GetPromise, type Getter, Once, Signal } from "@moq/signals import * as announce from "../announced.ts"; import type * as Auth from "../auth.ts"; import { AuthSession } from "../auth_session.ts"; -import type { Drain } from "../connection/goaway.ts"; +import type { Goaway as Received } from "../connection/goaway.ts"; import type { Session } from "../connection/session.ts"; import { detect, type Probe, type Stats, statsOf, type TransportStats, transportStats } from "../connection/stats.ts"; import { type Transport, transportOf } from "../connection/transport.ts"; @@ -166,7 +166,7 @@ export class Connection implements Session { readonly peerImplementation = this.#peerImplementation.promise; // The peer's GOAWAY. Lite carries no deadline, so only the URI is set. - #goaway = new Once(); + #goaway = new Once(); /** * The {@link Role} the peer advertised in its SETUP, for a server deciding whether the @@ -648,7 +648,7 @@ export class Connection implements Session { } /** Settles with the peer's GOAWAY; see {@link Session.draining}. */ - get draining(): GetPromise { + get draining(): GetPromise { return this.#goaway; } diff --git a/js/net/src/lite/routed.ts b/js/net/src/lite/routed.ts index ec22cf68edd..c4b80ce1470 100644 --- a/js/net/src/lite/routed.ts +++ b/js/net/src/lite/routed.ts @@ -10,7 +10,7 @@ import { type Dispose, type GetPromise, race } from "@moq/signals"; import type * as announce from "../announced.ts"; -import type { Drain } from "../connection/goaway.ts"; +import type { Goaway } from "../connection/goaway.ts"; import type { BroadcastCache } from "../consume.ts"; import { error, ProtocolViolation } from "../error.ts"; import { Cost, type Hop, routesEqual, WireRoute } from "../hop.ts"; @@ -61,7 +61,7 @@ export interface Session { slots: Slots; /** Rank a route last once the peer sent GOAWAY. */ drained(route: WireRoute): WireRoute; - goaway?: GetPromise; + goaway?: GetPromise; /** What covers a path changed with no claim reporting it: every selection is checked anew. */ revalidate(): void; /** Send one message on this session's Announce Stream. */ diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index c6510aa72a0..883f9aaf299 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -4,7 +4,7 @@ import * as announce from "../announced.ts"; import type { Grant } from "../auth.ts"; import type { AuthSession } from "../auth_session.ts"; import * as broadcast from "../broadcast.ts"; -import type { Drain } from "../connection/goaway.ts"; +import type { Goaway as Received } from "../connection/goaway.ts"; import type { Probe as ProbeStats } from "../connection/stats.ts"; import { BroadcastCache } from "../consume.ts"; import type * as datagram from "../datagram.ts"; @@ -510,7 +510,7 @@ export class Subscriber { #peerSetup?: Signal; // Settles when the peer sends GOAWAY, ranking this session's routes last. - #goaway?: GetPromise; + #goaway?: GetPromise; // Distinguishes failures from streams torn down by Subscriber.close(). #closed = new AbortController(); @@ -539,7 +539,7 @@ export class Subscriber { hop: Hop, probe?: Signal, peerSetup?: Signal, - goaway?: GetPromise, + goaway?: GetPromise, grant?: Getter, limits: Limits = LIMITS, auth?: AuthSession, diff --git a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt index 7584e82076b..3d031a3e538 100644 --- a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt +++ b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt @@ -388,8 +388,10 @@ typealias ConnectionState = uniffi.moq.MoqConnectionState /** * Retry pacing for the automatic reconnect: initial delay, multiplier, ceiling, and give-up window. - * Until a session has stayed up past 1s or been drained by a GOAWAY, a failed dial or a closed - * session, such as a refused credential, ends the connection with its error rather than retrying. + * A null field keeps its default, zero means zero (retry at once, or give up at the first failure), + * and `maxUnlimited` lifts the ceiling. Until a session has stayed up past 1s or been drained by a + * GOAWAY, a failed dial or a closed session, such as a refused credential, ends the connection with + * its error rather than retrying. */ typealias Backoff = uniffi.moq.MoqBackoff diff --git a/py/moq-rs/moq/_records.py b/py/moq-rs/moq/_records.py index f6414926bf4..ff9ea1383b3 100644 --- a/py/moq-rs/moq/_records.py +++ b/py/moq-rs/moq/_records.py @@ -9,7 +9,7 @@ from __future__ import annotations from collections.abc import Sequence -from dataclasses import dataclass +from dataclasses import dataclass, field from datetime import timedelta from importlib import metadata from typing import ClassVar, cast @@ -351,14 +351,16 @@ class Backoff: rather than retrying. The delay starts at ``initial``, multiplies by ``multiplier`` after each failed - attempt, and caps at ``max``. After ``timeout`` of consecutive failures the - connection gives up; ``timedelta(0)`` retries forever. ``None`` keeps each - default: 1s, x2, 5s, and no give-up. + attempt, and caps at ``max``; ``max_unlimited=True`` lifts the cap instead. After + ``timeout`` of consecutive failures the connection gives up. ``None`` keeps each + default: 1s, x2, 5s, and no give-up. Zero means zero: ``timedelta(0)`` retries + at once as ``initial`` or ``max``, and gives up at the first failure as ``timeout``. """ initial: timedelta | None = None multiplier: int | None = None max: timedelta | None = None + max_unlimited: bool = field(default=False, kw_only=True) timeout: timedelta | None = None def _ffi(self) -> MoqBackoff: @@ -366,6 +368,7 @@ def _ffi(self) -> MoqBackoff: initial=self.initial, multiplier=self.multiplier, max=self.max, + max_unlimited=self.max_unlimited, timeout=self.timeout, ) diff --git a/py/moq-rs/tests/test_local.py b/py/moq-rs/tests/test_local.py index 096af13b292..003ec4b7782 100644 --- a/py/moq-rs/tests/test_local.py +++ b/py/moq-rs/tests/test_local.py @@ -569,6 +569,13 @@ def test_default_subscription_skips_superseded_groups_at_once(): assert (delay.min, delay.max) == (None, None) +def test_backoff_keeps_its_positional_timeout(): + """``max_unlimited`` is keyword-only, so the fourth positional argument is still ``timeout``.""" + backoff = moq.Backoff(None, None, None, timedelta(seconds=3))._ffi() + assert (backoff.timeout, backoff.max_unlimited) == (timedelta(seconds=3), False) + assert moq.Backoff(max_unlimited=True)._ffi().max_unlimited + + def test_string_sequences_refuse_a_bare_string(): """A bare str would split into characters, so the config refuses it up front.""" with pytest.raises(TypeError, match="roots"): diff --git a/py/moq-rs/tests/test_server.py b/py/moq-rs/tests/test_server.py index 4d473c273c9..8d2716cb32f 100644 --- a/py/moq-rs/tests/test_server.py +++ b/py/moq-rs/tests/test_server.py @@ -130,7 +130,7 @@ async def accept_loop() -> None: initial=timedelta(milliseconds=50), multiplier=2, max=timedelta(milliseconds=200), - timeout=timedelta(0), + timeout=None, ), ) as client: session = client.session @@ -191,7 +191,7 @@ async def test_session_state_names_a_failing_first_dial(): initial=timedelta(milliseconds=50), multiplier=2, max=timedelta(milliseconds=200), - timeout=timedelta(0), + timeout=None, ), ) async with client: diff --git a/quest/m2/README.md b/quest/m2/README.md index 2ae229bd0ba..a5bf90ac3d8 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -36,8 +36,8 @@ removal, a rename, or a wire, catalog, or format change that is free until [switch](/quest/m2/multicast/switch.md) also settles the public route API and retires the moq-cluster draft. - **moq-net API.** [Groups](/quest/m2/group-api.md), - [reconnect](/quest/m2/reconnect-off.md), [names](/quest/m2/net-renames.md), - and [session codes](/quest/m2/session-codes.md). + [names](/quest/m2/net-renames.md), and + [session codes](/quest/m2/session-codes.md). [Socket endpoints](/quest/m2/sockets/README.md) leave for Kixel, and the demos return to their earlier format. - **Unused API.** [One sweep per area](/quest/m2/unused-api/README.md), each diff --git a/quest/m2/js-net-api.md b/quest/m2/js-net-api.md index 0b256320f19..b8edd9cd31c 100644 --- a/quest/m2/js-net-api.md +++ b/quest/m2/js-net-api.md @@ -82,6 +82,5 @@ compatible with code that awaits them. Wire: none. ## Related -- [Reconnect, absent means off](/quest/m2/reconnect-off.md) - changes the same reconnect options and names GOAWAY once - [@moq/net addresses groups by start](/quest/m2/time-addressing/js.md) - the other `@moq/net` rename before the cut - [Unused @moq/net exports](/quest/m2/unused-api/js.md) - the removals that land beside these diff --git a/quest/m2/reconnect-off.md b/quest/m2/reconnect-off.md deleted file mode 100644 index de285c05be3..00000000000 --- a/quest/m2/reconnect-off.md +++ /dev/null @@ -1,87 +0,0 @@ -# [M] Reconnect settings mean the same in every layer, and GOAWAY has one name - -## Goal - -A reconnect or GOAWAY duration reads the same in moq-net, moq-tokio's flags -and config, `@moq/net`, moq-ffi, and moq-c: an absent value turns the bound -off, and zero means zero (give up at the first failure, retry at once, close -now). `Restart::Wait` is gone. The message a peer sends to drain a session -has one name in Rust and JS, and a session sends it with one call. - -## Plan - -Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): ✅ one -quest before the cut that deletes `Restart::Wait`, makes an absent value mean -"off" and zero mean zero in every layer, and gives GOAWAY one name and one -send call (rejected: only the variant and the zero rule; leaving it to -[the connect policy's state](/quest/m4/connect-policy-state.md) in m4, where -it would be a break after the cut). The same field name means opposite -things in a browser and in Swift today, and the cut would publish both. - -Facts found by the audit, checked at `80ca9787e3`: - -- `Restart::Wait` (`rs/moq-net/src/reconnect.rs:156`) is set only by tests - (`rs/moq-relay/tests/integration/handoff.rs:115`, - `rs/moq-tokio/src/connection.rs:2861`, `rs/moq-net/src/reconnect/tests.rs`). - It was the cluster's policy until - [moq-dev/moq#8596](https://github.com/moq-dev/moq/pull/8596) moved the - relay to `Handoff` (`rs/moq-relay/src/cluster.rs:2291`), which covers the - same address at the cost of one session. -- `moq_net::reconnect::Backoff::timeout`: "Zero gives up at the first - failure", `None` retries forever (`rs/moq-net/src/reconnect.rs:89`), and - JS agrees (`js/net/src/connection/reload.ts`: `null` unlimited, zero gives - up). moq-tokio reads zero as off: "A configured zero turns the bound off" - (`rs/moq-tokio/src/connection.rs:191`), and so does moq-ffi: "0 retries - forever", "0 leaves it uncapped" (`rs/moq-ffi/src/session.rs:211-224`). - [moq-dev/moq#8265](https://github.com/moq-dev/moq/pull/8265) moved the core - to `Option` and left these two layers behind. -- `Goaway::with_timeout(Duration::ZERO)` means no deadline - (`rs/moq-net/src/goaway.rs:79`), and its one production caller - special-cases it: "the opposite of what was asked for" - (`rs/moq-relay/src/shutdown.rs:391`). `Server::with_timeout` in the same - crate takes `Option`. -- Rust has three public types named `Goaway`: the message - (`rs/moq-net/src/goaway.rs:34`), the reaction policy - (`rs/moq-net/src/reconnect.rs:116`), and moq-tokio's flags - (`rs/moq-tokio/src/connection.rs:290`). JS calls the message `Drain` and - the policy `GoawayProps` (`js/net/src/connection/goaway.ts:14,36`). -- `Session::drain()` returns a `goaway::Producer` whose only method is - `send`, and it hands out another each call (`rs/moq-net/src/session.rs:422`). - -For the PR: - -- Delete `Restart::Wait`, its branch (`reconnect.rs:603`), and its tests; the - relay handoff test that wants the old session to stay ignores the GOAWAY - instead. Run `/verify-reconnect`, since the model covers the restart rule. -- Each layer needs three states for `max` and `timeout`: unset (the - default), off, and a value. JS already has them (`undefined`, `null`, a - duration). Recommended: moq-tokio's flags and config keep "unset" as the - default and take a word (`none`) for off, refusing nothing that parses - today except that `0` now means zero; moq-ffi's `MoqBackoff` and moq-c's - `has_*` flags spell off as the absent value and document the default. - Whatever the spelling, write it once in `doc/` and mirror it. -- `connect.backoff.{max,timeout} = 0` changes meaning in existing config - files. The migration note names it; a warning at load when either is zero - is worth one release. -- `Goaway::with_timeout(Option)`, with zero closing at once. -- One name for the received message in Rust and JS, and one send call. - Recommended: `Goaway` stays the message in Rust and JS renames `Drain` to - it; the policy type becomes `reconnect::Handover` (its fields are - `redirect` and `handover`); `session.goaway(Goaway)` replaces - `drain().send(..)`, and `draining()` stays. The names are the PR's to - settle, mirrored in moq-ffi and moq-c. -- Keep [moq-dev/moq#8817](https://github.com/moq-dev/moq/pull/8817)'s rule: - a first session that fails or is refused before it is healthy ends the loop. - -Tests: one table of (unset, off, zero, value) per field run against -moq-net, moq-tokio's config parse, JS, and moq-ffi, asserting the same -behavior from each. - -Public API: breaking in Rust (`Restart`, `Goaway::with_timeout`, the -`Goaway` names, `Session::drain`), JS (`Drain`), moq-ffi, and moq-c. -Config and CLI: `connect.backoff.max` and `connect.backoff.timeout` of zero -change meaning. Wire: none. - -## Related - -- [The connect policy's state lives in one place, in both languages](/quest/m4/connect-policy-state.md) - reworks the same policy after the cut diff --git a/rs/moq-c/src/api.rs b/rs/moq-c/src/api.rs index 7e29823d7e9..9fc56f8079e 100644 --- a/rs/moq-c/src/api.rs +++ b/rs/moq-c/src/api.rs @@ -1585,11 +1585,13 @@ pub struct moq_client_config { pub backoff_multiplier: u32, /// Whether `backoff_multiplier` is present. pub has_backoff_multiplier: bool, - /// The maximum reconnect delay in microseconds; zero leaves it uncapped. + /// The maximum reconnect delay in microseconds; zero retries at once. Absent caps it + /// at 5s, and `backoff_max_unlimited` lifts the cap. pub backoff_max_us: u64, /// Whether `backoff_max_us` is present. pub has_backoff_max: bool, - /// How long reconnection keeps trying before giving up for good; zero retries forever. + /// How long reconnection keeps trying before giving up for good; zero gives up at the + /// first failure. Absent never gives up. pub backoff_timeout_us: u64, /// Whether `backoff_timeout_us` is present. pub has_backoff_timeout: bool, @@ -1649,6 +1651,10 @@ pub struct moq_client_config { /// Dial once instead of redialing with backoff whenever the transport drops; the /// transport's close then ends the session. False redials. pub once: bool, + + /// Lift the cap on the reconnect delay, which then grows without bound. Refused + /// with `has_backoff_max`. + pub backoff_max_unlimited: bool, } /// The settings [moq_session_connect] dials with when given NULL. @@ -1694,10 +1700,17 @@ pub extern "C" fn moq_client_defaults() -> moq_client_config { dst.has_backoff_initial = true; dst.backoff_multiplier = config.connect.backoff.multiplier; dst.has_backoff_multiplier = true; - dst.backoff_max_us = micros(config.connect.backoff.max().unwrap_or_default()); - dst.has_backoff_max = true; - dst.backoff_timeout_us = micros(config.connect.backoff.timeout().unwrap_or_default()); - dst.has_backoff_timeout = true; + match config.connect.backoff.max() { + Some(max) => { + dst.backoff_max_us = micros(max); + dst.has_backoff_max = true; + } + None => dst.backoff_max_unlimited = true, + } + if let Some(timeout) = config.connect.backoff.timeout() { + dst.backoff_timeout_us = micros(timeout); + dst.has_backoff_timeout = true; + } let quic = config.quic.resolve(); dst.quic_max_streams = quic.max_streams; diff --git a/rs/moq-c/src/client.rs b/rs/moq-c/src/client.rs index 4e522c73b79..1fc154e7b1d 100644 --- a/rs/moq-c/src/client.rs +++ b/rs/moq-c/src/client.rs @@ -164,9 +164,17 @@ pub unsafe fn parse_client(config: Option<&moq_client_config>) -> Result, - /// Multiplier applied to the delay after each failure. + /// Multiplier applied to the delay after each failure. 0 or 1 keeps it constant. #[uniffi(default = None)] pub multiplier: Option, - /// Maximum delay between reconnect attempts. 0 leaves it uncapped. + /// Maximum delay between reconnect attempts. Null uses 5s, and 0 retries at once. #[uniffi(default = None)] pub max: Option, - /// Time spent retrying before giving up. 0 retries forever. + /// Lift the cap on the delay, which then grows without bound. Refused with `max` set. + #[uniffi(default = false)] + pub max_unlimited: bool, + /// Time spent retrying before giving up. Null never gives up, and 0 gives up at the + /// first failure. #[uniffi(default = None)] pub timeout: Option, } +impl MoqBackoff { + /// Set each field given on moq-tokio's `backoff`, leaving the rest at their defaults. + pub(crate) fn apply(self, backoff: &mut moq_tokio::connection::Backoff) -> Result<(), MoqError> { + if let Some(initial) = self.initial { + backoff.initial = Some(initial.into()); + } + if let Some(multiplier) = self.multiplier { + backoff.multiplier = multiplier; + } + backoff.max = match (self.max, self.max_unlimited) { + (Some(_), true) => { + return Err(MoqError::Config { + reason: "backoff max and max_unlimited are exclusive".into(), + }); + } + (Some(max), false) => Some(max.into()), + (None, true) => Some(moq_tokio::cli::Bound::NONE), + (None, false) => None, + }; + backoff.timeout = self.timeout.map(Into::into); + Ok(()) + } +} + /// What sessions opened through this crate advertise in SETUP: the caller's `prefix`, then /// moq-ffi, then its core. pub(crate) fn implementation(prefix: Option<&str>) -> Result { @@ -275,8 +304,9 @@ pub fn moq_client_defaults() -> MoqClientConfig { backoff: MoqBackoff { initial: Some(backoff.initial()), multiplier: Some(backoff.multiplier), - max: Some(backoff.max().unwrap_or_default()), - timeout: Some(backoff.timeout().unwrap_or_default()), + max: backoff.max(), + max_unlimited: backoff.max().is_none(), + timeout: backoff.timeout(), }, ..Default::default() } @@ -352,19 +382,7 @@ impl Client { connect.websocket.delay = Some(delay.into()); } - let backoff = config.backoff; - if let Some(initial) = backoff.initial { - connect.backoff.initial = Some(initial.into()); - } - if let Some(multiplier) = backoff.multiplier { - connect.backoff.multiplier = multiplier; - } - if let Some(max) = backoff.max { - connect.backoff.max = Some(max.into()); - } - if let Some(timeout) = backoff.timeout { - connect.backoff.timeout = Some(timeout.into()); - } + config.backoff.apply(&mut connect.backoff)?; let quic = config.quic.into(); @@ -564,6 +582,7 @@ mod tests { once: true, backoff: MoqBackoff { timeout: Some(Duration::ZERO), + max_unlimited: true, ..Default::default() }, ..Default::default() diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 2b6794256bf..cf307b42c3f 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -131,7 +131,8 @@ fn fast_backoff() -> MoqBackoff { initial: Some(Duration::from_millis(50)), multiplier: Some(2), max: Some(Duration::from_millis(200)), - timeout: Some(Duration::ZERO), + max_unlimited: false, + timeout: None, } } @@ -6703,8 +6704,15 @@ async fn client_defaults_are_what_an_unset_config_dials_with() { assert_eq!(defaults.websocket.enabled, Some(true)); let backoff = moq_tokio::connection::Backoff::default(); assert_eq!(defaults.backoff.initial, Some(backoff.initial())); - assert_eq!(defaults.backoff.max, Some(backoff.max().unwrap_or_default())); - assert_eq!(defaults.backoff.timeout, Some(backoff.timeout().unwrap_or_default())); + assert_eq!(defaults.backoff.max, backoff.max()); + assert!(!defaults.backoff.max_unlimited); + assert_eq!(defaults.backoff.timeout, backoff.timeout()); + let mut reapplied = moq_tokio::connection::Backoff::default(); + defaults.backoff.clone().apply(&mut reapplied).unwrap(); + assert_eq!( + (reapplied.initial(), reapplied.max(), reapplied.timeout()), + (backoff.initial(), backoff.max(), backoff.timeout()) + ); // The backend picks these, so there is no single value to report. assert_eq!(defaults.tls.system_roots, None); @@ -6717,6 +6725,54 @@ async fn client_defaults_are_what_an_unset_config_dials_with() { assert!(MoqClient::new(defaults).is_ok()); } +/// `max` and `timeout` each read unset as the default, off as no bound, and zero as zero, +/// resolving to what moq-tokio's flags `none` and `0s` do. +#[test] +fn a_backoff_bound_is_unset_off_zero_or_a_value() { + let resolve = |backoff: MoqBackoff| { + let mut out = moq_tokio::connection::Backoff::default(); + backoff.apply(&mut out).map(|()| (out.max(), out.timeout())) + }; + let defaults = moq_tokio::connection::Backoff::default(); + let three = Duration::from_secs(3); + for (backoff, expected) in [ + (MoqBackoff::default(), (defaults.max(), defaults.timeout())), + ( + MoqBackoff { + max_unlimited: true, + ..Default::default() + }, + (None, None), + ), + ( + MoqBackoff { + max: Some(Duration::ZERO), + timeout: Some(Duration::ZERO), + ..Default::default() + }, + (Some(Duration::ZERO), Some(Duration::ZERO)), + ), + ( + MoqBackoff { + max: Some(three), + timeout: Some(three), + ..Default::default() + }, + (Some(three), Some(three)), + ), + ] { + assert_eq!(resolve(backoff.clone()).unwrap(), expected, "{backoff:?}"); + } + assert!(defaults.timeout().is_none(), "unset is already off"); + + let both = MoqBackoff { + max: Some(three), + max_unlimited: true, + ..Default::default() + }; + assert!(matches!(resolve(both), Err(MoqError::Config { .. }))); +} + /// Every listed version is one `versions` accepts. #[tokio::test] async fn versions_are_accepted_by_the_client() { diff --git a/rs/moq-gst/src/connection.rs b/rs/moq-gst/src/connection.rs index c8ea54c83ff..5c31a2f4bed 100644 --- a/rs/moq-gst/src/connection.rs +++ b/rs/moq-gst/src/connection.rs @@ -113,7 +113,7 @@ impl Settings { .collect::>()?; connect.websocket.enabled = Some(self.websocket_enabled); connect.tls.insecure = Some(self.tls_disable_verify); - connect.backoff.timeout = Some(Duration::ZERO.into()); + connect.backoff.timeout = Some(moq_tokio::cli::Bound::NONE); let mut quic = moq_tokio::quic::Config::default(); quic.idle_timeout = self.quic_idle_timeout.map(Into::into); quic.keep_alive = self.quic_keep_alive.map(Into::into); @@ -194,7 +194,7 @@ mod tests { ["moq-transport-18".parse().unwrap(), "moq-lite-06".parse().unwrap()] ); assert_eq!(connect.tls.insecure, Some(true)); - assert_eq!(connect.backoff.timeout, Some(Duration::ZERO.into())); + assert_eq!(connect.backoff.timeout, Some(moq_tokio::cli::Bound::NONE)); let quic = quic.resolve(); assert_eq!(quic.idle_timeout, Some(Duration::from_secs(15))); assert_eq!(quic.keep_alive, None); diff --git a/rs/moq-gst/src/sink/imp.rs b/rs/moq-gst/src/sink/imp.rs index 6a3e50c1ecf..b564cdfd963 100644 --- a/rs/moq-gst/src/sink/imp.rs +++ b/rs/moq-gst/src/sink/imp.rs @@ -2661,11 +2661,7 @@ mod tests { sink.set_property("broadcast", "test"); let resolved = ResolvedSettings::try_from(sink.imp().settings.lock().unwrap().clone()).unwrap(); let config = super::super::session::connect_config(&resolved); - assert_eq!( - config.backoff.timeout, - Some((Duration::ZERO).into()), - "zero is unlimited" - ); + assert_eq!(config.backoff.timeout, Some(moq_tokio::cli::Bound::NONE), "no give-up"); } #[test] diff --git a/rs/moq-gst/src/source/imp.rs b/rs/moq-gst/src/source/imp.rs index 514ae40e072..fc0cb003329 100644 --- a/rs/moq-gst/src/source/imp.rs +++ b/rs/moq-gst/src/source/imp.rs @@ -3804,11 +3804,7 @@ mod session_tests { #[test] fn the_source_retries_without_a_budget() { let config = super::connect_config(&unreachable()); - assert_eq!( - config.backoff.timeout, - Some((Duration::ZERO).into()), - "zero is unlimited" - ); + assert_eq!(config.backoff.timeout, Some(moq_tokio::cli::Bound::NONE), "no give-up"); } /// `quic-keylog` reaches the client's QUIC config, which refuses it without `SSLKEYLOGFILE`. diff --git a/rs/moq-net/src/auth.rs b/rs/moq-net/src/auth.rs index b88b3830f8c..49f224b806b 100644 --- a/rs/moq-net/src/auth.rs +++ b/rs/moq-net/src/auth.rs @@ -3332,7 +3332,7 @@ mod tests { )); let (goaway, _protocol) = crate::goaway::Handle::new(false); - goaway.producer().send(crate::goaway::Goaway::new()).unwrap(); + goaway.send(crate::goaway::Goaway::new()).unwrap(); link.drain_with(std::sync::Arc::new(goaway)); assert!(matches!( link.present(Bytes::from_static(b"late"), false), @@ -3347,7 +3347,7 @@ mod tests { let handle = Handle::new(); let draining = Link::attach(&handle, false, false).unwrap(); let (goaway, _protocol) = crate::goaway::Handle::new(false); - goaway.producer().send(crate::goaway::Goaway::new()).unwrap(); + goaway.send(crate::goaway::Goaway::new()).unwrap(); draining.drain_with(std::sync::Arc::new(goaway)); let successor = Link::attach(&handle, true, false).unwrap(); @@ -3370,7 +3370,7 @@ mod tests { let auth = handle.add(Bytes::from_static(b"jwt")).unwrap(); let id = opened(&draining); let (goaway, _protocol) = crate::goaway::Handle::new(false); - goaway.producer().send(crate::goaway::Goaway::new()).unwrap(); + goaway.send(crate::goaway::Goaway::new()).unwrap(); draining.drain_with(std::sync::Arc::new(goaway)); let successor = Link::attach(&handle, true, false).unwrap(); @@ -3391,7 +3391,7 @@ mod tests { let auth = handle.add(Bytes::from_static(b"jwt")).unwrap(); let id = opened(&draining); let (goaway, _protocol) = crate::goaway::Handle::new(false); - goaway.producer().send(crate::goaway::Goaway::new()).unwrap(); + goaway.send(crate::goaway::Goaway::new()).unwrap(); draining.drain_with(std::sync::Arc::new(goaway)); let successor = Link::attach(&handle, true, false).unwrap(); @@ -3424,7 +3424,7 @@ mod tests { let busy = Error::Session(SessionError::TooManyRequests); assert_eq!(link.retry(id, &busy), Some(RETRY_INITIAL)); - goaway.producer().send(crate::goaway::Goaway::new()).unwrap(); + goaway.send(crate::goaway::Goaway::new()).unwrap(); assert!(!link.active()); let err = link.reopen(id).unwrap_err(); assert!(matches!(err, Error::GoingAway), "{err:?}"); @@ -3481,7 +3481,7 @@ mod tests { let _auth = handle.add(Bytes::from_static(b"jwt")).unwrap(); let id = opened(&draining); let (goaway, _protocol) = crate::goaway::Handle::new(false); - goaway.producer().send(crate::goaway::Goaway::new()).unwrap(); + goaway.send(crate::goaway::Goaway::new()).unwrap(); draining.drain_with(std::sync::Arc::new(goaway)); let busy = Error::Session(SessionError::TooManyRequests); diff --git a/rs/moq-net/src/goaway.rs b/rs/moq-net/src/goaway.rs index 87366390b98..76ac2ba2b3f 100644 --- a/rs/moq-net/src/goaway.rs +++ b/rs/moq-net/src/goaway.rs @@ -1,9 +1,7 @@ -//! GOAWAY, the graceful drain signal, split into a [Producer] and [Consumer] handle. +//! GOAWAY, the graceful drain signal. //! -//! A session sends its own GOAWAY through the [Producer] from -//! `Session::drain`, and observes the peer's on the -//! [Consumer] from `Session::draining`. A session -//! sends at most one, which is why [`Producer::send`] consumes the handle. +//! A session sends its own [Goaway] with `Session::goaway`, at most once, and +//! observes the peer's on the [Consumer] from `Session::draining`. //! //! Draining works the same on a version with no GOAWAY message: the deadline is //! the sender's own timer, so it force-closes on schedule whether or not the peer @@ -71,14 +69,17 @@ impl Goaway { } /// Force-close the session with [`Error::GoawayTimeout`] if the peer is still - /// around after `timeout`. + /// around after `timeout`, or never with `None` (the default). /// - /// Zero means no deadline. Positive values round up to the next millisecond - /// and saturate at the wire's 62-bit millisecond limit, so the local deadline - /// agrees with what a moq-transport peer receives. - pub fn with_timeout(mut self, timeout: Duration) -> Self { - let millis = timeout.as_nanos().div_ceil(1_000_000).min(u128::from(MAX_TIMEOUT_MS)) as u64; - self.timeout = (millis != 0).then(|| Duration::from_millis(millis)); + /// Zero closes the session at once, without waiting for the GOAWAY to be written, so + /// the peer may never receive it or its redirect URI; a redirect needs a nonzero + /// timeout to arrive. Other values round up to the next millisecond and saturate at + /// the wire's 62-bit millisecond limit, so the local deadline agrees with what a + /// moq-transport peer receives. + pub fn with_timeout(mut self, timeout: Option) -> Self { + self.timeout = timeout.map(|timeout| { + Duration::from_millis(timeout.as_nanos().div_ceil(1_000_000).min(u128::from(MAX_TIMEOUT_MS)) as u64) + }); self } @@ -93,56 +94,6 @@ impl Goaway { } } -/// Sends this session's single GOAWAY. -/// -/// Obtained from `Session::drain`. -pub struct Producer { - trigger: kio::Producer>, - /// Whether this side may name a redirect URI. A client may not, - /// since it cannot tell a server to open connections (draft-22 section 9.2). - redirect: bool, -} - -impl Producer { - /// Send the GOAWAY, consuming the handle: a session may only send one. - /// - /// The timeout starts at the driver poll this send wakes, independent of the - /// GOAWAY's own write. The session must stay driven for the message to reach the wire. With - /// [`Goaway::timeout`] set, the driver force-closes the session once the - /// deadline passes, so awaiting `Session::closed` - /// is how a caller observes the drain finishing either way. That holds on a - /// version with no GOAWAY message too, where the deadline is all the peer gets. - /// - /// # Errors - /// - /// Returns [`Error::ProtocolViolation`] when a client names a - /// redirect URI, which the peer would be required to close the session over, - /// or when the URI exceeds the 8,192-byte wire cap. - /// - /// Returns [`Error::Duplicate`] if this session already sent one. The wire - /// carries at most one either way, so the second message is dropped rather - /// than replacing the first, but the caller is told rather than left guessing. - pub fn send(self, goaway: Goaway) -> Result<()> { - if !self.redirect && !goaway.uri.is_empty() { - return Err(Error::ProtocolViolation); - } - // Both wires cap the URI, so sending a longer one would just get the session - // closed by the peer. Refuse it here, the one chokepoint both wires funnel - // through, rather than at each encoder. - if goaway.uri.len() > MAX_URI { - return Err(Error::ProtocolViolation); - } - // The session closing first is not an error: there is nothing left to drain. - if let Ok(mut state) = self.trigger.write() { - if state.is_some() { - return Err(Error::Duplicate); - } - *state = Some(goaway); - } - Ok(()) - } -} - /// Observes the peer's GOAWAY. /// /// Obtained from `Session::draining`. Cheap to @@ -311,7 +262,7 @@ impl Deadline { /// The halves held by the public [`crate::Session`]. pub(crate) struct Handle { - /// Cloned into each [`Producer`] handed out by `crate::Session::drain`. + /// Fired once by [`Self::send`]. trigger: kio::Producer>, /// Whether this side may name a redirect URI in its own GOAWAY. redirect: bool, @@ -348,14 +299,25 @@ impl Handle { (handle, protocol) } - /// A send half. Handing one out is always allowed; sending twice is what - /// [`Producer::send`] refuses, so a caller never has to ask whether the slot - /// is still free before deciding how to shut down. - pub fn producer(&self) -> Producer { - Producer { - trigger: self.trigger.clone(), - redirect: self.redirect, + /// Send this session's GOAWAY; see `crate::Session::goaway`. + pub fn send(&self, goaway: Goaway) -> Result<()> { + if !self.redirect && !goaway.uri.is_empty() { + return Err(Error::ProtocolViolation); + } + // Both wires cap the URI, so sending a longer one would just get the session + // closed by the peer. Refuse it here, the one chokepoint both wires funnel + // through, rather than at each encoder. + if goaway.uri.len() > MAX_URI { + return Err(Error::ProtocolViolation); } + // The session closing first is not an error: there is nothing left to drain. + if let Ok(mut state) = self.trigger.write() { + if state.is_some() { + return Err(Error::Duplicate); + } + *state = Some(goaway); + } + Ok(()) } /// Watch the local deadline on the driver's `runtime` without holding up the protocol's writes. @@ -393,8 +355,7 @@ mod tests { assert!(deadline.poll(&waiter).is_pending()); handle - .producer() - .send(Goaway::new().with_timeout(Duration::from_secs(1))) + .send(Goaway::new().with_timeout(Some(Duration::from_secs(1)))) .unwrap(); let sent = crate::time::Instant::ZERO.saturating_add(Duration::from_secs(5)); runtime.advance(sent); @@ -415,15 +376,9 @@ mod tests { fn only_the_first_send_wins() { let (handle, protocol) = Handle::new(true); - handle - .producer() - .send(Goaway::redirect("https://first.example/")) - .unwrap(); + handle.send(Goaway::redirect("https://first.example/")).unwrap(); - let err = handle - .producer() - .send(Goaway::redirect("https://second.example/")) - .unwrap_err(); + let err = handle.send(Goaway::redirect("https://second.example/")).unwrap_err(); assert!(matches!(err, Error::Duplicate)); assert_eq!( @@ -439,7 +394,6 @@ mod tests { fn client_redirect_is_refused() { let (handle, _protocol) = Handle::new(false); let err = handle - .producer() .send(Goaway { uri: "https://elsewhere.example/".to_string(), ..Default::default() @@ -449,36 +403,47 @@ mod tests { // An empty URI ("I am going away") is still allowed. let (handle, _protocol) = Handle::new(false); - handle.producer().send(Goaway::default()).unwrap(); + handle.send(Goaway::default()).unwrap(); } #[test] fn timeout_matches_wire_milliseconds() { - assert_eq!(Goaway::new().with_timeout(Duration::ZERO).timeout(), None); + assert_eq!(Goaway::new().with_timeout(None).timeout(), None); assert_eq!( - Goaway::new().with_timeout(Duration::from_micros(1)).timeout(), + Goaway::new().with_timeout(Some(Duration::ZERO)).timeout(), + Some(Duration::ZERO) + ); + assert_eq!( + Goaway::new().with_timeout(Some(Duration::from_micros(1))).timeout(), Some(Duration::from_millis(1)) ); assert_eq!( - Goaway::new().with_timeout(Duration::MAX).timeout(), + Goaway::new().with_timeout(Some(Duration::MAX)).timeout(), Some(Duration::from_millis(MAX_TIMEOUT_MS)) ); } + /// A zero timeout closes the session at once, not never. + #[test] + fn a_zero_timeout_closes_at_once() { + let runtime = crate::time::Deadlines::new(crate::time::Instant::ZERO); + let (handle, _protocol) = Handle::new(true); + let mut deadline = handle.deadline(&runtime); + let waiter = kio::Waiter::noop(); + handle.send(Goaway::new().with_timeout(Some(Duration::ZERO))).unwrap(); + assert_eq!(deadline.poll(&waiter), Poll::Ready(Some(Duration::ZERO))); + } + /// A URI past the wire cap is refused at the send chokepoint rather than /// reaching an encoder that would frame it wrong. #[test] fn oversized_uri_is_refused() { let (handle, _protocol) = Handle::new(true); - let err = handle - .producer() - .send(Goaway::redirect("x".repeat(MAX_URI + 1))) - .unwrap_err(); + let err = handle.send(Goaway::redirect("x".repeat(MAX_URI + 1))).unwrap_err(); assert!(matches!(err, Error::ProtocolViolation)); let (handle, _protocol) = Handle::new(true); handle - .producer() .send(Goaway::redirect("x".repeat(MAX_URI))) .expect("exactly at the cap is fine"); } diff --git a/rs/moq-net/src/ietf/session.rs b/rs/moq-net/src/ietf/session.rs index 9f0a07cef7d..30ce8adc259 100644 --- a/rs/moq-net/src/ietf/session.rs +++ b/rs/moq-net/src/ietf/session.rs @@ -3517,7 +3517,7 @@ mod tests { control.clone(), Default::default(), ); - handle.producer().send(crate::goaway::Goaway::new()).unwrap(); + handle.send(crate::goaway::Goaway::new()).unwrap(); assert!(setup.poll(&kio::Waiter::noop()).is_pending()); let sent = goaway(VERSION, Some(first + 6)); assert_eq!(occurrences(&log, &sent), 1, "client={client}"); diff --git a/rs/moq-net/src/lite/session.rs b/rs/moq-net/src/lite/session.rs index c7ad7097d98..6d05bbcf1eb 100644 --- a/rs/moq-net/src/lite/session.rs +++ b/rs/moq-net/src/lite/session.rs @@ -1495,7 +1495,7 @@ mod tests { let transport = ScriptedSession::new(Vec::new()).without_credit(); let log = transport.log.clone(); let mut server = raw_session(transport.clone(), Version::Lite07, false, None); - server.goaway.producer().send(crate::goaway::Goaway::new()).unwrap(); + server.goaway.send(crate::goaway::Goaway::new()).unwrap(); let _ = server.driver.poll(&kio::Waiter::noop()); assert_eq!(log.bi_opens(), 0, "GOAWAY waits for stream credit"); transport.grant_credit(); @@ -1513,7 +1513,6 @@ mod tests { matches!( client .goaway - .producer() .send(crate::goaway::Goaway::redirect("https://other.example/")), Err(Error::ProtocolViolation) ), @@ -1523,7 +1522,6 @@ mod tests { assert!( server .goaway - .producer() .send(crate::goaway::Goaway::redirect("https://other.example/")) .is_ok(), "{version}" diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs index b4fb6f21951..f1e3cde9e8d 100644 --- a/rs/moq-net/src/reconnect.rs +++ b/rs/moq-net/src/reconnect.rs @@ -116,10 +116,10 @@ impl Default for Backoff { } } -/// How a connection reacts to a peer's GOAWAY. +/// How a connection hands over to a new session after a peer's GOAWAY. #[derive(Clone, Copy, Debug, PartialEq, Eq)] #[non_exhaustive] -pub struct Goaway { +pub struct Handover { /// What to do with the URI the GOAWAY names. Defaults to [`Redirect::SameHost`]. pub redirect: Redirect, /// The longest the old session keeps serving after a GOAWAY. Defaults to 10 s. @@ -130,7 +130,7 @@ pub struct Goaway { pub handover: Duration, } -impl Default for Goaway { +impl Default for Handover { fn default() -> Self { Self { redirect: Redirect::default(), @@ -139,7 +139,7 @@ impl Default for Goaway { } } -impl Goaway { +impl Handover { /// The handover window for a GOAWAY naming `deadline`: whichever is sooner. fn handover(&self, deadline: Option) -> Duration { deadline.map_or(self.handover, |deadline| deadline.min(self.handover)) @@ -155,20 +155,16 @@ pub enum Restart { /// off the drained one. The default. #[default] Redial, - /// Keep serving on the drained session until the peer closes it, then reconnect as after - /// any drop. Right for an address that names one node, where a redial only reaches the - /// draining process again. The peer must close the session itself, as a relay does at - /// its drain deadline. - Wait, /// Redial the same addresses once, as [`Self::Redial`] does, and keep serving on the /// drained session meanwhile. A redial that reaches the draining process again, which - /// drains it before it is [healthy](Backoff::healthy), then serves as under - /// [`Self::Wait`] until the peer closes it. + /// drains it before it is [healthy](Backoff::healthy), then keeps serving until the peer + /// closes it, and the connection reconnects as after any drop. /// /// Right for an address that names one node which may start its replacement beside the /// draining process, as a relay with `[handoff]` does: the one redial reaches the /// replacement. A node that restarts in place costs a single extra session to the - /// draining process rather than one per backoff. + /// draining process rather than one per backoff. The peer must close the session + /// itself, as a relay does at its drain deadline. Handoff, } @@ -179,7 +175,7 @@ pub struct Config { /// When to redial, and when to give up. pub backoff: Backoff, /// How to follow a peer's GOAWAY. - pub goaway: Goaway, + pub goaway: Handover, /// What a GOAWAY naming no URI does. pub restart: Restart, /// The deadline for one dial and its handshake. `None` waits forever. Defaults to 30 s. @@ -197,7 +193,7 @@ impl Default for Config { fn default() -> Self { Self { backoff: Backoff::default(), - goaway: Goaway::default(), + goaway: Handover::default(), restart: Restart::default(), timeout: Some(Duration::from_secs(30)), once: false, @@ -612,7 +608,6 @@ impl + From> Reco // only reach it again. let wait = match self.config.restart { Restart::Redial => false, - Restart::Wait => true, Restart::Handoff => now.saturating_duration_since(live.since) < self.config.backoff.healthy, }; // A one-shot policy ends on any GOAWAY instead, as `once` promises. @@ -751,7 +746,7 @@ impl + From> Reco }; // A GOAWAY naming no URI: the peer learns we are moving, and the session stops // taking new work, which frees the tokens for its replacement. - if old.session.drain().send(goaway::Goaway::new()).is_err() { + if old.session.goaway(goaway::Goaway::new()).is_err() { tracing::debug!("the fallback session was already draining"); } // The fallback session served all along, so a long one still earns the healthy reset. diff --git a/rs/moq-net/src/reconnect/redirect.rs b/rs/moq-net/src/reconnect/redirect.rs index b10dae7dea6..9b967ebbbc4 100644 --- a/rs/moq-net/src/reconnect/redirect.rs +++ b/rs/moq-net/src/reconnect/redirect.rs @@ -384,7 +384,7 @@ mod tests { "a peer-named host is refused without resolving it" ); assert_eq!( - super::super::Goaway::default().redirect, + super::super::Handover::default().redirect, Redirect::SameHost, "and the default config carries that default" ); diff --git a/rs/moq-net/src/reconnect/tests.rs b/rs/moq-net/src/reconnect/tests.rs index 671da3ff67a..3d501deeabd 100644 --- a/rs/moq-net/src/reconnect/tests.rs +++ b/rs/moq-net/src/reconnect/tests.rs @@ -168,7 +168,7 @@ async fn drive( true => goaway::Goaway::new(), false => goaway::Goaway::redirect(uri.clone()), }; - server.drain().send(goaway).unwrap(); + server.goaway(goaway).unwrap(); run.servers.push(server); } _ => run.servers.push(server), @@ -396,7 +396,7 @@ async fn a_healthy_session_establishes_the_policy() { /// then exits, the replacement's close and the failed walks after it retry rather than end. #[kio::test] async fn a_goaway_before_healthy_establishes_the_policy() { - for restart in [Restart::Redial, Restart::Wait] { + for restart in [Restart::Redial, Restart::Handoff] { let peer = url("tcp://peer:1/"); let mut config = config(INITIAL, None); config.restart = restart; @@ -407,8 +407,9 @@ async fn a_goaway_before_healthy_establishes_the_policy() { }) .await; - run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); - // `Redial` follows it to a replacement on the draining relay; `Wait` serves on. + run.servers[0].goaway(goaway::Goaway::new()).unwrap(); + // `Redial` follows it to a replacement on the draining relay; `Handoff` serves on, + // since the session is not yet healthy. if restart == Restart::Redial { drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { run.accepts.len() == 2 @@ -750,10 +751,7 @@ async fn a_healthy_goaway_hands_over_to_the_redirect() { // Past the healthy bar, the GOAWAY is a handover, not a failure. kio::clock::sleep(Duration::from_secs(2)).await; - run.servers[0] - .drain() - .send(goaway::Goaway::redirect(b.to_string())) - .unwrap(); + run.servers[0].goaway(goaway::Goaway::redirect(b.to_string())).unwrap(); let peers = [(a.clone(), Peer::Serve), (b.clone(), Peer::Serve)]; let migrated = kio::clock::now(); drive(&mut machine, &peers, &mut run, |run| run.events.contains(&"retired")).await; @@ -798,8 +796,7 @@ async fn the_handover_takes_the_earlier_deadline() { kio::clock::sleep(Duration::from_secs(2)).await; run.servers[0] - .drain() - .send(goaway::Goaway::new().with_timeout(Duration::from_secs(3600))) + .goaway(goaway::Goaway::new().with_timeout(Some(Duration::from_secs(3600)))) .unwrap(); let drained = kio::clock::now(); drive(&mut machine, &[(peer, Peer::Hang)], &mut run, |run| { @@ -813,7 +810,7 @@ async fn the_handover_takes_the_earlier_deadline() { "a longer peer deadline must not extend our cap" ); - let goaway = Goaway::default(); + let goaway = Handover::default(); assert_eq!(goaway.handover(None), Duration::from_secs(10), "no deadline: ours"); assert_eq!( goaway.handover(Some(Duration::from_secs(3))), @@ -838,7 +835,7 @@ async fn a_connect_cancels_the_offline_a_retirement_queued() { .await; kio::clock::sleep(Duration::from_secs(2)).await; - run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + run.servers[0].goaway(goaway::Goaway::new()).unwrap(); assert!(matches!(next(&mut machine).await, Action::Migrating)); assert!(matches!(next(&mut machine).await, Action::Dial(_))); // The dial outlasts the handover, so the predecessor retires with nothing serving. @@ -856,44 +853,11 @@ async fn a_connect_cancels_the_offline_a_retirement_queued() { assert!(machine.session().is_some()); } -/// Under [`Restart::Wait`] an empty-URI GOAWAY changes nothing while the peer keeps the -/// session: no migration and no redial. Its close then reconnects like any drop. -#[kio::test] -async fn a_waited_restart_redials_only_once_the_session_closes() { - let peer = url("tcp://peer:1/"); - let mut config = config(INITIAL, Some(Duration::from_secs(1))); - config.restart = Restart::Wait; - let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); - let mut run = Run::default(); - drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { - !run.accepts.is_empty() - }) - .await; - - kio::clock::sleep(Duration::from_secs(2)).await; - run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); - kio::clock::sim::idle().await; - assert!( - matches!( - machine.poll(kio::clock::now(), &kio::Waiter::noop()), - Action::Wait(None) - ), - "the policy left the session" - ); - - run.servers.remove(0).abort(Error::GoawayTimeout); - drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |run| { - run.accepts.len() == 2 - }) - .await; - assert_eq!(run.events, ["offline"]); -} - /// A one-shot policy ends cleanly on a GOAWAY naming no URI, as on any other, under /// every restart policy and whether or not the session was healthy. #[kio::test] async fn a_one_shot_policy_ends_on_a_restart() { - for restart in [Restart::Redial, Restart::Wait, Restart::Handoff] { + for restart in [Restart::Redial, Restart::Handoff] { let peer = url("tcp://peer:1/"); let mut config = config(INITIAL, Some(Duration::from_secs(1))); config.restart = restart; @@ -940,7 +904,7 @@ async fn a_handoff_restart_moves_to_a_replacement() { kio::clock::sleep(Duration::from_secs(2)).await; let goaway = kio::clock::now(); - run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + run.servers[0].goaway(goaway::Goaway::new()).unwrap(); drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { run.accepts.len() == 2 }) @@ -968,7 +932,7 @@ async fn a_handoff_restart_redials_once_until_the_session_closes() { .await; kio::clock::sleep(Duration::from_secs(2)).await; - run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + run.servers[0].goaway(goaway::Goaway::new()).unwrap(); drive( &mut machine, &[(peer.clone(), Peer::Goaway(String::new()))], @@ -1096,8 +1060,7 @@ async fn a_peer_goaway_wins_over_a_probe() { let (mut machine, fallback_server) = on_fallback(Config::default()).await; probe(&mut machine).await; fallback_server - .drain() - .send(goaway::Goaway::new()) + .goaway(goaway::Goaway::new()) .expect("the fallback peer sends a GOAWAY"); kio::clock::sim::idle().await; assert!( @@ -1122,8 +1085,7 @@ async fn a_redirect_after_a_probe_got_through_is_followed() { let elsewhere = url("https://peer:2/"); fallback_server - .drain() - .send(goaway::Goaway::redirect(elsewhere.to_string())) + .goaway(goaway::Goaway::redirect(elsewhere.to_string())) .expect("the fallback peer sends a GOAWAY"); assert!(matches!(next(&mut machine).await, Action::Abandon)); let Action::Dial(attempt) = next(&mut machine).await else { @@ -1147,8 +1109,7 @@ async fn a_fallback_peer_redirect_establishes_the_policy() { assert!(matches!(next(&mut machine).await, Action::Migrating)); fallback_server - .drain() - .send(goaway::Goaway::redirect(url("https://peer:2/").to_string())) + .goaway(goaway::Goaway::redirect(url("https://peer:2/").to_string())) .expect("the fallback peer sends a GOAWAY"); assert!(matches!(next(&mut machine).await, Action::Abandon)); assert!(matches!(next(&mut machine).await, Action::Dial(_))); @@ -1173,8 +1134,7 @@ async fn a_redirect_during_the_backoff_after_a_probe_is_followed() { let elsewhere = url("https://peer:2/"); fallback_server - .drain() - .send(goaway::Goaway::redirect(elsewhere.to_string())) + .goaway(goaway::Goaway::redirect(elsewhere.to_string())) .expect("the fallback peer sends a GOAWAY"); let Action::Dial(attempt) = next(&mut machine).await else { panic!("expected a redial"); @@ -1254,7 +1214,7 @@ async fn a_dropped_replacement_migrates_back_to_the_serving_predecessor() { machine.connected(kio::clock::now(), client); kio::clock::sleep(Duration::from_secs(2)).await; - server.drain().send(goaway::Goaway::redirect(b.to_string())).unwrap(); + server.goaway(goaway::Goaway::redirect(b.to_string())).unwrap(); assert!(matches!(next(&mut machine).await, Action::Migrating)); let Action::Dial(attempt) = next(&mut machine).await else { panic!("expected the redirect's dial"); @@ -1300,7 +1260,7 @@ async fn a_dropped_replacement_goes_offline_over_a_closing_predecessor() { .await; kio::clock::sleep(Duration::from_secs(2)).await; - run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + run.servers[0].goaway(goaway::Goaway::new()).unwrap(); assert!(matches!(next(&mut machine).await, Action::Migrating)); assert!(matches!(next(&mut machine).await, Action::Dial(_))); let (replacement, replacement_server) = handshake().await; @@ -1478,6 +1438,14 @@ fn each_pacing_knob_turns_off() { assert_eq!(delay, defaults.initial * 1024, "no max leaves the delay uncapped"); assert_eq!(pacing.settled(), defaults.healthy); + // Zero is a real cap, not "uncapped": every redial goes at once. + let pacing = Pacing::new(&Backoff { + max: Some(Duration::ZERO), + ..defaults + }); + assert_eq!(pacing.initial, Duration::ZERO); + assert_eq!(pacing.next(defaults.initial), Duration::ZERO, "zero max never waits"); + let pacing = Pacing::new(&Backoff { multiplier: 0, ..defaults diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index ed13434111f..800bef31e6e 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -404,23 +404,34 @@ impl Session { } } - /// Drain the peer gracefully: the handle for sending this session's single - /// GOAWAY. + /// Drain the peer gracefully: send this session's single GOAWAY. /// - /// The graceful counterpart to [`abort`](Self::abort). Send the message with - /// [`goaway::Producer::send`], then await [`closed`](Self::closed) to observe - /// the peer leaving. + /// The graceful counterpart to [`abort`](Self::abort). Await + /// [`closed`](Self::closed) afterwards to observe the peer leaving. /// /// Only a [`Goaway`](goaway::Goaway) carrying a [`timeout`](goaway::Goaway::timeout) /// schedules a close of our own, so without one this waits for a peer that may - /// never leave. Set a deadline when the drain has to finish. + /// never leave. Set a deadline when the drain has to finish. It starts at the driver + /// poll this send wakes, so the session must stay driven for the message to reach + /// the wire and for the deadline to close it. /// /// Available on every version. A version with no GOAWAY message (moq-lite-03 /// and earlier) simply carries no explanation to the peer; the deadline is the /// sender's own timer either way, so the session still closes on schedule and /// the caller does not branch on the negotiated version. - pub fn drain(&self) -> goaway::Producer { - self.goaway.producer() + /// + /// # Errors + /// + /// Returns [`Error::ProtocolViolation`] when a client names a redirect URI, which + /// the peer would be required to close the session over (draft-22 section 9.2), or + /// when the URI exceeds the 8,192-byte wire cap. + /// + /// Returns [`Error::Duplicate`] if this session already sent one. The wire carries + /// at most one, so the second is dropped rather than replacing the first, but the + /// caller is told rather than left guessing. A session that already closed has + /// nothing left to drain, which is no error. + pub fn goaway(&self, goaway: goaway::Goaway) -> Result<(), Error> { + self.goaway.send(goaway) } /// Observe a GOAWAY from the peer, telling us to migrate elsewhere. diff --git a/rs/moq-net/tests/integration/auth.rs b/rs/moq-net/tests/integration/auth.rs index 8f2088e5d9d..f9d80b49303 100644 --- a/rs/moq-net/tests/integration/auth.rs +++ b/rs/moq-net/tests/integration/auth.rs @@ -2265,7 +2265,7 @@ async fn a_handle_backs_one_active_session(version: &'static str) { .await; assert!(matches!(concurrent, Err(Error::Duplicate)), "{:?}", concurrent.err()); - first.server.drain().send(moq_net::goaway::Goaway::new()).unwrap(); + first.server.goaway(moq_net::goaway::Goaway::new()).unwrap(); support::Draining::recv(&first.client.draining()) .await .expect("the GOAWAY"); @@ -2321,7 +2321,7 @@ async fn a_token_queued_before_our_goaway_rides_only_the_successor(version: &'st // Its first poll queues the token; the driver runs only once this side drains. let mut adding = Box::pin(added(&handle, "late")); assert!(adding.as_mut().now_or_never().is_none()); - first.client.drain().send(moq_net::goaway::Goaway::new()).unwrap(); + first.client.goaway(moq_net::goaway::Goaway::new()).unwrap(); kio::clock::sleep(Duration::from_secs(1)).await; assert!( first_tokens.next().now_or_never().is_none(), diff --git a/rs/moq-net/tests/integration/drained_claim.rs b/rs/moq-net/tests/integration/drained_claim.rs index 369a389060b..3039dc19722 100644 --- a/rs/moq-net/tests/integration/drained_claim.rs +++ b/rs/moq-net/tests/integration/drained_claim.rs @@ -117,7 +117,7 @@ async fn drained_claim(version: &str) { .borrow() .iter() .for_each(|(output, _)| output.broadcast().close()); - a_link.server.drain().send(Goaway::new()).unwrap(); + a_link.server.goaway(Goaway::new()).unwrap(); // Past the linger, which is as long as the cache window, nothing reads the relay's // front for the path. kio::clock::sleep(moq_net::cache::DEFAULT_EXPIRY * 2).await; diff --git a/rs/moq-net/tests/integration/goaway.rs b/rs/moq-net/tests/integration/goaway.rs index dae27219687..1ec1d6df0d6 100644 --- a/rs/moq-net/tests/integration/goaway.rs +++ b/rs/moq-net/tests/integration/goaway.rs @@ -48,7 +48,7 @@ async fn closes_stalled_goaway(version: Version, writes: bool) { } let started = kio::clock::now(); let timeout = Duration::from_secs(1); - pair.server.drain().send(Goaway::new().with_timeout(timeout)).unwrap(); + pair.server.goaway(Goaway::new().with_timeout(Some(timeout))).unwrap(); let reason = kio::clock::timeout(timeout * 2, pair.server.closed()) .await .unwrap_or_else(|_| panic!("{version}: a stalled GOAWAY held the session past its deadline")); @@ -97,8 +97,7 @@ async fn goaway_send_receive_all_versions() { // Server -> client. pair.server - .drain() - .send(Goaway::redirect("https://new.example.com")) + .goaway(Goaway::redirect("https://new.example.com")) .expect("send goaway"); let goaway = pair @@ -116,8 +115,7 @@ async fn goaway_send_receive_all_versions() { // swapped in behind the peer's back. let err = pair .server - .drain() - .send(Goaway::redirect("https://second.example.com")) + .goaway(Goaway::redirect("https://second.example.com")) .expect_err("a session sends one GOAWAY"); assert!(matches!(err, moq_net::Error::Duplicate), "version {version}"); @@ -139,8 +137,7 @@ async fn goaway_wire_timeout_moq_transport() { let pair = connect_mock(MockConnectOptions::new(version)).await; pair.server - .drain() - .send(Goaway::redirect("moqt://relay.example/").with_timeout(Duration::from_secs(5))) + .goaway(Goaway::redirect("moqt://relay.example/").with_timeout(Some(Duration::from_secs(5)))) .expect("send goaway"); let goaway = pair @@ -166,7 +163,7 @@ async fn goaway_client_to_server_moq_lite_04() { let version: Version = "moq-lite-04".parse().unwrap(); let pair = connect_mock(MockConnectOptions::new(version)).await; - pair.client.drain().send(Goaway::new()).expect("send goaway"); + pair.client.goaway(Goaway::new()).expect("send goaway"); let goaway = pair .server @@ -194,8 +191,7 @@ async fn goaway_drains_without_a_wire_message_moq_lite_03() { let pair = connect_mock(MockConnectOptions::new(version)).await; pair.server - .drain() - .send(Goaway::new().with_timeout(Duration::from_secs(1))) + .goaway(Goaway::new().with_timeout(Some(Duration::from_secs(1)))) .expect("drain is available on every version"); // Nothing reaches the peer, since this version has no GOAWAY message. @@ -222,17 +218,16 @@ async fn goaway_client_redirect_refused_moq_transport_18() { let err = pair .client - .drain() - .send(Goaway::redirect("https://elsewhere.example/")) + .goaway(Goaway::redirect("https://elsewhere.example/")) .expect_err("a client may not name a redirect URI"); assert!(matches!(err, moq_net::Error::ProtocolViolation)); // The refusal is local, so nothing was claimed: the client may still send the // empty-URI GOAWAY it was always allowed to send. - pair.client.drain().send(Goaway::new()).expect("an empty URI is legal"); + pair.client.goaway(Goaway::new()).expect("an empty URI is legal"); // An empty URI ("I am leaving") is what a client may send. Check it from the // server side, which is the direction allowed to redirect at all. - pair.server.drain().send(Goaway::new()).expect("send goaway"); + pair.server.goaway(Goaway::new()).expect("send goaway"); let goaway = pair .client .draining() @@ -254,8 +249,7 @@ async fn goaway_timeout_force_close_moq_transport_17() { let pair = connect_mock(MockConnectOptions::new(version)).await; pair.server - .drain() - .send(Goaway::redirect("moqt://relay.example/").with_timeout(Duration::from_millis(100))) + .goaway(Goaway::redirect("moqt://relay.example/").with_timeout(Some(Duration::from_millis(100)))) .expect("send goaway"); // The client observes the GOAWAY but deliberately does NOT leave. @@ -425,8 +419,7 @@ async fn goaway_keeps_new_subscribes(version: Version) { // Server drains; the client observes the GOAWAY. server - .drain() - .send(moq_net::goaway::Goaway::redirect("https://elsewhere.example/")) + .goaway(moq_net::goaway::Goaway::redirect("https://elsewhere.example/")) .expect("send goaway"); client.draining().recv().await.expect("goaway"); assert!(client.draining().peek().is_some()); @@ -523,7 +516,7 @@ async fn goaway_drains_routes(version: Version) { let MockPair { client, server, .. } = connect_mock(opts).await; route_at(&mut announced, |route| *route.instance() != standby).await; - server.drain().send(Goaway::new()).expect("send goaway"); + server.goaway(Goaway::new()).expect("send goaway"); client.draining().recv().await.expect("goaway"); // Nothing else is sent on the announce stream, so this only resolves if the diff --git a/rs/moq-net/verify/reconnect/Handover.tla b/rs/moq-net/verify/reconnect/Handover.tla index 8a5b534de12..9f5d29eac7a 100644 --- a/rs/moq-net/verify/reconnect/Handover.tla +++ b/rs/moq-net/verify/reconnect/Handover.tla @@ -29,7 +29,7 @@ CONSTANTS Endpoints, \* where a dial can land Start, \* the configured target Refused, \* a GOAWAY URI `Redirect::target` refuses - Restart, \* `Config::restart`: "redial", "wait", or "handoff" + Restart, \* `Config::restart`: "redial" or "handoff" MaxSessions \* sessions that connect in one behaviour Sessions == 1..MaxSessions @@ -350,13 +350,13 @@ EndClosed == \* `Phase::Live`: a GOAWAY is checked before the close. Goaway(s) == watch /\ sess[s].goaway # NoGoaway -\* `Restart::Wait` serves out a restart in place; `Restart::Handoff` does so -\* only on a session drained before it was healthy, and redials an older one -\* once, as `Restart::Redial` does. +\* `Restart::Handoff` serves out a restart in place on a session drained +\* before it was healthy, and redials an older one once, as `Restart::Redial` +\* does. WaitOut(s) == /\ sess[s].goaway = "" - /\ \/ Restart = "wait" - \/ Restart = "handoff" /\ sess[s].age = 0 + /\ Restart = "handoff" + /\ sess[s].age = 0 LiveEnds == /\ queue = <<>> diff --git a/rs/moq-net/verify/reconnect/RestartWait.cfg b/rs/moq-net/verify/reconnect/RestartWait.cfg deleted file mode 100644 index da9611b396e..00000000000 --- a/rs/moq-net/verify/reconnect/RestartWait.cfg +++ /dev/null @@ -1,20 +0,0 @@ -\* Handover.cfg's rules under `Restart::Wait`, where a GOAWAY naming no URI -\* keeps the session serving until the peer closes it. -CONSTANTS - Endpoints = {"a", "b"} - Start = "a" - Refused = "far" - Restart = "wait" - MaxSessions = 3 - -SPECIFICATION Spec -VIEW View - -INVARIANTS - NoStaleSession - StatusTruthful - OfflineMeansNothingServes - NoOrphan - PresenceBalanced - FirstWalkFailsFast - RedirectFollowed diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index dcf85285dce..8c3bc6c95fb 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -2309,8 +2309,8 @@ impl Cluster { /// redial moves to a fresh socket rather than retrying one the path may be /// dropping. /// - /// Every peer client's reconnect loop gives up eventually: a zero backoff - /// timeout, which would retry forever, takes [`PEER_RETRY`] instead. The + /// Every peer client's reconnect loop gives up eventually: an unset or `none` + /// backoff timeout, which would retry forever, takes [`PEER_RETRY`] instead. The /// redial loop already retries forever, and only it re-reads `cluster.token`, /// so an unending inner loop could keep presenting a rotated-out token. /// diff --git a/rs/moq-relay/src/shutdown.rs b/rs/moq-relay/src/shutdown.rs index 876657d22c8..72fb6bd95a3 100644 --- a/rs/moq-relay/src/shutdown.rs +++ b/rs/moq-relay/src/shutdown.rs @@ -373,7 +373,8 @@ impl Observer { /// and no grace. /// /// With no time left (a zero [`drain_timeout`](Self::drain_timeout), or a - /// session accepted after the deadline) the session is closed at once. + /// session accepted after the deadline) the GOAWAY's zero timeout closes the + /// session at once with `GoawayTimeout`, likely before the GOAWAY is written. pub async fn drain_session(&self, session: &moq_net::Session) { self.drain(session, true).await; } @@ -386,21 +387,10 @@ impl Observer { // first), so the count holds whichever way the session goes. let _draining = counted.then(|| self.draining(deadline)); - // No grace left, so there is nothing to drain. Close now rather than send a - // GOAWAY: the peer would have no time to act on it, and a zero timeout means - // "no deadline" on the wire, which is the opposite of what was asked for. - if remaining.is_zero() { - session.abort(moq_net::Error::GoingAway); - return; - } - // "Reconnect to me": the relay is restarting, not moving. An empty URI is // legal from either side and this runs once per session, so neither error // is reachable; the deadline closes the session regardless. - if let Err(err) = session - .drain() - .send(moq_net::goaway::Goaway::new().with_timeout(remaining)) - { + if let Err(err) = session.goaway(moq_net::goaway::Goaway::new().with_timeout(Some(remaining))) { tracing::warn!(%err, "failed to drain session"); } diff --git a/rs/moq-relay/tests/integration/cluster_fresh_port.rs b/rs/moq-relay/tests/integration/cluster_fresh_port.rs index 0105a9fbcf9..9c8ac9b571d 100644 --- a/rs/moq-relay/tests/integration/cluster_fresh_port.rs +++ b/rs/moq-relay/tests/integration/cluster_fresh_port.rs @@ -119,19 +119,19 @@ async fn spawn_proxy(upstream: SocketAddr) -> (SocketAddr, Arc, effective: Duration) { let _ = rind_rustls::default_provider().unwrap().install_default(); let peer = spawn_peer().await; diff --git a/rs/moq-relay/tests/integration/goaway_cluster.rs b/rs/moq-relay/tests/integration/goaway_cluster.rs index c19e9a6c4da..7c36d416b19 100644 --- a/rs/moq-relay/tests/integration/goaway_cluster.rs +++ b/rs/moq-relay/tests/integration/goaway_cluster.rs @@ -99,12 +99,8 @@ fn drain_session_with_zero_timeout_closes_at_once() { memory::run(drain_session_with_zero_timeout_closes_at_once_inner); } -/// A zero drain window means no grace, so `drain_session` closes the session -/// rather than sending a GOAWAY the peer has no time to act on. -/// -/// The regression it guards: `Goaway::with_timeout(ZERO)` means "no deadline" on -/// the wire, so passing the window straight through left the session with no -/// force-close at all and `drain_session` awaiting a peer that never leaves. +/// A zero drain window means no grace, so `drain_session` closes the session at +/// once rather than awaiting a peer under no deadline to leave. async fn drain_session_with_zero_timeout_closes_at_once_inner() { let net = Net::new(); let mut accepted = net.upstream("upstream", &sim::origin()); @@ -119,8 +115,7 @@ async fn drain_session_with_zero_timeout_closes_at_once_inner() { let (_trigger, shutdown) = shutdown::Observer::new(Duration::ZERO); - // The assertion is that this resolves at all: with the window passed through as - // a wire timeout it would await a peer under no deadline to leave. + // The assertion is that this resolves at all. within("drain_session returns", shutdown.drain_session(&server_session)).await; // The peer observes the close rather than being left connected. @@ -195,8 +190,7 @@ async fn cluster_continues_a_group_split_by_goaway_inner() { other => panic!("expected A's route, got {other:?}"), }; session_a - .drain() - .send(moq_net::goaway::Goaway::redirect(net.url("b"))) + .goaway(moq_net::goaway::Goaway::redirect(net.url("b"))) .expect("send goaway"); let _session_b = accept("B accepts", &mut accepted_b).await; within("routed via B", async { @@ -301,8 +295,7 @@ async fn cluster_migrates_on_upstream_goaway_inner() { // ── sibling A drains with a redirect to sibling B ──────────────── session_a - .drain() - .send(moq_net::goaway::Goaway::redirect(net.url("b"))) + .goaway(moq_net::goaway::Goaway::redirect(net.url("b"))) .expect("send goaway"); // The cluster reconnects: sibling B accepts a session. @@ -499,8 +492,7 @@ async fn cluster_diamond_goaway_seamless_failover_inner() { // ── TRIGGER: MID-A drains BOTTOM with a redirect to MID-B ─────────── session_bottom_on_a - .drain() - .send(moq_net::goaway::Goaway::redirect(net.url("mid-b")).with_timeout(Duration::from_secs(5))) + .goaway(moq_net::goaway::Goaway::redirect(net.url("mid-b")).with_timeout(Some(Duration::from_secs(5)))) .expect("send goaway"); // Positive gate: MID-B's first inbound connection can only be BOTTOM's @@ -683,8 +675,7 @@ async fn goaway_handover_is_enforced_while_the_replacement_dial_hangs_inner() { // Send the client somewhere that will never finish handshaking. No wire timeout: // our own `--connect-goaway-handover` is then the only thing bounding the old session. server_session - .drain() - .send(moq_net::goaway::Goaway::redirect(net.url("black-hole"))) + .goaway(moq_net::goaway::Goaway::redirect(net.url("black-hole"))) .expect("send goaway"); let sent = kio::clock::now(); diff --git a/rs/moq-relay/tests/integration/handoff.rs b/rs/moq-relay/tests/integration/handoff.rs index 8ad848f43d1..6554fda0632 100644 --- a/rs/moq-relay/tests/integration/handoff.rs +++ b/rs/moq-relay/tests/integration/handoff.rs @@ -110,9 +110,19 @@ async fn connect(url: &str) -> moq_tokio::Connection { established(dial().init(Default::default()).unwrap().with_reconnect(false), url).await } -/// A session that stays through the relay's drain until the relay closes it. +/// A session that ignores the relay's GOAWAY and stays through its drain until the relay +/// closes it. +/// +/// Under [`moq_tokio::Restart::Handoff`] a GOAWAY that drains a session before it is +/// healthy changes nothing, and this healthy bar outlasts the test. async fn stays(url: &str, quic: moq_tokio::quic::Config) -> moq_tokio::Connection { - established(dial().init(quic).unwrap().with_restart(moq_tokio::Restart::Wait), url).await + let mut config = dial(); + config.backoff.healthy = Some(Duration::from_secs(3600).into()); + established( + config.init(quic).unwrap().with_restart(moq_tokio::Restart::Handoff), + url, + ) + .await } /// Wait for the GOAWAY a draining relay sends `connection`. @@ -500,7 +510,7 @@ async fn a_killed_relay_s_clients_are_reset_by_the_one_it_replaced() { let mut new = relay(dir.path(), &cert, &key, port, ¬ify_path); ready(¬ify).await; - let mut orphan = stays(&url, quic).await; + let mut orphan = established(dial().init(quic).unwrap(), &url).await; // Past the default healthy bar, so the reset reconnects rather than ending the loop. kio::clock::sleep(Duration::from_secs(1)).await; diff --git a/rs/moq-tokio/src/cli.rs b/rs/moq-tokio/src/cli.rs index 66028165693..739acda5a9b 100644 --- a/rs/moq-tokio/src/cli.rs +++ b/rs/moq-tokio/src/cli.rs @@ -6,8 +6,9 @@ //! never reaches that code, so it has to answer them itself. Both shapes exist here: //! a TOML merge that layers CLI, env, and file with recorded provenance, and //! moq-cli's repeated `--` stage grammar. [`Duration`] parses the human-readable -//! values used by those flags and TOML keys. +//! values used by those flags and TOML keys, and [`Bound`] a limit `none` turns off. +pub(crate) mod bound; #[path = "deprecated.rs"] mod deprecated; pub(crate) mod duration; @@ -23,6 +24,7 @@ use usage_rs::config::{ SourceKind, Value, resolve, }; +pub use bound::Bound; pub use deprecated::Deprecated; pub use duration::Duration; diff --git a/rs/moq-tokio/src/cli/bound.rs b/rs/moq-tokio/src/cli/bound.rs new file mode 100644 index 00000000000..76bdb172944 --- /dev/null +++ b/rs/moq-tokio/src/cli/bound.rs @@ -0,0 +1,111 @@ +//! A duration limit that can be turned off. + +use std::fmt; +use std::str::FromStr; +use std::sync::atomic::{AtomicBool, Ordering}; + +/// TODO(next release): delete. A zero backoff `max` or `timeout` turned the bound off +/// until `none` took that job. Set when a flag or config key spells a zero, so +/// [`warn_zero`] warns a setting written for the old meaning. Parsing runs before +/// logging is up, so the warning waits for the client; an API caller's deliberate +/// zero is never parsed and stays quiet. +static ZERO_PARSED: AtomicBool = AtomicBool::new(false); + +/// Warn once per process, if a zero was parsed, that the `zero` keys no longer turn their bound off. +pub(crate) fn warn_zero(zero: &[String]) { + if !zero.is_empty() && ZERO_PARSED.swap(false, Ordering::Relaxed) { + let zero = zero.join(" and "); + tracing::warn!("a zero {zero} no longer turns the bound off; write \"none\" for no bound"); + } +} + +/// A limit written as a duration (`5s`, `0s`), or `none` to turn it off. +/// +/// An unset flag or key takes the knob's default instead, so a setting has three +/// states: unset, `none`, and a duration, where zero means zero. +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, serde_with::SerializeDisplay, serde_with::DeserializeFromStr)] +pub struct Bound(Option); + +impl Bound { + /// No limit, written `none`. + pub const NONE: Self = Self(None); +} + +impl From for Bound { + fn from(value: std::time::Duration) -> Self { + Self(Some(value)) + } +} + +impl From> for Bound { + fn from(value: Option) -> Self { + Self(value) + } +} + +impl From for Option { + fn from(value: Bound) -> Self { + value.0 + } +} + +impl FromStr for Bound { + type Err = humantime::DurationError; + + fn from_str(value: &str) -> Result { + match value { + "none" => Ok(Self::NONE), + value => { + let value = humantime::parse_duration(value)?; + // TODO(next release): delete, with `ZERO_PARSED`. + if value.is_zero() { + ZERO_PARSED.store(true, Ordering::Relaxed); + } + Ok(value.into()) + } + } + } +} + +impl fmt::Display for Bound { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self.0 { + Some(value) => humantime::format_duration(value).fmt(f), + None => f.write_str("none"), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_none_zero_and_a_duration() { + assert_eq!("none".parse::().unwrap(), Bound::NONE); + assert_eq!("0s".parse::().unwrap(), Bound::from(std::time::Duration::ZERO)); + assert_eq!( + "1m 5s".parse::().unwrap(), + Bound::from(std::time::Duration::from_secs(65)) + ); + assert!("never".parse::().is_err()); + } + + /// What a config file holds reads back as the same bound. + #[test] + fn round_trips_through_toml() { + #[derive(serde::Serialize, serde::Deserialize, PartialEq, Debug)] + struct File { + bound: Bound, + } + for bound in [ + Bound::NONE, + std::time::Duration::ZERO.into(), + std::time::Duration::from_millis(1500).into(), + ] { + let file = File { bound }; + let text = toml::to_string(&file).unwrap(); + assert_eq!(toml::from_str::(&text).unwrap(), file, "{text}"); + } + } +} diff --git a/rs/moq-tokio/src/client.rs b/rs/moq-tokio/src/client.rs index 4f0026ebce2..4fd44b4e4c4 100644 --- a/rs/moq-tokio/src/client.rs +++ b/rs/moq-tokio/src/client.rs @@ -813,8 +813,7 @@ mod tests { redirect.set_ip_host(peer.ip()).unwrap(); } server_session - .drain() - .send(moq_net::goaway::Goaway::redirect(redirect.to_string())) + .goaway(moq_net::goaway::Goaway::redirect(redirect.to_string())) .unwrap(); let result = connection.closed().await; if pinned { diff --git a/rs/moq-tokio/src/connect.rs b/rs/moq-tokio/src/connect.rs index 51830c57e5d..97c0c59d0bc 100644 --- a/rs/moq-tokio/src/connect.rs +++ b/rs/moq-tokio/src/connect.rs @@ -4,7 +4,7 @@ //! accept side lives in [`crate::listen`]. use crate::Backoff; -use crate::connection::Goaway; +use crate::connection::Handover; use std::net; use url::Url; @@ -672,7 +672,7 @@ pub struct Config { /// How [`crate::Client::connect`] reacts to a peer's GOAWAY (`--connect-goaway-*`). #[usage(flatten)] #[serde(default)] - pub goaway: Goaway, + pub goaway: Handover, /// WebSocket fallback settings (`--websocket-*`), used when QUIC is /// blocked. diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index 7390eca15e8..2fbbc333114 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -74,7 +74,8 @@ pub struct Backoff { )] pub multiplier: u32, - /// Cap on the delay between reconnect attempts. Defaults to 5s; zero leaves it uncapped. + /// Cap on the delay between reconnect attempts. Defaults to 5s; `none` leaves it uncapped, + /// and zero retries at once. /// /// A cap below `initial` caps the first delay too. #[usage( @@ -84,7 +85,7 @@ pub struct Backoff { setting = "connect.backoff.max" )] #[serde(default, skip_serializing_if = "Option::is_none")] - pub max: Option, + pub max: Option, /// How long a session must stay up to count as healthy. Defaults to 1s; zero is refused. /// @@ -95,7 +96,8 @@ pub struct Backoff { #[serde(skip)] pub healthy: Option, - /// Maximum time to spend retrying before giving up. Unset or zero retries forever. + /// Maximum time to spend retrying before giving up. Unset or `none` retries forever, and + /// zero gives up at the first failure. /// /// No new walk of the peer's addresses starts once it has passed, but one under way finishes: /// each candidate but the last gets up to 5s, and the last runs to `--connect-timeout`, @@ -110,7 +112,7 @@ pub struct Backoff { setting = "connect.backoff.timeout" )] #[serde(default, skip_serializing_if = "Option::is_none")] - pub timeout: Option, + pub timeout: Option, #[usage( name = "backoff-initial", long = "backoff-initial", @@ -188,11 +190,10 @@ impl Backoff { *knob = set.into(); } } - // A configured zero turns the bound off. let bounds = [(&mut backoff.max, self.max), (&mut backoff.timeout, self.timeout)]; for (bound, set) in bounds { if let Some(set) = set { - *bound = Some(set.into()).filter(|set: &Duration| !set.is_zero()); + *bound = set.into(); } } backoff @@ -204,6 +205,13 @@ impl Backoff { if self.resolve().healthy.is_zero() { return Err(Error::BackoffHealthy); } + // TODO(next release): delete, with `cli::bound::warn_zero`. + let zero: Vec<_> = [("max", self.max), ("timeout", self.timeout)] + .into_iter() + .filter(|(_, set)| set.and_then(Into::into).is_some_and(|set: Duration| set.is_zero())) + .map(|(name, _)| format!("connect.backoff.{name}")) + .collect(); + crate::cli::bound::warn_zero(&zero); Ok(()) } @@ -282,12 +290,12 @@ impl From for moq_net::reconnect::Redirect { } } -/// How a reconnect loop reacts to a peer's GOAWAY. +/// How a reconnect loop hands over to a new session after a peer's GOAWAY. #[derive(Clone, Debug, usage_rs::Args, serde::Serialize, serde::Deserialize)] #[usage(unknown_flags = "error", args_override_self = false)] #[serde(default, deny_unknown_fields)] #[non_exhaustive] -pub struct Goaway { +pub struct Handover { /// What to do with the URI a peer names in its GOAWAY. `same-host` (the /// default) lets it move us between ports and schemes on the host we already /// chose, `follow` also lets it name the host, and `ignore` redials the @@ -306,8 +314,7 @@ pub struct Goaway { /// "500ms". It closes sooner, once its replacement is up and it carries no /// subscription or fetch. A GOAWAY naming a shorter deadline wins, since the peer /// force-closes at its own deadline regardless, but a longer one does not extend - /// it. Defaults to 10 seconds. - /// `None` uses the default; zero disables this duration. + /// it. Defaults to 10 seconds; zero closes the old session at once. #[usage( name = "connect-goaway-handover", long, @@ -336,7 +343,7 @@ pub struct Goaway { legacy_handover: Option, } -impl Default for Goaway { +impl Default for Handover { fn default() -> Self { Self { redirect: Redirect::SameHost, @@ -347,7 +354,7 @@ impl Default for Goaway { } } -impl Goaway { +impl Handover { pub(crate) fn deprecated(&self) -> crate::cli::Deprecated { let mut found = crate::cli::Deprecated::default(); if self.legacy_redirect.is_some() { @@ -368,8 +375,8 @@ impl Goaway { } /// The policy these knobs configure, with each unset one at its default. - pub(crate) fn resolve(&self) -> moq_net::reconnect::Goaway { - let mut goaway = moq_net::reconnect::Goaway::default(); + pub(crate) fn resolve(&self) -> moq_net::reconnect::Handover { + let mut goaway = moq_net::reconnect::Handover::default(); goaway.redirect = self.redirect.into(); if let Some(handover) = self.handover { goaway.handover = handover.into(); @@ -1592,13 +1599,13 @@ mod tests { parsed.backoff.initial = Some((Duration::from_secs(7)).into()); parsed.backoff.multiplier = 5; parsed.backoff.max = Some((Duration::from_secs(11)).into()); - parsed.backoff.timeout = Some((Duration::ZERO).into()); + parsed.backoff.timeout = Some(crate::cli::Bound::NONE); parsed.update_from(&[]); assert_eq!(parsed.backoff.initial, Some((Duration::from_secs(7)).into())); assert_eq!(parsed.backoff.multiplier, 5); assert_eq!(parsed.backoff.max, Some((Duration::from_secs(11)).into())); - assert_eq!(parsed.backoff.timeout(), None, "0 means retry forever"); + assert_eq!(parsed.backoff.timeout(), None, "none means retry forever"); // With no flags, the parser defaults resolve to the policy's defaults. let parsed = Wrapper::parse_from(&[]).unwrap(); @@ -1613,6 +1620,46 @@ mod tests { assert_eq!(parsed.backoff.resolve().initial, Duration::from_secs(3)); } + /// `max` and `timeout` each read unset as the default, `none` as off, and zero as zero, + /// from a flag and from a config file alike. + #[test] + fn a_backoff_bound_is_unset_off_zero_or_a_value() { + #[derive(usage_rs::Cli)] + #[usage(unknown_flags = "error", args_override_self = false)] + #[usage(settings)] + struct Wrapper { + #[usage(flatten)] + backoff: Backoff, + } + + let defaults = moq_net::reconnect::Backoff::default(); + for field in ["max", "timeout"] { + let read = |backoff: &moq_net::reconnect::Backoff| match field { + "max" => backoff.max, + _ => backoff.timeout, + }; + let flag = format!("--connect-backoff-{field}"); + for (value, expected) in [ + (None, read(&defaults)), + (Some("none"), None), + (Some("0s"), Some(Duration::ZERO)), + (Some("3s"), Some(Duration::from_secs(3))), + ] { + let args = value.map_or_else(Vec::new, |value| vec![flag.as_str(), value]); + let flagged = Wrapper::parse_from(&args.iter().map(std::ffi::OsStr::new).collect::>()) + .unwrap() + .backoff; + assert_eq!(read(&flagged.resolve()), expected, "{flag} {value:?}"); + + let file = value.map_or_else(String::new, |value| format!("{field} = \"{value}\"")); + let filed: Backoff = toml::from_str(&file).unwrap(); + assert_eq!(read(&filed.resolve()), expected, "{file:?}"); + let saved: Backoff = toml::Value::try_from(&filed).unwrap().try_into().unwrap(); + assert_eq!(read(&saved.resolve()), expected, "{file:?} round-trip"); + } + } + } + /// GOAWAY typed defaults remain overrideable by a standing TOML layer. #[test] fn cli_does_not_clobber_toml_goaway() { @@ -1621,13 +1668,13 @@ mod tests { #[usage(settings)] struct Wrapper { #[usage(flatten)] - goaway: Goaway, + goaway: Handover, } // No flags passed: the typed defaults are present. let parsed = Wrapper::parse_from(&[]).unwrap(); assert_eq!(parsed.goaway.redirect, Redirect::SameHost); - assert_eq!(parsed.goaway.resolve(), moq_net::reconnect::Goaway::default()); + assert_eq!(parsed.goaway.resolve(), moq_net::reconnect::Handover::default()); // Flags passed: they land where the merge can see them. let parsed = Wrapper::parse_from(&[ @@ -2062,7 +2109,7 @@ mod tests { async fn an_unsupported_scheme_ends_unlimited_reconnects() { let mut config = crate::connect::Config::default(); config.tls.insecure = Some(true); - config.backoff.timeout = Some((Duration::ZERO).into()); + config.backoff.timeout = Some(crate::cli::Bound::NONE); let client = config.init(Default::default()).expect("build client"); let started = kio::clock::now(); let connection = client.connect(url("moqz://localhost/")); @@ -2078,7 +2125,7 @@ mod tests { async fn an_invalid_tls_server_name_ends_unlimited_reconnects() { let mut config = crate::connect::Config::default(); config.tls.insecure = Some(true); - config.backoff.timeout = Some((Duration::ZERO).into()); + config.backoff.timeout = Some(crate::cli::Bound::NONE); let client = config.init(Default::default()).expect("build client"); for target in ["https://a!b:1/", "wss://a!b:1/", "moqt://a!b:1/", "tls://a!b:1/"] { let connection = client.connect(url(target)); @@ -2115,8 +2162,8 @@ mod tests { let mut config = crate::connect::Config::default(); config.backoff.initial = Some((FAST).into()); - // Zero retries forever, so only the first walk failing fast ends this loop. - config.backoff.timeout = Some((Duration::ZERO).into()); + // No give-up, so only the first walk failing fast ends this loop. + config.backoff.timeout = Some(crate::cli::Bound::NONE); // No timer may be pending while a dial is in flight, since the paused clock // jumps to it whenever the runtime idles. Linux and macOS refuse a loopback // connect inside the call, but Windows reports it later, after retrying the @@ -2184,7 +2231,7 @@ mod tests { let mut config = crate::connect::Config::default(); config.backoff.initial = Some((FAST).into()); config.backoff.max = Some((FAST).into()); - config.backoff.timeout = Some((Duration::ZERO).into()); + config.backoff.timeout = Some(crate::cli::Bound::NONE); let (mut connection, live, mut accepted, _task) = redialing(HOST, config).await; let monitor = connection.monitor(); @@ -2226,7 +2273,7 @@ mod tests { let mut config = crate::connect::Config::default(); config.backoff.initial = Some((FAST).into()); config.backoff.max = Some((FAST).into()); - config.backoff.timeout = Some((Duration::ZERO).into()); + config.backoff.timeout = Some(crate::cli::Bound::NONE); let (connection, _live, _accepted, _task) = redialing(HOST, config).await; let mut monitor = connection.monitor(); @@ -2707,8 +2754,7 @@ mod tests { crate::resolve::hosts::point(HOST, [addr_b]); let drained = kio::clock::now(); session_a - .drain() - .send(moq_net::goaway::Goaway::new().with_timeout(DEADLINE)) + .goaway(moq_net::goaway::Goaway::new().with_timeout(Some(DEADLINE))) .unwrap(); let _session_b = kio::clock::timeout(WAIT, accepted_b.recv()) @@ -2798,7 +2844,7 @@ mod tests { crate::resolve::hosts::point(HOST, [addr_b]); outlast_fast().await; - session_a.drain().send(moq_net::goaway::Goaway::new()).unwrap(); + session_a.goaway(moq_net::goaway::Goaway::new()).unwrap(); let session_b = kio::clock::timeout(WAIT, accepted_b.recv()).await.unwrap().unwrap(); let mut state = monitor.state(); while !matches!(state.phase, Phase::Connected { .. }) || state.connects < 2 { @@ -2856,10 +2902,11 @@ mod tests { assert_eq!(g1.start, 1, "the drained session stopped delivering"); } - /// Under [`Restart::Wait`] an empty-URI GOAWAY changes nothing while the peer keeps - /// the session: no migration and no redial. Its close then reconnects like any drop. + /// Under [`Restart::Handoff`] an empty-URI GOAWAY that drains a session before it is + /// healthy changes nothing while the peer keeps the session: no migration and no redial. + /// Its close then reconnects like any drop. #[tokio::test] - async fn a_waited_restart_redials_only_once_the_session_closes() { + async fn a_handoff_restart_of_a_young_session_redials_only_once_it_closes() { use futures::FutureExt; const WAIT: Duration = Duration::from_secs(10); @@ -2869,18 +2916,18 @@ mod tests { let subscriber = crate::origin::spawn(); let mut config = crate::connect::Config::default(); config.backoff.initial = Some((FAST).into()); - config.backoff.healthy = Some((FAST).into()); + config.backoff.healthy = Some(Duration::from_secs(3600).into()); let client = config .init(Default::default()) .unwrap() .with_subscriber(subscriber.clone()) - .with_restart(Restart::Wait); + .with_restart(Restart::Handoff); let mut connection = client.connect(url(&format!("tcp://{addr}/"))); let status = kio::clock::timeout(WAIT, connection.status()).await.unwrap(); assert_eq!(status.unwrap(), Status::Connected); let session = kio::clock::timeout(WAIT, accepted.recv()).await.unwrap().unwrap(); - session.drain().send(moq_net::goaway::Goaway::new()).unwrap(); + session.goaway(moq_net::goaway::Goaway::new()).unwrap(); // A broadcast announced after the GOAWAY still arrives: the drained session serves. let _late = origin.route("late", moq_net::origin::RouteOptions::default()).unwrap(); @@ -3236,7 +3283,7 @@ mod duration_tests { max, "--connect-backoff-max", |c: &Backoff| c.max(), - |d: Duration| Some(d).filter(|d| !d.is_zero()) + Some ); crate::cli::duration::test_config_duration!( timeout_set_in_code_beats_parser_default, @@ -3244,14 +3291,14 @@ mod duration_tests { timeout, "--connect-backoff-timeout", |c: &Backoff| c.timeout(), - |d: Duration| Some(d).filter(|d| !d.is_zero()) + Some ); crate::cli::duration::test_config_duration!( handover_set_in_code_beats_parser_default, - Goaway, + Handover, handover, "--connect-goaway-handover", - |c: &Goaway| c.resolve().handover, + |c: &Handover| c.resolve().handover, |d| d ); } diff --git a/rs/moq-tokio/tests/integration/backend.rs b/rs/moq-tokio/tests/integration/backend.rs index 03954f49bf4..f208de1828c 100644 --- a/rs/moq-tokio/tests/integration/backend.rs +++ b/rs/moq-tokio/tests/integration/backend.rs @@ -853,7 +853,7 @@ async fn quic_client_close_drains_migrated_predecessor() { .expect("the subscription never reached the client") .expect("client thread panicked"); - first.drain().send(moq_net::goaway::Goaway::new()).expect("send goaway"); + first.goaway(moq_net::goaway::Goaway::new()).expect("send goaway"); let request = kio::clock::timeout(TIMEOUT, server.accept()) .await .expect("the replacement never dialed") @@ -974,7 +974,7 @@ async fn quic_client_close_keeps_predecessor_handover() { .expect("the subscription never reached the client") .expect("client thread panicked"); - first.drain().send(moq_net::goaway::Goaway::new()).expect("send goaway"); + first.goaway(moq_net::goaway::Goaway::new()).expect("send goaway"); let (err, elapsed) = tokio::task::spawn_blocking(move || client.join()) .await diff --git a/rs/moq-tokio/tests/integration/broadcast.rs b/rs/moq-tokio/tests/integration/broadcast.rs index 688694ecdd6..0b61cf02d70 100644 --- a/rs/moq-tokio/tests/integration/broadcast.rs +++ b/rs/moq-tokio/tests/integration/broadcast.rs @@ -2630,10 +2630,7 @@ async fn one_shot_goaway_ends_the_connection() { let session = request.ok().await?; // No deadline, so nothing on the peer's side ever force-closes us. - session - .drain() - .send(moq_net::goaway::Goaway::default()) - .expect("send goaway"); + session.goaway(moq_net::goaway::Goaway::default()).expect("send goaway"); // Wait for the client to leave, exactly as a draining peer does. If the // client ignores the GOAWAY, both sides wait here forever. @@ -2747,7 +2744,7 @@ async fn a_dead_session_unannounces_while_the_reconnect_retries() { client_config.tls.insecure = Some(true); // Retry forever with a fast cadence: the worst case for a stale announce. client_config.backoff.initial = Some((Duration::from_millis(10)).into()); - client_config.backoff.timeout = Some((Duration::ZERO).into()); + client_config.backoff.timeout = Some(moq_tokio::cli::Bound::NONE); let client = client_config.init(Default::default()).expect("failed to init client"); let connection = kio::clock::timeout(TIMEOUT, client.with_subscriber(sub_origin).connect(url).established()) @@ -3218,8 +3215,10 @@ async fn goaway_test(scheme: &str, version: &str, expect_wire_timeout: bool) { start_drain_rx.await.expect("drain signal"); session - .drain() - .send(moq_net::goaway::Goaway::redirect("https://elsewhere.example/").with_timeout(Duration::from_secs(5))) + .goaway( + moq_net::goaway::Goaway::redirect("https://elsewhere.example/") + .with_timeout(Some(Duration::from_secs(5))), + ) .expect("send goaway"); // Keep producers alive while the client finishes reading. @@ -3344,8 +3343,7 @@ async fn goaway_timeout_force_close_moq_transport_22_quic() { let session = request.with_publisher(&pub_origin).ok().await?; session - .drain() - .send(moq_net::goaway::Goaway::new().with_timeout(Duration::from_millis(200))) + .goaway(moq_net::goaway::Goaway::new().with_timeout(Some(Duration::from_millis(200)))) .expect("send goaway"); // The client deliberately overstays; this resolves via the force-close. session.closed().await; diff --git a/rs/moq-tokio/tests/integration/fresh_port.rs b/rs/moq-tokio/tests/integration/fresh_port.rs index db2ebe35f7d..6c3e0e174f3 100644 --- a/rs/moq-tokio/tests/integration/fresh_port.rs +++ b/rs/moq-tokio/tests/integration/fresh_port.rs @@ -98,7 +98,7 @@ fn client(bind: SocketAddr) -> Client { config.tls.insecure = Some(true); config.timeout = Some(DIAL.into()); config.backoff.initial = Some(Duration::ZERO.into()); - config.backoff.timeout = Some(Duration::ZERO.into()); + config.backoff.timeout = Some(moq_tokio::cli::Bound::NONE); config.websocket.enabled = Some(false); config.init(Default::default()).unwrap() } diff --git a/rs/moq-tokio/tests/integration/migrate.rs b/rs/moq-tokio/tests/integration/migrate.rs index b2aff6d1d91..ac165cfffde 100644 --- a/rs/moq-tokio/tests/integration/migrate.rs +++ b/rs/moq-tokio/tests/integration/migrate.rs @@ -196,7 +196,7 @@ async fn a_migrated_session_closes_once_nothing_flows() { read(&mut upstream, GROUPS, &mut up).await; read(&mut downstream, GROUPS, &mut down).await; - old.drain().send(moq_net::goaway::Goaway::new()).expect("send goaway"); + old.goaway(moq_net::goaway::Goaway::new()).expect("send goaway"); // Held, since dropping the last handle closes the replacement. let _replacement = kio::clock::timeout(TIMEOUT, sessions.recv()) .await @@ -231,7 +231,7 @@ async fn a_predecessor_serves_until_its_last_subscription_leaves() { let mut down = Vec::new(); read(&mut downstream, GROUPS, &mut down).await; - old.drain().send(moq_net::goaway::Goaway::new()).expect("send goaway"); + old.goaway(moq_net::goaway::Goaway::new()).expect("send goaway"); // Held, since dropping the last handle closes the replacement. let _replacement = kio::clock::timeout(TIMEOUT, sessions.recv()) .await @@ -273,7 +273,7 @@ async fn a_replacement_does_not_check_what_its_predecessor_learned() { let mut down = Vec::new(); read(&mut downstream, GROUPS, &mut down).await; - old.drain().send(moq_net::goaway::Goaway::new()).expect("send goaway"); + old.goaway(moq_net::goaway::Goaway::new()).expect("send goaway"); let replacement = kio::clock::timeout(TIMEOUT, sessions.recv()) .await .expect("the client never redialed") diff --git a/rs/moq-tokio/tests/integration/reconnect.rs b/rs/moq-tokio/tests/integration/reconnect.rs index e78db06a0d0..e7d36b6f1d5 100644 --- a/rs/moq-tokio/tests/integration/reconnect.rs +++ b/rs/moq-tokio/tests/integration/reconnect.rs @@ -170,7 +170,7 @@ fn quick_client(redirect: moq_tokio::Redirect) -> moq_tokio::Client { let mut config = moq_tokio::connect::Config::default(); config.backoff.initial = Some((Duration::from_millis(20)).into()); config.backoff.max = Some((Duration::from_millis(40)).into()); - config.backoff.timeout = Some((Duration::ZERO).into()); + config.backoff.timeout = Some(moq_tokio::cli::Bound::NONE); config.goaway.redirect = redirect; config.init(Default::default()).expect("failed to init client") } @@ -216,8 +216,7 @@ async fn a_redirect_to_another_host_is_refused_by_default() { .expect("server A stopped accepting"); first - .drain() - .send(moq_net::goaway::Goaway::redirect(format!("tcp://127.0.0.1:{port_b}/"))) + .goaway(moq_net::goaway::Goaway::redirect(format!("tcp://127.0.0.1:{port_b}/"))) .expect("send goaway"); let err = kio::clock::timeout(Duration::from_secs(10), connection.closed()) @@ -250,8 +249,7 @@ async fn follow_still_honors_a_cross_host_redirect() { .expect("server A stopped accepting"); first - .drain() - .send(moq_net::goaway::Goaway::redirect(format!("tcp://127.0.0.1:{port_b}/"))) + .goaway(moq_net::goaway::Goaway::redirect(format!("tcp://127.0.0.1:{port_b}/"))) .expect("send goaway"); kio::clock::timeout(Duration::from_secs(10), sessions_b.recv()) @@ -278,8 +276,7 @@ async fn a_refused_redirect_skips_configured_fallbacks() { task_a.abort(); assert!(task_a.await.expect_err("listener was aborted").is_cancelled()); first - .drain() - .send(moq_net::goaway::Goaway::redirect("tcp://127.0.0.1:1/")) + .goaway(moq_net::goaway::Goaway::redirect("tcp://127.0.0.1:1/")) .expect("send goaway"); let err = kio::clock::timeout(Duration::from_secs(10), connection.closed()) @@ -310,7 +307,7 @@ async fn an_empty_goaway_preserves_configured_fallbacks() { // Stop accepting before GOAWAY so the migration must use the configured fallback. task_a.abort(); assert!(task_a.await.expect_err("listener was aborted").is_cancelled()); - first.drain().send(moq_net::goaway::Goaway::new()).expect("send goaway"); + first.goaway(moq_net::goaway::Goaway::new()).expect("send goaway"); kio::clock::timeout(Duration::from_secs(10), sessions_b.recv()) .await @@ -372,8 +369,7 @@ async fn rides_out_a_drain(backoff: moq_tokio::Backoff, drained: usize) { }; if held.len() < drained { session - .drain() - .send(moq_net::goaway::Goaway::new().with_timeout(Duration::from_secs(30))) + .goaway(moq_net::goaway::Goaway::new().with_timeout(Some(Duration::from_secs(30)))) .expect("send goaway"); held.push(session); } else { @@ -413,7 +409,7 @@ fn bounce( let uri = uri(n); tokio::spawn(async move { kio::clock::sleep(LIFE).await; - let _ = session.drain().send(moq_net::goaway::Goaway::redirect(uri)); + let _ = session.goaway(moq_net::goaway::Goaway::redirect(uri)); session.closed().await; }); } @@ -451,7 +447,7 @@ async fn ping_pong( /// A backoff starting at 50ms that doubles up to `max`, giving up after `timeout`, and that /// counts a session as healthy once it outlives the initial delay. -fn bounced_backoff(max: Duration, timeout: Duration) -> moq_tokio::Backoff { +fn bounced_backoff(max: Duration, timeout: Option) -> moq_tokio::Backoff { let mut backoff = moq_tokio::Backoff::default(); backoff.initial = Some((Duration::from_millis(50)).into()); backoff.healthy = Some((Duration::from_millis(50)).into()); @@ -488,7 +484,7 @@ async fn redial_waits( async fn two_peers_redirecting_to_each_other_escalate_the_backoff() { for rotate in [false, true] { let (url, mut accepts, _tasks) = ping_pong(rotate).await; - let backoff = bounced_backoff(Duration::from_millis(400), Duration::ZERO); + let backoff = bounced_backoff(Duration::from_millis(400), None); let _connection = client(backoff).connect(url); let delays = [0, 50, 100, 200, 400, 400].map(Duration::from_millis); @@ -507,7 +503,7 @@ async fn two_peers_redirecting_to_each_other_escalate_the_backoff() { #[tokio::test] async fn a_redirect_loop_stops_at_the_retry_window() { let (url, mut accepts, _tasks) = ping_pong(false).await; - let backoff = bounced_backoff(Duration::from_millis(400), Duration::from_millis(300)); + let backoff = bounced_backoff(Duration::from_millis(400), Some(Duration::from_millis(300))); let connection = client(backoff).connect(url); let closed = connection.closed(); tokio::pin!(closed); @@ -549,7 +545,7 @@ async fn migrates_without_backing_off( async fn a_loop_no_faster_than_the_capped_delay_migrates_without_backing_off() { let (url, mut accepts, _tasks) = ping_pong(false).await; // The cap is the initial delay, well under each session's life. - let backoff = bounced_backoff(Duration::from_millis(50), Duration::from_millis(60)); + let backoff = bounced_backoff(Duration::from_millis(50), Some(Duration::from_millis(60))); migrates_without_backing_off(client(backoff).connect(url), &mut accepts, 6).await; } @@ -569,7 +565,7 @@ async fn a_one_way_rolling_restart_migrates_without_backing_off() { _ => String::new(), }); - let backoff = bounced_backoff(Duration::from_millis(400), Duration::from_millis(60)); + let backoff = bounced_backoff(Duration::from_millis(400), Some(Duration::from_millis(60))); let url: url::Url = format!("tcp://127.0.0.1:{port_a}/").parse().expect("parse url"); migrates_without_backing_off(client(backoff).connect(url), &mut accepts, 6).await; } diff --git a/swift/Sources/Moq/Aliases.swift b/swift/Sources/Moq/Aliases.swift index 8b292e5c50e..fdb22dacf50 100644 --- a/swift/Sources/Moq/Aliases.swift +++ b/swift/Sources/Moq/Aliases.swift @@ -89,9 +89,11 @@ public typealias Compression = MoqFFI.MoqCompression public typealias ConnectionStats = MoqFFI.MoqConnectionStats /// Retry pacing for the automatic reconnect; see `Client(backoff:)`. A `nil` -/// field keeps its default. Until a session has stayed up past 1s or been drained -/// by a GOAWAY, a failed dial or a closed session, such as a refused credential, -/// ends the connection with its error rather than retrying. +/// field keeps its default, zero means zero (retry at once, or give up at the +/// first failure), and `maxUnlimited` lifts the ceiling. Until a session has +/// stayed up past 1s or been drained by a GOAWAY, a failed dial or a closed +/// session, such as a refused credential, ends the connection with its error +/// rather than retrying. public typealias Backoff = MoqFFI.MoqBackoff /// Certificate trust and the mTLS identity for a `Client`. diff --git a/test/interop/clients/rust/goaway.rs b/test/interop/clients/rust/goaway.rs index 9f86f478caa..884e62a2968 100644 --- a/test/interop/clients/rust/goaway.rs +++ b/test/interop/clients/rust/goaway.rs @@ -61,7 +61,7 @@ async fn main() -> anyhow::Result<()> { output.next_line().await?.as_deref() == Some("first group"), "JS did not read first group" ); - session.drain().send(moq_net::goaway::Goaway::redirect(&second_url))?; + session.goaway(moq_net::goaway::Goaway::redirect(&second_url))?; let replacement = second.accept().await.context("replacement accept")?.ok().await?; ensure!(replacement.version() == version, "wrong replacement negotiated version"); // First exercise the replacement while the drained route still serves. A JS