diff --git a/quest/m1/README.md b/quest/m1/README.md index e4586ae079e..aa8f227da8e 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -133,7 +133,6 @@ tracks, so per-group stats stop costing cluster announces. - [Rust resume checks compression](/quest/m1/resume-compress.md) - a same-epoch route copy with a different `compress` is refused instead of corrupting framing - [Rust resume backs single-route groups](/quest/m1/resume-handed-out.md) - a group first read from one route keeps the later copy that holds it, with a passthrough benchmark - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS -- [Interop finish](/quest/m1/interop-finish.md) - the interop matrix proves a clean broadcast finish across implementations - [Exporters follow a rendition to a new track](/quest/m1/export-rendition-rebind.md) - the fMP4, TS, MKV and FLV exporters rebuild a rendition whose resolved `track` or `broadcast` changes under the same ID - [HLS cursor failure](/quest/m1/hls-segments-failure.md) - the export segment cursor returns a failed timeline as an error after draining, so a truncated recording never looks complete - [Track live signal](/quest/m1/track-live.md) - a reader can wait for its track's feed to be live, so a player skips old cached groups while a fresh upstream subscription starts diff --git a/quest/m1/interop-finish.md b/quest/m1/interop-finish.md deleted file mode 100644 index d10b4c092b9..00000000000 --- a/quest/m1/interop-finish.md +++ /dev/null @@ -1,19 +0,0 @@ -# [S] Interop covers a clean broadcast finish - -## Goal - -The interop matrix has a media "finish" case: a publisher in each -implementation finishes its broadcast cleanly, and every subscriber drains -the tail and then sees the broadcast end, rather than an abort. A regression -in any implementation's linger shows up as a red cell. - -## Plan - -Rust, JS ([moq-dev/moq#6151](https://github.com/moq-dev/moq/pull/6151)), -and the bindings ([moq-dev/moq#6157](https://github.com/moq-dev/moq/pull/6157)) -each have a finish. Drive it from the existing -interop clients rather than adding a new harness, and assert both halves: -the late tail arrives and the end is clean. The new case's cells show up -in the nightly report at . - -Public API: none. Wire: none. diff --git a/quest/m1/release-audit/ffi-finished.md b/quest/m1/release-audit/ffi-finished.md index fd6549da9a0..b4e95449c47 100644 --- a/quest/m1/release-audit/ffi-finished.md +++ b/quest/m1/release-audit/ffi-finished.md @@ -31,7 +31,3 @@ Measure retained resources as retired-track count grows; avoid a brittle RSS threshold. No public API or wire change is expected. If FFI API shape changes, update every binding consumer and migration entry together. Current consumers are long-lived binding publishers that replace or retire audio tracks. - -## Related - -- [Interop finish](/quest/m1/interop-finish.md) - preserve delivery while releasing resources diff --git a/quest/m2/README.md b/quest/m2/README.md index 0be30afc246..f950870b5c2 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -215,6 +215,7 @@ ranked just below deterministic TS legs, which it requires. - [Relay metrics through prometheus-client](/quest/m2/relay-prometheus-client.md) - `/metrics` renders through the maintained client with the same series - [IETF request order](/quest/m2/ietf-request-order.md) - d14/15 requests take their ID after opening, legacy `control.sending()` races are gone, and SUBSCRIBE_NAMESPACE never closes over its own close - [JS IETF request streams](/quest/m2/js-request-stream.md) - `RequestStream` owns deadlines and cancellation, and TRACK trailing bytes close the session like Rust +- [moq-transport finish interop](/quest/m2/ietf-finish.md) - the interop `finish` case runs on every even moq-transport draft instead of being skipped - [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - a pre-draft-20 joining FETCH that arrives after its subscription finished is still answered, within a bound the peer already negotiates - [ashpd uses maintained D-Bus crates](/quest/m2/ashpd-maintained.md) - wait for a stable zbus 6 port, then remove the portal dependency exceptions - [Askama uses a maintained TOML parser](/quest/m2/askama-maintained.md) - wait for a stable parser migration, then remove the generator dependency exception diff --git a/quest/m2/ietf-finish.md b/quest/m2/ietf-finish.md new file mode 100644 index 00000000000..8a3eb7c7d3f --- /dev/null +++ b/quest/m2/ietf-finish.md @@ -0,0 +1,35 @@ +# [M] moq-transport finish interop + +## Goal + +The interop `finish` case passes on every even moq-transport draft, not only +moq-lite: each reader that joins a finished broadcast during its linger gets +the whole ten-frame tail, natural media and catalog EOF, the exact broadcast's +withdrawal, and the publisher's later health broadcast through the same +session. Until then `interop.sh` records those cells as skipped; delete that +skip in `run_finishes` and its README note. + +## Plan + +Observed on 2026-10-22 with `just test interop --finish --all --versions ietf` +(lite passes 45/45): + +- Drafts 14 to 19: late readers see the finished catalog end empty, a short + tail, or a refused joining FETCH (`publish done: internal error`). This is + the pre-draft-20 join after a finish, which + [join after a quick finish](/quest/m2/ietf-join-after-finish.md) fixes; rerun + these drafts once it lands. +- Drafts 20 and 22: several readers time out after the withdrawal (C and C++ + on the health broadcast), JS readers see `publish done: internal error`, and + native JS refuses a "malformed request stream message" from the relay. + Reproduce each between two clients before fixing; they may be separate bugs. + +A failing reader fails its whole publisher row, since the publisher waits for +every reader's acknowledgement, so judge each cell from its reader's log. + +Public API: none. Wire: none expected; any fix that changes framing updates +its draft. + +## Related + +- [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - the pre-draft-20 half of this diff --git a/rs/moq-cli/Cargo.toml b/rs/moq-cli/Cargo.toml index 354199d790d..19159747620 100644 --- a/rs/moq-cli/Cargo.toml +++ b/rs/moq-cli/Cargo.toml @@ -179,6 +179,10 @@ harness = false name = "interop-tail" path = "../../test/interop/clients/rust/tail.rs" +[[example]] +name = "interop-finish" +path = "../../test/interop/clients/rust/finish.rs" + [[example]] name = "interop-audio" path = "../../test/interop/clients/rust/audio.rs" diff --git a/test/interop/README.md b/test/interop/README.md index fa3e2104940..ef8fc451977 100644 --- a/test/interop/README.md +++ b/test/interop/README.md @@ -46,6 +46,27 @@ shutdown. No sleep stands in for drain completion. A missing group, error, or stall fails the lane; the timeout only bounds failure. QUIC on localhost rarely reorders, so the ordering race remains covered by transport unit tests. +The in-tree matrix also has a `finish` case. `just test interop --finish --all` +runs it alone. Readers establish their sessions first, then wait on a private +stdin barrier without subscribing to media. Each publisher imports ten nonempty +H.264 frames in one GOP (the browser encodes the same finite fixture), finishes +all media and its catalog, and obtains the normal broadcast finish handle. Only +then does the runner release readers. They require all ten frames, natural media +and catalog EOF, and the exact broadcast's announcement END. After the finish +handle's real linger expires, the publisher announces a separate untimed health +track. Reading its payload and clean EOF on the original session proves that +withdrawal was not a transport shutdown. The final FIFO acknowledgement permits +teardown only; it never ends the linger. Each publisher also measures monotonic +time from before creating its finish handle and refuses a zero linger or a wait +shorter than that handle promises, without a fixed duration or tolerance. + +GStreamer checks the late tail alone. `moqsrc` keeps watching a finished +broadcast, since an announcement means online/offline, so it has no end, +withdrawal, or session to report. It joins once the broadcast has finished and +requires exactly ten nonempty video buffers; its cell passes with the end check +marked N/A. A pinned moq-transport pass records the case as skipped until +[`quest/m2/ietf-finish.md`](../../quest/m2/ietf-finish.md) makes it pass. + Every client closes its session on the way out: publishers stop when their input ends (the browser on SIGTERM), and subscribers close once they have their frame. After each publisher's round, the harness waits for the relay to log the close of diff --git a/test/interop/clients/c/subscribe.c b/test/interop/clients/c/subscribe.c index 0b6bc57eea5..36660993927 100644 --- a/test/interop/clients/c/subscribe.c +++ b/test/interop/clients/c/subscribe.c @@ -6,6 +6,8 @@ // and a callback fires as each frame arrives. Once a non-empty video frame has // landed and --audio-ms of audio has been decoded to --audio-out (for tone.py to // judge) we close and drain every registration, then return 0; a timeout exits 1. +// --finish instead verifies the complete finite tail, clean media/catalog EOF, +// withdrawal, and a live session. // Publishing isn't wired up: the raw-stream importer that the other clients use // to publish isn't part of this subscribe-only client. // @@ -15,6 +17,8 @@ // --version offers only that version, through moq_client_config. #include +#include +#include #include #include #include @@ -24,10 +28,20 @@ #include #include +typedef struct { + int enabled, failed, announce_end, done_announce, done_health_wait, done_health; + unsigned expected, frames, health_frames; + int connected; + int32_t frame_end, catalog_end, health_end; + int32_t announcements, health_wait, health_broadcast; + const char *path; +} finish_t; + typedef struct { pthread_mutex_t mu; pthread_cond_t cv; int got; // a non-empty frame arrived + finish_t finish; int32_t origin; int32_t session; int32_t broadcast_wait; @@ -44,7 +58,7 @@ typedef struct { int audio_got; // One flag per registration handed &ctx, set by that registration's terminal - // (<= 0) callback. main returns only once all five are set: that terminal is + // (<= 0) callback. main returns only once every registration is done: that terminal is // moq-c's documented last touch of user_data, so it is what makes main's // frame safe to destroy. int done_status, done_broadcast, done_catalog, done_frame, done_audio; @@ -66,23 +80,40 @@ static void done(ctx_t *c, int *flag) { static void on_status(void *ud, int32_t code) { ctx_t *c = (ctx_t *)ud; fprintf(stderr, "session status: %d\n", code); + if (c->finish.enabled && code > 0) { + pthread_mutex_lock(&c->mu); + c->finish.connected = code; + if (code > 1) c->finish.failed = 1; + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); + } if (code <= 0) done(c, &c->done_status); } static void on_frame(void *ud, int32_t frame) { ctx_t *c = (ctx_t *)ud; if (frame <= 0) { // 0 = ended, negative = error + pthread_mutex_lock(&c->mu); + c->finish.frame_end = frame; + if (c->finish.enabled && frame < 0) { + fprintf(stderr, "finish media error: %d %s\n", frame, moq_error()); + c->finish.failed = 1; + } + pthread_mutex_unlock(&c->mu); done(c, &c->done_frame); return; } moq_frame f; memset(&f, 0, sizeof(f)); - if (moq_consume_frame((uint32_t)frame, &f) == 0 && f.payload_size > 0) { - pthread_mutex_lock(&c->mu); + int32_t read = moq_consume_frame((uint32_t)frame, &f); + pthread_mutex_lock(&c->mu); + if (read < 0 && c->finish.enabled) c->finish.failed = 1; + if (read == 0 && f.payload_size > 0) { c->got = 1; - pthread_cond_broadcast(&c->cv); - pthread_mutex_unlock(&c->mu); + if (c->finish.enabled && ++c->finish.frames > c->finish.expected) c->finish.failed = 1; } + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); moq_consume_frame_free((uint32_t)frame); } @@ -120,6 +151,12 @@ static void on_audio(void *ud, int32_t frame) { static void on_broadcast(void *ud, int32_t broadcast) { ctx_t *c = (ctx_t *)ud; if (broadcast <= 0) { // 0 = ended, negative = error + if (c->finish.enabled) { + if (broadcast < 0) fprintf(stderr, "finish broadcast error: %d %s\n", broadcast, moq_error()); + pthread_mutex_lock(&c->mu); + c->finish.failed = 1; + pthread_mutex_unlock(&c->mu); + } done(c, &c->done_broadcast); return; } @@ -130,19 +167,35 @@ static void on_broadcast(void *ud, int32_t broadcast) { pthread_cond_broadcast(&c->cv); } pthread_mutex_unlock(&c->mu); - if (!first) moq_consume_close((uint32_t)broadcast); + if (!first) { + if (c->finish.enabled) { + pthread_mutex_lock(&c->mu); + c->finish.failed = 1; // A replacement instance cannot finish this fixture. + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); + } + moq_consume_close((uint32_t)broadcast); + } } static void on_catalog(void *ud, int32_t catalog) { ctx_t *c = (ctx_t *)ud; if (catalog <= 0) { + pthread_mutex_lock(&c->mu); + c->finish.catalog_end = catalog; + if (c->finish.enabled && (catalog < 0 || c->video_track <= 0)) { + if (catalog < 0) fprintf(stderr, "finish catalog error: %d %s\n", catalog, moq_error()); + c->finish.failed = 1; + } + pthread_mutex_unlock(&c->mu); done(c, &c->done_catalog); return; } pthread_mutex_lock(&c->mu); int start = c->video_track <= 0; - int start_audio = c->audio_track <= 0; + // A finish reader checks the video tail alone. + int start_audio = !c->finish.enabled && c->audio_track <= 0; pthread_mutex_unlock(&c->mu); if (start_audio) { @@ -180,12 +233,95 @@ static void on_catalog(void *ud, int32_t catalog) { pthread_mutex_lock(&c->mu); c->video_track = track; pthread_mutex_unlock(&c->mu); + } else if (c->finish.enabled) { + fprintf(stderr, "finish video registration error: %d %s\n", track, moq_error()); + pthread_mutex_lock(&c->mu); + c->finish.failed = 1; + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); } } } moq_consume_catalog_free((uint32_t)catalog); } +// An END is a route withdrawal, not the terminal of the announcement listener. +static void on_finish_announce(void *ud, int32_t event) { + ctx_t *c = ud; + if (event <= 0) { + if (event < 0) { + fprintf(stderr, "finish announcement error: %d %s\n", event, moq_error()); + pthread_mutex_lock(&c->mu); + c->finish.failed = 1; + pthread_mutex_unlock(&c->mu); + } + done(c, &c->finish.done_announce); + return; + } + moq_announce_event info; + int32_t read = moq_origin_announced_info((uint32_t)event, &info); + pthread_mutex_lock(&c->mu); + if (read < 0) c->finish.failed = 1; + else if (info.prefix_len == strlen(c->finish.path) && + memcmp(info.prefix, c->finish.path, info.prefix_len) == 0 && + info.kind == MOQ_ANNOUNCE_KIND_END) c->finish.announce_end = 1; + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); + moq_origin_announced_free((uint32_t)event); +} + +static void on_finish_health_frame(void *ud, int32_t frame) { + ctx_t *c = ud; + if (frame <= 0) { + pthread_mutex_lock(&c->mu); + c->finish.health_end = frame; + if (frame < 0) { + fprintf(stderr, "finish health error: %d %s\n", frame, moq_error()); + c->finish.failed = 1; + } + pthread_mutex_unlock(&c->mu); + done(c, &c->finish.done_health); + return; + } + moq_frame info; + int32_t read = moq_consume_track_frame((uint32_t)frame, &info); + const char payload[] = "session-alive"; + pthread_mutex_lock(&c->mu); + if (read < 0 || info.payload_size != sizeof(payload) - 1 || + memcmp(info.payload, payload, sizeof(payload) - 1) != 0 || + info.timestamp_present || ++c->finish.health_frames != 1) c->finish.failed = 1; + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); + moq_consume_track_frame_free((uint32_t)frame); +} + +static void on_finish_health(void *ud, int32_t broadcast) { + ctx_t *c = ud; + if (broadcast <= 0) { + if (broadcast < 0) fprintf(stderr, "finish health broadcast error: %d %s\n", broadcast, moq_error()); + pthread_mutex_lock(&c->mu); + c->finish.failed = 1; + pthread_mutex_unlock(&c->mu); + done(c, &c->finish.done_health_wait); + return; + } + pthread_mutex_lock(&c->mu); + int first = c->finish.health_broadcast == 0; + if (first) c->finish.health_broadcast = broadcast; + else c->finish.failed = 1; + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); + if (!first) { + moq_consume_close((uint32_t)broadcast); + return; + } + int32_t track = moq_consume_track((uint32_t)broadcast, "live", 4, NULL, on_finish_health_frame, ud); + pthread_mutex_lock(&c->mu); + if (track <= 0) c->finish.failed = 1; + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); +} + // The context whose session a failure closes, once connected. static ctx_t *failing; @@ -218,6 +354,40 @@ static _Noreturn void fail(const char *fmt, ...) { _exit(1); } +// These assertions precede every local cancel, which would also produce terminal 0. +static void finish(ctx_t *c, const struct timespec *deadline) { + pthread_mutex_lock(&c->mu); + while (!c->finish.failed && !c->done_status && !c->finish.done_announce && + !(c->done_frame && c->done_catalog && c->finish.announce_end)) { + if (pthread_cond_timedwait(&c->cv, &c->mu, deadline) != 0) break; + } + int ok = !c->finish.failed && !c->done_status && !c->finish.done_announce && + c->done_frame && c->done_catalog && + c->finish.frame_end == 0 && c->finish.catalog_end == 0 && c->finish.announce_end && + c->finish.frames == c->finish.expected; + unsigned frames = c->finish.frames; + pthread_mutex_unlock(&c->mu); + if (!ok) fail("error: finish did not drain naturally (frames=%u expected=%u)\n", frames, c->finish.expected); + + size_t length = strlen(c->finish.path) + sizeof(".health"); + char *path = malloc(length); + if (!path) fail("error: cannot allocate health path\n"); + snprintf(path, length, "%s.health", c->finish.path); + c->finish.health_wait = moq_origin_announced_broadcast((uint32_t)c->origin, path, strlen(path), on_finish_health, c); + free(path); + if (c->finish.health_wait <= 0) fail("error: health registration failed\n"); + + pthread_mutex_lock(&c->mu); + while (!c->finish.failed && !c->done_status && !c->finish.done_health) { + if (pthread_cond_timedwait(&c->cv, &c->mu, deadline) != 0) break; + } + ok = !c->finish.failed && !c->done_status && c->finish.done_health && + c->finish.health_end == 0 && c->finish.health_frames == 1; + pthread_mutex_unlock(&c->mu); + if (!ok) fail("error: session health failed after broadcast finish\n"); + fprintf(stderr, "finish clean frames=%u catalog=eof announce=end health=ok\n", frames); +} + // Close every registration holding &ctx and wait for each one's terminal // callback. Only then may main return: main's frame is ctx's storage, and // exit()'s own call frames reuse those addresses, so a callback that arrives @@ -232,15 +402,21 @@ static void drain(ctx_t *c) { pthread_mutex_lock(&c->mu); int32_t track = c->video_track; int32_t audio = c->audio_track; + int cancel_frame = !c->finish.enabled || !c->done_frame; + int cancel_catalog = !c->finish.enabled || !c->done_catalog; // A track that was never started registered nothing, so it owes no terminal. if (track <= 0) c->done_frame = 1; if (audio <= 0) c->done_audio = 1; pthread_mutex_unlock(&c->mu); - if (track > 0) moq_consume_video_cancel((uint32_t)track); + if (track > 0 && cancel_frame) moq_consume_video_cancel((uint32_t)track); if (audio > 0) moq_decode_audio_cancel((uint32_t)audio); - moq_consume_catalog_cancel((uint32_t)c->catalog); + if (cancel_catalog) moq_consume_catalog_cancel((uint32_t)c->catalog); moq_origin_announced_broadcast_cancel((uint32_t)c->broadcast_wait); + if (c->finish.enabled) { + moq_origin_announced_cancel((uint32_t)c->finish.announcements); + moq_origin_announced_broadcast_cancel((uint32_t)c->finish.health_wait); + } moq_session_close((uint32_t)c->session); struct timespec deadline; @@ -248,10 +424,12 @@ static void drain(ctx_t *c) { deadline.tv_sec += 10; pthread_mutex_lock(&c->mu); - while (!(c->done_frame && c->done_audio && c->done_catalog && c->done_broadcast && c->done_status)) { + while (!(c->done_frame && c->done_audio && c->done_catalog && c->done_broadcast && c->done_status && + (!c->finish.enabled || (c->finish.done_announce && c->finish.done_health_wait && c->finish.done_health)))) { if (pthread_cond_timedwait(&c->cv, &c->mu, &deadline) != 0) break; // timed out } - int ok = c->done_frame && c->done_audio && c->done_catalog && c->done_broadcast && c->done_status; + int ok = c->done_frame && c->done_audio && c->done_catalog && c->done_broadcast && c->done_status && + (!c->finish.enabled || (c->finish.done_announce && c->finish.done_health_wait && c->finish.done_health)); int f = c->done_frame, a = c->done_audio, ct = c->done_catalog, b = c->done_broadcast, s = c->done_status; pthread_mutex_unlock(&c->mu); @@ -264,6 +442,8 @@ static void drain(ctx_t *c) { int main(int argc, char **argv) { const char *url = NULL, *broadcast = NULL, *version = NULL, *audio_out = NULL; double timeout_s = 20.0; + int finish_mode = 0; + unsigned frames = 0; long audio_ms = 0; for (int i = 1; i < argc; i++) { if (!strcmp(argv[i], "--url") && i + 1 < argc) url = argv[++i]; @@ -272,28 +452,41 @@ int main(int argc, char **argv) { else if (!strcmp(argv[i], "--version") && i + 1 < argc) version = argv[++i]; else if (!strcmp(argv[i], "--audio-out") && i + 1 < argc) audio_out = argv[++i]; else if (!strcmp(argv[i], "--audio-ms") && i + 1 < argc) audio_ms = atol(argv[++i]); + else if (!strcmp(argv[i], "--finish")) finish_mode = 1; + else if (!strcmp(argv[i], "--frames") && i + 1 < argc) { + char *end; + errno = 0; + unsigned long value = strtoul(argv[++i], &end, 10); + if (errno || !*argv[i] || argv[i][0] == '-' || *end || value == 0 || value > UINT_MAX) return 2; + frames = (unsigned)value; + } // a leading "subscribe" positional (and anything else) is ignored. } - if (!url || !broadcast || !audio_out || audio_ms <= 0) { - fprintf(stderr, "usage: c-interop subscribe --url U --broadcast B --audio-out PREFIX --audio-ms MS [--timeout S] [--version V]\n"); + if (!url || !broadcast || (finish_mode ? frames == 0 || audio_out != NULL : frames != 0 || !audio_out || audio_ms <= 0)) { + fprintf(stderr, "usage: c-interop subscribe --url U --broadcast B (--audio-out PREFIX --audio-ms MS | --finish --frames N) [--timeout S] [--version V]\n"); return 2; } ctx_t c; memset(&c, 0, sizeof(c)); + c.finish.enabled = finish_mode; + c.finish.expected = frames; + c.finish.path = broadcast; pthread_mutex_init(&c.mu, NULL); pthread_cond_init(&c.cv, NULL); - char path[4096]; - snprintf(path, sizeof(path), "%s.txt", audio_out); - c.audio_log = fopen(path, "w"); - snprintf(path, sizeof(path), "%s.pcm", audio_out); - c.audio_pcm = fopen(path, "wb"); - if (!c.audio_log || !c.audio_pcm) { - fprintf(stderr, "error: cannot write %s.{txt,pcm}\n", audio_out); - return 1; + if (!finish_mode) { + char path[4096]; + snprintf(path, sizeof(path), "%s.txt", audio_out); + c.audio_log = fopen(path, "w"); + snprintf(path, sizeof(path), "%s.pcm", audio_out); + c.audio_pcm = fopen(path, "wb"); + if (!c.audio_log || !c.audio_pcm) { + fprintf(stderr, "error: cannot write %s.{txt,pcm}\n", audio_out); + return 1; + } + c.audio_ms = (uint64_t)audio_ms; } - c.audio_ms = (uint64_t)audio_ms; c.origin = moq_origin_create(); if (c.origin <= 0) { @@ -326,6 +519,33 @@ int main(int argc, char **argv) { clock_gettime(CLOCK_REALTIME, &deadline); deadline.tv_sec += (time_t)timeout_s; + if (finish_mode) { + pthread_mutex_lock(&c.mu); + while (!c.finish.connected && !c.done_status && !c.finish.failed) { + if (pthread_cond_timedwait(&c.cv, &c.mu, &deadline) != 0) break; + } + int ready = c.finish.connected == 1 && !c.done_status && !c.finish.failed; + pthread_mutex_unlock(&c.mu); + if (!ready) fail("error: finish reader did not establish its first session\n"); + puts("finish reader ready"); + fflush(stdout); + char command[16]; + if (!fgets(command, sizeof(command), stdin) || strcmp(command, "read\n")) { + fail("error: finish reader expected read command\n"); + } + // Cold reader startup precedes publication; only the released read spends the linger. + clock_gettime(CLOCK_REALTIME, &deadline); + deadline.tv_sec += (time_t)timeout_s; + + moq_announce_config announce; + memset(&announce, 0, sizeof(announce)); + announce.prefix = broadcast; + announce.prefix_len = strlen(broadcast); + announce.filter = ""; // Exact path only, excluding the health sibling. + c.finish.announcements = moq_origin_announced((uint32_t)c.origin, &announce, on_finish_announce, &c); + if (c.finish.announcements <= 0) fail("error: finish announcement registration failed\n"); + } + // The broadcast arrives over the network after connect, so wait for it to be // announced. moq_origin_announced_broadcast resolves via on_broadcast once it's // available; we block on the condvar until then (or the deadline). @@ -350,27 +570,33 @@ int main(int argc, char **argv) { fail("error: moq_consume_catalog failed: %d\n", c.catalog); } - pthread_mutex_lock(&c.mu); - while (!c.got || !c.audio_got) { - if (pthread_cond_timedwait(&c.cv, &c.mu, &deadline) != 0) break; // timed out - } - int got = c.got, audio_got = c.audio_got; - pthread_mutex_unlock(&c.mu); + if (finish_mode) finish(&c, &deadline); + else { + pthread_mutex_lock(&c.mu); + while (!c.got || !c.audio_got) { + if (pthread_cond_timedwait(&c.cv, &c.mu, &deadline) != 0) break; // timed out + } + int got = c.got, audio_got = c.audio_got; + pthread_mutex_unlock(&c.mu); - if (!got || !audio_got) { - fail("error: timed out waiting for data (video=%d audio=%d)\n", got, audio_got); - } + if (!got || !audio_got) { + fail("error: timed out waiting for data (video=%d audio=%d)\n", got, audio_got); + } - fprintf(stderr, "received a frame and %llu audio samples from %s\n", (unsigned long long)c.audio_samples, - broadcast); + fprintf(stderr, "received a frame and %llu audio samples from %s\n", (unsigned long long)c.audio_samples, + broadcast); + } // The data path succeeded, which is all this interop client verifies. Returning // (rather than _exit) is the other half of what it verifies: an embedder that // closes and drains gets a clean process exit, with moq-c's runtime thread // still live behind its LazyLock. drain(&c); - fclose(c.audio_log); - fclose(c.audio_pcm); + if (finish_mode) moq_consume_close((uint32_t)c.finish.health_broadcast); + else { + fclose(c.audio_log); + fclose(c.audio_pcm); + } moq_consume_close((uint32_t)bc); moq_origin_close((uint32_t)c.origin); fflush(stderr); diff --git a/test/interop/clients/cpp/main.cpp b/test/interop/clients/cpp/main.cpp index 64831d6f13d..07ba24d24ce 100644 --- a/test/interop/clients/cpp/main.cpp +++ b/test/interop/clients/cpp/main.cpp @@ -25,6 +25,7 @@ #include #include #include +#include #include #include #include @@ -180,7 +181,10 @@ void publish_tone( } } -int publish(const std::string &url, const std::string &path, const Dial &dial, const std::vector &tones) { +int publish( + const std::string &url, const std::string &path, const Dial &dial, const std::vector &tones, + const std::string &finish_ack +) { // No deadline to publish: the harness stops the publisher when the round ends. const auto forever = Clock::now() + std::chrono::hours(24); auto session = connect(url, dial, forever); @@ -230,7 +234,34 @@ int publish(const std::string &url, const std::string &path, const Dial &dial, c tone.join(); ok(audio->finish(), "audio finish"); ok(media->finish(), "video finish"); - ok(broadcast->close(), "broadcast close"); + if (!finish_ack.empty()) { + const auto started = Clock::now(); + auto finish = ok(broadcast->finish(), "broadcast finish"); + const auto linger = std::chrono::microseconds(finish->linger_us()); + if (linger.count() <= 0) fail("finished media has no linger"); + std::printf("finish ready linger_us=%llu\n", static_cast(finish->linger_us())); + std::fflush(stdout); + ok(finish->wait().get(), "finish linger"); + if (Clock::now() - started < linger) fail("finish wait ended before its linger"); + std::puts("finish ended"); + std::fflush(stdout); + auto health = ok(session->publish()->create_broadcast(path + ".health"), "health broadcast"); + auto track = ok(health->publish_track("live", std::nullopt), "health track"); + const std::string payload = "session-alive"; + ok(track->write_frame({std::vector(payload.begin(), payload.end()), std::nullopt}), "health write"); + ok(track->finish(), "health finish"); + ok(health->announce({}), "health announce"); + // Only teardown waits on the FIFO; linger has already completed on its own clock. + std::ifstream ack(finish_ack); + std::string line; + if (!std::getline(ack, line) || line != "complete" || session->connects() != 1) { + fail("finish acknowledgement or session continuity failed"); + } + ok(health->close(), "health close"); + std::puts("finish acknowledged"); + } else { + ok(broadcast->close(), "broadcast close"); + } close_session(session); moq::shutdown(); return 0; @@ -269,22 +300,33 @@ void decode_audio( int subscribe( const std::string &url, const std::string &path, const Dial &dial, double timeout, - const std::string &audio_out, uint64_t audio_ms + const std::string &audio_out, uint64_t audio_ms, bool finish, size_t expected_frames ) { const auto deadline = Clock::now() + std::chrono::duration_cast(std::chrono::duration(timeout)); auto session = connect(url, dial, deadline); + std::shared_ptr routes; + if (finish) { + std::puts("finish reader ready"); + std::fflush(stdout); + std::string line; + if (!std::getline(std::cin, line) || line != "read") fail("finish read phase was not released"); + routes = ok(session->consume()->announced({}), "finish announcements"); + } + const auto read_deadline = finish + ? Clock::now() + std::chrono::duration_cast(std::chrono::duration(timeout)) + : deadline; auto announced = ok(session->consume()->announced_broadcast(path), "announced_broadcast"); - auto consumer = until(announced->available(), deadline, "available"); + auto consumer = until(announced->available(), read_deadline, "available"); // The catalog is a live track. A lazy publisher (e.g. the browser, which only encodes on // demand) may announce a rendition in a later update rather than the first snapshot, so - // wait for a catalog that has both video and audio. - auto catalogs = until(moq::MediaCatalogConsumer::subscribe(consumer), deadline, "catalog consumer"); + // wait for a catalog that has both video and audio. A finished fixture has video only. + auto catalogs = until(moq::MediaCatalogConsumer::subscribe(consumer), read_deadline, "catalog consumer"); std::optional catalog; - while (!catalog || catalog->video.empty() || catalog->audio.empty()) { - catalog = until(catalogs->next(), deadline, "catalog"); + while (!catalog || catalog->video.empty() || (!finish && catalog->audio.empty())) { + catalog = until(catalogs->next(), read_deadline, "catalog"); if (!catalog) { fail("catalog stream ended without a video and an audio track"); } @@ -292,29 +334,56 @@ int subscribe( const auto &[id, video] = *catalog->video.begin(); moq::Subscription subscription; - subscription.max_delay_us = MAX_DELAY_US; + subscription.max_delay_us = finish ? static_cast(timeout * 1e6) : MAX_DELAY_US; auto media = until( - moq::MediaTrackConsumer::subscribe(consumer, {moq::Rendition::kVideo{id, video}, subscription}), deadline, + moq::MediaTrackConsumer::subscribe(consumer, {moq::Rendition::kVideo{id, video}, subscription}), read_deadline, "container consumer" ); // Audio first: it has to arrive without a gap, and the video consumer keeps its first frame // (or skips to a newer one) while this reads. - const auto &[audio_id, audio] = *catalog->audio.begin(); - decode_audio(consumer, audio_id, audio, audio_out, audio_ms, deadline); + if (!finish) { + const auto &[audio_id, audio] = *catalog->audio.begin(); + decode_audio(consumer, audio_id, audio, audio_out, audio_ms, deadline); + } - size_t total = 0; - while (total == 0) { - auto frame = until(media->next(), deadline, "next frame"); + size_t total = 0, frames = 0; + while (finish || total == 0) { + auto frame = until(media->next(), read_deadline, "next frame"); if (!frame) { break; } total += frame->payload.size(); + if (!frame->payload.empty()) ++frames; } if (total == 0) { fail("no frame data received"); } std::printf("received %zu bytes from \"%s\"\n", total, path.c_str()); + if (finish) { + if (frames != expected_frames) fail("wrong finished media frame count"); + while (until(catalogs->next(), read_deadline, "catalog EOF")) {} + for (;;) { + auto event = until(routes->next(), read_deadline, "announcement END"); + if (!event) fail("origin ended before broadcast withdrawal"); + const auto *end = std::get_if(&event->get_variant()); + if (end && end->announce.prefix == path) break; + } + auto watch = ok(session->consume()->announced_broadcast(path + ".health"), "health watch"); + auto health = until(watch->available(), read_deadline, "health available"); + auto track = until(health->subscribe_track("live", std::nullopt), read_deadline, "health subscribe"); + auto group = until(track->next_group(), read_deadline, "health group"); + if (!group) fail("missing health group"); + auto frame = until(group->read_frame(), read_deadline, "health frame"); + if (!frame || std::string(frame->payload.begin(), frame->payload.end()) != "session-alive" || frame->timestamp_us) { + fail("health payload or timestamp"); + } + if (until(group->read_frame(), read_deadline, "health group EOF") || + until(track->next_group(), read_deadline, "health track EOF") || session->connects() != 1) { + fail("health EOF or session continuity"); + } + std::printf("finish clean frames=%zu catalog=eof announce=end health=ok\n", frames); + } close_session(session); moq::shutdown(); return 0; @@ -333,12 +402,19 @@ int main(int argc, char **argv) { std::string url; std::string broadcast; Dial dial; + bool finish = false; + std::string finish_ack; + size_t frames = 10; std::vector tones; std::string audio_out; uint64_t audio_ms = 0; double timeout = 20; for (int i = 2; i < argc; i++) { const std::string flag = argv[i]; + if (flag == "--finish") { + finish = true; + continue; + } if (flag == "--grant") { dial.grant = true; continue; @@ -353,6 +429,11 @@ int main(int argc, char **argv) { broadcast = value; } else if (flag == "--timeout") { timeout = std::atof(value.c_str()); + } else if (flag == "--finish-ack") { + finish_ack = value; + } else if (flag == "--frames") { + frames = std::strtoull(value.c_str(), nullptr, 10); + if (frames == 0) fail("frames must be positive"); } else if (flag == "--version") { dial.version = value; } else if (flag == "--tone") { @@ -378,13 +459,13 @@ int main(int argc, char **argv) { if (tones.empty()) { fail("publish needs --tone"); } - return publish(url, broadcast, dial, tones); + return publish(url, broadcast, dial, tones, finish_ack); } if (role == "subscribe") { - if (audio_out.empty() || audio_ms == 0) { + if (!finish && (audio_out.empty() || audio_ms == 0)) { fail("subscribe needs --audio-out and --audio-ms"); } - return subscribe(url, broadcast, dial, timeout, audio_out, audio_ms); + return subscribe(url, broadcast, dial, timeout, audio_out, audio_ms, finish, frames); } fail(("unknown role " + role).c_str()); } diff --git a/test/interop/clients/go/main.go b/test/interop/clients/go/main.go index b6ab1333db4..1b988744e3f 100644 --- a/test/interop/clients/go/main.go +++ b/test/interop/clients/go/main.go @@ -12,11 +12,15 @@ // ffmpeg ... -f h264 - | go-interop publish --url http://localhost:4443 --broadcast b.hang --tone 440,660 // go-interop subscribe --url http://localhost:4443 --broadcast b.hang --timeout 20 --audio-out /tmp/audio --audio-ms 600 // +// --finish verifies a finite tail, natural media/catalog EOF, announcement END, and +// same-session health; --frames sets its exact size. --finish-ack selects publisher +// finish mode and names the FIFO releasing teardown after the real linger. // --version offers only that version. --grant waits for the grant the relay sends // over AUTH and prints it as an `auth granted` line before the client starts. package main import ( + "bufio" "context" "encoding/binary" "encoding/json" @@ -126,7 +130,7 @@ func patterns(list []string) string { return string(out) } -func publish(ctx context.Context, url, broadcast string, opts dialOptions, timeout time.Duration, tones []float64) error { +func publish(ctx context.Context, url, broadcast string, opts dialOptions, timeout time.Duration, tones []float64, finishAck string) error { client, err := dial(ctx, url, opts, timeout) if err != nil { return err @@ -201,7 +205,65 @@ func publish(ctx context.Context, url, broadcast string, opts dialOptions, timeo if err := audio.Finish(); err != nil { return err } - return media.Finish() + if err := media.Finish(); err != nil { + return err + } + if finishAck == "" { + return nil + } + // Finish also finalizes the catalog. ACK never controls the real linger. + started := time.Now() + finished, err := producer.Finish() + if err != nil { + return err + } + defer finished.Close() + linger := finished.Linger() + if linger <= 0 { + return errors.New("finished media has no linger") + } + fmt.Printf("finish ready linger_us=%d\n", linger.Microseconds()) + if err := finished.Wait(ctx); err != nil { + return err + } + if time.Since(started) < linger { + return errors.New("finish wait ended before its linger") + } + health, err := client.CreateBroadcast(broadcast + ".health") + if err != nil { + return err + } + defer health.Close() + live, err := health.PublishTrack("live", nil) + if err != nil { + return err + } + if err := live.WriteFrame(moq.Frame{Payload: []byte("session-alive")}); err != nil { + return err + } + if err := live.Finish(); err != nil { + return err + } + if err := health.Announce(moq.Route{}); err != nil { + return err + } + ack, err := os.Open(finishAck) + if err != nil { + return err + } + defer ack.Close() + command, err := bufio.NewReader(ack).ReadString('\n') + if err != nil { + return err + } + if command != "complete\n" { + return errors.New("expected finish completion acknowledgement") + } + if client.Session().Connects() != 1 { + return errors.New("publisher reconnected during finish") + } + fmt.Println("finish acknowledged") + return nil } // The catalog is a live track. A lazy publisher (e.g. the browser, which only @@ -228,6 +290,153 @@ func catalogWithMedia(ctx context.Context, consumer *moq.BroadcastConsumer) (*mo } } +// finishSubscribe retains each reader until its natural EOF, then proves the session still serves. +func finishSubscribe(ctx context.Context, client *moq.Client, broadcast string, frames int) error { + announcements, err := client.Announced(moq.AnnounceOptions{Prefix: broadcast}) + if err != nil { + return err + } + defer announcements.Cancel() + announced, err := client.AnnouncedBroadcast(broadcast) + if err != nil { + return err + } + defer announced.Cancel() + consumer, err := announced.Available(ctx) + if err != nil { + return err + } + catalogs, err := moqmedia.NewCatalogConsumer(ctx, consumer) + if err != nil { + return err + } + defer catalogs.Cancel() + var catalog *moqmedia.Catalog + for { + catalog, err = catalogs.Next(ctx) + if err != nil { + return err + } + if catalog == nil { + return errors.New("catalog ended without video") + } + if len(catalog.Video) != 0 { + break + } + } + var id string + var video moqmedia.Video + for id, video = range catalog.Video { + break + } + media, err := moqmedia.NewTrackConsumer(ctx, consumer, moqmedia.TrackConfig{Rendition: moqmedia.VideoRendition(id, video), Subscription: &moq.Subscription{MaxDelay: maxDelay}}) + if err != nil { + return err + } + defer media.Cancel() + count := 0 + for { + frame, err := media.Next(ctx) + if err != nil { + return err + } + if frame == nil { + break + } + // Empty payloads carry endpoint metadata, not codec frames. + if len(frame.Payload) == 0 { + continue + } + count++ + if count > frames { + return fmt.Errorf("expected %d frames, received more", frames) + } + } + if count != frames { + return fmt.Errorf("expected %d frames, received %d", frames, count) + } + for { + catalog, err := catalogs.Next(ctx) + if err != nil { + return err + } + if catalog == nil { + break + } + } + started := false + for { + event, err := announcements.Next(ctx) + if err != nil { + return err + } + if event == nil { + return errors.New("announcement stream ended before broadcast END") + } + if start, ok := event.(moq.AnnounceEventStart); ok && start.Announce.Prefix == broadcast { + started = true + } + if restart, ok := event.(moq.AnnounceEventRestart); ok && restart.Announce.Prefix == broadcast { + return errors.New("broadcast restarted during finish") + } + if end, ok := event.(moq.AnnounceEventEnd); ok && end.Announce.Prefix == broadcast { + if !started { + return errors.New("broadcast ended without announcement") + } + break + } + } + healthAnnounced, err := client.AnnouncedBroadcast(broadcast + ".health") + if err != nil { + return err + } + defer healthAnnounced.Cancel() + health, err := healthAnnounced.Available(ctx) + if err != nil { + return err + } + live, err := health.SubscribeTrack(ctx, "live", nil) + if err != nil { + return err + } + defer live.Cancel() + group, err := live.NextGroup(ctx) + if err != nil { + return err + } + if group == nil { + return errors.New("missing session health group") + } + defer group.Cancel() + frame, err := group.ReadFrame(ctx) + if err != nil { + return err + } + if frame == nil || string(frame.Payload) != "session-alive" || frame.Timestamp != nil { + return errors.New("unexpected session health payload or timestamp") + } + frame, err = group.ReadFrame(ctx) + if err != nil { + return err + } + if frame != nil { + return errors.New("session health group did not end after one frame") + } + next, err := live.NextGroup(ctx) + if err != nil { + return err + } + if next != nil { + next.Cancel() + return errors.New("session health track did not end after one group") + } + if client.Session().Connects() != 1 { + return errors.New("subscriber reconnected during finish") + } + fmt.Printf("finish clean frames=%d catalog=eof announce=end health=ok\n", frames) + return nil +} + // decodeAudio decodes the first audioMs of the rendition to out.{txt,pcm}, the layout tone.py reads. func decodeAudio(ctx context.Context, consumer *moq.BroadcastConsumer, id string, rendition moqmedia.Audio, out string, audioMs int) error { decoder, err := moqaudio.Subscribe(ctx, consumer, id, rendition, moqaudio.DecoderOutput{ @@ -271,15 +480,30 @@ func decodeAudio(ctx context.Context, consumer *moq.BroadcastConsumer, id string return nil } -func subscribe(ctx context.Context, url, broadcast string, opts dialOptions, timeout time.Duration, audioOut string, audioMs int) error { +func subscribe(ctx context.Context, url, broadcast string, opts dialOptions, timeout time.Duration, audioOut string, audioMs int, finish bool, frames int) error { + parent := ctx ctx, cancel := context.WithTimeout(ctx, timeout) defer cancel() - client, err := dial(ctx, url, opts, timeout) if err != nil { return err } defer client.Close() + if finish { + fmt.Println("finish reader ready") + command, err := bufio.NewReader(os.Stdin).ReadString('\n') + if err != nil { + return err + } + if command != "read\n" { + return errors.New("expected finish read command") + } + } + if finish { + readCtx, cancel := context.WithTimeout(parent, timeout) + defer cancel() + return finishSubscribe(readCtx, client, broadcast, frames) + } announced, err := client.AnnouncedBroadcast(broadcast) if err != nil { @@ -350,6 +574,9 @@ func run() error { timeout := flags.Float64("timeout", 20, "seconds to wait for data") version := flags.String("version", "", "offer only this version") grant := flags.Bool("grant", false, "wait for the grant the relay sends over AUTH and print it") + finishAck := flags.String("finish-ack", "", "publisher completion FIFO; enables clean finish") + finish := flags.Bool("finish", false, "verify a late tail and clean broadcast end") + frames := flags.Int("frames", 10, "expected finish video frame count") tone := flags.String("tone", "", "publish: comma-separated frequency of each channel, in Hz") audioOut := flags.String("audio-out", "", "subscribe: decode audio to this prefix's .txt and .pcm") audioMs := flags.Int("audio-ms", 0, "subscribe: how much audio to decode, in milliseconds") @@ -360,6 +587,10 @@ func run() error { return errors.New("--url and --broadcast are required") } + if *frames <= 0 || (*finish && role != "subscribe") || (*finishAck != "" && role != "publish") { + return errors.New("finish requires a subscriber, finish-ack a publisher, and frames must be positive") + } + // Warnings and errors from the Rust layer, such as a publication outside the grant. if err := moq.LogLevel("warn"); err != nil { return err @@ -377,12 +608,12 @@ func run() error { } tones = append(tones, hz) } - return publish(ctx, *url, *broadcast, opts, wait, tones) + return publish(ctx, *url, *broadcast, opts, wait, tones, *finishAck) case "subscribe": - if *audioOut == "" || *audioMs <= 0 { + if !*finish && (*audioOut == "" || *audioMs <= 0) { return errors.New("subscribe needs --audio-out and --audio-ms") } - return subscribe(ctx, *url, *broadcast, opts, wait, *audioOut, *audioMs) + return subscribe(ctx, *url, *broadcast, opts, wait, *audioOut, *audioMs, *finish, *frames) default: return fmt.Errorf("unknown role %q", role) } diff --git a/test/interop/clients/js-native/finish.ts b/test/interop/clients/js-native/finish.ts new file mode 100644 index 00000000000..9410e834946 --- /dev/null +++ b/test/interop/clients/js-native/finish.ts @@ -0,0 +1,96 @@ +/** Shared finish assertions for native JS and the existing browser client. @module */ +import type { Milli } from "@moq/clock"; +import * as Catalog from "@moq/hang/catalog"; +import * as Container from "@moq/hang/container"; +import * as Moq from "@moq/net"; + +/** Drain a finished media tail, then prove the same one-shot session can still receive. */ +export async function readFinish( + origin: Moq.Origin.Producer, + path: string, + frames: number, + maxDelay: Milli, +): Promise { + const source = origin.consume(); + const routes = source.announced(); + const request = origin.request(Moq.Path.from(path), { announced: true }); + const healthRequest = origin.request(Moq.Path.from(`${path}.health`), { announced: true }); + try { + let broadcast = request.active.peek(); + while (!broadcast) { + await request.active.changed(); + broadcast = request.active.peek(); + } + const catalogs = Catalog.watch(broadcast)[Symbol.asyncIterator](); + try { + let catalog = await catalogs.next(); + while (!catalog.done && !Object.keys(catalog.value.video?.renditions ?? {}).length) + catalog = await catalogs.next(); + if (catalog.done) throw new Error("finished catalog had no video"); + const [name, config] = Object.entries(catalog.value.video?.renditions ?? {})[0]; + const format = + config.container.kind === "legacy" + ? new Container.Legacy.Format(config) + : config.container.kind === "loc" + ? new Container.Loc.Format("video") + : undefined; + if (!format) throw new Error("unsupported finish media container"); + const subscription = broadcast + .track(Catalog.track(name, config)) + .subscribe({ maxDelay, groups: { start: { included: 0 } } }); + const media = new Container.Consumer(subscription, { format, maxDelay }); + let count = 0; + try { + for (;;) { + const item = await media.next(); + if (!item) break; + if (item.kind === "gap") throw new Error("gap in finished tail"); + if (item.kind === "frame" && item.frame.payload.byteLength) count++; + } + if ((await subscription.closed) !== null) throw new Error("media did not end cleanly"); + if (count !== frames) throw new Error(`wrong finished tail: ${count} frames, expected ${frames}`); + } finally { + media.close(); + } + while (!(await catalogs.next()).done) {} + for (;;) { + const event = await routes.next(); + if (!event) throw new Error("origin ended before broadcast withdrawal"); + if (event.kind === "end" && event.prefix === path) break; + } + let health = healthRequest.active.peek(); + while (!health) { + await healthRequest.active.changed(); + health = healthRequest.active.peek(); + } + const track = health.track("live").subscribe(); + try { + const group = await track.recvGroup(); + if (!group) throw new Error("no health group"); + try { + const frame = await group.readFrame(); + if ( + !frame || + new TextDecoder().decode(frame.payload) !== "session-alive" || + frame.timestamp !== undefined + ) + throw new Error("health frame"); + if ((await group.readFrame()) !== undefined || (await track.recvGroup()) !== undefined) + throw new Error("health EOF"); + } finally { + group.close(); + } + if ((await track.closed) !== null) throw new Error("health did not end cleanly"); + } finally { + track.close(); + } + return `finish clean frames=${count} catalog=eof announce=end health=ok`; + } finally { + await catalogs.return?.(); + } + } finally { + routes.close(); + request.close(); + healthRequest.close(); + } +} diff --git a/test/interop/clients/js-native/subscribe.ts b/test/interop/clients/js-native/subscribe.ts index 2fac8b1f230..74248b158ee 100644 --- a/test/interop/clients/js-native/subscribe.ts +++ b/test/interop/clients/js-native/subscribe.ts @@ -19,7 +19,9 @@ * * @module */ + import { writeFileSync } from "node:fs"; +import { createInterface } from "node:readline/promises"; import { parseArgs } from "node:util"; import { Milli } from "@moq/clock"; import * as Catalog from "@moq/hang/catalog"; @@ -29,6 +31,7 @@ import { Hex, Libav } from "@moq/hang/util"; import * as Json from "@moq/json"; import * as Moq from "@moq/net"; import { install } from "@moq/web-transport"; +import { readFinish } from "./finish"; // globalThis.WebTransport = the polyfill (no-op if a native one already exists). // @moq/net's connect() reads globalThis.WebTransport at call time, so this just @@ -53,6 +56,8 @@ const { positionals, values } = parseArgs({ broadcast: { type: "string" }, timeout: { type: "string", default: "20" }, version: { type: "string" }, + finish: { type: "boolean", default: false }, + frames: { type: "string", default: "10" }, "audio-out": { type: "string" }, "audio-ms": { type: "string" }, }, @@ -61,6 +66,8 @@ const { positionals, values } = parseArgs({ const role = positionals[0]; const url = values.url; const broadcast = values.broadcast; +const frames = Number(values.frames); +if (!Number.isSafeInteger(frames) || frames <= 0) throw new Error("frames must be a positive integer"); const audioOut = values["audio-out"]; const audioMs = Number.parseInt(values["audio-ms"] ?? "", 10); const timeoutMs = Number.parseFloat(values.timeout ?? "20") * 1000; @@ -68,13 +75,12 @@ if ( role !== "subscribe" || !url || !broadcast || - !audioOut || - !(audioMs > 0) || + (!values.finish && (!audioOut || !(audioMs > 0))) || !Number.isFinite(timeoutMs) || timeoutMs <= 0 ) { console.error( - "usage: subscribe.ts subscribe --url U --broadcast B --audio-out PREFIX --audio-ms MS [--timeout S>0] [--version V]", + "usage: subscribe.ts subscribe --url U --broadcast B (--audio-out PREFIX --audio-ms MS | --finish [--frames N]) [--timeout S>0] [--version V]", ); process.exit(2); } @@ -230,6 +236,17 @@ async function run(): Promise { const unwatch = connection.auth.grant.subscribe(printGrant); let requested: Moq.Origin.Requesting | undefined; try { + if (values.finish) { + console.error("finish reader ready"); + const input = createInterface({ input: process.stdin }); + try { + if ((await input.question("")) !== "read") throw new Error("expected finish read release"); + } finally { + input.close(); + } + console.error(await readFinish(origin, broadcast as string, frames, Milli(timeoutMs))); + return; + } const path = Moq.Path.from(broadcast as string); requested = origin.request(path, { announced: true }); let bc = requested.active.peek(); @@ -257,7 +274,7 @@ async function run(): Promise { } const timeout = new Promise((_, reject) => - setTimeout(() => reject(new Error("timed out waiting for data")), timeoutMs), + setTimeout(() => reject(new Error("timed out waiting for data")), timeoutMs * (values.finish ? 2 : 1)), ); try { diff --git a/test/interop/clients/js/driver.ts b/test/interop/clients/js/driver.ts index 88072dfdb79..53cadfc7324 100644 --- a/test/interop/clients/js/driver.ts +++ b/test/interop/clients/js/driver.ts @@ -15,6 +15,8 @@ * * @module */ +import { createReadStream } from "node:fs"; +import { createInterface } from "node:readline/promises"; import { parseArgs } from "node:util"; import type { Page } from "playwright"; import { @@ -59,6 +61,9 @@ const { positionals, values } = parseArgs({ url: { type: "string" }, broadcast: { type: "string" }, version: { type: "string" }, + finish: { type: "boolean", default: false }, + "finish-ack": { type: "string" }, + frames: { type: "string", default: "10" }, timeout: { type: "string", default: "20" }, "audio-file": { type: "string" }, tone: { type: "string" }, @@ -69,6 +74,7 @@ const role = positionals[0]; const url = values.url; const broadcast = values.broadcast; const timeoutMs = Number.parseFloat(values.timeout ?? "20") * 1000; +const finishing = values.finish || values["finish-ack"] !== undefined; const audioFile = values["audio-file"]; const tones = (values.tone ?? "").split(",").filter(Boolean).map(Number); if ( @@ -77,11 +83,11 @@ if ( !broadcast || !Number.isFinite(timeoutMs) || timeoutMs <= 0 || - (role === "publish" && !audioFile) || - (role === "subscribe" && (!tones.length || tones.some((hz) => !(hz > 0)))) + (!finishing && role === "publish" && !audioFile) || + (!finishing && role === "subscribe" && (!tones.length || tones.some((hz) => !(hz > 0)))) ) { console.error( - "usage: driver.ts publish --url U --broadcast B --audio-file WAV | subscribe --url U --broadcast B --tone HZ,... [--version V] [--timeout S>0]", + "usage: driver.ts publish --url U --broadcast B (--audio-file WAV | --finish-ack FIFO) | subscribe --url U --broadcast B (--tone HZ,... | --finish [--frames N]) [--version V] [--timeout S>0]", ); process.exit(2); } @@ -133,18 +139,61 @@ let code = 1; try { const [page, errors] = await open( browser, - pageUrl(server.origin, role, { + pageUrl(server.origin, finishing ? `finish-${role}` : role, { url, broadcast, + ...(finishing ? { frames: values.frames ?? "10", timeout: String(timeoutMs) } : {}), ...(values.version ? { versions: values.version } : {}), - ...(role === "subscribe" ? { tone: tones.join(",") } : {}), + ...(!finishing && role === "subscribe" ? { tone: tones.join(",") } : {}), }), role, role === "subscribe", ); - if (role === "subscribe") await waitForWatch(page); + if (!finishing && role === "subscribe") await waitForWatch(page); - if (role === "publish") { + if (finishing) { + const wait = async (field: "ready" | "ended" | "result") => { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + throwPageErrors(errors); + const state = await page.evaluate(() => { + const s = window.interopFinish; + return ( + s && { ready: s.ready, ended: s.ended, result: s.result, error: s.error, lingerUs: s.lingerUs } + ); + }); + if (state?.error) throw new Error(state.error); + if (state?.[field]) return state; + await sleep(POLL_INTERVAL_MS); + } + throw new Error(`finish ${field} timed out`); + }; + const ready = await wait("ready"); + if (role === "subscribe") { + console.error("finish reader ready"); + const input = createInterface({ input: process.stdin }); + try { + if ((await input.question("")) !== "read") throw new Error("expected finish read release"); + } finally { + input.close(); + } + await page.evaluate(() => window.interopFinish?.read()); + } else { + console.error(`finish ready linger_us=${ready.lingerUs}`); + await wait("ended"); + console.error("finish ended"); + const input = createInterface({ input: createReadStream(values["finish-ack"] as string) }); + try { + if ((await input[Symbol.asyncIterator]().next()).value !== "complete") + throw new Error("expected finish acknowledgement"); + } finally { + input.close(); + } + await page.evaluate(() => window.interopFinish?.acknowledge()); + } + console.error((await wait("result")).result); + code = 0; + } else if (role === "publish") { console.error(`publishing ${broadcast} (fake camera + ${audioFile} as the microphone) to ${url}`); // Stream until the orchestrator stops us. await new Promise((resolve) => process.once("SIGTERM", resolve)); @@ -245,6 +294,7 @@ try { // Close every session before the browser: closing it, or even the page, can end them without // a close, leaving the relay to time them out. for (const page of browser.contexts().flatMap((context) => context.pages())) { + await page.evaluate(() => window.interopFinish?.close()).catch(() => {}); await closeSessions(page); await page.close().catch(() => {}); } diff --git a/test/interop/clients/js/src/finish.ts b/test/interop/clients/js/src/finish.ts new file mode 100644 index 00000000000..b80fa18e03a --- /dev/null +++ b/test/interop/clients/js/src/finish.ts @@ -0,0 +1,155 @@ +// Finite media fixture and finish observations for the existing browser client. +import { Instant, Micro, Milli, Nano, now } from "@moq/clock"; +import * as Catalog from "@moq/hang/catalog"; +import * as Container from "@moq/hang/container"; +import * as Moq from "@moq/net"; +import { CatalogProducer } from "@moq/publish"; +import { Effect } from "@moq/signals"; +import { readFinish } from "../../js-native/finish"; + +interface State { + ready: boolean; + ended: boolean; + lingerUs?: number; + result?: string; + error?: string; + read(): void; + acknowledge(): void; + close(): void; +} + +declare global { + interface Window { + interopFinish?: State; + } +} + +export function finish( + role: string, + url: string, + path: string, + version: string | null, + frames: number, + timeout: number, +): void { + let releaseRead = () => {}; + let releaseAck = () => {}; + const read = new Promise((resolve) => { + releaseRead = resolve; + }); + const ack = new Promise((resolve) => { + releaseAck = resolve; + }); + const origin = new Moq.Origin.Producer(); + let connection: Moq.Connection.Established | undefined; + const state: State = { + ready: false, + ended: false, + read: () => releaseRead(), + acknowledge: () => releaseAck(), + close: () => { + connection?.close(); + origin.close(); + }, + }; + window.interopFinish = state; + void (async () => { + connection = await Moq.Connection.connect({ + url: new URL(url), + consume: origin, + publish: origin.consume(), + websocket: { enabled: false }, + ...(version ? { versions: [version] } : {}), + }); + if (role === "finish-subscribe") { + state.ready = true; + await read; + state.result = await readFinish(origin, path, frames, Milli(timeout)); + return; + } + const broadcast = origin.createBroadcast(Moq.Path.from(path)); + const config: Catalog.VideoConfig = { + codec: "avc1.42001e", + codedWidth: Catalog.u53(320), + codedHeight: Catalog.u53(240), + container: { kind: "legacy" }, + }; + const track = broadcast.createTrack("video", { + timescale: Moq.Time.Timescale.MICRO, + maxAge: Milli(Number.MAX_SAFE_INTEGER), + priority: Catalog.PRIORITY.video, + }); + const media = new Container.Legacy.Producer(track, new Container.Legacy.Format(config)); + let encodingError: Error | undefined; + const encoder = new VideoEncoder({ + output: (chunk) => media.encode(chunk, Micro(chunk.timestamp), chunk.type === "key"), + error: (error) => { + encodingError = error; + }, + }); + try { + encoder.configure({ + codec: config.codec, + width: 320, + height: 240, + bitrate: 300_000, + framerate: 30, + avc: { format: "annexb" }, + }); + const canvas = new OffscreenCanvas(320, 240); + const context = canvas.getContext("2d"); + if (!context) throw new Error("no fixture canvas"); + for (let index = 0; index < frames; index++) { + context.fillStyle = `rgb(${index * 20},80,160)`; + context.fillRect(0, 0, 320, 240); + const frame = new VideoFrame(canvas, { timestamp: Math.round((index * 1_000_000) / 30) }); + try { + encoder.encode(frame, { keyFrame: index === 0 }); + } finally { + frame.close(); + } + } + await encoder.flush(); + if (encodingError) throw encodingError; + } finally { + encoder.close(); + } + media.close(); + const catalog = new CatalogProducer(); + catalog.mutate((root) => { + root.video = { renditions: { video: config } }; + }); + const serving = new Effect(); + catalog.serve( + broadcast.createTrack(Catalog.TRACK, { + timescale: Moq.Time.Timescale.MILLI, + maxAge: Milli(Number.MAX_SAFE_INTEGER), + priority: Catalog.PRIORITY.catalog, + }), + serving, + ); + serving.close(); + broadcast.announce(); + const started = now(); + const finishing = broadcast.finish(); + const linger = Nano.fromMilli(finishing.linger); + if (linger <= 0) throw new Error("finished media has no linger"); + state.lingerUs = finishing.linger * 1000; + state.ready = true; + await finishing.wait(); + if (Instant.durationSince(now(), started) < linger) throw new Error("finish wait ended before its linger"); + const health = origin.createBroadcast(Moq.Path.from(`${path}.health`)); + const healthTrack = health.createTrack("live"); + const group = healthTrack.appendGroup(); + group.writeFrame({ payload: new TextEncoder().encode("session-alive") }); + group.close(); + healthTrack.close(); + health.announce(); + state.ended = true; + await ack; + health.close(); + state.result = "finish acknowledged"; + })().catch((error) => { + state.error = error instanceof Error ? error.message : String(error); + }); +} diff --git a/test/interop/clients/js/src/setup.ts b/test/interop/clients/js/src/setup.ts index 7976c6c7fa9..bcb20dcc7ab 100644 --- a/test/interop/clients/js/src/setup.ts +++ b/test/interop/clients/js/src/setup.ts @@ -6,6 +6,7 @@ import type MoqPublish from "@moq/publish/element"; import type MoqWatch from "@moq/watch/element"; import { refused, refusedWebSocket } from "./close"; import { type CaptureState, FAULTS, type Fault, publish, SAMPLE_MS } from "./contract"; +import { finish } from "./finish"; import { Fixture } from "./fixture"; import { attach, watchResources } from "./probe"; @@ -23,7 +24,9 @@ function parseFault(value: string | null): Fault { return fault; } -if (role === "publish") { +if (role === "finish-publish" || role === "finish-subscribe") { + finish(role, url, broadcast, versions, Number(params.get("frames") ?? 10), Number(params.get("timeout") ?? 60_000)); +} else if (role === "publish") { const el = document.createElement("moq-publish") as MoqPublish; el.setAttribute("url", url); if (versions) el.setAttribute("versions", versions); diff --git a/test/interop/clients/python/interop.py b/test/interop/clients/python/interop.py index 1e83fa3c643..c1a43a9963a 100644 --- a/test/interop/clients/python/interop.py +++ b/test/interop/clients/python/interop.py @@ -8,6 +8,9 @@ non-empty video frame has arrived and --audio-ms of audio has been decoded to --audio-out for tone.py to judge (exit 1 on timeout / no data). +--finish verifies a finite tail, natural media/catalog EOF, announcement END, and +same-session health; --frames sets its exact size. --finish-ack selects publisher +finish mode and names the FIFO releasing teardown after the real linger. --version offers only that version. --grant waits for the grant the relay sends over AUTH and prints it as an `auth granted` line before the client starts. @@ -23,6 +26,7 @@ import math import struct import sys +import time from datetime import timedelta import moq @@ -88,12 +92,18 @@ async def first() -> moq.Grant: async def publish( - url: str, broadcast: str, version: str | None, grant: bool, timeout: float, tones: list[int] + url: str, + broadcast: str, + version: str | None, + grant: bool, + timeout: float, + tones: list[int], + finish_ack: str | None, ) -> None: async with _client(url, version) as client: if grant: await _print_grant(client, timeout) - # Hold the producer for the lifetime of the publish loop; finish() unpublishes. + # Hold the producer for the lifetime of the publish loop. producer = client.create_broadcast(broadcast) catalog = moq.media.CatalogProducer(producer) media = moq.media.TrackStreamProducer.video( @@ -131,6 +141,32 @@ async def publish( await tone audio.finish() media.finish() + if finish_ack is not None: + # finish() also finalizes the catalog. ACK never controls the real linger. + started = time.monotonic_ns() + finished = producer.finish() + linger_us = finished.linger // timedelta(microseconds=1) + if linger_us <= 0: + raise RuntimeError("finished media has no linger") + print(f"finish ready linger_us={linger_us}", flush=True) + await finished.wait() + if time.monotonic_ns() - started < linger_us * 1000: + raise RuntimeError("finish wait ended before its linger") + health = client.create_broadcast(f"{broadcast}.health") + live = health.publish_track("live") + live.write_frame(b"session-alive") + live.finish() + health.announce() + + def acknowledge() -> None: + with open(finish_ack) as ack: + if ack.readline() != "complete\n": + raise RuntimeError("expected finish completion acknowledgement") + + await loop.run_in_executor(None, acknowledge) + if client.session is None or client.session.connects() != 1: + raise RuntimeError("publisher reconnected during finish") + print("finish acknowledged", flush=True) async def _catalog_with_media(consumer: moq.BroadcastConsumer) -> moq.media.Catalog: @@ -144,6 +180,65 @@ async def _catalog_with_media(consumer: moq.BroadcastConsumer) -> moq.media.Cata raise RuntimeError("catalog stream ended without a video and an audio track") +async def _finish_subscribe(client: moq.Client, broadcast: str, frames: int) -> None: + # Watch the exact path before resolving it, retaining END while media drains. + async with client.announced(broadcast) as announcements: + async with client.announced_broadcast(broadcast) as announced: + consumer = await announced + async with await moq.media.CatalogConsumer.subscribe(consumer) as catalogs: + async for catalog in catalogs: + if catalog.video: + break + else: + raise RuntimeError("catalog ended without video") + id, video = next(iter(catalog.video.items())) + async with await moq.media.TrackConsumer.subscribe( + consumer, id, video, subscription=moq.Subscription(max_delay=MAX_DELAY) + ) as media: + count = 0 + async for frame in media: + # Empty payloads carry endpoint metadata, not codec frames. + if not frame.payload: + continue + count += 1 + if count > frames: + raise RuntimeError(f"expected {frames} frames, received more") + if count != frames: + raise RuntimeError(f"expected {frames} frames, received {count}") + # Do not cancel the catalog after its first snapshot: EOF is part of the case. + async for _ in catalogs: + pass + + started = False + async for event in announcements: + if isinstance(event, moq.AnnounceEventRestart) and event.announce.prefix == broadcast: + raise RuntimeError("broadcast restarted during finish") + if isinstance(event, moq.AnnounceEventStart) and event.announce.prefix == broadcast: + started = True + if isinstance(event, moq.AnnounceEventEnd) and event.announce.prefix == broadcast: + if not started: + raise RuntimeError("broadcast ended without announcement") + break + else: + raise RuntimeError("announcement stream ended before broadcast END") + + async with client.announced_broadcast(f"{broadcast}.health") as announced_health: + health = await announced_health + async with await health.subscribe_track("live") as live: + seen = False + async for group in live: + async with group: + async for frame in group: + if seen or frame.payload != b"session-alive" or frame.timestamp is not None: + raise RuntimeError("unexpected session health payload or timestamp") + seen = True + if not seen: + raise RuntimeError("missing session health payload") + if client.session is None or client.session.connects() != 1: + raise RuntimeError("subscriber reconnected during finish") + print(f"finish clean frames={frames} catalog=eof announce=end health=ok", flush=True) + + async def _decode_audio( consumer: moq.BroadcastConsumer, id: str, audio: moq.media.Audio, out: str, audio_ms: int ) -> None: @@ -167,11 +262,27 @@ async def _decode_audio( async def subscribe( - url: str, broadcast: str, version: str | None, grant: bool, timeout: float, audio_out: str, audio_ms: int + url: str, + broadcast: str, + version: str | None, + grant: bool, + timeout: float, + audio_out: str | None, + audio_ms: int | None, + finish: bool, + frames: int, ) -> None: async with _client(url, version) as client: if grant: await _print_grant(client, timeout) + if finish: + print("finish reader ready", flush=True) + command = await asyncio.wait_for(asyncio.to_thread(sys.stdin.readline), timeout) + if command != "read\n": + raise RuntimeError("expected finish read command") + await asyncio.wait_for(_finish_subscribe(client, broadcast, frames), timeout) + return + assert audio_out is not None and audio_ms is not None consumer = await asyncio.wait_for(client.announced_broadcast(broadcast), timeout) catalog = await asyncio.wait_for(_catalog_with_media(consumer), timeout) @@ -211,10 +322,15 @@ def main() -> None: parser.add_argument("--tone", help="publish: comma-separated frequency of each channel, in Hz") parser.add_argument("--audio-out", help="subscribe: decode audio to this prefix's .txt and .pcm") parser.add_argument("--audio-ms", type=int, help="subscribe: how much audio to decode") + parser.add_argument("--finish-ack", help="publisher completion FIFO; enables clean finish") + parser.add_argument("--finish", action="store_true", help="verify a late tail and clean broadcast end") + parser.add_argument("--frames", type=int, default=10, help="expected finish video frame count") args = parser.parse_args() + if args.frames <= 0 or (args.finish and args.role != "subscribe") or (args.finish_ack and args.role != "publish"): + parser.error("finish requires a subscriber, finish-ack a publisher, and frames must be positive") if args.role == "publish" and not args.tone: parser.error("publish needs --tone") - if args.role == "subscribe" and (not args.audio_out or not args.audio_ms): + if args.role == "subscribe" and not args.finish and (not args.audio_out or not args.audio_ms): parser.error("subscribe needs --audio-out and --audio-ms") # Warnings and errors from the Rust layer, such as a publication outside the grant. @@ -222,11 +338,21 @@ def main() -> None: try: if args.role == "publish": tones = [int(hz) for hz in args.tone.split(",")] - asyncio.run(publish(args.url, args.broadcast, args.version, args.grant, args.timeout, tones)) + asyncio.run( + publish(args.url, args.broadcast, args.version, args.grant, args.timeout, tones, args.finish_ack) + ) else: asyncio.run( subscribe( - args.url, args.broadcast, args.version, args.grant, args.timeout, args.audio_out, args.audio_ms + args.url, + args.broadcast, + args.version, + args.grant, + args.timeout, + args.audio_out, + args.audio_ms, + args.finish, + args.frames, ) ) except KeyboardInterrupt: diff --git a/test/interop/clients/rust/finish.rs b/test/interop/clients/rust/finish.rs new file mode 100644 index 00000000000..4fdd6b69ff6 --- /dev/null +++ b/test/interop/clients/rust/finish.rs @@ -0,0 +1,184 @@ +//! Finite media finish case in the existing interop runner. +use std::path::PathBuf; +use std::time::Duration; + +use anyhow::{Context, ensure}; +use moq_mux::catalog::Stream; +use moq_tokio::moq_net; +use tokio::io::{AsyncBufReadExt, AsyncReadExt}; + +struct Args { + role: String, + url: url::Url, + broadcast: String, + version: Option, + timeout: Duration, + frames: usize, + finish: bool, + finish_ack: Option, +} + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + let mut input = std::env::args().skip(1); + let role = input.next().context("missing role")?; + let mut options = std::collections::HashMap::new(); + let mut finish = false; + while let Some(flag) = input.next() { + if flag == "--finish" { + finish = true; + continue; + } + ensure!( + [ + "--url", + "--broadcast", + "--version", + "--timeout", + "--frames", + "--finish-ack" + ] + .contains(&flag.as_str()), + "unknown flag: {flag}" + ); + options.insert(flag, input.next().context("missing flag value")?); + } + let args = Args { + role, + finish, + url: options.remove("--url").context("missing --url")?.parse()?, + broadcast: options.remove("--broadcast").context("missing --broadcast")?, + version: options.remove("--version"), + timeout: Duration::try_from_secs_f64(options.remove("--timeout").unwrap_or_else(|| "60".into()).parse()?)?, + frames: options.remove("--frames").unwrap_or_else(|| "10".into()).parse()?, + finish_ack: options.remove("--finish-ack").map(PathBuf::from), + }; + ensure!( + !args.timeout.is_zero() && args.frames > 0, + "timeout and frames must be positive" + ); + let mut config = moq_tokio::connect::Config::default(); + if let Some(version) = &args.version { + config.version.push(version.parse().map_err(anyhow::Error::msg)?); + } + config.websocket.enabled = Some(false); + let client = config.init(Default::default())?.with_reconnect(false); + let origin = moq_tokio::origin::spawn(); + let consumer = origin.consume(); + let session = client + .clone() + .with_publisher(&origin) + .with_subscriber(origin.clone()) + .connect(args.url) + .established() + .await?; + let mut stdin = tokio::io::BufReader::new(tokio::io::stdin()); + if args.role == "publish" { + let mut broadcast = origin.create_broadcast(&args.broadcast)?; + let mut catalog = moq_mux::catalog::Producer::<()>::new(&mut broadcast, Default::default())?; + let mut media = moq_mux::import::TrackStream::video( + broadcast.reserve_track("video")?, + catalog.reserve(), + moq_mux::import::VideoInit::new(moq_mux::import::VideoFormat::Avc3, Vec::new()), + )?; + let mut data = Vec::new(); + stdin.read_to_end(&mut data).await?; + media.decode(&data)?; + media.finish()?; + catalog.finish()?; + broadcast.announce(Default::default())?; + let started = moq_clock::now(); + let finish = broadcast.finish(); + let linger = finish.linger(); + ensure!(!linger.is_zero(), "finished media has no linger"); + println!("finish ready linger_us={}", finish.linger().as_micros()); + finish.wait().await; + ensure!(started.elapsed() >= linger, "finish wait ended before its linger"); + println!("finish ended"); + let health = origin.create_broadcast(format!("{}.health", args.broadcast))?; + let mut track = health.create_track("live", moq_net::track::Info::default())?; + track.write_frame(None, b"session-alive".to_vec())?; + track.finish()?; + health.announce(Default::default())?; + let ack = args.finish_ack.context("publisher needs --finish-ack")?; + let mut complete = String::new(); + tokio::io::BufReader::new(tokio::fs::File::open(ack).await?) + .read_line(&mut complete) + .await?; + ensure!(complete == "complete\n", "finish acknowledgement was not complete"); + health.close(); + println!("finish acknowledged"); + } else { + ensure!(args.role == "subscribe" && args.finish, "expected finish subscriber"); + println!("finish reader ready"); + let mut command = String::new(); + stdin.read_line(&mut command).await?; + ensure!(command == "read\n", "finish read phase was not released"); + moq_clock::timeout(args.timeout, async { + let mut routes = consumer.announced(); + let broadcast = consumer.routed_broadcast(&args.broadcast).await?; + let mut catalogs = + moq_mux::catalog::Consumer::<()>::new(&broadcast, moq_mux::catalog::CatalogFormat::Hang).await?; + let catalog = loop { + let catalog = catalogs.next().await?.context("finished catalog had no video")?; + if !catalog.video.renditions.is_empty() { + break catalog; + } + }; + let (name, config) = catalog.video.renditions.iter().next().context("no video")?; + let track = broadcast + .track(name)? + .subscribe(Some( + moq_net::track::Subscription::default() + .with_max_delay(args.timeout) + .with_groups(0..), + )) + .await?; + let format = moq_mux::catalog::hang::Container::try_from(config)?; + let mut media = moq_mux::container::Consumer::new(track, format); + let mut count = 0; + while let Some(item) = media.read().await? { + match item { + moq_mux::container::Read::Frame { frame, .. } => { + if !frame.payload.is_empty() { + count += 1; + } + } + moq_mux::container::Read::Gap { .. } => anyhow::bail!("gap in finished tail"), + moq_mux::container::Read::End { .. } => {} + } + } + ensure!(count == args.frames, "wrong finished tail: {count} frames"); + while catalogs.next().await?.is_some() {} + loop { + let event = routes + .next() + .await + .context("origin ended before broadcast withdrawal")?; + if let moq_net::announce::Event::End(announce) = event + && announce.prefix.as_str() == args.broadcast + { + break; + } + } + let health = consumer.routed_broadcast(format!("{}.health", args.broadcast)).await?; + let mut track = health.track("live")?.subscribe(None).await?; + let mut group = track.recv_group().await?.context("no health group")?; + let frame = group.read_frame().await?.context("no health frame")?; + ensure!( + frame.payload.as_ref() == b"session-alive" && frame.timestamp.is_none(), + "health payload or timestamp" + ); + ensure!( + group.read_frame().await?.is_none() && track.recv_group().await?.is_none(), + "health EOF" + ); + println!("finish clean frames={count} catalog=eof announce=end health=ok"); + Ok::<_, anyhow::Error>(()) + }) + .await??; + } + session.close().await?; + client.close().await; + Ok(()) +} diff --git a/test/interop/interop.sh b/test/interop/interop.sh index d84ffb4930c..6743d6950b0 100755 --- a/test/interop/interop.sh +++ b/test/interop/interop.sh @@ -68,6 +68,7 @@ MEDIA=0 MEDIA_CASES="" MEDIA_ENGINES="chromium" TAIL_ONLY=0 +FINISH_ONLY=0 # The versions the matrix runs on, one pass each. `auto` leaves every client its defaults, # except the clients the harness reads a grant from, which offer $AUTH_VERSION alone. Any other # entry pins every client in its pass to that version, and `ietf` expands to IETF_VERSIONS. @@ -113,6 +114,10 @@ while [[ $# -gt 0 ]]; do NEGATIVE=1 shift ;; + --finish) + FINISH_ONLY=1 + shift + ;; --tail) TAIL_ONLY=1 shift @@ -168,6 +173,13 @@ fi exit 2 } +if [[ "$FINISH_ONLY" -eq 1 ]]; then + if [[ "$NEGATIVE" -eq 1 || "$MEDIA" -eq 1 || "$TAIL_ONLY" -eq 1 || "$IN_TREE" -eq 0 ]]; then + echo "error: --finish requires in-tree clients and is separate from --tail, --media, and --negative" >&2 + exit 2 + fi +fi + if [[ "$TAIL_ONLY" -eq 1 ]]; then if [[ "$NEGATIVE" -eq 1 || "$MEDIA" -eq 1 ]]; then echo "error: --tail, --media, and --negative are separate runs" >&2 @@ -384,7 +396,7 @@ plan_cases() { # One version's share of plan_cases. plan_pass() { local pub sub runtime version - if [[ "$TAIL_ONLY" -eq 0 ]]; then + if [[ "$TAIL_ONLY" -eq 0 && "$FINISH_ONLY" -eq 0 ]]; then for pub in "${PUB_LIST[@]}"; do for sub in "${SUB_LIST[@]}"; do record "not run" case=media publisher="$pub" subscriber="$sub"; done [[ "$IN_TREE" -eq 0 ]] || record "not run" case="relay close" publisher="$pub" @@ -402,6 +414,12 @@ plan_pass() { [[ "$VERSION" != auto ]] || ! needs js || record "not run" case="refused session" subscriber=js fi [[ "$IN_TREE" -eq 1 ]] || return 0 + if [[ "$TAIL_ONLY" -eq 0 ]]; then + for pub in "${PUB_LIST[@]}"; do + for sub in "${SUB_LIST[@]}"; do record "not run" case=finish publisher="$pub" subscriber="$sub"; done + done + fi + [[ "$FINISH_ONLY" -eq 0 ]] || return 0 record "not run" case="CLI FETCH" publisher=rust subscriber=rust record "not run" case=tail publisher=rust subscriber=rust for runtime in js-native-node js-native-bun; do @@ -493,7 +511,7 @@ build_relay_cli() { exit 1 } if [[ "$MEDIA" -eq 0 ]]; then - (cd "$WORKSPACE" && cargo build --locked ${flag[@]+"${flag[@]}"} -p moq-cli --example interop-tail --example interop-audio) + (cd "$WORKSPACE" && cargo build --locked ${flag[@]+"${flag[@]}"} -p moq-cli --example interop-tail --example interop-audio --example interop-finish) fi [[ -n "$RELAY" ]] || RELAY="$TARGET_BASE/$PROFILE/moq-relay" # The `moq-cli` crate ships its binary as `moq` (a `[[bin]]` override). @@ -1358,6 +1376,225 @@ run_round() { fi } +# Finish readers connect before publication, but make no media request until finish is fixed. +# shellcheck disable=SC2329 # invoked indirectly via harness_spawn +run_finish_client() { + local lang="$1" role="$2" broadcast="$3" url="$4" ack="${5:-}" version limit read_limit + version=$(cell_version "$lang") + # Setup and the read phase each have a deadline. The latter includes the real media linger. + read_limit=$(awk -v t="$TIMEOUT" 'BEGIN { print t * 2 }') + limit=$(awk -v t="$TIMEOUT" 'BEGIN { print t * 4 }') + local args=("$role" --url "$url" --broadcast "$broadcast" --timeout "$read_limit") + [[ -z "$version" ]] || args+=(--version "$version") + if [[ "$role" == publish ]]; then args+=(--finish-ack "$ack"); else args+=(--finish --frames 10); fi + case "$lang/$role" in python/publish | go/publish | cpp/publish) args+=(--tone "$(publisher_tone "$lang")") ;; esac + case "$lang" in + rust) timeout -k 3 "$limit" "$TARGET_BASE/$PROFILE/examples/interop-finish" "${args[@]}" ;; + python) timeout -k 3 "$limit" "$PY" "$CLIENTS/python/interop.py" "${args[@]}" ;; + go) timeout -k 3 "$limit" "$GO_INTEROP" "${args[@]}" ;; + cpp) timeout -k 3 "$limit" "$CPP_INTEROP" "${args[@]}" ;; + c) timeout -k 3 "$limit" "$C_INTEROP" "${args[@]}" ;; + js) (cd "$CLIENTS/js" && timeout -k 3 "$limit" bun driver.ts "${args[@]}") ;; + js-native-node) (cd "$CLIENTS/js-native" && timeout -k 3 "$limit" node --import tsx subscribe.ts "${args[@]}") ;; + js-native-bun) (cd "$CLIENTS/js-native" && timeout -k 3 "$limit" bun subscribe.ts "${args[@]}") ;; + *) + echo "finish assertions unavailable for $lang" >&2 + return 1 + ;; + esac +} + +# shellcheck disable=SC2329 # invoked indirectly via harness_spawn +run_finish_publisher() { + local lang="$1" broadcast="$2" url="$3" ack="$4" + if [[ "$lang" == js ]]; then + run_finish_client "$lang" publish "$broadcast" "$url" "$ack" + else + ffmpeg -hide_banner -loglevel error -f lavfi -i "testsrc2=size=$SIZE:rate=$FPS" \ + -frames:v 10 -c:v libx264 -preset ultrafast -tune zerolatency -pix_fmt yuv420p \ + -g 10 -keyint_min 10 -sc_threshold 0 -f h264 - | run_finish_client "$lang" publish "$broadcast" "$url" "$ack" + fi +} + +# moqsrc's watch is persistent: a finished broadcast keeps its pads, since an announcement means +# online/offline, so GStreamer has no end, withdrawal, or session to report. It joins once the +# broadcast has finished and checks the late tail alone: ten nonempty video buffers, and no more +# by the time an interrupt stops the pipeline. +# shellcheck disable=SC2329 # invoked indirectly via harness_spawn +run_finish_gst() { + local broadcast="$1" url="$2" out="$3" version pid deadline count + version=$(cell_version gst) + # The read phase's deadline, as the other readers get; SECONDS counts whole seconds. + deadline=$((SECONDS + ${TIMEOUT%.*} * 2)) + [[ "$TIMEOUT" != *.* ]] || deadline=$((deadline + 2)) + GST_PLUGIN_PATH_1_0="$GST_PLUGIN_DIR" GST_REGISTRY_1_0="$HARNESS_RUN/gst-run-registry.bin" \ + gst-launch-1.0 -v moqsrc name=s url="$url" broadcast="$broadcast" ${version:+versions="$version"} \ + s.video_0 ! identity name=frames silent=false ! fakesink >"$out" 2>&1 & + pid=$! + count=0 + until ((count >= 10)); do + if ! kill -0 "$pid" 2>/dev/null || ((SECONDS >= deadline)); then + kill -KILL "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + echo "gst: $count of 10 finished frames before the deadline" + tail -n 20 "$out" + return 1 + fi + sleep 0.1 + count=$(grep -cE 'GstIdentity:frames: last-message = chain.*\([1-9][0-9]* bytes' "$out" || true) + done + kill -INT "$pid" + while kill -0 "$pid" 2>/dev/null && ((SECONDS < deadline + 3)); do sleep 0.1; done + kill -KILL "$pid" 2>/dev/null || true + wait "$pid" || { + echo "gst-launch did not stop cleanly on an interrupt" + tail -n 20 "$out" + return 1 + } + count=$(grep -c 'GstIdentity:frames: last-message = chain' "$out" || true) + [[ "$count" -eq 10 ]] || { + echo "gst: $count buffers, want exactly 10" + return 1 + } + echo "finish tail frames=10 end=n/a" +} + +finish_ready() { + local pid="$1" log="$2" marker="$3" deadline + deadline=$(awk -v now="$SECONDS" -v t="$TIMEOUT" 'BEGIN { print int(now + t + 1) }') + while ((SECONDS < deadline)); do + grep -q "$marker" "$log" && return 0 + harness_exited "$pid" && return 1 + sleep 0.05 + done + return 1 +} + +# shellcheck disable=SC2329 # invoked indirectly via harness_spawn +run_finish_reader() { + local fifo="$1" + shift + run_finish_client "$@" <"$fifo" +} + +# The line a finish reader prints once every check it can make has passed. +finish_marker() { + if [[ "$1" == gst ]]; then + echo 'finish tail frames=10 end=n/a' + else + echo 'finish clean frames=10 catalog=eof announce=end health=ok' + fi +} + +run_finishes() { + local pub sub name broadcast grant pid ack ack_fd reader_fd i started late + # moq-transport does not yet deliver a finished broadcast's late tail, withdrawal, and + # same-session health across clients, so a pinned moq-transport pass records the case as + # skipped rather than red. + if [[ "$VERSION" == moq-transport-* ]]; then + for pub in "${PUB_LIST[@]}"; do + for sub in "${SUB_LIST[@]}"; do + record skip case=finish publisher="$pub" subscriber="$sub" detail="moq-transport finish: quest/m2/ietf-finish.md" + done + done + return + fi + for pub in "${PUB_LIST[@]}"; do + name="finish-$pub" + # The fixture and its .health sibling share a parent, so one prefix grant covers both: + # moq-transport's AUTH extension carries prefixes alone. + broadcast="$name-$$-$RANDOM/fixture.hang" + grant="${broadcast%/*}/**" + ack="$LOGS/$name-ack" + local pids=() fds=() names=() all_ready=1 all_clean=1 publisher_ok=1 + late=0 + started=$SECONDS + echo "=== finished media: $pub ===" + if is_broken "$pub"; then + for sub in "${SUB_LIST[@]}"; do record fail case=finish publisher="$pub" subscriber="$sub" detail="publisher unavailable"; done + overall=1 + continue + fi + mkfifo "$ack" + exec {ack_fd}<>"$ack" + for sub in "${SUB_LIST[@]}"; do + if is_broken "$sub"; then + record fail case=finish publisher="$pub" subscriber="$sub" detail="finish assertions unavailable for $sub" + echo " FAIL finish $pub -> $sub (finish assertions unavailable)" + overall=1 + continue + fi + # GStreamer has no stdin barrier; it joins after the finish instead. + if [[ "$sub" == gst ]]; then + late=1 + continue + fi + local fifo="$LOGS/$name-$sub-read" + mkfifo "$fifo" + exec {reader_fd}<>"$fifo" + harness_spawn "$name-$sub" "$LOGS/$name-$sub.log" run_finish_reader "$fifo" "$sub" subscribe "$broadcast" "$(token_url --subscribe "$grant")" + pids+=("$HARNESS_PID") + fds+=("$reader_fd") + names+=("$sub") + done + for i in "${!pids[@]}"; do + if ! finish_ready "${pids[$i]}" "$LOGS/$name-${names[$i]}.log" '^finish reader ready$'; then all_ready=0; fi + done + if [[ "$all_ready" -eq 1 ]]; then + harness_spawn "$name-pub" "$LOGS/$name-pub.log" run_finish_publisher "$pub" "$broadcast" "$(token_url --publish "$grant")" "$ack" + pid="$HARNESS_PID" + if ! finish_ready "$pid" "$LOGS/$name-pub.log" '^finish ready linger_us='; then all_ready=0; fi + else pid=""; fi + if [[ "$all_ready" -eq 1 ]]; then + for reader_fd in "${fds[@]}"; do printf 'read\n' >&"$reader_fd"; done + if [[ "$late" -eq 1 ]]; then + harness_spawn "$name-gst" "$LOGS/$name-gst.log" run_finish_gst "$broadcast" "$(token_url --subscribe "$grant")" "$LOGS/$name-gst.out" + pids+=("$HARNESS_PID") + fds+=("") + names+=(gst) + fi + for i in "${!pids[@]}"; do + harness_wait "${pids[$i]}" || all_clean=0 + if ! grep -qx "$(finish_marker "${names[$i]}")" "$LOGS/$name-${names[$i]}.log"; then all_clean=0; fi + done + else all_clean=0; fi + if [[ "$all_clean" -eq 1 ]]; then + printf 'complete\n' >&"$ack_fd" + exec {ack_fd}>&- + harness_wait "$pid" || publisher_ok=0 + grep -qx 'finish acknowledged' "$LOGS/$name-pub.log" || publisher_ok=0 + else + publisher_ok=0 + [[ -z "$pid" ]] || harness_reap "$pid" + exec {ack_fd}>&- + fi + for i in "${!pids[@]}"; do + sub="${names[$i]}" + local cell=(case=finish publisher="$pub" subscriber="$sub" seconds=$((SECONDS - started)) log="$LOGS/$name-$sub.log") + if [[ "$all_ready" -eq 1 && "$publisher_ok" -eq 1 ]] && grep -qx "$(finish_marker "$sub")" "$LOGS/$name-$sub.log"; then + if [[ "$sub" == gst ]]; then + echo " PASS finish $pub -> $sub (late tail; end N/A, moqsrc keeps watching)" + record pass "${cell[@]}" detail="late tail only; end N/A: moqsrc keeps watching a finished broadcast" + else + echo " PASS finish $pub -> $sub (late tail, EOF, END, live session)" + record pass "${cell[@]}" + fi + else + record fail "${cell[@]}" detail="missing finish tail/EOF/withdrawal/health or acknowledgement" + echo " FAIL finish $pub -> $sub" + overall=1 + cat "$LOGS/$name-$sub.log" "$LOGS/$name-pub.log" 2>/dev/null || true + harness_reap "${pids[$i]}" + fi + [[ "$sub" != gst ]] || continue + reader_fd="${fds[$i]}" + exec {reader_fd}>&- + rm "$LOGS/$name-$sub-read" + done + rm "$ack" + done +} + # Raw-track clients have no codecs: every byte, group, and the declared end is checked. # shellcheck disable=SC2329 # invoked indirectly via harness_spawn run_tail_client() { @@ -1726,8 +1963,11 @@ else LOGS="$HARNESS_RUN/$VERSION" mkdir -p "$LOGS" fi - [[ "$TAIL_ONLY" -eq 1 ]] || run_matrix - [[ "$IN_TREE" -eq 0 ]] || run_tails + [[ "$TAIL_ONLY" -eq 1 || "$FINISH_ONLY" -eq 1 ]] || run_matrix + if [[ "$IN_TREE" -eq 1 ]]; then + [[ "$TAIL_ONLY" -eq 1 ]] || run_finishes + [[ "$FINISH_ONLY" -eq 1 ]] || run_tails + fi done fi