diff --git a/quest/m0/2676-libmoq-process-exit-can-abort-in-glibcs-pthread-tpp.md b/quest/m0/2676-libmoq-process-exit-can-abort-in-glibcs-pthread-tpp.md deleted file mode 100644 index 98b27976e04..00000000000 --- a/quest/m0/2676-libmoq-process-exit-can-abort-in-glibcs-pthread-tpp.md +++ /dev/null @@ -1,50 +0,0 @@ -# [M] libmoq: process exit can abort in glibc's __pthread_tpp_change_priority - -## Goal - -A C consumer that closes its session, waits for the terminal status, and -returns from `main` exits cleanly on Linux/glibc. The success path of -`test/smoke/clients/c/subscribe.c` returns instead of calling `_exit` to dodge -teardown. - -## Plan - -A program linking `libmoq.a` that consumed video can abort in atexit teardown -with glibc's priority-protected-mutex assertion, after the data path worked. -The smoke client works around it with `_exit(0)` the moment it has its frame -(17bf0d950, "fixes the flaky rust -> c leg"), with no diagnosis. `cpp/obs` -links `libmoq` the same way and follows the documented contract exactly, so -this is a real bug for embedders, not a flake. - -What is known: `libmoq` runs a tokio current-thread runtime on a detached -`libmoq` thread behind a `LazyLock` (`rs/libmoq/src/ffi.rs`) that is never -shut down, and its global `State` is likewise never torn down, so worker -threads are live when C's atexit handlers and C++ static destructors run. -openh264 (vendored C++) is always linked. Nothing in our Rust creates a -priority-protected mutex, so either a `pthread_mutex_t` is used after free and -a garbage `__kind` routes an ordinary unlock down the TPP path, or a still -running thread touches state whose destructor already ran. - -- Reproduce with a minimal C program (connect, consume video, close, wait for - the terminal status, `return 0`) in a loop to get a hit rate. Take a - backtrace from the core: `pthread_mutex_destroy`, `unlock`, or a static - destructor narrows it a lot. -- Build `libmoq` under AddressSanitizer (the fuzz recipe in `rs/justfile` - shows how to step off the pinned toolchain for `-Zsanitizer`) and run the - loop; if a freed mutex is the cause ASan names it. Check whether it still - reproduces with openh264 out of the link. -- Fix the mechanism where it lives. If it is teardown ordering, an atexit - handler that quiesces the runtime thread before static destructors run is - the likely shape; do not add a `moq_shutdown` the contract does not ask for. -- Remove `_exit` from the smoke client's success path. Returning is only safe - once every registration that holds the stack `ctx_t` has fired its terminal - callback: closing the session ends its status registration alone, while - `moq_origin_consume_announced`, `moq_consume_catalog`, and - `moq_consume_video` each keep the pointer until their own terminal, so close - and drain all of them before `return`, or a late frame callback reproduces - a second lifetime bug instead of isolating atexit teardown. The failure - paths keep `_exit` for the `user_data` lifetime reason from #2675. - -## Closes - -- [#2676](https://github.com/moq-dev/moq/issues/2676) - close this issue when the quest finishes diff --git a/quest/m0/README.md b/quest/m0/README.md index 5b2197d2551..20df1c722e3 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -30,7 +30,6 @@ regression test per Root Cause First. - [#2799](/quest/m0/2799-moq-video-capture-negotiates-twice-so-a-window-resize.md) - moq-transcode: the ladder follows a source resolution change instead of keeping the one it started with - [Group charge](/quest/m0/group-charge.md) - charge real per-group cost so MOQ_CACHE_CAPACITY bounds real memory - [H.265 suffix SEI ownership](/quest/m0/h265-suffix.md) - suffix SEI stays with the access unit it follows, including at EOF -- [#2676](/quest/m0/2676-libmoq-process-exit-can-abort-in-glibcs-pthread-tpp.md) - libmoq: process exit can abort in glibc's __pthread_tpp_change_priority - [#3123](/quest/m0/3123-moq-bench-a-lagged-group-permanently-ends-the.md) - moq-bench: a lagged group ends the subscription, so offered load silently decays mid-run - [#2838](/quest/m0/2838-js-flate-codec-rejects-frames-that-inflate-past-the.md) - js/flate: the inflate-cap test times out under parallel load - [#2798](/quest/m0/2798-moq-import-ts-an-audio-resync-is-silent-no-log-no-counter.md) - moq import ts: an audio resync leaves no trace, no log, no counter diff --git a/test/smoke/clients/c/subscribe.c b/test/smoke/clients/c/subscribe.c index 0d155258862..806540a3e25 100644 --- a/test/smoke/clients/c/subscribe.c +++ b/test/smoke/clients/c/subscribe.c @@ -3,9 +3,10 @@ // // libmoq is a handle + callback API: connect, consume a broadcast, get a // catalog snapshot via callback, start the video track, and a frame callback -// fires as frames arrive. We exit 0 the moment a non-empty frame lands, 1 on -// timeout. Publishing isn't wired up: the raw-stream importer that the other -// clients use to publish isn't part of this subscribe-only client. +// fires as frames arrive. Once a non-empty frame lands we close and drain every +// registration, then return 0; a timeout exits 1. Publishing isn't wired up: the +// raw-stream importer that the other clients use to publish isn't part of this +// subscribe-only client. // // c-smoke subscribe --url http://127.0.0.1:4443 --broadcast b.hang --timeout 20 #include @@ -22,54 +23,82 @@ typedef struct { pthread_mutex_t mu; pthread_cond_t cv; - int got; // a non-empty frame arrived - int video_started; // guard: start the video track only once + int got; // a non-empty frame arrived + int32_t origin; + int32_t session; + int32_t broadcast_wait; int32_t broadcast; // handle delivered by moq_origin_consume_announced (0 until it arrives) + int32_t catalog; + int32_t video_track; // handle from moq_consume_video (0 until on_catalog starts it) + + // One flag per registration handed &ctx, set by that registration's terminal + // (<= 0) callback. main returns only once all four are set: that terminal is + // libmoq's documented last touch of user_data, so it is what makes main's + // frame safe to destroy. + int done_status, done_broadcast, done_catalog, done_frame; } ctx_t; -// Callbacks run on libmoq's runtime thread; main waits on the condvar. ctx -// lives on main's stack, and libmoq keeps the pointer until each registration's -// terminal (<= 0) callback fires, which this client never waits for. So once -// anything is registered, main must not return: every path from there on ends in -// _exit, which keeps the frame alive for as long as the callbacks can run. +// Callbacks run on libmoq's runtime thread; main waits on the condvar. ctx lives +// on main's stack and libmoq keeps the pointer until each registration's +// terminal (<= 0) callback fires, so main must not return until every one of +// them has. Closing the session ends its status registration alone; +// moq_origin_consume_announced, moq_consume_catalog and moq_consume_video each +// keep the pointer until their own terminal. See drain() at the bottom. +static void done(ctx_t *c, int *flag) { + pthread_mutex_lock(&c->mu); + *flag = 1; + pthread_cond_broadcast(&c->cv); + pthread_mutex_unlock(&c->mu); +} + static void on_status(void *ud, int32_t code) { - (void)ud; + ctx_t *c = (ctx_t *)ud; fprintf(stderr, "session status: %d\n", code); + if (code <= 0) done(c, &c->done_status); } static void on_frame(void *ud, int32_t frame) { ctx_t *c = (ctx_t *)ud; - if (frame <= 0) return; // 0 = ended, negative = error + if (frame <= 0) { // 0 = ended, negative = error + done(c, &c->done_frame); + return; + } moq_frame f; memset(&f, 0, sizeof(f)); if (moq_consume_frame((uint32_t)frame, &f) == 0 && f.payload_size > 0) { pthread_mutex_lock(&c->mu); c->got = 1; - pthread_cond_signal(&c->cv); + pthread_cond_broadcast(&c->cv); pthread_mutex_unlock(&c->mu); } moq_consume_frame_free((uint32_t)frame); } // Delivers the broadcast handle once it's announced, then once more with a -// terminal code (<= 0) we ignore. Store the first positive handle and wake main. +// terminal code (<= 0). Store the first positive handle and wake main. static void on_broadcast(void *ud, int32_t broadcast) { ctx_t *c = (ctx_t *)ud; - if (broadcast <= 0) return; // 0 = ended, negative = error + if (broadcast <= 0) { // 0 = ended, negative = error + done(c, &c->done_broadcast); + return; + } pthread_mutex_lock(&c->mu); if (c->broadcast <= 0) { c->broadcast = broadcast; - pthread_cond_signal(&c->cv); + pthread_cond_broadcast(&c->cv); } pthread_mutex_unlock(&c->mu); } static void on_catalog(void *ud, int32_t catalog) { ctx_t *c = (ctx_t *)ud; - if (catalog <= 0) return; + if (catalog <= 0) { + done(c, &c->done_catalog); + return; + } pthread_mutex_lock(&c->mu); - int start = !c->video_started; + int start = c->video_track <= 0; pthread_mutex_unlock(&c->mu); if (start) { @@ -81,7 +110,7 @@ static void on_catalog(void *ud, int32_t catalog) { int32_t track = moq_consume_video((uint32_t)catalog, 0, 1000, on_frame, ud); if (track > 0) { pthread_mutex_lock(&c->mu); - c->video_started = 1; + c->video_track = track; pthread_mutex_unlock(&c->mu); } } @@ -89,15 +118,13 @@ static void on_catalog(void *ud, int32_t catalog) { moq_consume_catalog_free((uint32_t)catalog); } -// Report a failure and exit 1, for the paths where libmoq still holds &c. +// Report a failure and exit 1, for the paths where libmoq still holds &ctx. // -// _exit rather than a return, for two reasons. It leaves main's frame (and so -// ctx) intact for the callbacks still pointing at it, and it skips the atexit -// teardown: libmoq statically bundles moq-video (openh264/cuda), whose worker -// threads use priority-protected mutexes, and tearing them down at normal exit -// can trip a glibc pthread priority assertion and abort. An abort here would -// replace the message we just printed with a SIGABRT, which is exactly the -// wrong thing to do on the path that explains why the test failed. +// _exit rather than a return: it leaves main's frame (and so ctx) intact for the +// callbacks still pointing at it, which is what the failure paths want. Draining +// first would work too, but a client that has already failed should print its +// reason and go, not block for up to ten seconds on a teardown that may be exactly +// what is broken. static _Noreturn void fail(const char *fmt, ...) { va_list ap; va_start(ap, fmt); @@ -107,6 +134,45 @@ static _Noreturn void fail(const char *fmt, ...) { _exit(1); } +// Close every registration holding &ctx and wait for each one's terminal +// callback. Only then may main return: main's frame is ctx's storage, and +// exit()'s own call frames reuse those addresses, so a callback that arrives +// after the return writes ctx->mu and ctx->cv over the live exit path. That is +// a use-after-return, and what it corrupts decides the symptom (a SIGSEGV or +// SIGBUS on a clobbered return address, or the glibc __pthread_tpp_change_priority +// assertion when the mutex's __kind is the field that got garbage). +// +// Each *_close returns immediately and does NOT deliver the terminal itself; the +// task delivers it, so every one of them has to be waited for. +static void drain(ctx_t *c) { + pthread_mutex_lock(&c->mu); + int32_t track = c->video_track; + // A video track that was never started registered nothing, so it owes no terminal. + if (track <= 0) c->done_frame = 1; + pthread_mutex_unlock(&c->mu); + + if (track > 0) moq_consume_video_close((uint32_t)track); + moq_consume_catalog_close((uint32_t)c->catalog); + moq_origin_consume_announced_close((uint32_t)c->broadcast_wait); + moq_session_close((uint32_t)c->session); + + struct timespec deadline; + clock_gettime(CLOCK_REALTIME, &deadline); + deadline.tv_sec += 10; + + pthread_mutex_lock(&c->mu); + while (!(c->done_frame && c->done_catalog && c->done_broadcast && c->done_status)) { + if (pthread_cond_timedwait(&c->cv, &c->mu, &deadline) != 0) break; // timed out + } + int ok = c->done_frame && c->done_catalog && c->done_broadcast && c->done_status; + int f = c->done_frame, ct = c->done_catalog, b = c->done_broadcast, s = c->done_status; + pthread_mutex_unlock(&c->mu); + + if (!ok) { + fail("error: teardown never quiesced (frame=%d catalog=%d broadcast=%d status=%d)\n", f, ct, b, s); + } +} + int main(int argc, char **argv) { const char *url = NULL, *broadcast = NULL; double timeout_s = 20.0; @@ -122,24 +188,22 @@ int main(int argc, char **argv) { } ctx_t c; + memset(&c, 0, sizeof(c)); pthread_mutex_init(&c.mu, NULL); pthread_cond_init(&c.cv, NULL); - c.got = 0; - c.video_started = 0; - c.broadcast = 0; - int32_t origin = moq_origin_create(); - if (origin <= 0) { - fprintf(stderr, "error: moq_origin_create failed: %d\n", origin); + c.origin = moq_origin_create(); + if (c.origin <= 0) { + fprintf(stderr, "error: moq_origin_create failed: %d\n", c.origin); return 1; } // origin_publish = 0 disables publishing; consume via our origin. - int32_t session = moq_session_connect(url, strlen(url), 0, (uint32_t)origin, on_status, &c); - if (session <= 0) { + c.session = moq_session_connect(url, strlen(url), 0, (uint32_t)c.origin, on_status, &c); + if (c.session <= 0) { // A registration that fails never invokes its callback, so &c isn't held // yet and returning is still safe here. - fprintf(stderr, "error: moq_session_connect failed: %d\n", session); + fprintf(stderr, "error: moq_session_connect failed: %d\n", c.session); return 1; } @@ -150,9 +214,10 @@ int main(int argc, char **argv) { // The broadcast arrives over the network after connect, so wait for it to be // announced. moq_origin_consume_announced resolves via on_broadcast once it's // available; we block on the condvar until then (or the deadline). - int32_t wait = moq_origin_consume_announced((uint32_t)origin, broadcast, strlen(broadcast), on_broadcast, &c); - if (wait <= 0) { - fail("error: moq_origin_consume_announced failed: %d\n", wait); + c.broadcast_wait = + moq_origin_consume_announced((uint32_t)c.origin, broadcast, strlen(broadcast), on_broadcast, &c); + if (c.broadcast_wait <= 0) { + fail("error: moq_origin_consume_announced failed: %d\n", c.broadcast_wait); } pthread_mutex_lock(&c.mu); @@ -165,9 +230,9 @@ int main(int argc, char **argv) { fail("error: broadcast never announced\n"); } - int32_t cat = moq_consume_catalog((uint32_t)bc, on_catalog, &c); - if (cat <= 0) { - fail("error: moq_consume_catalog failed: %d\n", cat); + c.catalog = moq_consume_catalog((uint32_t)bc, on_catalog, &c); + if (c.catalog <= 0) { + fail("error: moq_consume_catalog failed: %d\n", c.catalog); } pthread_mutex_lock(&c.mu); @@ -177,13 +242,19 @@ int main(int argc, char **argv) { int got = c.got; pthread_mutex_unlock(&c.mu); - if (got) { - fprintf(stderr, "received a frame from %s\n", broadcast); - // The data path succeeded, which is all this smoke client verifies. - // _exit for the reasons on fail() above: the callbacks still point at - // ctx, and the atexit teardown can abort. - fflush(stderr); - _exit(0); + if (!got) { + fail("error: timed out waiting for data\n"); } - fail("error: timed out waiting for data\n"); + + fprintf(stderr, "received a frame from %s\n", broadcast); + + // The data path succeeded, which is all this smoke client verifies. Returning + // (rather than _exit) is the other half of what it verifies: an embedder that + // closes and drains gets a clean process exit, with libmoq's runtime thread + // still live behind its LazyLock. + drain(&c); + moq_consume_close((uint32_t)bc); + moq_origin_close((uint32_t)c.origin); + fflush(stderr); + return 0; }