From fd9eb316fc21cb05e3ed25e972ad6312d83660cc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 15:24:21 -0700 Subject: [PATCH] quest: resolve TS live clock and same-instance resume findings Co-Authored-By: Codex --- quest/m1/README.md | 2 + quest/m1/ts-live-clock.md | 59 ++++++++++++++++++++++++++ quest/m1/ts-same-instance-resume.md | 65 +++++++++++++++++++++++++++++ 3 files changed, 126 insertions(+) create mode 100644 quest/m1/ts-live-clock.md create mode 100644 quest/m1/ts-same-instance-resume.md diff --git a/quest/m1/README.md b/quest/m1/README.md index ed84224460b..88eb352c714 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -69,6 +69,8 @@ session outcomes, and embedder refusals inside session outcomes. - [Dogfood hosted worklets](/quest/m1/dogfood-assets.md) - the moq.pro dashboard hosts the worklets and calls `assets()` after the release - [TS pair acquisition](/quest/m1/ts-pair-acquisition.md) - fix the clean baseline and prove the grader's controls before #5328 enables its scheduled hard gate - [Initial catalog routing](/quest/m1/initial-catalog-routing.md) - a usable fallback does not strand initial catalog acquisition with a spurious Unroutable +- [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls +- [Same-instance TS resume](/quest/m1/ts-same-instance-resume.md) - prove valid continuation and fix the unexpected final exporter failure without weakening identity - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [END_OF_TRACK placement](/quest/m1/ietf-end-of-track-placement.md) - END_OF_TRACK rides the upstream's Location, and Rust's header stops claiming END_OF_GROUP - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone diff --git a/quest/m1/ts-live-clock.md b/quest/m1/ts-live-clock.md new file mode 100644 index 00000000000..61015b78848 --- /dev/null +++ b/quest/m1/ts-live-clock.md @@ -0,0 +1,59 @@ +# [M] Live TS clock-rate validation identifies and fixes timing defects + +## Goal + +Resolve the local live PCR-rate failure seen during automated TS validation. +The exporter and the grader agree on the supported release-clock contract, +with reproducible evidence and automated controls that distinguish valid +output from actual timing defects. + +## Plan + +Decided with the maintainer on 2026-10-11: investigate and fix this finding +independently of the same-instance resume failure. Preserve strict publisher +identity, the existing validation gates, and production negotiation defaults. +Keep validation PR #5328 draft while its local readiness findings remain +unresolved; a passing CI run alone does not explain the local failure. + +The epoch-capable 120-second local run completed with publisher and exporter +exit zero, but the grader reported -1382.60 ppm against its +/-30 ppm limit. +Release timing passed with 224.4 ms accumulated lag and -0.003 ms/s over the +last 38.7 seconds; byte-position scheduling passed at 100%. Logs recorded +WebSocket fallback, old-session forced closure, and a later video deadline +miss. These are observations, not an established causal chain. The same-head +full CI run passed with -0.24 ppm and no release-interval violations. + +`test/ts/pcr-timing.py` already excludes the first third before fitting PCR +against arrival time. Do not attribute the failure to startup merely because +the tail is nearly flat. The retained local report contains aggregates, not +the raw arrival samples needed to replay that fit. Retain enough clock, +packet-position, discontinuity and arrival evidence to reproduce the verdict +and separate source timing, clock recovery, export release, pipe backpressure +and measurement behavior. Compare a direct connection with controlled +fallback/handover where useful; do not infer transport causality from timing. + +Reproduce the cause before changing production code or the oracle. Correct +the layer that is wrong, including the grader if its measurement does not +support its claim. Never obtain a pass by widening the limit, choosing a +convenient measurement window, disabling fallback, or retrying. A necessary +change to the externally promised timing contract needs a maintainer decision. +Check current main and the related open work before duplicating a fix. + +Add deterministic positive and negative timing controls with the intended +failure reason asserted. They should separate valid bounded delay and +measurement effects from the clock-rate or release defect being fixed. +Replay evidence where possible and mock time in unit tests. Keep byte-position +PCR accuracy distinct from arrival-clock measurements. Run short controls in +PR CI and real live timing at least nightly, retaining diagnostic artifacts. +Completion requires a causal regression and the corrected live validation, +not a single successful rerun. Update the harness documentation inline; +public API and wire changes are not expected. + +## Related + +- [Validation PR #5328](https://github.com/moq-dev/moq/pull/5328#issuecomment-6114456593) - retained local failure and process-status evidence +- [Passing full CI](https://github.com/moq-dev/moq/actions/runs/38178905198) - same-head contrast, including real live timing +- [Same-instance resume](/quest/m1/ts-same-instance-resume.md) - independent full-check failure +- [Transport upgrade](/quest/m1/transport-upgrade/README.md) - existing session handover and epoch contract +- [Mux-rate hold](/quest/m1/tstd/mux-rate-hold.md) - #5287 owns initial rate publication and its finite-source behavior +- [TS AC-3 scheduling](/quest/m1/ts-atsc-ac3.md) - #5306 owns decoder deadlines and unknown-rate startup overflow diff --git a/quest/m1/ts-same-instance-resume.md b/quest/m1/ts-same-instance-resume.md new file mode 100644 index 00000000000..fbab9ce0504 --- /dev/null +++ b/quest/m1/ts-same-instance-resume.md @@ -0,0 +1,65 @@ +# [M] Same-instance TS resume has a valid, reliable regression + +## Goal + +A valid same-instance TS return within the configured linger resumes output +and finishes with the publisher's clean final verdict. Resolve the existing +CLI regression's unexpected exporter failure while preserving explicit +refusal of a different publisher instance. + +## Plan + +Decided with the maintainer on 2026-10-11: investigate and fix this finding +independently of the live PCR-rate discrepancy. Preserve strict identity: +matching epochs only permit continuation of the same named content. Reusing +an epoch never licenses different groups or catalog content under the same +names. Do not add implicit stitching, automatic replacement acquisition, or +change production protocol defaults. + +The integrated validation branch's `just check` stopped after 3,566 passes +at `moq-cli::export::the_same_instance_within_the_linger_continues`. The test +observed resumed output, then expected a clean final exit and received `exit 1`. +Its captured exporter stderr is discarded, so the underlying error is not +established. This is distinct from initial catalog acquisition, whose +same-epoch success and epochless refusal controls pass in #5333. + +First retain the exporter's error and reproduce on current main. The fixture +interrupts an importer, then starts a fresh process replaying the same clip +under the same explicit epoch. Establish whether its catalog, tracks, group +numbers and contents satisfy the identity and continuation contract before +calling the exporter's refusal a production bug. Correct an invalid fixture +at its source; do not make the exporter accept invalid identity reuse. If a +valid continuation fails, isolate the responsible ordering and fix that layer. +Do not label it a load flake without evidence. + +Check #5305 and the same-epoch importer work before duplicating implementation. +A pending return catalog, a clean final finish, a replacement switch and an +invalid same-epoch replay are different cases until reproduction connects them. +If an existing quest owns the cause, put the regression and fix there and +reconcile this quest's scope. Do not expand this into the full redundant +importer redesign just to obtain a green CLI test. + +Make the regression depend on controlled events and mocked time where +possible. Retain real relay coverage for the CLI outcome. Prove resumed +content belongs to the intended instance and preserve controls for a genuine +replacement, a missing return, and a failed final end as applicable to the +cause. Assert useful error evidence on failure. Do not increase sleeps, +linger, or timeouts, add retries, or treat the success fixture's nonzero exit +as acceptable. Every regression runs in CI, at least nightly for long integration +coverage. Completion requires the causal test and the relevant full checks, +not a successful isolated rerun after a loaded failure. + +Keep validation PR #5328 and catalog PR #5333 draft until their readiness +findings are resolved. Update affected test documentation and comments +inline. No new public API, wire format, or user guide is expected; a newly +required behavior or API change returns to the maintainer for a decision. + +## Related + +- [Validation PR #5328](https://github.com/moq-dev/moq/pull/5328#issuecomment-6114456593) - failing full-check evidence +- [Catalog identity controls #5333](https://github.com/moq-dev/moq/pull/5333) - initial acquisition preserves the selected contract +- [Live clock validation](/quest/m1/ts-live-clock.md) - independent local live failure +- [Replacement catalog linger](/quest/m1/ts-follow-catalog-bound.md) - #5305 owns the bound on a mid-stream switch's first catalog +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - shared catalogs, tracks and source-derived group identity +- [Closed fallback](/quest/m1/transport-upgrade/closed-fallback.md) - #5284 owns early WebSocket closure and its transport-test fix +- [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - diagnose a shared end-of-stream cause only if reproduced