Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,12 @@ slow-timeout = { period = "60s", terminate-after = 2 }
retries = 0
# Repeat each failure at the end of the run, so it isn't buried in the log.
failure-output = "immediate-final"

# `just rs miri` interprets every instruction, orders of magnitude slower than
# native, so a test that takes a second here takes minutes there. The cap still
# turns a lost wakeup into a TIMEOUT; keep a test under it by shrinking its
# workload under `cfg(miri)`, as with the native ceiling above.
[profile.miri]
slow-timeout = { period = "300s", terminate-after = 3 }
retries = 0
failure-output = "immediate-final"
11 changes: 11 additions & 0 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,11 @@ jobs:
# virtual camera, a module load no pull request job should make.
# `js worklet-matrix` loads `?worklet` output through the bundlers pinned in test/worklet.
recipe: ["gate", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test drain", "test json-properties", "test resume-model", "rs quic-sim", "test drill impaired", "rs cli-export", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"]
include:
# Miri interprets kio's and moq-net's in-memory unsafe; too slow for the merge gate.
- recipe: rs miri
nightly: true
components: miri
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand All @@ -380,6 +385,12 @@ jobs:
with:
access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }}
secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }}
- name: Install nightly Rust
if: matrix.nightly
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
toolchain: nightly
components: ${{ matrix.components }}
- name: Add a second IPv6 loopback address
if: matrix.recipe == 'rs uring'
# moq-uring's local address test reaches a wildcard socket on it.
Expand Down
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ lite-07; bitrate enforcement is not part of this release.
- [Untrusted media parsers have fuzz targets](/quest/m2/media-fuzz.md) - fMP4, MKV, FLV, TS, Annex-B, catalog, and AMF0 parsing fuzzed with a bounded-work invariant
- [One policy for oversized container metadata](/quest/m2/import-metadata-policy.md) - fMP4 and MKV refuse or stream huge metadata by one documented rule
- [Wire fuzz coverage](/quest/m2/wire-fuzz-coverage.md) - the cmsg walks in moq-sock and moq-uring have invariant fuzz targets, and the unsafe peer input reaches runs under Miri
- [In-memory unsafe runs under Miri](/quest/m2/miri-memory.md) - kio's and moq-net's memory-only unsafe runs under a nightly Miri job
- [VA-API renderer tests](/quest/m2/vaapi-render-tests.md) - the VA-API renderer, decode, and transcode tests run in `just rs gpu` on AMD and Intel and fail instead of skipping, and hosted CI reports them ignored
- [Weekly mutation testing on critical modules](/quest/m2/mutants.md) - cargo-mutants runs weekly on moq-auth and moq-net's permission, resume, and expiry code
- [Video quality](/quest/m2/video-quality/README.md) - a nightly VMAF harness grades every encoder backend and on-demand GoP pops against floors, and moq-boy's encode is fixed under it
Expand Down
36 changes: 0 additions & 36 deletions quest/m2/miri-memory.md

This file was deleted.

5 changes: 2 additions & 3 deletions quest/m2/wire-fuzz-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The moq-net half is done: the `ingest` and `request` targets cover the increment
Decided 2026-10-22 from the security audit, with the maintainer:

- Add a target for `moq-sock`'s cmsg walk (`cmsg::Iter` and `ControlMetadata::decode` in `rs/moq-sock/src/udp/unix.rs`) and for `moq-uring`'s `RecvMeta::parse` (`rs/moq-uring/src/udp.rs`). Both walks must stop at a malformed length and keep what came before it: build well-formed control messages from the input, append a malformed header and arbitrary bytes, and assert the walk reads the same metadata as the well-formed prefix, and the metadata a reference model expects from it.
- Add a Miri job for the crates whose unsafe is reachable from a peer.
- Add the crates whose unsafe is reachable from a peer to `just rs miri` (`sh/rs/miri.sh`), which Nightly already runs for `kio` and `moq-net`.
- Wire every target and the Miri run into the nightly gate, and keep the merge-gate replay of committed regressions.

The point is the invariant, not the coverage number: a target that only checks for a panic would miss the bugs this audit is about.
Expand All @@ -21,13 +21,12 @@ Each crate follows moq-net's shape (`rs/moq-net/fuzz/README.md`): the target bod
Ordering:

- `moq-sock`'s `CMsg::decode` still reads `size_of::<T>()` past a short message (only a `debug_assert_eq!` checks it), which [Bound cmsg payload reads](https://github.com/moq-dev/moq/pull/7394) fixes. A moq-sock target landing first would fail nightly on that known overread, so land it after, or with, that fix. The moq-uring target has no such dependency.
- [In-memory unsafe under Miri](/quest/m2/miri-memory.md) shares one `just rs miri` recipe and nightly job with this quest: whichever lands first creates it, and the other adds its crates.
- `just rs miri` and its nightly job already exist ([moq-dev/moq#7315](https://github.com/moq-dev/moq/pull/7315)); this quest adds its crates to them.

Public API: none. Wire: none.

## Related

- [CI cost and latency](/quest/m0/ci/README.md) - where the nightly budget lives
- [Bound cmsg payload reads](https://github.com/moq-dev/moq/pull/7394) - fixes the moq-sock overread a cmsg target would hit
- [In-memory unsafe under Miri](/quest/m2/miri-memory.md) - shares the Miri recipe and nightly job
- [Untrusted media parsers have fuzz targets](/quest/m2/media-fuzz.md) - the media parser targets beside these
8 changes: 8 additions & 0 deletions rs/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,14 @@ play *args:
doctest *args:
bash sh/local.sh cargo test --locked --doc "$@"

# Needs nightly with the miri component; executes only in Nightly. See sh/rs/miri.sh
# for what runs and why it splits in two.

# Run kio's and moq-net's hand-written in-memory unsafe under Miri.
[positional-arguments]
miri *args:
bash sh/rs/miri.sh "$@"

# Run the moq-quic simulation's random workloads from a fresh seed, printing the settings that replay it, then the bandwidth probe's acceptance suite and the receive-timestamps comparison.
quic-sim:
#!/usr/bin/env bash
Expand Down
2 changes: 1 addition & 1 deletion rs/kio/src/channel.rs
Original file line number Diff line number Diff line change
Expand Up @@ -495,7 +495,7 @@ mod tests {
#[test]
fn racing_publishes_and_closes_wake_every_waiter() {
const READERS: usize = 4;
const STEPS: u32 = 20_000;
const STEPS: u32 = if cfg!(miri) { 1_000 } else { 20_000 };

struct Unpark(std::thread::Thread);
impl Wake for Unpark {
Expand Down
2 changes: 2 additions & 0 deletions rs/kio/src/clock/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@ mod tests {
/// A paused runtime counting from its own origin anchors on its own clock, not
/// on one another runtime advanced.
#[test]
#[cfg_attr(miri, ignore = "re-runs itself in a fresh process, which Miri cannot spawn")]
fn paused_runtimes_own_their_anchors() {
// A fresh process keeps other tests from taking the shared origin first.
if !test_env::Env::default().set("MOQ_CLOCK_TEST_ORIGIN", "fresh").isolate() {
Expand Down Expand Up @@ -316,6 +317,7 @@ mod tests {
/// A paused runtime sharing the origin cannot skew the anchor of unpaused
/// callers by advancing its clock before the anchor is first taken.
#[test]
#[cfg_attr(miri, ignore = "reads the wall clock, which Miri's isolation refuses")]
fn a_later_paused_runtime_does_not_skew_the_anchor() {
let _ = crate::clock::now();
// Built after the origin, so it counts from the shared origin.
Expand Down
6 changes: 4 additions & 2 deletions rs/kio/src/log.rs
Original file line number Diff line number Diff line change
Expand Up @@ -977,6 +977,8 @@ mod test {
/// it evicts behind them.
#[test]
fn concurrent_readers_see_every_entry() {
// Miri takes minutes over the full run; a tenth still evicts under the readers.
const ENTRIES: u64 = if cfg!(miri) { 1_000 } else { 10_000 };
let _collector = COLLECTOR.lock().unwrap();
let mut writer = Writer::new(0);
let readers: Vec<_> = (0..4)
Expand All @@ -985,7 +987,7 @@ mod test {
std::thread::spawn(move || {
let mut index = 0;
let mut evicted = 0;
while index < 10_000 {
while index < ENTRIES {
match reader.read().get(index) {
Ok(Some(value)) => assert_eq!(*value, index),
Ok(None) => {
Expand All @@ -1000,7 +1002,7 @@ mod test {
})
})
.collect();
for i in 0..10_000u64 {
for i in 0..ENTRIES {
writer.push(i);
if i % 100 == 0 {
writer.evict(i.saturating_sub(500));
Expand Down
5 changes: 4 additions & 1 deletion rs/kio/src/waiter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -607,8 +607,11 @@ mod tests {
waiter.register(&mut list);
}

// Miri takes over twenty minutes for the full run; a thousand rounds still far
// outnumber the bound.
let rounds = if cfg!(miri) { 1_000 } else { 100_000 };
let mut seed = 1u64;
for _ in 0..100_000 {
for _ in 0..rounds {
seed = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
let i = (seed >> 33) as usize % LIVE;
waiters[i] = Waiter::noop();
Expand Down
1 change: 1 addition & 0 deletions rs/moq-net/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,4 @@ The wire layer. Generic over the transport and media-agnostic: the relay never i

- Tests never use tokio or the wall clock. Prefer a plain `#[test]` that polls with explicit instants; async bodies use `#[kio::test]`, whose simulated time jumps only when every task is stalled. tokio is for the benches.
- `just rs fuzz <target>` (one per file in `fuzz/fuzz_targets/`) needs nightly. The target bodies live in `src/fuzz.rs`; `fuzz/regressions/<target>/` replays under `just check`, so commit every crash input there.
- `just rs miri` runs kio's and this crate's model tests under Miri, in Nightly (`sh/rs/miri.sh` lists them). A test Miri cannot run (foreign calls, sockets, the wall clock) carries `#[cfg_attr(miri, ignore = "...")]` naming why; a heavy workload shrinks under `cfg(miri)` instead. A test whose point is a full-size production limit (such as filling `MAX_CACHE_BYTES`) and which can't finish under the `miri` profile's 15-minute cap is ignored the same way, with the reason.
19 changes: 15 additions & 4 deletions rs/moq-net/src/model/frame.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,17 @@ impl Frame {
/// one, so a whole read of them never copies.
pub(crate) const SEGMENT: usize = 1024 * 1024;

/// `len` test bytes counting up from `seed` modulo 251, a prime, so the pattern never
/// lines up with a [`SEGMENT`] and a segment out of place shows. Repeated rather than
/// generated byte by byte, which takes Miri minutes for a few segments.
#[cfg(test)]
pub(crate) fn pattern(seed: usize, len: usize) -> Vec<u8> {
let period: Vec<u8> = (0..251).map(|i| ((seed + i) % 251) as u8).collect();
let mut bytes = period.repeat(len.div_ceil(period.len()));
bytes.truncate(len);
bytes
}

/// A completed frame's payload as its group stores it: one buffer, or the sealed
/// segments of a frame larger than [`SEGMENT`] written in pieces.
///
Expand Down Expand Up @@ -1680,7 +1691,7 @@ mod tests {
#[test]
fn a_conflicting_delivery_leaves_shared_content_and_progress_unchanged() {
let size = SEGMENT + 8;
let payload: Vec<_> = (0..size).map(|i| (i % 251) as u8).collect();
let payload = pattern(0, size);
let a = FrameBuf::growing(size);
a.append(&payload[..SEGMENT + 4]).unwrap();
let b = a.fork();
Expand Down Expand Up @@ -1725,7 +1736,7 @@ mod tests {
fn simultaneous_deliveries_publish_one_segmented_payload() {
const SIZE: usize = 2 * SEGMENT + 37;
const ROUNDS: usize = 12;
let payload: Vec<_> = (0..SIZE).map(|i| (i % 251) as u8).collect();
let payload = pattern(0, SIZE);
for grow in [false, true] {
let a = FrameBuf::with_growth(SIZE, grow);
let b = a.fork();
Expand Down Expand Up @@ -1944,7 +1955,7 @@ mod tests {
}
});

let payload: Vec<u8> = (0..SIZE).map(|i| (i % 251) as u8).collect();
let payload = pattern(0, SIZE);
for chunk in payload.chunks(1000) {
buf.append(chunk).unwrap();
}
Expand All @@ -1960,7 +1971,7 @@ mod tests {
const SIZE: usize = 2 * SEGMENT + 100;
// Straddles every segment boundary.
const CHUNK: usize = 300_000;
let payload: Vec<u8> = (0..SIZE).map(|i| (i % 251) as u8).collect();
let payload = pattern(0, SIZE);

for grow in [false, true] {
let buf = FrameBuf::with_growth(SIZE, grow);
Expand Down
10 changes: 9 additions & 1 deletion rs/moq-net/src/model/group.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4812,6 +4812,7 @@ mod test {
/// A head's slots count against the live tail, so a head of empty frames that would
/// take the group's slots past it is not adopted.
#[test]
#[cfg_attr(miri, ignore = "fills the 32 MiB group tail, past Miri's 15-minute cap")]
fn an_adopted_head_counts_its_slots() {
let pool = cache::Pool::unbounded();
let cache = cache::Track::new(pool, Default::default(), kio::Weak::new());
Expand Down Expand Up @@ -5026,6 +5027,7 @@ mod test {
/// A group grows past its live tail by shedding whole frames from its front: a reader
/// that keeps up reads every frame, a late one gets `Evicted`, and nothing aborts.
#[test]
#[cfg_attr(miri, ignore = "fills the 32 MiB group tail, past Miri's 15-minute cap")]
#[cfg(feature = "async")]
fn a_long_group_sheds_its_front() {
let pool = cache::Pool::unbounded();
Expand Down Expand Up @@ -5061,6 +5063,7 @@ mod test {

/// Empty frames still cost their slot, so a group of them sheds its front too.
#[test]
#[cfg_attr(miri, ignore = "fills the 32 MiB group tail, past Miri's 15-minute cap")]
fn empty_frames_shed_by_their_slots() {
let mut producer = Info { sequence: 0 }.produce();
let slot = size_of::<Entry>();
Expand All @@ -5077,6 +5080,7 @@ mod test {
/// Batches of empty frames count their new slots too, so they shed the front as
/// single writes do.
#[test]
#[cfg_attr(miri, ignore = "fills the 32 MiB group tail, past Miri's 15-minute cap")]
fn empty_frame_batches_shed_by_their_slots() {
let mut producer = Info { sequence: 0 }.produce();
let count = 2 * MAX_CACHE_BYTES as usize / size_of::<Entry>();
Expand Down Expand Up @@ -5318,6 +5322,7 @@ mod test {
/// Frames a cursor already prefetched are its own, so shedding them from the group
/// does not make `finished` report the cursor evicted.
#[test]
#[cfg_attr(miri, ignore = "fills the 32 MiB group tail, past Miri's 15-minute cap")]
#[cfg(feature = "async")]
fn finished_counts_prefetched_frames_as_read() {
let mut producer = Info { sequence: 0 }.produce();
Expand Down Expand Up @@ -5519,6 +5524,7 @@ mod test {

/// An explicit current timestamp is converted to the group's scale.
#[test]
#[cfg_attr(miri, ignore = "reads the wall clock, which Miri's isolation refuses")]
fn create_frame_converts_current_timestamp() {
use crate::Timescale;

Expand Down Expand Up @@ -5913,7 +5919,7 @@ mod test {
let payloads: Vec<Bytes> = (0..Prefetch::CAP)
.map(|i| match i % 2 {
0 => Bytes::from(vec![i as u8; 3]),
_ => Bytes::from((0..2 * frame::SEGMENT + 7).map(|b| (b + i) as u8).collect::<Vec<_>>()),
_ => Bytes::from(frame::pattern(i, 2 * frame::SEGMENT + 7)),
})
.collect();
for payload in &payloads {
Expand Down Expand Up @@ -5993,6 +5999,7 @@ mod test {
/// front, and sheds its own front past the tail; a plain or raw reader that starts after
/// frame 0 was shed gets `Evicted`.
#[test]
#[cfg_attr(miri, ignore = "fills the 32 MiB group tail, past Miri's 15-minute cap")]
#[cfg(feature = "async")]
fn a_long_compressed_group_sheds_its_front() {
let pool = cache::Pool::unbounded();
Expand Down Expand Up @@ -6039,6 +6046,7 @@ mod test {
/// A batch of large frames that deflate to almost nothing stays within one tail of
/// inflated bytes, and the copy sheds what the reader took past its tail.
#[test]
#[cfg_attr(miri, ignore = "fills the 32 MiB group tail, past Miri's 15-minute cap")]
#[cfg(feature = "async")]
fn a_batch_of_large_compressible_frames_stays_in_the_tail() {
let pool = cache::Pool::unbounded();
Expand Down
5 changes: 3 additions & 2 deletions rs/moq-net/src/path/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1065,8 +1065,9 @@ mod tests {
#[test]
#[should_panic(expected = "longer than")]
fn test_from_static_too_long() {
let literal: &'static str = Box::leak("a".repeat(Path::MAX_BYTES + 1).into_boxed_str());
Path::from_static(literal);
// A static rather than a leaked string, which Miri would report.
static LITERAL: [u8; Path::MAX_BYTES + 1] = [b'a'; Path::MAX_BYTES + 1];
Path::from_static(std::str::from_utf8(&LITERAL).unwrap());
}

#[test]
Expand Down
Loading