From 56fc692674b456be0584aa564d09a5e2a75cadd3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 20 Nov 2026 17:13:03 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m1/relay-memory-default.md Quest-Claim: 3330fdbdf68b9bf3 From 2ea3daffe865694c1a2f81bbf81f81d3104ec817 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 20 Nov 2026 18:23:34 -0800 Subject: [PATCH 2/3] feat(relay): a stock relay keeps 10% of memory free cache.headroom defaults to 10%, so an authenticated publisher can't grow a relay without bound by opening tracks. headroom = "0" removes the governor and leaves the cache unbounded unless capacity is set. The governor and the percentage sizing read the process's own cgroup, so a systemd MemoryMax counts, not only a container's cgroup namespace. Co-Authored-By: Claude Sonnet 5.5 --- doc/bin/relay/config.md | 8 ++- quest/m1/relay-memory-default.md | 31 --------- quest/m3/release/cut.md | 4 +- rs/moq-relay/benches/session.rs | 2 + rs/moq-relay/src/cache.rs | 78 +++++++++++++++++----- rs/moq-relay/src/config.rs | 4 +- rs/moq-relay/src/sim.rs | 2 + rs/moq-relay/tests/integration/quic_sim.rs | 2 + 8 files changed, 79 insertions(+), 52 deletions(-) delete mode 100644 quest/m1/relay-memory-default.md diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index c9615c69654..db450efea21 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -314,10 +314,16 @@ replacement is up and it carries no subscription or fetch. ```toml [cache] capacity = "8GiB" # Target bytes of cached groups and buffered datagrams. "75%" of memory also works. -headroom = "2GiB" # Or: keep this much system memory free and grow into the rest. +headroom = "10%" # Keep this much system memory free and grow into the rest (the default). duration = "30s" # Reclaim groups idle this long, and cap every track's max age at it. ``` +With no settings, the relay keeps 10% of system memory free (the cgroup limit +when set), so a publisher that opens many tracks can't grow it without bound. +`headroom` takes bytes or a percentage, and `capacity` also caps the cache at an +absolute size. `headroom = "0"` removes the governor, so the cache grows without +bound unless `capacity` is set: only run that where every publisher is trusted. + `capacity` is a byte target, repaid as active publishers write. `duration` (30s by default) bounds memory by age, and sweeps on a timer, so a publisher that stalls but stays connected still has its idle groups reclaimed. Set diff --git a/quest/m1/relay-memory-default.md b/quest/m1/relay-memory-default.md deleted file mode 100644 index fe6add77137..00000000000 --- a/quest/m1/relay-memory-default.md +++ /dev/null @@ -1,31 +0,0 @@ -# [S] A stock moq-relay bounds its cache memory - -## Goal - -moq-relay with no cache settings sizes its group cache to keep 10% of system -memory free (the cgroup limit when set), as `cache.headroom = "10%"` does -today, so an authenticated publisher can't grow the relay without bound by -opening tracks. Setting `capacity` or `headroom` explicitly still works, and -an operator can still choose an unbounded pool explicitly. - -## Plan - -Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control -audit): ✅ default `headroom` to 10% (rejected: a 50% `capacity`, which -ignores the rest of the process's memory; both). - -Receive credit bounds only what a connection holds in the transport and in -unfinished frames. A finished frame moves into the shared cache, which is -unbounded in bytes when neither knob is set (`rs/moq-relay/src/cache.rs`). -Each track's live group is exempt from the 30 s idle window and holds up to -its 32 MiB tail, so a publisher holds about 32 MiB per track it opens. Under -pool debt a group sheds its front sooner, so a byte budget bounds it. - -Decide how an operator spells "unbounded" without a magic value in the -setting, per AGENTS.md (for example an explicit `capacity = "100%"`, or -`headroom = "0%"`), and say it in the relay docs and the config reference. -Measure process RSS with several publishers opening many tracks against a -small headroom, with the default, so the bound is shown on the built binary. - -Behavior change, no `!`. Public API: the relay config default. Wire: none. -`## Caused by`: as old as the cache pool, no single PR. diff --git a/quest/m3/release/cut.md b/quest/m3/release/cut.md index 4bf0548d3cd..6c9a8ace5ec 100644 --- a/quest/m3/release/cut.md +++ b/quest/m3/release/cut.md @@ -235,8 +235,7 @@ Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control audit): ✅ the cut waits on the [credit model](/quest/m1/credit-model.md), [#9210](https://github.com/moq-dev/moq/pull/9210)'s [paused-stream allowance](/quest/m1/paused-allowance.md), which merges once -the model passes on it, and a [bounded relay cache by -default](/quest/m1/relay-memory-default.md). Rejected: cutting with flow +the model passes on it, and a bounded relay cache by default. Rejected: cutting with flow control unverified; landing #9210 after the cut. The [JS frame cap](/quest/m1/js-frame-cap.md) ships with the cut only if it lands first. @@ -281,5 +280,4 @@ candidate: - [Nightly passes on main](/quest/m0/nightly/README.md) - the release gate runs `features.yml`, red in the 2026-11-12 Nightly - [Credit model](/quest/m1/credit-model.md) - receive credit, pausing, and the sender's credit wait never deadlock - [Paused-stream allowance](/quest/m1/paused-allowance.md) - #9210 merges with its design checked -- [Relay memory default](/quest/m1/relay-memory-default.md) - a stock relay's cache is bounded - [Claim new npm names](/quest/m3/release/npm-first-publish.md) - trusted publishing can publish every new `@moq/*` package diff --git a/rs/moq-relay/benches/session.rs b/rs/moq-relay/benches/session.rs index e209463787f..bc333cc6d72 100644 --- a/rs/moq-relay/benches/session.rs +++ b/rs/moq-relay/benches/session.rs @@ -73,6 +73,8 @@ impl Backend for Relay { fn relay(id: u64, capacity: Option) -> (Node, cache::Pool) { let mut config = moq_relay::cache::Config::default(); + // Only the swept capacity bounds the pool, not the machine's free memory. + config.headroom = Some("0".to_string()); if std::env::var_os("SESSION_EXPIRY_ONLY").is_none() { config.capacity = capacity.map(|capacity| capacity.to_string()); } diff --git a/rs/moq-relay/src/cache.rs b/rs/moq-relay/src/cache.rs index fc154dc54c4..4cb9bdd7b52 100644 --- a/rs/moq-relay/src/cache.rs +++ b/rs/moq-relay/src/cache.rs @@ -10,6 +10,10 @@ use futures::FutureExt; use moq_net::cache; use serde::{Deserialize, Serialize}; +/// The headroom a relay keeps free when `headroom` is unset, so an authenticated +/// publisher opening tracks can't grow the relay without bound. +const DEFAULT_HEADROOM: &str = "10%"; + /// How often the headroom governor re-samples system memory. const GOVERNOR_INTERVAL: Duration = Duration::from_secs(5); @@ -18,8 +22,9 @@ const GOVERNOR_INTERVAL: Duration = Duration::from_secs(5); /// Non-latest groups stay cached until they sit unaccessed past the wall-clock /// LRU window (`duration`, 30s by default) or the pool runs out of room, /// whichever comes first (the latest group of every track is always retained). -/// With none of the knobs set the pool is unbounded in bytes and only the -/// default LRU window bounds memory. +/// With none of the knobs set the pool keeps 10% of system memory free +/// ([`DEFAULT_HEADROOM`]); `headroom = "0"` with no `capacity` leaves it unbounded in +/// bytes, so only the LRU window bounds memory. #[derive(usage_rs::Args, Clone, Debug, Default, Deserialize, Serialize)] #[usage(unknown_flags = "error", args_override_self = false)] #[serde(default, deny_unknown_fields)] @@ -27,20 +32,22 @@ const GOVERNOR_INTERVAL: Duration = Duration::from_secs(5); pub struct Config { /// Target bytes of cached group payload, e.g. "8GiB", "512MB", or a /// percentage of memory like "75%" (respecting the cgroup limit when set). - /// Unbounded when unset. + /// Unbounded when unset, apart from what `headroom` frees. /// /// A target that usage converges toward as tracks write, not a hard limit, and it /// counts cached group bytes (payload plus a fixed cost per group), not process - /// RSS; leave some slack below physical memory or combine with `headroom`. + /// RSS; leave some slack below physical memory. `headroom` still applies on top. #[usage(long = "cache-capacity", env = "MOQ_CACHE_CAPACITY", setting = "cache.capacity")] pub capacity: Option, /// Keep at least this much system memory available, e.g. "2GiB" or "10%". + /// Defaults to "10%" (respecting the cgroup limit when set); "0" disables it. /// - /// Enables a background governor that re-sizes the cache every few seconds + /// Runs a background governor that re-sizes the cache every few seconds /// so the cache soaks up idle memory but is the first thing reclaimed when - /// the rest of the system needs it. Combine with `capacity` to also cap the - /// absolute size. + /// the rest of the system needs it. `capacity` also caps the absolute size. + /// "0" removes the governor, leaving the cache unbounded unless `capacity` + /// is set: an authenticated publisher can then grow the relay without limit. #[usage(long = "cache-headroom", env = "MOQ_CACHE_HEADROOM", setting = "cache.headroom")] pub headroom: Option, @@ -107,14 +114,15 @@ impl Config { .with_expiry(duration.unwrap_or(cache::DEFAULT_EXPIRY)); let pool = cache::Pool::new(config); + let headroom = parse_limit(self.headroom.as_deref().unwrap_or(DEFAULT_HEADROOM))?; let mut governed = None; - if let Some(headroom) = self.headroom.as_deref() { - let headroom = parse_limit(headroom)?; - anyhow::ensure!(headroom > 0, "cache headroom must be non-zero"); + if headroom > 0 { tracing::info!(?capacity, headroom, "cache governor enabled"); governed = Some(governor(system()?, pool.downgrade(), capacity, headroom).boxed()); } else if let Some(capacity) = capacity { tracing::info!(capacity, "cache capacity set"); + } else { + tracing::warn!("cache headroom is 0 and no capacity is set: the cache is unbounded"); } if let Some(duration) = duration { @@ -131,13 +139,14 @@ impl Config { /// Parse a size limit: either absolute bytes ("8GiB", "512MB", "1073741824") or /// a percentage of total memory ("75%"), honoring the cgroup limit when set. +/// Zero is a valid limit, which `headroom` reads as no governor. fn parse_limit(value: &str) -> anyhow::Result { let value = value.trim(); if let Some(percent) = value.strip_suffix('%') { let percent: f64 = percent.trim().parse()?; anyhow::ensure!( - percent > 0.0 && percent <= 100.0, - "memory percentage must be within (0, 100], got {percent}" + (0.0..=100.0).contains(&percent), + "memory percentage must be within [0, 100], got {percent}" ); return Ok((total_memory()? as f64 * percent / 100.0) as u64); } @@ -152,12 +161,27 @@ fn parse_limit(value: &str) -> anyhow::Result { fn total_memory() -> anyhow::Result { let mut sys = system()?; sys.refresh_memory(); - Ok(match sys.cgroup_limits() { + Ok(match cgroup_limits(&mut sys) { Some(limits) => limits.total_memory, None => sys.total_memory(), }) } +/// The memory limits of this process's own cgroup, such as a systemd unit's `MemoryMax`. +/// +/// Not [`sysinfo::System::cgroup_limits`], which reads the cgroup root this process can +/// see: that is the container's own only under a private cgroup namespace, so a service +/// limited by its parent cgroup would be sized against the whole host. +fn cgroup_limits(sys: &mut sysinfo::System) -> Option { + let pid = sysinfo::get_current_pid().ok()?; + sys.refresh_processes_specifics( + sysinfo::ProcessesToUpdate::Some(&[pid]), + false, + sysinfo::ProcessRefreshKind::nothing(), + ); + sys.process(pid)?.cgroup_limits() +} + fn system() -> anyhow::Result { sysinfo::System::new().map_err(|err| anyhow::anyhow!("failed to read system memory: {err}")) } @@ -184,7 +208,7 @@ async fn governor(mut sys: sysinfo::System, pool: cache::PoolWeak, capacity: Opt sys.refresh_memory(); - let available = match sys.cgroup_limits() { + let available = match cgroup_limits(&mut sys) { Some(limits) => limits.free_memory, None => sys.available_memory(), }; @@ -218,6 +242,29 @@ mod tests { assert_eq!(cache.duration, None); } + /// A stock relay is governed; only an explicit zero headroom leaves the pool unbounded. + #[test] + fn headroom_defaults_to_a_governor_and_zero_disables_it() { + assert!(Config::default().init().unwrap().governor.is_some()); + + for zero in ["0", "0%", "0GiB"] { + let config = Config { + headroom: Some(zero.to_string()), + ..Default::default() + }; + assert!(config.init().unwrap().governor.is_none(), "headroom {zero:?}"); + } + + let capped = Config { + capacity: Some("1GiB".to_string()), + ..Default::default() + }; + assert!( + capped.init().unwrap().governor.is_some(), + "an explicit capacity keeps the default headroom" + ); + } + #[test] fn explicit_duration_configures_both_bounds() { let duration = Duration::from_secs(5); @@ -239,6 +286,7 @@ mod tests { #[test] fn parse_limit_percent() { + assert_eq!(parse_limit("0%").unwrap(), 0); let half = parse_limit("50%").unwrap(); let full = parse_limit("100%").unwrap(); assert!(half > 0); @@ -249,7 +297,7 @@ mod tests { #[test] fn parse_limit_rejects_garbage() { assert!(parse_limit("lots").is_err()); - assert!(parse_limit("0%").is_err()); + assert!(parse_limit("-1%").is_err()); assert!(parse_limit("150%").is_err()); } diff --git a/rs/moq-relay/src/config.rs b/rs/moq-relay/src/config.rs index 47cf577f210..e6af108a3c3 100644 --- a/rs/moq-relay/src/config.rs +++ b/rs/moq-relay/src/config.rs @@ -83,8 +83,8 @@ pub struct Config { #[serde(default)] pub stats: stats::Config, - /// Group cache sizing. Unbounded unless `cache.capacity` or `cache.headroom` - /// is set. + /// Group cache sizing. Keeps 10% of memory free unless `cache.capacity` or + /// `cache.headroom` says otherwise; `cache.headroom = "0"` removes that bound. #[usage(flatten)] #[serde(default)] pub cache: cache::Config, diff --git a/rs/moq-relay/src/sim.rs b/rs/moq-relay/src/sim.rs index aa6d0669606..ad5688c627d 100644 --- a/rs/moq-relay/src/sim.rs +++ b/rs/moq-relay/src/sim.rs @@ -169,6 +169,8 @@ impl Net { let cache = crate::cache::Config { // Past what the pool counts in milliseconds, so it never expires idle groups. duration: Some(Duration::MAX.into()), + // No governor either: its timer would keep the mesh from settling. + headroom: Some("0".to_string()), ..Default::default() }; let mut options = cluster::Options::new(config).with_cache(cache.init().expect("cache")); diff --git a/rs/moq-relay/tests/integration/quic_sim.rs b/rs/moq-relay/tests/integration/quic_sim.rs index f130593f1fc..9d9309f5449 100644 --- a/rs/moq-relay/tests/integration/quic_sim.rs +++ b/rs/moq-relay/tests/integration/quic_sim.rs @@ -70,6 +70,8 @@ fn two_relays() { let mut cache = moq_relay::cache::Config::default(); // Past what the pool counts in milliseconds, so it never expires idle groups. cache.duration = Some(Duration::MAX.into()); + // And no governor, whose timer would never let the simulation settle. + cache.headroom = Some("0".to_string()); let mut options = cluster::Options::new(config).with_cache(cache.init().expect("cache")); options.sim = Some(network.host(b.ip())); // A fixed port, which the static peer dials share, so the test can impair that path. From d73ff5d49617424ffde6a545ea9faebabd38f397 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 20 Nov 2026 18:26:40 -0800 Subject: [PATCH 3/3] quest: a publisher's live edge across many tracks needs a byte bound Found by the review of the relay's default headroom. Co-Authored-By: Claude Sonnet 5.5 --- quest/m2/relay-live-edge-bound.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 quest/m2/relay-live-edge-bound.md diff --git a/quest/m2/relay-live-edge-bound.md b/quest/m2/relay-live-edge-bound.md new file mode 100644 index 00000000000..56f58a57388 --- /dev/null +++ b/quest/m2/relay-live-edge-bound.md @@ -0,0 +1,30 @@ +# [M] A publisher's live edge across many tracks is bounded in bytes + +## Goal + +A publisher holding many live tracks, each with one finished group and nothing +more written, cannot make the relay hold more than a bound set by the +relay's own settings. The cache pool's byte budget, which the headroom +governor moves, today reaches only what a writing track sheds; the newest +frame of every live track is kept whatever the budget. + +## Plan + +Found by the review of the relay's default headroom: `Pool::resize` only +moves the target, a track repays debt when it next writes, a group keeps its +newest frame while open, and the latest group of every track is exempt from +idle expiry. So N tracks that each wrote one large frame and went quiet hold +N frames past any capacity, until the publisher or a subscriber ends them. The +default headroom bounds a track that keeps writing, not this shape. + +Decide the bound: reclaiming a quiet track's live edge (a late joiner then +needs a flight or a new group), a per-session or per-broadcast track limit, or +charging the held live edge against the session's credit. Measure it with +many tracks of one frame each against a small headroom on the built binary. + +## Waiting + +- Maintainer: pick the bound. Recommend a per-session live-edge byte budget + (charge a track's retained newest frame to the session that opened it), since + evicting a live edge changes what a subscriber is served and a track-count + limit does not bound bytes; alternative is reclaiming idle live edges.