Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ test:
python3 sh/rs/model-markers.test.py
python3 sh/gh/nightly.test.py
python3 sh/gh/bench-report.test.py
python3 sh/gh/ts-paths.test.py
sh/go/publish-wrapper.test.sh
bash .claude/hooks/direnv.test.sh
python3 sh/rs/uring.test.py
Expand Down
30 changes: 19 additions & 11 deletions .github/workflows/ts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,24 +11,32 @@ on:
- "test/ts/**"
- "test/lib/**"
- "test/justfile"
- "rs/moq-mux/**"
# Exactly the workspace crates run.sh builds; sh/gh/ts-paths.py fails Check on drift.
- "rs/hang/**"
- "rs/moq-cli/**"
- "rs/moq-net/**"
- "rs/moq-tokio/**"
- "rs/moq-relay/**"
- "rs/moq-loc/**"
- "rs/moq-json/**"
# The lossy-path step's transport and shaper.
- "rs/moq-quic/**"
- "rs/moq-shaper/**"
# Workspace dependencies of moq-net, moq-tokio and moq-mux on the TS path.
- "rs/kio/**"
- "rs/moq-archive/**"
- "rs/moq-auth/**"
- "rs/moq-cli/**"
- "rs/moq-clock/**"
- "rs/moq-flate/**"
- "rs/moq-hls/**"
- "rs/moq-ice/**"
- "rs/moq-json/**"
- "rs/moq-loc/**"
- "rs/moq-msf/**"
- "rs/moq-mux/**"
- "rs/moq-net/**"
- "rs/moq-pattern/**"
- "rs/moq-quic/**"
- "rs/moq-relay/**"
- "rs/moq-rtc/**"
- "rs/moq-rtmp/**"
- "rs/moq-rtsp/**"
- "rs/moq-shaper/**"
- "rs/moq-sock/**"
- "rs/moq-srt/**"
- "rs/moq-stats/**"
- "rs/moq-tokio/**"
- "Cargo.toml"
- "Cargo.lock"
- "rust-toolchain.toml"
Expand Down
1 change: 0 additions & 1 deletion quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,5 +67,4 @@ The historical samples locate candidates; they are not controlled benchmarks.
- [Require platform gates](/quest/m0/ci/platform-queue.md) - activate the consolidated platform checks in the main and release rulesets after the workflow lands
- [mbx on R2](/quest/m0/ci/mbx-r2.md) - Rust builds on every OS read an R2 cache that only `main` writes, escaping the 10 GB Actions cache
- [Nix cache](/quest/m0/ci/nix-cache.md) - `nix.moq.dev` on R2 replaces Cachix for CI and releases
- [TS path filter](/quest/m0/ci/ts-path-filter.md) - TS validation runs whenever a crate it builds changes, checked against `cargo metadata`
- [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the first `moq-ffi-v*` tag that runs `Release FFI`, and a `Shipping sizes` run; verify the wrappers and size keys
25 changes: 0 additions & 25 deletions quest/m0/ci/ts-path-filter.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m1/ts-reliability/grader-determinism.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Two sources of run-to-run variance, found by #5810 and #5854:
repeated timestamp read as a hole; `decode.py` has the same exposure.

Decided 2026-10-17 (maintainer): one quest for the grader, with the CI path
filter as a separate [m0 quest](/quest/m0/ci/ts-path-filter.md). The harness
filter as a separate m0 quest. The harness
never gates on a random-loss percentage (see
[bufferbloat](/quest/m1/bufferbloat.md)).

Expand Down
7 changes: 6 additions & 1 deletion sh/dispatch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,9 @@ declare -A scope=(
# nixpkgs, and the last moves on a flake.lock bump alone.
[obs]='^(cpp/obs/|sh/obs/|flake\.(nix|lock)$)'
[flake]='(^rs/|^Cargo\.(toml|lock)$|^flake\.lock$|\.nix$)'
# ts.yml's crate filters against the closure test/ts/run.sh builds, which
# moves only with a manifest, the lockfile, or the harness's packages.
[ts_paths]='^(rs/.*Cargo\.toml$|Cargo\.(toml|lock)$|test/ts/run\.sh$|\.github/workflows/ts\.yml$|sh/gh/ts-paths\.py$)'
[markdown]=''
[shell]=''
[toml]=''
Expand Down Expand Up @@ -149,6 +152,7 @@ declare -A tools=(
[obs_compile]='cargo cmake pkg-config uniffi-bindgen-cpp'
[obs]='clang-format cmake gersemi jq'
[flake]='nix'
[ts_paths]='cargo python3'
[markdown]='bun'
[shell]='shellcheck shfmt'
[toml]='taplo'
Expand All @@ -171,7 +175,7 @@ declare -A compiles=(
)

case "$action" in
check | ci-check) modules=(js workers moq_sh drafts rs bench quest drill py kt swift go dart cpp obs_compile obs flake markdown shell toml nix justfile gh) ;;
check | ci-check) modules=(js workers moq_sh drafts rs bench quest drill py kt swift go dart cpp obs_compile obs flake ts_paths markdown shell toml nix justfile gh) ;;
fix) modules=(js rs py dart obs markdown shell toml nix justfile) ;;
ci-test) modules=(js rs py) ;;
esac
Expand Down Expand Up @@ -251,6 +255,7 @@ for module in "${selected[@]}"; do
check:drill) just test drill-sensitivity --apply-only ;;
check:obs_compile) just obs compile ;;
check:flake) nix flake check ;;
check:ts_paths) python3 sh/gh/ts-paths.py ;;
check:markdown)
bun install --frozen-lockfile
sh/markdown.sh check
Expand Down
64 changes: 64 additions & 0 deletions sh/gh/ts-paths.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""Fail when ts.yml's crate path filters differ from the workspace crates test/ts/run.sh builds.

The filter covers exactly the activated closure of run.sh's `-p` packages with their
default features: a crate outside it must not trigger the expensive workflow, and a
crate inside it must. crates.io dependencies are covered by the Cargo.lock filter.
"""

import json
import re
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[2]
WORKFLOW = ROOT / ".github/workflows/ts.yml"
HARNESS = ROOT / "test/ts/run.sh"
# ts.yml's ubuntu-latest runner, so target-specific dependencies resolve the same on any host.
TARGET = "x86_64-unknown-linux-gnu"


def built_dirs(tree: str, metadata: dict) -> set[str]:
"""Workspace-relative directories of the workspace packages in `cargo tree --format {p}` output.

Packages are matched by name and version, never by the printed source path, which can contain
spaces or parentheses. Paths come from `cargo metadata --no-deps`, in cargo's own syntax, which
is `C:\\...` on Windows even under MSYS or Cygwin.
"""
root = metadata["workspace_root"].replace("\\", "/").rstrip("/") + "/"
members = {
(package["name"], f"v{package['version']}"):
package["manifest_path"].replace("\\", "/").removeprefix(root).removesuffix("/Cargo.toml")
for package in metadata["packages"]
}
return {members[key] for key in (tuple(line.split()[:2]) for line in tree.splitlines()) if key in members}


def cargo(*args: str) -> str:
return subprocess.run(["cargo", *args], cwd=ROOT, check=True, stdout=subprocess.PIPE, text=True).stdout


def main() -> None:
packages = sorted(set(re.findall(r"(?<![\w-])-p ([\w-]+)", HARNESS.read_text())))
if not packages:
sys.exit(f"error: no `-p` packages found in {HARNESS.relative_to(ROOT)}")

tree = cargo("tree", "--locked", "--target", TARGET, "--edges", "normal,build", "--prefix", "none",
"--format", "{p}", *(arg for p in packages for arg in ("-p", p)))
built = built_dirs(tree, json.loads(cargo("metadata", "--locked", "--no-deps", "--format-version", "1")))

# Every `rs/<dir>/**` filter, even one whose crate is gone; other filters are the harness's own inputs.
filtered = set(re.findall(r'^\s+- "(rs/[^"*]+)/\*\*"$', WORKFLOW.read_text(), re.MULTILINE))

missing, extra = sorted(built - filtered), sorted(filtered - built)
for path in missing:
print(f"error: {WORKFLOW.relative_to(ROOT)} misses \"{path}/**\", which `just test ts` builds", file=sys.stderr)
for path in extra:
print(f"error: {WORKFLOW.relative_to(ROOT)} filters \"{path}/**\", which `just test ts` does not build", file=sys.stderr)
if missing or extra:
sys.exit(1)


if __name__ == "__main__":
main()
46 changes: 46 additions & 0 deletions sh/gh/ts-paths.test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
"""Map cargo tree output to workspace crate directories whatever the checkout path looks like."""

import importlib.util
from pathlib import Path
import unittest

spec = importlib.util.spec_from_file_location("ts_paths", Path(__file__).with_name("ts-paths.py"))
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)


def metadata(root: str, sep: str) -> dict:
return {
"workspace_root": root,
"packages": [
{"name": name, "version": version, "manifest_path": sep.join([root, "rs", name, "Cargo.toml"])}
for name, version in (("moq-cli", "0.1.0"), ("moq-lite", "0.2.0"), ("moq-macro", "0.1.0"), ("moq-unused", "0.1.0"))
],
}


class BuiltDirsTest(unittest.TestCase):
def check(self, root: str, sep: str):
tree = "".join(f"{line}\n" for line in (
f"moq-cli v0.1.0 ({sep.join([root, 'rs', 'moq-cli'])})",
f"moq-lite v0.2.0 ({sep.join([root, 'rs', 'moq-lite'])}) (*)",
f"moq-macro v0.1.0 ({sep.join([root, 'rs', 'moq-macro'])}) (proc-macro)",
"moq-lite v0.1.0",
"serde v1.0.0",
"serde_derive v1.0.0 (proc-macro)",
"forked v0.1.0 (https://github.com/example/forked#abc123)",
))
self.assertEqual(module.built_dirs(tree, metadata(root, sep)), {"rs/moq-cli", "rs/moq-lite", "rs/moq-macro"})

def test_posix(self):
self.check("/repo", "/")

def test_windows(self):
self.check("C:\\repo", "\\")

def test_parentheses(self):
self.check("/repo (copy)", "/")


if __name__ == "__main__":
unittest.main()
3 changes: 2 additions & 1 deletion test/ts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -800,7 +800,8 @@ pending acquisition. `rs/moq-net/tests/integration/catalog_acquisition.rs` cover
outcomes deterministically in Rust CI, without retries or implicit `--stitch`.

[`.github/workflows/ts.yml`](../../.github/workflows/ts.yml) is independent of
browser interop. Relevant mux, Hang, CLI, transport, harness and toolchain PRs run
browser interop. PRs touching a workspace crate `run.sh` builds (kept in sync by
`sh/gh/ts-paths.py`), the harness, or the toolchain run
`just test ts-controls`, `just test ts-pair-controls`, the default and 2 Mb/s round trips, HRD, leak, full, headroom,
open-GOP, passthrough (generated, Kyrion and MPTS), EIT, and T-STD controls. Each step runs even after another test fails.
All recipes use the same pinned Nix FFmpeg and TSDuck locally and in CI.
Expand Down
Loading