Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,43 @@ jobs:
NEXTEST_PROFILE: ci
MOQ_STRICT: 1

# Media QA with non-Chromium players against a Chromium fixture. The Chromium
# player already runs in interop.yml, on PRs and nightly.
media-engines:
name: Media (${{ matrix.engine }} player)
runs-on: ubuntu-24.04-arm
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
engine: [firefox, webkit]
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: DeterminateSystems/nix-installer-action@1d87d45818068401a10cf16bdc5f00b24994a83f # main
with:
determinate: false
extra-conf: |
extra-substituters = https://kixelated.cachix.org
extra-trusted-public-keys = kixelated.cachix.org-1:CmFcV0lyM6KuVM2m9mih0q4SrAa0XyCsiM7GHrz3KKk=
- name: Rust cache
uses: ./.github/actions/rust-cache
- name: Install publisher and player engines
run: nix develop --command bash -c "bun install --frozen-lockfile && bunx playwright install --with-deps chromium ${{ matrix.engine }}"
- name: Media and negative controls
run: nix develop --command just test media --media-engines ${{ matrix.engine }}
env:
MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs
- name: Preserve measurements and traces
if: ${{ failure() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: media-${{ matrix.engine }}
path: ${{ runner.temp }}/moq-test-runs
if-no-files-found: error

# Full platform coverage without rebuilding after every main queue merge.
platform:
uses: ./.github/workflows/platform.yml
Expand Down
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,6 @@ ranked just below deterministic TS legs, which it requires.
- [Keep-alive from the idle timeout](/quest/m2/quic-keep-alive.md) - the keep-alive interval defaults to a fraction of the negotiated idle timeout; an explicit `quic.keep_alive` overrides
- [Socket close](/quest/m2/noq-socket-close.md) - moq-tokio's QUIC endpoint releases its socket on close, so it drops its wrapper
- [TS health stats](/quest/m2/ts-health-stats.md) - the TS counters ride the stats plumbing beside the media counters
- [Media QA on other engines](/quest/m2/browser-media-qa-engines.md) - the media harness measures a Firefox or WebKit player over the fallback and names what each engine lacks
- [Windows.Graphics.Capture](/quest/m2/capture-wgc.md) - the WGC display and window backend verified on real Windows hardware
- [Windows capture parity](/quest/m2/capture-windows.md) - system audio and a settled app-capture policy
- [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits
Expand Down
38 changes: 0 additions & 38 deletions quest/m2/browser-media-qa-engines.md

This file was deleted.

24 changes: 20 additions & 4 deletions test/interop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,19 @@ refresh each reason before committing.
The matrix asks "did bytes arrive and did a pixel light up". That passes on a
frozen picture, on silence, and on audio a second out of step, so
`just test media` measures the media itself, browser to browser.
Use `just test media --media-engines chromium,firefox,webkit` to run each pinned
Playwright player against a Chromium publisher. The harness installs the requested
engines and runs the same lifecycle cases, content tolerances, and five negative
controls per engine. A missing runtime or required API fails that engine by name.
Firefox and WebKit players explicitly use WebSocket; Chromium keeps normal
transport selection. This does not measure Firefox's QUIC path.

The capability report names the engine version and separately lists missing
publisher and player APIs. Firefox 155 in Playwright 1.63 exposes WebTransport and
WebCodecs but lacks `MediaStreamTrackProcessor`, so it can play this fixture but
cannot publish it. Capabilities are measured on every run rather than assumed
from an engine name. Playwright WebKit is not Safari; Safari device/transport
bugs and real Firefox transport negotiation still require manual evidence.

The publisher is a fixture, not a fake camera: a canvas painting a frame counter
as black/white blocks, and a tone stepping through a fixed frequency table. Both
Expand All @@ -242,17 +255,20 @@ samples, not that a machine played them.

Each run covers, against a real local relay:

- **capabilities** - probes every platform API the player needs, and fails
- **capabilities** - probes native APIs separately for the publisher and player, and fails
naming what is missing rather than skipping a case.
- **cold start** - the publisher reports when it is announced and encoding, then
a fresh page joins. No reload, unlike the matrix driver: a subscriber that
needs a second page load is an initialization bug, not a race.
- **user gesture** - Chromium is launched with
`--autoplay-policy=document-user-activation-required`, which applies to
top-level Web Audio contexts. The fixture and player graphs must be suspended
before either page is clicked, then both must carry audio afterwards. Harness
state probes use CDP with `userGesture: false`: Playwright's usual page reads
themselves grant activation and would invalidate this assertion.
before either page is clicked, then both must carry audio afterwards. Firefox
is launched with its Web Audio autoplay block enabled. Playwright's usual page
reads grant activation and would invalidate this assertion, so the player logs
each sample to the console and the harness never evaluates in a player page
before its click. The Chromium fixture is read over CDP with
`userGesture: false`, and capability probes run in a separate page.
- **capture permission** - the camera case uses a fake device for deterministic
input, while Playwright denies and then grants permissions. Both source errors
must be visible; neither a full nor a microphone-only denial may announce a
Expand Down
6 changes: 3 additions & 3 deletions test/interop/clients/js/driver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ import {
finishTraces,
launch,
open,
type PlayerState,
POLL_INTERVAL_MS,
pageUrl,
pause,
Expand All @@ -32,6 +31,7 @@ import {
waitForState,
waitForWatch,
} from "./harness";
import type { Sample } from "./src/contract";

/** Remove the page's players, which closes their sessions, and wait until every transport has. */
async function closeSessions(page: Page): Promise<void> {
Expand Down Expand Up @@ -78,7 +78,7 @@ if (

const PAUSE_STABILITY_MS = 750;

async function waitForStablePause(page: Page, errors: BrowserErrors, deadline: number): Promise<PlayerState> {
async function waitForStablePause(page: Page, errors: BrowserErrors, deadline: number): Promise<Sample> {
let previous = await readPlayerState(page);
let stableSince = Date.now();
while (Date.now() < deadline) {
Expand Down Expand Up @@ -123,7 +123,7 @@ try {
const start = Date.now();
const startupDeadline = start + timeoutMs;
let reloaded = false;
let playing: PlayerState | undefined;
let playing: Sample | undefined;
while (Date.now() < startupDeadline) {
throwPageErrors(errors);
const state = await readPlayerState(page);
Expand Down
47 changes: 47 additions & 0 deletions test/interop/clients/js/harness.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
import { expect, test } from "bun:test";
import type { Browser, Page } from "playwright";
import { open, readPlayerState, waitForWatch } from "./harness";
import { SAMPLE_EVENT } from "./src/contract";

// A page that logs samples on demand and refuses evaluation, since evaluating would grant
// user activation before the gesture checks.
function fakePage() {
const listeners = new Map<string, (arg: unknown) => void>();
const frame = {};
const page = {
on(event: string, listener: (arg: unknown) => void) {
listeners.set(event, listener);
},
mainFrame: () => frame,
evaluate() {
throw new Error("evaluation would grant user activation");
},
goto() {},
} as unknown as Page;
const log = (sample: object) => listeners.get("console")?.({ text: () => SAMPLE_EVENT + JSON.stringify(sample) });
const navigate = () => listeners.get("framenavigated")?.(frame);
return { page, log, navigate, browser: { newPage: () => page } as unknown as Browser };
}

test("player observations never evaluate in the page", async () => {
const { page, log, browser } = fakePage();
await open(browser, "http://localhost");
log({ seq: 1, userActivated: false, centerPlayVisible: true, controlLabel: "Play" });
await waitForWatch(page);
expect(await readPlayerState(page)).toMatchObject({
seq: 1,
userActivated: false,
centerPlayVisible: true,
controlLabel: "Play",
});
});

test("a navigation discards the previous document's sample", async () => {
const { page, log, navigate, browser } = fakePage();
await open(browser, "http://localhost");
log({ seq: 7 });
navigate();
await expect(readPlayerState(page)).rejects.toThrow("has not published a sample");
log({ seq: 0 });
expect((await readPlayerState(page)).seq).toBe(0);
});
102 changes: 48 additions & 54 deletions test/interop/clients/js/harness.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/**
* Shared Playwright plumbing for the browser clients: serve the built page, launch Chromium, and
* Shared Playwright plumbing for the browser clients: serve the built page, launch an engine, and
* read back the state each role mirrors onto the DOM.
*
* Used by `driver.ts` (the interop matrix) and `media.ts` (the media-output and lifecycle checks).
Expand All @@ -14,9 +14,18 @@ import {
type BrowserContextOptions,
type CDPSession,
chromium,
firefox,
type Page,
webkit,
} from "playwright";
import { CONTROL, type FixtureState, type InteropControl, type Resources, type Sample } from "./src/contract";
import {
CONTROL,
type FixtureState,
type InteropControl,
type Resources,
SAMPLE_EVENT,
type Sample,
} from "./src/contract";

/**
* A failed check, named after the property it was measuring.
Expand All @@ -39,14 +48,6 @@ export function check(ok: boolean, assertion: string, detail: () => string): voi
if (!ok) throw new Failure(assertion, detail());
}

/** The page's own measurements plus the player chrome, which lives in a shadow root. */
export type PlayerState = Sample & {
/** `aria-label` of the primary control button: "Pause" while playing, "Play" while paused. */
controlLabel?: string;
/** Whether the big center play button is showing. */
centerPlayVisible: boolean;
};

/** Errors the page reported, collected so any wait can fail on them instead of timing out. */
export type BrowserErrors = {
page: string[];
Expand All @@ -57,7 +58,6 @@ export type BrowserErrors = {
export const SELECTORS = {
watch: "moq-watch",
ui: "moq-watch-ui",
control: "button.control[aria-label]",
pauseControl: 'button.control[aria-label="Pause"]',
centerPlay: "button.center-play",
fixture: "#fixture",
Expand Down Expand Up @@ -103,19 +103,31 @@ export function pageUrl(origin: string, role: string, params: Record<string, str
return `${origin}/?${query}`;
}

/**
* Launch headless Chromium.
*
* The "chromium" channel is full Chromium (new headless); the headless shell lacks WebTransport and
* WebCodecs. `args` is empty by default: the media checks deliberately run without the fake-device
* and autoplay overrides, so those belong to the caller that needs them.
*/
export function launch(args: string[] = []): Promise<Browser> {
/** Browser engines the media harness can launch. */
export const ENGINES = ["chromium", "firefox", "webkit"] as const;
/** One of the pinned Playwright engines. */
export type Engine = (typeof ENGINES)[number];

/** Launch an engine with its normal gesture gate and caller-supplied Chromium arguments. */
export function launch(args: string[] = [], engine: Engine = "chromium"): Promise<Browser> {
// Playwright closes the browser on SIGTERM by default, which ends its sessions without a
// close. The drivers close it themselves; the harness reaps whatever a signal leaves behind.
return chromium.launch({ channel: "chromium", headless: true, args, handleSIGTERM: false });
// Full Chromium includes WebTransport and WebCodecs; the headless shell does not.
if (engine === "chromium")
return chromium.launch({ channel: "chromium", headless: true, args, handleSIGTERM: false });
if (engine === "firefox")
return firefox.launch({
headless: true,
handleSIGTERM: false,
firefoxUserPrefs: { "media.autoplay.default": 1, "media.autoplay.block-webaudio": true },
});
return webkit.launch({ headless: true, handleSIGTERM: false });
}

// The latest sample each player page logged. Cleared on navigation so a reload cannot read the
// previous document's state.
const samples = new WeakMap<Page, Sample>();

/** Contexts tracing this process, saved by {@link finishTraces} when the run fails. */
const traces: Array<{ context: BrowserContext; name: string }> = [];

Expand Down Expand Up @@ -161,7 +173,14 @@ export async function open(
): Promise<[Page, BrowserErrors]> {
const page = await browser.newPage(options);
const errors: BrowserErrors = { page: [], console: [] };
page.on("framenavigated", (frame) => {
if (frame === page.mainFrame()) samples.delete(page);
});
page.on("console", (message) => {
if (message.text().startsWith(SAMPLE_EVENT)) {
samples.set(page, JSON.parse(message.text().slice(SAMPLE_EVENT.length)) as Sample);
return;
}
console.error(`[${label}] ${message.text()}`);
if (message.type() === "error") errors.console.push(message.text());
});
Expand All @@ -183,11 +202,11 @@ export function throwPageErrors(errors: BrowserErrors): void {
if (messages.length > 0) throw new Error(messages.join("\n"));
}

// Playwright's page.evaluate and locator reads grant user activation in Chromium. A state probe
// Playwright's page.evaluate and locator reads grant user activation. A state probe
// before the deliberate click would therefore unlock Web Audio and invalidate the gesture test.
const sessions = new WeakMap<Page, CDPSession>();

/** Inspect page state without granting user activation to the document. */
/** Inspect a Chromium page without granting user activation to the document. Players log samples instead. */
export async function inspect<A, T>(page: Page, fn: (arg: A) => T, arg: A): Promise<Awaited<T>> {
let session = sessions.get(page);
if (!session) {
Expand All @@ -208,42 +227,17 @@ export async function inspect<A, T>(page: Page, fn: (arg: A) => T, arg: A): Prom
export async function waitForWatch(page: Page): Promise<void> {
const deadline = Date.now() + 30_000;
while (Date.now() < deadline) {
const ready = await inspect(
page,
(tag) => document.querySelector(`${tag}[data-interop-ready]`) !== null,
SELECTORS.watch,
);
if (ready) return;
if (samples.has(page)) return;
await sleep(POLL_INTERVAL_MS);
}
throw new Error("the player did not publish its first sample");
}

/** Read one sample plus the player chrome. Throws until the page has sampled at least once. */
export async function readPlayerState(page: Page): Promise<PlayerState> {
const state = await inspect(
page,
(selectors) => {
const watch = document.querySelector<HTMLElement>(selectors.watch);
const ui = document.querySelector(selectors.ui);
const control = ui?.shadowRoot?.querySelector<HTMLButtonElement>(selectors.control);
const centerPlay = ui?.shadowRoot?.querySelector<HTMLButtonElement>(selectors.centerPlay);

return {
sample: watch?.dataset.interopState,
controlLabel: control?.getAttribute("aria-label") ?? undefined,
centerPlayVisible: centerPlay ? getComputedStyle(centerPlay).display !== "none" : false,
};
},
SELECTORS,
);

if (!state.sample) throw new Error("the player has not published a sample");
return {
...(JSON.parse(state.sample) as Sample),
controlLabel: state.controlLabel,
centerPlayVisible: state.centerPlayVisible,
};
/** Read the latest player sample. Throws until the page has sampled at least once. */
export async function readPlayerState(page: Page): Promise<Sample> {
const sample = samples.get(page);
if (!sample) throw new Error("the player has not published a sample");
return sample;
}

/** Read what the fixture publisher says about itself. Throws until it has published anything. */
Expand Down Expand Up @@ -309,7 +303,7 @@ export async function waitFor<T>(
}

/** Poll the player until `predicate` holds. See {@link waitFor}. */
export function waitForState(page: Page, errors: BrowserErrors, props: WaitProps<PlayerState>): Promise<PlayerState> {
export function waitForState(page: Page, errors: BrowserErrors, props: WaitProps<Sample>): Promise<Sample> {
return waitFor(page, errors, readPlayerState, props);
}

Expand Down
Loading
Loading