Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,11 @@ permissions:

on:
workflow_dispatch:
inputs:
bench-base:
description: Revision the paired Rust benchmarks compare against; the run's own commit gives an A/A run. Empty is the previous scheduled Nightly's commit.
type: string
default: ""
schedule:
- cron: "0 7 * * *"

Expand Down Expand Up @@ -80,12 +85,19 @@ jobs:
# build environment, so a benchmark reuses the dependencies its group already
# compiled. The Rust benchmarks outgrew one hour, so they split by harness:
# nextest's ignored benchmark tests and cargo's bench targets share few builds.
# A paired group also builds and runs the previous nightly's commit beside this
# one on the same runner, which is what fails the job on a regression; the
# cross-night history on the benchmark site stays informational.
nightly:
name: Nightly (${{ matrix.name }})
runs-on: ${{ matrix.runner || 'ubuntu-24.04-arm' }}
# `main` and its merge queue write the Rust cache; see .github/actions/rust-cache.
environment: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/gh-readonly-queue/main/')) && 'cache-write' || '' }}
timeout-minutes: 60
permissions:
contents: read
# Paired groups look up the previous scheduled Nightly's commit.
actions: read
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -124,16 +136,25 @@ jobs:
artifact: bench-rust
artifact-path: target/bench/nightly/rust-bench
benchmark: true
paired: true
- name: Rust shared bench targets
recipe: gh nightly rust-shared
artifact: bench-rust-shared
artifact-path: target/bench/nightly/rust-shared
benchmark: true
paired: true
- name: Rust supplemental benchmarks
recipe: gh nightly rust-extra
artifact: bench-rust-extra
artifact-path: target/bench/nightly/rust-extra
benchmark: true
paired: true
- name: io_uring benchmarks
recipe: gh nightly rust-uring
artifact: bench-rust-uring
artifact-path: target/bench/nightly/rust-uring
benchmark: true
paired: true
- name: JS benchmarks
recipe: gh nightly js
artifact: bench-js
Expand Down Expand Up @@ -210,6 +231,8 @@ jobs:
run: nix develop ${{ matrix.shell }} --command just ${{ matrix.recipe }}
env:
NEXTEST_PROFILE: default
MOQ_BENCH_BASE: ${{ matrix.paired && (inputs.bench-base || 'previous-nightly') || '' }}
GH_TOKEN: ${{ matrix.paired && github.token || '' }}

- name: Record benchmark outcome
if: ${{ matrix.benchmark && !cancelled() }}
Expand All @@ -223,6 +246,12 @@ jobs:
path: ${{ matrix.artifact-path }}
retention-days: 45

# After the upload, so the benchmark site still records tonight's history and
# publishes the comparison that failed this job before the alert links to it.
- name: Fail on paired regressions
if: ${{ matrix.paired && steps.workload.outcome == 'success' }}
run: nix develop --command just gh benchmark-report gate "${{ matrix.artifact-path }}"

# gh-pages is the one Pages site, and each directory has one writer: this job owns
# bench/ (the regression table and its history) and the root index listing every
# report, smoke.yml owns relays/, and interop.yml owns interop/. Each commits only
Expand Down Expand Up @@ -295,7 +324,7 @@ jobs:
comment-on-alert: false
fail-on-alert: false
- name: Publish shared history
run: nix shell --inputs-from . nixpkgs#just --command just gh benchmark-publish finish benchmark-site
run: nix shell --inputs-from . nixpkgs#just --command just gh benchmark-publish finish benchmark-site benchmark-reports/paired.json
- name: Drop a stale Pages artifact
run: nix shell --inputs-from . nixpkgs#just --command just gh pages-clean
env:
Expand Down
45 changes: 40 additions & 5 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,15 +398,15 @@ The table includes these measurements:
same host, including for throughput, where the ratio is Rust/wrapper.
- Rust Criterion mean ns per iteration; the history also keeps the confidence interval.
The `net-cpu` source and the moq-quic targets time thread CPU; the others
time elapsed time. The Rust bench targets job measures them with a short warmup
time elapsed time. The Rust bench target jobs measure them with a short warmup
and Criterion's quick mode. Per-group benchmark settings can override the default warmup. Quick estimates
trade precision for nightly runtime; use full paired runs to assess a change.
trade precision for nightly runtime; the paired comparison below assesses a change.
- JS benchmark CSV time columns (`ns`, `us`, `ms`, and their per-operation
variants) and `size_bytes` columns, retaining every other column as a workload
dimension in each series name.

Hosted runners vary across nights, so a flag prompts a paired local run rather
than proving a regression. Binding ratios compare languages within one run.
Hosted runners vary across nights, so a trend flag prompts a look at the paired
comparison rather than proving a regression. Binding ratios compare languages within one run.
Malformed, empty, nonfinite, or duplicate measurements fail validation. The
nightly artifacts retain the reports, raw JS output, and each Criterion run's
metadata and estimates for 45 days. Relay load, audio quality, and the Rust
Expand Down Expand Up @@ -445,4 +445,39 @@ within a source; dimensions belong in the name. Supported units are `ns`, `us`,
`ops/s`, and `frames/s`. Optional `range` and `extra` fields retain uncertainty
and context. Validate and split sources into cost and throughput histories with
`just gh benchmark-report prepare INPUT_DIRECTORY OUTPUT_DIRECTORY`; extending
the table requires no new renderer or data branch.
the table requires no new renderer or data branch. A source's `paired.json`, as
`just gh benchmark-report rust` writes for a paired run, joins the paired comparison.

### Paired comparison

Each Nightly's Rust bench groups (`rust-bench`, `rust-shared`, `rust-extra` and
`rust-uring`) also build the previous scheduled Nightly's commit beside their
own, on the same runner, so night-to-night runner variance drops out. Each
built target runs once per side in the trend's quick mode, alternating which
side goes first; the current side is the trend point. A case more than 10%
slower in that screen reruns, worst first and at most 20 per group, in three
alternating rounds with the BASE comparison's settings (1 s warmup, 2 s
measurement, 30 samples). A case whose rounds are all slower with a median
beyond 10% is a confirmed regression and fails its job after the upload, so the
Alert workflow posts the failed Nightly. A target the base cannot build or run
fails the job too, rather than passing uncompared; a bench target the base does
not have yet is only reported as new. The threshold is a starting point until
A/A runs on the hosted runner measure its noise floor.

The table page shows tonight's comparison above the trend: confirmed
regressions, targets not compared, cases faster in the screen, slowdowns the
rounds rejected or the budget skipped, and the rest. `bench/paired.json` on the
site holds every case. A dispatched Nightly's `bench-base` input compares
against another revision; its own commit gives an A/A run.

Reproduce a paired group locally against any revision; `previous-nightly`
resolves the last scheduled Nightly's commit through `gh`:

```bash
nix develop --command env MOQ_BENCH_BASE=previous-nightly just gh nightly rust-bench
nix develop --command just gh benchmark-report gate target/bench/nightly/rust-bench
```

Each run's evidence directory keeps both builds' estimates (`criterion` and
`criterion-base`), the confirmation rounds under `confirm/`, and the run's
`paired.json`.
124 changes: 117 additions & 7 deletions bench/compare.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
"""Reduce paired relay rounds; retain dispersion instead of implying single-run precision."""
"""Reduce paired relay rounds and Criterion runs; retain dispersion instead of implying single-run precision."""
import argparse
from collections import Counter
import json
import math
import shutil
Expand All @@ -14,6 +15,15 @@ def value(row, metric):
return row.get('generator', {}).get('cpu_cores') if metric == 'generator_cpu_cores' else row.get(metric)


def summarize(deltas):
"""The median paired percent delta, its median absolute deviation, and whether every pair agreed."""
median = statistics.median(deltas)
mad = statistics.median(abs(value - median) for value in deltas)
return {"delta_pct": median, "mad_pct": mad, "range_pct": [min(deltas), max(deltas)], "paired_deltas_pct": deltas,
"status": "insufficient samples" if len(deltas) < 3 else
"sign varies across pairs" if min(deltas) < 0 < max(deltas) else "consistent sign"}


Comment thread
kixelated marked this conversation as resolved.
Resolved
def reduce(pairs):
if not pairs:
raise ValueError("no paired rounds")
Expand All @@ -37,12 +47,7 @@ def reduce(pairs):
if not deltas:
result["metrics"][metric] = {"delta_pct": None, "status": "undefined ratio to zero", "paired_absolute_deltas": absolute}
continue
median = statistics.median(deltas)
mad = statistics.median(abs(value - median) for value in deltas)
result["metrics"][metric] = {"delta_pct": median, "mad_pct": mad,
"range_pct": [min(deltas), max(deltas)], "paired_deltas_pct": deltas, "paired_absolute_deltas": absolute,
"status": "insufficient samples" if len(deltas) < 3 else
"sign varies across pairs" if min(deltas) < 0 < max(deltas) else "consistent sign"}
result["metrics"][metric] = {**summarize(deltas), "paired_absolute_deltas": absolute}
# Keep offered load and delivery/loss next to CPU; no normalized CPU claim when windows differ.
result["cpu_efficiency"] = "not estimated: verify offered load, delivery/loss and host/load window alignment before normalizing CPU"
return result
Expand Down Expand Up @@ -84,15 +89,120 @@ def criterion_baseline(directory, discovery, benchmark):
shutil.copytree(sources[0], destination, dirs_exist_ok=True)


# A paired case beyond this change is flagged, and fails the nightly only when its rounds agree.
# Loose until A/A runs on the hosted runner measure its noise floor.
THRESHOLD_PCT = 10
# Each alternating round costs seconds per case, so a noisy night confirms only its worst slowdowns.
CONFIRMATIONS = 20


def criterion_means(home):
"""Map each (target, full ID) measured under HOME/<package>/<target>/ to its mean nanoseconds."""
means = {}
for path in home.glob("*/*/**/new/estimates.json"):
benchmark = json.loads(path.with_name("benchmark.json").read_text())["full_id"]
if not isinstance(benchmark, str) or not benchmark or any(c in benchmark for c in "\t\r\n"):
raise ValueError(f"invalid Criterion full_id in {path}: {benchmark!r}")
key = ("/".join(path.relative_to(home).parts[:2]), benchmark)
if key in means:
raise ValueError(f"duplicate Criterion case {key} in {home}")
mean = json.loads(path.read_text())["mean"]["point_estimate"]
if isinstance(mean, bool) or not isinstance(mean, (int, float)) or not math.isfinite(mean) or mean <= 0:
raise ValueError(f"invalid Criterion mean in {path}: {mean!r}")
means[key] = mean
return means


def percent(base, current):
return 100 * (current / base - 1)


def criterion_screen(run):
"""The quick screen's slowdowns past the threshold, worst first, up to the confirmation budget."""
base, current = criterion_means(run / "criterion-base"), criterion_means(run / "criterion")
slower = sorted(((percent(base[key], current[key]), key) for key in base.keys() & current.keys()), reverse=True)
return [key for delta, key in slower if delta > THRESHOLD_PCT][:CONFIRMATIONS]


def criterion_paired(run):
"""Classify every case from the quick screen and the alternating confirmation rounds."""
metadata = json.loads((run / "metadata.json").read_text())
rounds = metadata["rounds"]
base, current = criterion_means(run / "criterion-base"), criterion_means(run / "criterion")
unavailable = []
if (run / "paired-unavailable.tsv").exists():
for line in (run / "paired-unavailable.tsv").read_text().splitlines():
target, reason = line.split("\t", 1)
unavailable.append({"target": target, "reason": reason})
blocked = {entry["target"] for entry in unavailable}
confirmed = {}
lines = (run / "confirm.tsv").read_text().splitlines() if (run / "confirm.tsv").exists() else []
for index, line in enumerate(lines, 1):
key = tuple(line.split("\t", 1))
if key[0] in blocked:
continue
deltas = []
for number in range(1, rounds + 1):
sides = [criterion_means(run / "confirm" / str(index) / f"round-{number}" / side).get(key)
for side in ("base", "current")]
if None in sides:
raise ValueError(f"confirmation round {number} did not measure {key}")
deltas.append(percent(*sides))
confirmed[key] = summarize(deltas)
cases = []
for key in sorted(base.keys() | current.keys()):
target, benchmark = key
case = {"target": target, "id": benchmark, "base_ns": base.get(key), "current_ns": current.get(key)}
if key not in current:
case["verdict"] = "removed"
elif key not in base:
case["verdict"] = "unavailable" if target in blocked else "new"
else:
case["delta_pct"] = delta = percent(base[key], current[key])
if key in confirmed:
case["rounds"] = confirmed[key]
# Every round strictly slower, so a tie or a single fast round rejects it at any round count.
slower = all(delta > 0 for delta in case["rounds"]["paired_deltas_pct"])
case["verdict"] = "regression" if slower and case["rounds"]["delta_pct"] > THRESHOLD_PCT else "noise"
elif delta > THRESHOLD_PCT:
case["verdict"] = "unavailable" if target in blocked else "unconfirmed"
elif delta < -THRESHOLD_PCT:
case["verdict"] = "faster"
else:
case["verdict"] = "unchanged"
cases.append(case)
return {"commit": metadata["revision"], "base": metadata["base"], "threshold_pct": THRESHOLD_PCT,
"rounds": rounds, "unavailable": unavailable, "cases": cases}


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--criterion-cases", action="store_true", help="list full IDs from isolated discovery samples")
parser.add_argument("--criterion-baseline", nargs=2, metavar=("DISCOVERY", "ID"), help="align an exact baseline with current discovery")
parser.add_argument("--criterion-screen", action="store_true", help="list a paired run's slowdowns to confirm")
parser.add_argument("--criterion-paired", action="store_true", help="classify a paired run's cases into paired.json")
parser.add_argument("directory", type=Path)
args = parser.parse_args()
if args.criterion_cases:
print("\n".join(criterion_cases(args.directory)))
return
if args.criterion_screen:
for target, benchmark in criterion_screen(args.directory):
print(f"{target}\t{benchmark}")
return
if args.criterion_paired:
result = criterion_paired(args.directory)
(args.directory / "paired.json").write_text(json.dumps(result, indent=2) + "\n")
verdicts = Counter(case["verdict"] for case in result["cases"])
print(f"\nPaired comparison against {result['base']}: "
+ ", ".join(f"{count} {verdict}" for verdict, count in sorted(verdicts.items())))
for case in result["cases"]:
if case["verdict"] == "regression":
print(f"REGRESSION {case['target']} {case['id']}: "
f"{case['rounds']['delta_pct']:+.1f}% over {result['rounds']} rounds")
for entry in result["unavailable"]:
print(f"UNAVAILABLE {entry['target']}: {entry['reason']}")
return
if args.criterion_baseline:
discovery, benchmark = args.criterion_baseline
criterion_baseline(args.directory, Path(discovery), benchmark)
Expand Down
Loading
Loading