Repository navigation
ci: retain nightly shipping and consumer size reports #5275
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
1479723
chore(quest): claim nightly artifact size report
kixelated 6847a8e
ci: retain nightly shipping and consumer size reports
kixelated 7d9f05e
Merge remote-tracking branch 'origin/main' into quest/m1/size-report
kixelated 34a1bb5
Keep nightly size history comparable and recoverable
kixelated 6be49cf
Merge remote-tracking branch 'origin/main' into quest/m1/size-report
kixelated 6bca03e
ci: measure nightly release uploads instead of rebuilding them
kixelated bc2e2f3
quest: complete the nightly size report
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Nightly artifact sizes | ||
|
|
||
| `just gh size-collect OUTPUT [RUN]` measures what a workflow run's release jobs | ||
| uploaded, plus the builds no release job makes: the minimal-feature FFI library | ||
| and real browser consumer bundles (raw, gzip and Brotli). `just gh size-report | ||
| OUTPUT [--history HISTORY]` compares the report with its previous report (5% | ||
| growth) and the newest report at least 30 days old (10% growth). | ||
|
|
||
| Nightly passes its own run, so the sizes are the shipping artifacts for every | ||
| release target, Linux `x86_64` and `aarch64` included, without rebuilding them. Any | ||
| retained run works locally too. Each release job's `Build` step durations are | ||
| recorded and summarized but never compared, since hosted build time is noisy. | ||
| The Go mirror's 100 MiB file limit is enforced by the Go FFI package step; | ||
| the report tracks those staticlibs. | ||
|
|
||
| Keys name the artifact (which carries the target) and the filename without its | ||
| release version, so a version bump keeps the history comparable. Duplicate keys | ||
| and missing release artifacts are refused. An artifact-set change is refused | ||
| too: review the new set, then start a fresh history by reporting without one. | ||
|
|
||
| A regression fails Nightly and reaches its existing alert workflow. The job | ||
| summary carries the byte counts, build times and FFI crate attribution, the | ||
| latter from the symbol-preserving `profiling` profile. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| // Measure consumer bundles from published dist entry points, including only static chunks. | ||
|
|
||
| import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; | ||
| import { join, resolve } from "node:path"; | ||
| import { brotliCompressSync, gzipSync } from "node:zlib"; | ||
| import { build } from "esbuild"; | ||
|
|
||
| const root = process.cwd(); | ||
| if (!process.argv[2]) throw new Error("usage: bundles.ts OUTPUT"); | ||
| const out = resolve(process.argv[2]); | ||
| mkdirSync(out, { recursive: true }); | ||
| const sizes: Record<string, number> = {}; | ||
| for (const [name, code] of Object.entries({ | ||
| net: 'export * from "@moq/net";', | ||
| watch: 'import "@moq/watch/element";', | ||
| "watch-ui": 'import "@moq/watch/element"; import "@moq/watch/ui";', | ||
| publish: 'import "@moq/publish/element";', | ||
| })) { | ||
| const result = await build({ | ||
| stdin: { contents: code, resolveDir: root, sourcefile: `${name}.js` }, | ||
| bundle: true, | ||
| minify: true, | ||
| splitting: true, | ||
| format: "esm", | ||
| platform: "browser", | ||
| metafile: true, | ||
| write: false, | ||
| outdir: join(out, name), | ||
| plugins: [ | ||
| { | ||
| name: "built-packages", | ||
| setup(b) { | ||
| b.onResolve({ filter: /^@moq\// }, ({ path }) => { | ||
| const [, pkg, ...subpath] = path.split("/"); | ||
| const dir = join(root, "js", pkg, "dist"); | ||
| // Only packages outside this workspace fall through to node_modules. | ||
| if (!existsSync(join(root, "js", pkg, "package.json"))) return; | ||
| const manifest: { exports: Record<string, string | { default: string }> } = JSON.parse( | ||
| readFileSync(join(dir, "package.json"), "utf8"), | ||
| ); | ||
| const entry = manifest.exports[subpath.length ? `./${subpath.join("/")}` : "."]; | ||
| if (!entry) throw new Error(`missing built export: ${path}`); | ||
| return { path: resolve(dir, typeof entry === "string" ? entry : entry.default) }; | ||
| }); | ||
| }, | ||
| }, | ||
| ], | ||
| }); | ||
| if (!result.metafile || !result.outputFiles) throw new Error(`bundle outputs missing: ${name}`); | ||
| writeFileSync(join(out, `${name}.metafile.json`), JSON.stringify(result.metafile, null, 2)); | ||
| const outputs = result.metafile.outputs; | ||
| const first = new Set<string>(); | ||
| const visit = (path: string) => { | ||
| if (first.has(path)) return; | ||
| first.add(path); | ||
| for (const dep of outputs[path].imports) if (!dep.external && dep.kind !== "dynamic-import") visit(dep.path); | ||
| }; | ||
| for (const [path, output] of Object.entries(outputs)) if (output.entryPoint === `${name}.js`) visit(path); | ||
| if (!first.size) throw new Error(`entry output missing: ${name}`); | ||
| for (const file of result.outputFiles) { | ||
| if (![...first].some((path) => resolve(root, path) === file.path)) continue; | ||
| for (const [format, bytes] of Object.entries({ | ||
| raw: file.contents.length, | ||
| gzip: gzipSync(file.contents).length, | ||
| brotli: brotliCompressSync(file.contents).length, | ||
| })) | ||
| sizes[`js/${name}/${format}`] = (sizes[`js/${name}/${format}`] ?? 0) + bytes; | ||
| } | ||
| if (!sizes[`js/${name}/raw`]) throw new Error(`no first-load bytes: ${name}`); | ||
| } | ||
| writeFileSync(join(out, "bytes.json"), JSON.stringify(sizes, null, 2)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| #!/usr/bin/env bash | ||
| # Measure a workflow run's release uploads and the builds no release job makes, retaining diagnostics. | ||
| set -euo pipefail | ||
| out=$(realpath -m "${1:?usage: collect.sh OUTPUT [RUN]}") | ||
| run=${2:-} | ||
| mkdir -p "$out/native" | ||
| target=$(cargo metadata --locked --format-version 1 --no-deps | jq -r .target_directory) | ||
| case $(uname -s) in Linux) dylib=so ;; Darwin) dylib=dylib ;; *) | ||
| echo 'native size capture requires Linux or macOS' >&2 | ||
| exit 1 | ||
| ;; | ||
| esac | ||
| # Release jobs ship default features only; this shows what the feature gates save. | ||
| cargo build --locked --release -p moq-ffi --no-default-features | ||
| cp "$target/release/libmoq_ffi.$dylib" "$out/native/libmoq_ffi-minimal.$dylib" | ||
| cp "$target/release/libmoq_ffi.a" "$out/native/libmoq_ffi-minimal.a" | ||
| bun install --frozen-lockfile | ||
| just js build | ||
| bun sh/size/bundles.ts "$out/js" | ||
| # Symbols are necessary only for attribution; they never replace the measured shipping sizes. | ||
| cargo bloat --locked --profile profiling -p moq-ffi --crates --lib >"$out/ffi-bloat.txt" | ||
| python3 - "$out" "$run" <<'PY' | ||
| import datetime, json, pathlib, subprocess, sys | ||
| sys.path.insert(0, "sh/size") | ||
| import release | ||
| out, run = pathlib.Path(sys.argv[1]), sys.argv[2] | ||
| host = next(line.split(': ', 1)[1] for line in subprocess.check_output(['rustc', '-vV'], text=True).splitlines() if line.startswith('host: ')) | ||
| sizes = {f'native/{host}/{p.name}': p.stat().st_size for p in (out / 'native').iterdir()} | ||
| sizes.update(json.loads((out / 'js/bytes.json').read_text())) | ||
| report = {'version': 1, 'date': datetime.date.today().isoformat(), | ||
| 'revision': subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip(), | ||
| 'bytes': sizes} | ||
| if run: | ||
| release.download(run, out / 'release') | ||
| sizes.update(release.sizes(out / 'release')) | ||
| report['seconds'] = release.seconds(run) | ||
| (out / 'sizes.json').write_text(json.dumps(report, indent=2) + '\n') | ||
| PY | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| #!/usr/bin/env bash | ||
| # Load the latest completed history, then retain the new report even if growth fails. | ||
| set -euo pipefail | ||
| out=${1:?usage: nightly.sh OUTPUT RUN} | ||
| run=${2:?usage: nightly.sh OUTPUT RUN} | ||
| mkdir -p "$out" | ||
| history=() | ||
| # A packaging or collector failure may leave no history artifact in the newest run. | ||
| # Search retained completed runs rather than silently discarding the older baseline. | ||
| runs=$(gh run list --workflow nightly.yml --branch main --limit 45 --status completed --json databaseId --jq '.[].databaseId') | ||
| while IFS= read -r prior; do | ||
| [[ -n "$prior" ]] || continue | ||
| artifact=$(gh api "repos/{owner}/{repo}/actions/runs/$prior/artifacts" --jq '.artifacts[] | select(.name == "nightly-size-history" and .expired == false) | .id') | ||
| if [[ -n "$artifact" ]]; then | ||
| previous="$out/previous/$prior" | ||
| gh run download "$prior" --name nightly-size-history --dir "$previous" | ||
| # Failed collectors can upload diagnostics without producing a history. | ||
| if [[ -f "$previous/history.json" ]]; then | ||
| history=(--history "$previous/history.json") | ||
| break | ||
| fi | ||
| fi | ||
| done <<<"$runs" | ||
| sh/size/collect.sh "$out" "$run" | ||
| status=0 | ||
| python3 sh/size/report.py "$out" "${history[@]}" || status=$? | ||
| { | ||
| cat "$out/summary.md" | ||
| printf '\n<details><summary>FFI crate attribution</summary>\n\n```text\n' | ||
| cat "$out/ffi-bloat.txt" | ||
| printf '\n```\n</details>\n' | ||
| } >>"${GITHUB_STEP_SUMMARY:-/dev/stdout}" | ||
| exit "$status" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| #!/usr/bin/env python3 | ||
| """Exercise history recovery without network access or native builds.""" | ||
| import json | ||
| import os | ||
| from pathlib import Path | ||
| import subprocess | ||
| import tempfile | ||
| import unittest | ||
|
|
||
| SCRIPT = Path(__file__).with_name("nightly.sh").resolve() | ||
| REPORT = Path(__file__).with_name("report.py").resolve() | ||
|
|
||
|
|
||
| class History(unittest.TestCase): | ||
| def test_skips_retained_artifact_without_history(self): | ||
| with tempfile.TemporaryDirectory() as directory: | ||
| root = Path(directory) | ||
| (root / "bin").mkdir() | ||
| (root / "sh/size").mkdir(parents=True) | ||
| (root / "sh/size/report.py").symlink_to(REPORT) | ||
| gh = root / "bin/gh" | ||
| gh.write_text('''#!/usr/bin/env python3 | ||
| import json, pathlib, sys | ||
| args = sys.argv[1:] | ||
| if args[:2] == ['run', 'list']: | ||
| print('2\\n1') | ||
| elif args[0] == 'api': | ||
| print('artifact') | ||
| elif args[:2] == ['run', 'download']: | ||
| out = pathlib.Path(args[args.index('--dir') + 1]) | ||
| out.mkdir(parents=True) | ||
| if args[2] == '1': | ||
| (out / 'history.json').write_text(json.dumps([{'version': 1, 'date': '2026-10-09', 'bytes': {'a': 100}}])) | ||
| else: | ||
| (out / 'sizes.json').write_text('{}') | ||
| else: | ||
| raise SystemExit(args) | ||
| ''') | ||
| gh.chmod(0o755) | ||
| collect = root / "sh/size/collect.sh" | ||
| collect.write_text('''#!/usr/bin/env bash | ||
| set -eu | ||
| printf '%s' '{"version":1,"date":"2026-10-10","bytes":{"a":100}}' >"$1/sizes.json" | ||
| printf 'attribution' >"$1/ffi-bloat.txt" | ||
| ''') | ||
| collect.chmod(0o755) | ||
| env = dict(os.environ, PATH=f"{root / 'bin'}:{os.environ['PATH']}", | ||
| GITHUB_STEP_SUMMARY=str(root / "summary")) | ||
| result = subprocess.run(["bash", str(SCRIPT), "out", "3"], cwd=root, | ||
| env=env, capture_output=True, text=True) | ||
| self.assertEqual(result.returncode, 0, result.stderr) | ||
| history = json.loads((root / "out/history.json").read_text()) | ||
| self.assertEqual([row['date'] for row in history], ['2026-10-09', '2026-10-10']) | ||
|
|
||
|
|
||
| unittest.main() |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The scheduled
sizesjob runs only onubuntu-24.04-arm, and this records only the host's minimal-feature artifacts. Consequently, an x86_64-specific size regression inmoq-ffi --no-default-featuresis invisible; the downloaded x86_64 release artifacts use default features and do not cover it. Build or cross-build the minimal cdylib/staticlib for both Linux architectures before treating the report as x86_64 and aarch64 coverage.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Declining. Nothing ships the minimal-feature build. It's there to show what the feature gates save, which hardly depends on the architecture. The x86_64 shipping builds (Go staticlib and cdylib, wheel, relay, CLI, moq-c) are all measured from the release uploads. Adding a cross-compiled x86_64 fat-LTO build to an aarch64 runner would cost about 10 minutes per night to watch an artifact nobody downloads.
(Written by Claude Opus 5.5)