1- # A/B Benchmark Runbook — baseline vs. standard mod set + JVM flags
2-
3- ** What this measures:** whether the ` claude/standard-install-mods-yfox41 ` changes
4- (required perf mods: Sodium, Lithium, FerriteCore, ModernFix-mVUS, Entity Culling,
5- ImmediatelyFast + Aikar-style JVM GC flags) actually improve FPS, memory headroom,
6- and smoothness at 1–4 concurrent players on a 16GB Steam Deck — before merging to main.
1+ # A/B Benchmark Runbook — pre- #94 baseline vs. current main (standard mod set + JVM flags)
2+
3+ ** What this measures:** PR #94 (` claude/standard-install-mods-yfox41 ` , required perf
4+ mods Sodium/Lithium/FerriteCore/ModernFix-mVUS/Entity Culling/ImmediatelyFast +
5+ Aikar-style JVM GC flags) ** already merged to ` main ` on 2026-07-18** and has shipped
6+ in every install since, without a formal before/after measurement. This runbook
7+ retroactively measures whether that change actually improved FPS, memory headroom,
8+ and smoothness at 1–4 concurrent players on a 16GB Steam Deck.
9+
10+ ** Because the change is already on ` main ` , "the existing install" is no longer a
11+ valid baseline** — it already has the mods/flags. ** Both phases now require a fresh
12+ install from a specific git ref:** Phase A installs from commit ` 2d5d321 ` (main, the
13+ commit immediately before PR #94 merged); Phase C installs from current ` main ` /HEAD.
14+ There is no separate branch left to check out — PR #94 's content lives permanently
15+ in ` main ` now.
716
817** Who does what:** a Claude Code session (the ** driver** ) runs on/SSH'd into the Deck
918from the repo checkout and executes this runbook top to bottom, including creating
@@ -50,7 +59,41 @@ incrementally into `$BENCH/RESULTS.md` (copied from `RESULTS-TEMPLATE.md`).
5059 ` bash tests/benchmark/sampler.sh run /tmp/sampler-smoke & sleep 5; bash tests/benchmark/sampler.sh stop /tmp/sampler-smoke; head -3 /tmp/sampler-smoke/sampler.csv `
5160 — expect populated ` gpu_busy_pct ` and ` apu_temp_mc ` columns on real Deck hardware.
5261
53- ## Phase A step 0 — Baseline inventory (BEFORE touching anything)
62+ ## Phase A step -1 — Back up the current world, then torch + install the pre- #94 baseline
63+
64+ ** Why this step exists:** if ` BenchWorld ` already exists on the currently-installed
65+ (post- #94 ) tree — e.g. from the pre-flight scouting pass — it must survive the torch
66+ below, since re-scouting bearings would be wasted work (the terrain is seed-determined,
67+ not mod/flag-determined).
68+
69+ 1 . ** Back up whatever world already exists, before touching anything:**
70+ ` cp -r ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves/BenchWorld $BENCH/world-backup/ `
71+ (skip if no world exists yet — Phase A's "World + settings standardization" step
72+ below creates one fresh instead). Verify: ` du -sh $BENCH/world-backup/BenchWorld `
73+ is non-trivial (>1MB) if it ran.
74+ 2 . ** Torch:** ` cd <repo checkout> && ./uninstall-minecraft-splitscreen.sh `
75+ - "Keep my data?" → ** ` n ` ** (full wipe — a reinstall over surviving instances
76+ silently enters * update mode* , which would invalidate the fresh-install
77+ comparison).
78+ - "Are you sure…" → ** ` y ` ** .
79+ - Verify: ` ~/.local/share/PolyMC ` and ` ~/.local/share/PrismLauncher ` are gone.
80+ 3 . ** Steam shortcut: leave it alone.** The uninstaller never removes it; it points at
81+ ` ~/.local/share/PolyMC/minecraftSplitscreen.sh ` , which the reinstall recreates at
82+ the same path.
83+ 4 . ** Checkout the pre- #94 baseline ref:**
84+ ` git fetch origin && git checkout 2d5d321 ` (detached HEAD — this is main immediately
85+ before PR #94 merged; do not commit anything on it).
86+ 5 . ** Install:** ` ./install-minecraft-splitscreen.sh ` (local checkout at ` 2d5d321 `
87+ supplies ` modules/ ` /` mods.conf ` /launcher, which at this ref has neither the 6 perf
88+ mods nor the GC flags). Prompt answers:
89+ - Minecraft version → record whatever is offered; this becomes the ** Phase A
90+ version** other phases must match or explicitly diverge from.
91+ - Custom mods → ` N ` . Steam integration → ` N ` (shortcut still exists). Desktop
92+ launcher → ` N ` .
93+ 6 . ** Restore the world, if step 1 backed one up:**
94+ ` mkdir -p ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves && cp -r $BENCH/world-backup/BenchWorld ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves/ `
95+
96+ ## Phase A step 0 — Baseline inventory (after the pre- #94 install, before the first cycle)
5497
5598```
5699for n in 1 2 3 4; do
63106
64107Record in RESULTS.md: the baseline mod list, the ** Minecraft version** (from
65108` mmc-pack.json ` — this is the version-match control for Phase B), and whether
66- ` instance.cfg ` contains GC flags in ` JvmArgs ` (expected: ** no** ).
109+ ` instance.cfg ` contains GC flags in ` JvmArgs ` (expected: ** no** — if this check
110+ finds them anyway, ` 2d5d321 ` was checked out wrong; STOP and re-verify the ref
111+ before proceeding, since the whole A/B depends on this being a real pre-change
112+ baseline).
67113
68114## MangoHud probe (Phase A step 1; repeated as Phase B step 7)
69115
@@ -78,9 +124,11 @@ Record in RESULTS.md: the baseline mod list, the **Minecraft version** (from
78124
79125## World + settings standardization (Phase A, before the first cycle)
80126
81- - Human creates world ** ` BenchWorld ` ** on Player 1's instance: ** seed ` 4815162342 ` ** ,
82- Creative, Normal difficulty, default world type, cheats ON. Enter it once, stand at
83- spawn ~ 2 min (initial worldgen), then quit.
127+ - ** If Phase A step -1 restored a backed-up ` BenchWorld ` ** , skip world creation —
128+ it already exists with cheats on and the confirmed seed. ** Otherwise** (first-ever
129+ run, no prior world), human creates world ** ` BenchWorld ` ** on Player 1's instance:
130+ ** seed ` 4815162342 ` ** , Creative, Normal difficulty, default world type, cheats ON.
131+ Enter it once, stand at spawn ~ 2 min (initial worldgen), then quit.
84132- Driver pins video settings identically in every instance (repeat after Phase B too):
85133
86134```
@@ -290,15 +338,17 @@ slot 1's player (cheats are on) runs e.g. `/tp @a 20000 120 20000` (Phase A 3P),
290338there. This guarantees every flight generates brand-new chunks (worldgen load, not
291339chunk-cache reload) despite reusing the same world.
292340
293- ## Phase A — Baseline (existing install, as-is )
341+ ## Phase A — Baseline (fresh install from ` 2d5d321 ` , pre- # 94 )
294342
295- 0 . Inventory (above). 1. MangoHud probe. 2. World + settings standardization.
343+ -1. Back up existing world, torch, install pre- #94 (above). 0. Inventory. 1. MangoHud
344+ probe. 2. World + settings standardization.
2963453–6. Cycles ` phaseA/1p ` → ` 2p ` → ` 3p ` → ` 4p ` per the protocol.
297- 7 . ** Back up the world + options** (MUST happen before Phase B):
346+ 7 . ** Back up the world + options** (MUST happen before Phase B — this re-backs-up
347+ the post-cycle state, superseding step -1's pre-cycle backup):
298348 ` cp -r ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves/BenchWorld $BENCH/world-backup/ `
299349 Verify: ` du -sh $BENCH/world-backup/BenchWorld ` is non-trivial (>1MB).
300350
301- ## Phase B — Torch + fresh install from the branch
351+ ## Phase B — Torch + fresh install from current ` main `
302352
3033531 . ** Pre-torch checklist** — driver verifies ALL, then asks the human to type ` TORCH ` :
304354 - [ ] ` $BENCH/world-backup/BenchWorld ` exists, >1MB
@@ -314,11 +364,12 @@ chunk-cache reload) despite reusing the same world.
3143643 . ** Steam shortcut: leave it alone.** The uninstaller never removes it; it points at
315365 ` ~/.local/share/PolyMC/minecraftSplitscreen.sh ` , which the reinstall recreates at
316366 the same path.
317- 4 . ** Checkout the branch:**
318- ` git fetch origin claude/standard-install-mods-yfox41 && git checkout claude/standard-install-mods-yfox41 && git pull `
319- 5 . ** Install:** ` REPO_REF=claude/standard-install-mods-yfox41 ./install-minecraft-splitscreen.sh `
320- (local checkout supplies modules/mods.conf/launcher; REPO_REF points the
321- always-remote accounts.json/token.enc at the branch). Prompt answers:
367+ 4 . ** Return to current ` main ` :**
368+ ` git checkout main && git fetch origin && git pull --ff-only ` (undoes Phase A
369+ step -1's detached-HEAD checkout at ` 2d5d321 ` ).
370+ 5 . ** Install:** ` ./install-minecraft-splitscreen.sh ` (local checkout at ` main `
371+ supplies the post- #94 ` modules/ ` /` mods.conf ` /launcher — the 6 perf mods + GC
372+ flags). Prompt answers:
322373 - Minecraft version → ** the Phase A version** if listed as supported; otherwise
323374 accept latest and record the version delta as a CONFOUND in RESULTS.md.
324375 - Custom mods → ` N ` . Steam integration → ` N ` (shortcut still exists).
@@ -336,7 +387,7 @@ chunk-cache reload) despite reusing the same world.
336387 Controlify + Fabric API); ` instance.cfg ` ` JvmArgs ` contains ` -XX:+UseG1GC ` . If
337388 either check fails, STOP — the A/B delta wouldn't measure what we think.
338389
339- ## Phase C — Branch benchmark
390+ ## Phase C — Current-main benchmark
340391
341392Cycles ` phaseC/1p ` → ` 2p ` → ` 3p ` → ` 4p ` , identical protocol, Phase C flight bearings.
342393
@@ -345,7 +396,7 @@ Cycles `phaseC/1p` → `2p` → `3p` → `4p`, identical protocol, Phase C fligh
345396For each N: ` bash tests/benchmark/summarize.sh $BENCH/phaseA/<N>p --compare $BENCH/phaseC/<N>p `
346397→ paste tables into RESULTS.md, then evaluate the gates:
347398
348- ** Hard gates — ALL must hold in Phase C, else NO merge :**
399+ ** Hard gates — ALL must hold in Phase C, else the change does not stand as validated :**
349400- 4P cycle completes with all 4 instances alive end-to-end; no oom-kill in
350401 dmesg/journal; no ` SLOT_DIED ` in the session log.
351402- 4P ` rss_sum_max_mb ` ≤ 12288 (12 GiB); ` memavail_min_mb ` ≥ 1024;
@@ -362,14 +413,18 @@ For each N: `bash tests/benchmark/summarize.sh $BENCH/phaseA/<N>p --compare $BEN
362413most (Sodium/Lithium/ModernFix), RSS should drop (FerriteCore), startup should feel
363414faster (ModernFix). A regression in any of these deserves investigation, not hand-waving.
364415
365- ** Decision:** all hard + soft pass → merge the branch to main. Any hard gate fails →
366- no merge; record the failing metric + cycle in RESULTS.md and open an issue.
416+ ** Decision:** all hard + soft pass → change stands validated on ` main ` , close #70 . Any
417+ hard gate fails → the already-shipped change on ` main ` has a real regression; record
418+ the failing metric + cycle in RESULTS.md and open an issue (revert or fix-forward is
419+ a maintainer call, not a merge decision, since it's already live).
367420
368421## Post-run recording (repo conventions)
369422
370- - Summary tables + verdict → commit as ` docs/BENCH-AB-<date>.md ` on the branch.
423+ - Summary tables + verdict → commit as ` docs/BENCH-AB-<date>.md ` directly on ` main `
424+ (there is no separate branch — PR #94 already merged; this is a retroactive
425+ validation of what's already shipped).
371426- Dated "Validation run" block in ` docs/SPEC.md ` §3b — a 4P pass also formally closes
372427 the D6 item "4 instances run concurrently without OOM (RAM within budget)".
373- - MEMORY.md: flip the two 2026-07-17 entries' Status lines with the verdict.
428+ - ` docs/ MEMORY.md` : flip the two 2026-07-17 entries' Status lines with the verdict.
374429- ` sessions/SESSION-<date>.md ` : narrative of the run.
375430- Raw CSVs stay on the Deck under ` $BENCH/ ` (not committed).
0 commit comments