Skip to content

Commit 9d75966

Browse files
Scottclaude
andcommitted
docs(#70): fix RUNBOOK for PR #94 having already merged to main
RUNBOOK.md described Phase A as "the existing install, as-is" and Phase C as "checkout out claude/standard-install-mods-yfox41" — but that branch merged to main on 2026-07-18 (PR #94) and no longer exists separately. The Deck's current install already carries the full perf mod set + Aikar GC flags, discovered while starting the baseline manifest inventory: instance.cfg already had -XX:+UseG1GC and all 6 required perf mods were already present, which should have been absent for a true baseline. Redefine the phases around git refs instead of "current install" vs "a branch": Phase A now does its own torch + install from 2d5d321 (main, immediately before PR #94 merged) after backing up any already-scouted world; Phase B/C install from current main/HEAD. Fixes the Decision/ post-run-recording language that assumed a branch to merge — this run retroactively validates a change that's already shipped, not a merge gate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012DrqhXXJ6ZxZw3Ry7oCTRV
1 parent 30719f2 commit 9d75966

2 files changed

Lines changed: 84 additions & 28 deletions

File tree

docs/MEMORY.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -148,8 +148,9 @@ on 16GB) already matches the doc's memory-budget guidance, so left untouched;
148148
kept `MCSS_MIN_MEM_MB=512` over the doc's `-Xms2G` because `-XX:+AlwaysPreTouch`
149149
would commit 4×2G = 8GiB at launch.
150150

151-
**Status:** pushed to `claude/standard-install-mods-yfox41`, awaiting Deck
152-
validation per this project's standing rule (SPEC §3a/§3b).
151+
**Status:** merged to `main` 2026-07-18 (PR #94), still awaiting the Deck A/B
152+
validation run per this project's standing rule (SPEC §3a/§3b) — the branch
153+
itself no longer exists as a separate ref; `main` already carries this change.
153154

154155
---
155156

tests/benchmark/RUNBOOK.md

Lines changed: 81 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,18 @@
1-
# A/B Benchmark Runbook — baseline vs. standard mod set + JVM flags
2-
3-
**What this measures:** whether the `claude/standard-install-mods-yfox41` changes
4-
(required perf mods: Sodium, Lithium, FerriteCore, ModernFix-mVUS, Entity Culling,
5-
ImmediatelyFast + Aikar-style JVM GC flags) actually improve FPS, memory headroom,
6-
and smoothness at 1–4 concurrent players on a 16GB Steam Deck — before merging to main.
1+
# A/B Benchmark Runbook — pre-#94 baseline vs. current main (standard mod set + JVM flags)
2+
3+
**What this measures:** PR #94 (`claude/standard-install-mods-yfox41`, required perf
4+
mods Sodium/Lithium/FerriteCore/ModernFix-mVUS/Entity Culling/ImmediatelyFast +
5+
Aikar-style JVM GC flags) **already merged to `main` on 2026-07-18** and has shipped
6+
in every install since, without a formal before/after measurement. This runbook
7+
retroactively measures whether that change actually improved FPS, memory headroom,
8+
and smoothness at 1–4 concurrent players on a 16GB Steam Deck.
9+
10+
**Because the change is already on `main`, "the existing install" is no longer a
11+
valid baseline** — it already has the mods/flags. **Both phases now require a fresh
12+
install from a specific git ref:** Phase A installs from commit `2d5d321` (main, the
13+
commit immediately before PR #94 merged); Phase C installs from current `main`/HEAD.
14+
There is no separate branch left to check out — PR #94's content lives permanently
15+
in `main` now.
716

817
**Who does what:** a Claude Code session (the **driver**) runs on/SSH'd into the Deck
918
from the repo checkout and executes this runbook top to bottom, including creating
@@ -50,7 +59,41 @@ incrementally into `$BENCH/RESULTS.md` (copied from `RESULTS-TEMPLATE.md`).
5059
`bash tests/benchmark/sampler.sh run /tmp/sampler-smoke & sleep 5; bash tests/benchmark/sampler.sh stop /tmp/sampler-smoke; head -3 /tmp/sampler-smoke/sampler.csv`
5160
— expect populated `gpu_busy_pct` and `apu_temp_mc` columns on real Deck hardware.
5261

53-
## Phase A step 0 — Baseline inventory (BEFORE touching anything)
62+
## Phase A step -1 — Back up the current world, then torch + install the pre-#94 baseline
63+
64+
**Why this step exists:** if `BenchWorld` already exists on the currently-installed
65+
(post-#94) tree — e.g. from the pre-flight scouting pass — it must survive the torch
66+
below, since re-scouting bearings would be wasted work (the terrain is seed-determined,
67+
not mod/flag-determined).
68+
69+
1. **Back up whatever world already exists, before touching anything:**
70+
`cp -r ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves/BenchWorld $BENCH/world-backup/`
71+
(skip if no world exists yet — Phase A's "World + settings standardization" step
72+
below creates one fresh instead). Verify: `du -sh $BENCH/world-backup/BenchWorld`
73+
is non-trivial (>1MB) if it ran.
74+
2. **Torch:** `cd <repo checkout> && ./uninstall-minecraft-splitscreen.sh`
75+
- "Keep my data?" → **`n`** (full wipe — a reinstall over surviving instances
76+
silently enters *update mode*, which would invalidate the fresh-install
77+
comparison).
78+
- "Are you sure…" → **`y`**.
79+
- Verify: `~/.local/share/PolyMC` and `~/.local/share/PrismLauncher` are gone.
80+
3. **Steam shortcut: leave it alone.** The uninstaller never removes it; it points at
81+
`~/.local/share/PolyMC/minecraftSplitscreen.sh`, which the reinstall recreates at
82+
the same path.
83+
4. **Checkout the pre-#94 baseline ref:**
84+
`git fetch origin && git checkout 2d5d321` (detached HEAD — this is main immediately
85+
before PR #94 merged; do not commit anything on it).
86+
5. **Install:** `./install-minecraft-splitscreen.sh` (local checkout at `2d5d321`
87+
supplies `modules/`/`mods.conf`/launcher, which at this ref has neither the 6 perf
88+
mods nor the GC flags). Prompt answers:
89+
- Minecraft version → record whatever is offered; this becomes the **Phase A
90+
version** other phases must match or explicitly diverge from.
91+
- Custom mods → `N`. Steam integration → `N` (shortcut still exists). Desktop
92+
launcher → `N`.
93+
6. **Restore the world, if step 1 backed one up:**
94+
`mkdir -p ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves && cp -r $BENCH/world-backup/BenchWorld ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves/`
95+
96+
## Phase A step 0 — Baseline inventory (after the pre-#94 install, before the first cycle)
5497

5598
```
5699
for n in 1 2 3 4; do
@@ -63,7 +106,10 @@ done
63106

64107
Record in RESULTS.md: the baseline mod list, the **Minecraft version** (from
65108
`mmc-pack.json` — this is the version-match control for Phase B), and whether
66-
`instance.cfg` contains GC flags in `JvmArgs` (expected: **no**).
109+
`instance.cfg` contains GC flags in `JvmArgs` (expected: **no** — if this check
110+
finds them anyway, `2d5d321` was checked out wrong; STOP and re-verify the ref
111+
before proceeding, since the whole A/B depends on this being a real pre-change
112+
baseline).
67113

68114
## MangoHud probe (Phase A step 1; repeated as Phase B step 7)
69115

@@ -78,9 +124,11 @@ Record in RESULTS.md: the baseline mod list, the **Minecraft version** (from
78124

79125
## World + settings standardization (Phase A, before the first cycle)
80126

81-
- Human creates world **`BenchWorld`** on Player 1's instance: **seed `4815162342`**,
82-
Creative, Normal difficulty, default world type, cheats ON. Enter it once, stand at
83-
spawn ~2 min (initial worldgen), then quit.
127+
- **If Phase A step -1 restored a backed-up `BenchWorld`**, skip world creation —
128+
it already exists with cheats on and the confirmed seed. **Otherwise** (first-ever
129+
run, no prior world), human creates world **`BenchWorld`** on Player 1's instance:
130+
**seed `4815162342`**, Creative, Normal difficulty, default world type, cheats ON.
131+
Enter it once, stand at spawn ~2 min (initial worldgen), then quit.
84132
- Driver pins video settings identically in every instance (repeat after Phase B too):
85133

86134
```
@@ -290,15 +338,17 @@ slot 1's player (cheats are on) runs e.g. `/tp @a 20000 120 20000` (Phase A 3P),
290338
there. This guarantees every flight generates brand-new chunks (worldgen load, not
291339
chunk-cache reload) despite reusing the same world.
292340

293-
## Phase A — Baseline (existing install, as-is)
341+
## Phase A — Baseline (fresh install from `2d5d321`, pre-#94)
294342

295-
0. Inventory (above). 1. MangoHud probe. 2. World + settings standardization.
343+
-1. Back up existing world, torch, install pre-#94 (above). 0. Inventory. 1. MangoHud
344+
probe. 2. World + settings standardization.
296345
3–6. Cycles `phaseA/1p``2p``3p``4p` per the protocol.
297-
7. **Back up the world + options** (MUST happen before Phase B):
346+
7. **Back up the world + options** (MUST happen before Phase B — this re-backs-up
347+
the post-cycle state, superseding step -1's pre-cycle backup):
298348
`cp -r ~/.local/share/PolyMC/instances/latestUpdate-1/.minecraft/saves/BenchWorld $BENCH/world-backup/`
299349
Verify: `du -sh $BENCH/world-backup/BenchWorld` is non-trivial (>1MB).
300350

301-
## Phase B — Torch + fresh install from the branch
351+
## Phase B — Torch + fresh install from current `main`
302352

303353
1. **Pre-torch checklist** — driver verifies ALL, then asks the human to type `TORCH`:
304354
- [ ] `$BENCH/world-backup/BenchWorld` exists, >1MB
@@ -314,11 +364,12 @@ chunk-cache reload) despite reusing the same world.
314364
3. **Steam shortcut: leave it alone.** The uninstaller never removes it; it points at
315365
`~/.local/share/PolyMC/minecraftSplitscreen.sh`, which the reinstall recreates at
316366
the same path.
317-
4. **Checkout the branch:**
318-
`git fetch origin claude/standard-install-mods-yfox41 && git checkout claude/standard-install-mods-yfox41 && git pull`
319-
5. **Install:** `REPO_REF=claude/standard-install-mods-yfox41 ./install-minecraft-splitscreen.sh`
320-
(local checkout supplies modules/mods.conf/launcher; REPO_REF points the
321-
always-remote accounts.json/token.enc at the branch). Prompt answers:
367+
4. **Return to current `main`:**
368+
`git checkout main && git fetch origin && git pull --ff-only` (undoes Phase A
369+
step -1's detached-HEAD checkout at `2d5d321`).
370+
5. **Install:** `./install-minecraft-splitscreen.sh` (local checkout at `main`
371+
supplies the post-#94 `modules/`/`mods.conf`/launcher — the 6 perf mods + GC
372+
flags). Prompt answers:
322373
- Minecraft version → **the Phase A version** if listed as supported; otherwise
323374
accept latest and record the version delta as a CONFOUND in RESULTS.md.
324375
- Custom mods → `N`. Steam integration → `N` (shortcut still exists).
@@ -336,7 +387,7 @@ chunk-cache reload) despite reusing the same world.
336387
Controlify + Fabric API); `instance.cfg` `JvmArgs` contains `-XX:+UseG1GC`. If
337388
either check fails, STOP — the A/B delta wouldn't measure what we think.
338389

339-
## Phase C — Branch benchmark
390+
## Phase C — Current-main benchmark
340391

341392
Cycles `phaseC/1p``2p``3p``4p`, identical protocol, Phase C flight bearings.
342393

@@ -345,7 +396,7 @@ Cycles `phaseC/1p` → `2p` → `3p` → `4p`, identical protocol, Phase C fligh
345396
For each N: `bash tests/benchmark/summarize.sh $BENCH/phaseA/<N>p --compare $BENCH/phaseC/<N>p`
346397
→ paste tables into RESULTS.md, then evaluate the gates:
347398

348-
**Hard gates — ALL must hold in Phase C, else NO merge:**
399+
**Hard gates — ALL must hold in Phase C, else the change does not stand as validated:**
349400
- 4P cycle completes with all 4 instances alive end-to-end; no oom-kill in
350401
dmesg/journal; no `SLOT_DIED` in the session log.
351402
- 4P `rss_sum_max_mb` ≤ 12288 (12 GiB); `memavail_min_mb` ≥ 1024;
@@ -362,14 +413,18 @@ For each N: `bash tests/benchmark/summarize.sh $BENCH/phaseA/<N>p --compare $BEN
362413
most (Sodium/Lithium/ModernFix), RSS should drop (FerriteCore), startup should feel
363414
faster (ModernFix). A regression in any of these deserves investigation, not hand-waving.
364415

365-
**Decision:** all hard + soft pass → merge the branch to main. Any hard gate fails →
366-
no merge; record the failing metric + cycle in RESULTS.md and open an issue.
416+
**Decision:** all hard + soft pass → change stands validated on `main`, close #70. Any
417+
hard gate fails → the already-shipped change on `main` has a real regression; record
418+
the failing metric + cycle in RESULTS.md and open an issue (revert or fix-forward is
419+
a maintainer call, not a merge decision, since it's already live).
367420

368421
## Post-run recording (repo conventions)
369422

370-
- Summary tables + verdict → commit as `docs/BENCH-AB-<date>.md` on the branch.
423+
- Summary tables + verdict → commit as `docs/BENCH-AB-<date>.md` directly on `main`
424+
(there is no separate branch — PR #94 already merged; this is a retroactive
425+
validation of what's already shipped).
371426
- Dated "Validation run" block in `docs/SPEC.md` §3b — a 4P pass also formally closes
372427
the D6 item "4 instances run concurrently without OOM (RAM within budget)".
373-
- MEMORY.md: flip the two 2026-07-17 entries' Status lines with the verdict.
428+
- `docs/MEMORY.md`: flip the two 2026-07-17 entries' Status lines with the verdict.
374429
- `sessions/SESSION-<date>.md`: narrative of the run.
375430
- Raw CSVs stay on the Deck under `$BENCH/` (not committed).

0 commit comments

Comments
 (0)