Switch CPU: frame-free leaves, hidden symbols, write-back elision, profile-guided build (+2-10% fps at 1020 MHz)

AGENTS.md

@@ -74,6 +74,20 @@ from the code. Performance tests state the GPU profile they ran at, and check th
 overrides the port's clocks. The target is 720p
 (1280x720 internal) at 30 fps.
 
+## Faster builds: PGO and the recompiler passes
+
+Optional profile-guided optimisation (`-DWWHD_PGO=GEN` to train, `USE` to build; [docs/pgo.md](docs/pgo.md)), **off by
+default**: a normal build and every release are not profiled (turning it on for players: issue #90). Results and dead ends in
+[docs/pgo-progress.md](docs/pgo-progress.md). A profile fits the exact
+code it was made from: remake it after a recompile with a changed recompiler or a change to a profiled runtime file. Never
+judge speed on a training build (several times slower), and never deploy one to a console nobody can restart: a crashing
+start-up needs the NRO put back by hand (FTP :5000 works when the game is not running). The write-back elision
+(`tools/recomp/calleeregs.py`, `python3 tools/recomp/test_calleeregs.py`) assumes function hooks read only argument
+registers; the recompiler checks that against the runtime's sources.
+
+Before any timing, check the log's `[dk] clocks:` line: sys-clk or a saved GPU profile other than the build's default makes
+numbers meaningless (docs/perf-test.md).
+
 ## The SD card without the debug server
 
 - Best: hekate's USB mass storage (UMS). The SD mounts at `/Volumes/SWITCH SD` (FAT32): plain `cp`, then

cmake/SwitchPgo.cmake

@@ -0,0 +1,83 @@
+# SwitchWakerHD (this fork): profile-guided optimisation of the Switch build (docs/pgo.md).
+#
+# The recompiled game is machine-translated PowerPC: its branches carry no layout information, so GCC lays
+# blocks out, allocates registers and decides what is cold from static guesses. A profile of the real game
+# on the console tells it which paths run.
+#
+#   WWHD_PGO=OFF  (default) nothing changes: a normal build.
+#   WWHD_PGO=GEN  a training build: the hot game files (code_hot_*.c) and the runtime's frame-thread code are
+#                 instrumented (-fprofile-generate, counters not atomic). It is slower and larger and is never
+#                 released. The debug server gets "pgo dump|reset|status" (runtime/src/platform/pgo_switch.cpp),
+#                 which writes the counters to sdmc:/switch/wwhd/pgo/*.gcda (tools/switch/pgo_train.py).
+#   WWHD_PGO=USE  the same files compiled with that profile (WWHD_PGO_DIR). Code the training never reached
+#                 is compiled as usual (-fprofile-partial-training), not for size.
+#
+# The profile only fits the exact sources it was made from: GCC compares a checksum of every function's flow
+# graph and ignores (with -Wno-coverage-mismatch) the profile of a function that changed. Make it again after a
+# recompile with a different recompiler, or a change to a profiled runtime file.
+set(WWHD_PGO "OFF" CACHE STRING "Switch: profile-guided optimisation: OFF, GEN (instrumented training build) or USE (docs/pgo.md)")
+set_property(CACHE WWHD_PGO PROPERTY STRINGS OFF GEN USE)
+set(WWHD_PGO_DIR "${CMAKE_SOURCE_DIR}/build/pgo" CACHE PATH "WWHD_PGO=USE: the folder with the .gcda files fetched from the console")
+set(WWHD_PGO_SCOPE "game;runtime" CACHE STRING "what the profile covers: game (code_hot_*.c) and/or runtime (runtime/src, the frame threads' code)")
+# extra flags for the USE build only, to try variants without editing this file (e.g. -fno-tracer -fno-peel-loops)
+set(WWHD_PGO_USE_FLAGS "" CACHE STRING "WWHD_PGO=USE: extra compiler flags for the profiled files")
+
+macro(switch_pgo_configure)
+  if(WWHD_SWITCH AND NOT WWHD_PGO STREQUAL "OFF")
+    if(NOT WWHD_PGO STREQUAL "GEN" AND NOT WWHD_PGO STREQUAL "USE")
+      message(FATAL_ERROR "WWHD_PGO must be OFF, GEN or USE")
+    endif()
+    set(_pgo_files)
+    if("game" IN_LIST WWHD_PGO_SCOPE)
+      file(GLOB _pgo_hot CONFIGURE_DEPENDS ${GEN_DIR}/code_hot_*.c)
+      list(APPEND _pgo_files ${_pgo_hot})
+    endif()
+    if("runtime" IN_LIST WWHD_PGO_SCOPE)
+      get_target_property(_pgo_all wwhd SOURCES)
+      foreach(_s ${_pgo_all})
+        # the runtime's own C++ (not the generated build info, the vendored libraries or the diagnostics)
+        if(_s MATCHES "runtime/src/.*\\.cpp$" AND NOT _s MATCHES "third_party")
+          list(APPEND _pgo_files ${_s})
+        endif()
+      endforeach()
+    endif()
+    # the object paths (and so the .gcda names) are relative to the build folder: a profile made in one build
+    # folder is found by another
+    set(_pgo_common -fprofile-prefix-path=${CMAKE_BINARY_DIR})
+    if(WWHD_PGO STREQUAL "GEN")
+      # not atomic: three cores increment the same counters, a few lost increments do not change a layout
+      # -fno-profile-values: no value profiling (indirect-call targets, divisors). Its counters are thread-local, and the
+      # runtime's first code (libnx's init hooks, the heap) runs before the console has set up thread-local storage;
+      # the generated code's indirect calls go through our own call-site caches anyway
+      set(_pgo_flags ${_pgo_common} -fprofile-generate=/pgo -fprofile-update=single -fno-profile-values)
+      target_compile_definitions(wwhd PRIVATE WWHD_PGO_GEN=1)
+      target_sources(wwhd PRIVATE runtime/src/platform/pgo_switch.cpp)
+      target_link_options(wwhd PRIVATE -fprofile-generate)
+    else()
+      if(NOT IS_DIRECTORY "${WWHD_PGO_DIR}")
+        message(FATAL_ERROR "WWHD_PGO=USE: ${WWHD_PGO_DIR} is not a folder (fetch the profile: tools/switch/pgo_train.py)")
+      endif()
+      # -fno-reorder-functions / -fno-profile-reorder-functions: the profile would rename the functions' sections
+      # (.text.hot.*, .text.sorted.*) and tools/switch/hot_sections.ld, which lists .text.<function>, would no
+      # longer put them together. Cold basic blocks are still moved out of the functions (.text.unlikely.*).
+      set(_pgo_flags ${_pgo_common} -fprofile-use=${WWHD_PGO_DIR} -fprofile-correction -fprofile-partial-training
+                     -fno-reorder-functions -fno-profile-reorder-functions -Wno-missing-profile -Wno-coverage-mismatch
+                     # -fprofile-use turns on loop unrolling, peeling and tail duplication: on one hot file (code_hot_012)
+                     # the object grew 29% (40,180 -> 51,900 bytes), and the hot code is already bigger than the A57's
+                     # 2 MiB L2. Without them it is +3%; the block partitioning moves cold blocks out (.text.unlikely).
+                     -fno-tracer -fno-unroll-loops -fno-peel-loops -freorder-blocks-and-partition
+                     ${WWHD_PGO_USE_FLAGS})
+    endif()
+    string(REPLACE ";" " " _pgo_flags_text "${_pgo_flags}")
+    foreach(_f ${_pgo_files})
+      set_property(SOURCE ${_f} APPEND_STRING PROPERTY COMPILE_FLAGS " ${_pgo_flags_text}")
+      # the instrumented allocator wrappers leave a relocation against posix_memalign in .debug_info, which --wrap does
+      # not redirect (undefined reference at the link): no debug information for that one file in a training build
+      if(WWHD_PGO STREQUAL "GEN" AND _f MATCHES "thread_heap_switch\\.cpp$")
+        set_property(SOURCE ${_f} APPEND_STRING PROPERTY COMPILE_FLAGS " -g0")
+      endif()
+    endforeach()
+    list(LENGTH _pgo_files _pgo_n)
+    message(STATUS "WWHD_PGO=${WWHD_PGO}: ${_pgo_n} files (${WWHD_PGO_SCOPE})")
+  endif()
+endmacro()

cmake/SwitchPort.cmake

@@ -7,6 +7,9 @@ else()
   set(WWHD_SWITCH OFF)
 endif()
 
+# (this fork) profile-guided optimisation: WWHD_PGO=GEN|USE (docs/pgo.md); off by default
+include(${CMAKE_SOURCE_DIR}/cmake/SwitchPgo.cmake)
+
 if(WWHD_SWITCH)
   # DEKO3D: deko3d (the Switch's own GPU API, docs/deko3d-plan.md) with the libnx host, the only renderer
   # there (the OpenGL renderer it replaced was removed after dev 207349b). Upstream's renderer choice is skipped.
@@ -95,6 +98,14 @@ macro(switch_port_executable)
     target_compile_options(wwhd PRIVATE -ffixed-x28)
     target_compile_definitions(wwhd PRIVATE PPC_BASE_REG=1)
   endif()
+  # (this fork) the runtime's extern symbols hidden: devkitPro's toolchain is -fPIC, which loads every extern symbol
+  # with default visibility through the GOT (runtime/include/switch_visibility.h); the game code gets the same
+  # from switch_code_align.h. WWHD_SWITCH_HIDDEN=OFF for a build without it.
+  option(WWHD_SWITCH_HIDDEN "Switch: hidden symbol visibility in the runtime (no GOT loads of extern symbols)" ON)
+  if(WWHD_SWITCH AND WWHD_SWITCH_HIDDEN)
+    target_compile_options(wwhd PRIVATE -fvisibility=hidden
+                           "SHELL:-include \"${CMAKE_SOURCE_DIR}/runtime/include/switch_visibility.h\"")
+  endif()
   # guest mods built into the NRO (tools/switch/guest_mods.py, docs/guest-mods-switch-plan.md): their modules are
   # compiled as game code (same flags, x28); without build/guest-mods/registry.c the NRO has none
   set(WWHD_GUEST_MODS_DIR "${CMAKE_SOURCE_DIR}/build/guest-mods" CACHE PATH "guest mods for the NRO (guest_mods.py)")
@@ -261,4 +272,5 @@ macro(switch_port_executable)
 
     nx_create_nro(wwhd NACP "${WWHD_NACP_FILE}" ${WWHD_ICON_ARG} OUTPUT ${WWHD_NRO_OUTPUT})
   endif()
+  switch_pgo_configure()  # (this fork) cmake/SwitchPgo.cmake: after all the target's sources are known
 endmacro()

docs/cpu-threading-plan.md

@@ -503,3 +503,10 @@ Newest last. Each entry: what was done, what was measured, what it decided.
   main f1f3132 27.9-28.0 / 24.4-24.5 / 28.9-29.0; this branch's build (O12) 29.1 / 25.35 / 29.0 (one boot so far).
   PR #60 is not the cause: per draw before and after it 4.4 us, the same components; its single-boot bisect step was
   the boot-to-boot spread.
+- 2026-10-23, **CPU work at the stock 1020 MHz: PR #87** (frame-free leaves, hidden symbols, interprocedural write-back elision,
+  PGO). Phase 2.2's "write-back elision" is done with the callees' real read/write sets (the earlier callee-saved-only version,
+  K/L above, was too small to see at the time; the A/B then ran on a console whose clocks were not stock, which is also why it
+  read as noise). Stock-clock A/B, 4 boots per build: the three code changes +1.8% to +3.3% fps in the CPU-bound scenes,
+  PGO on top another +2.6% to +7.5% (outset_village 25.35 -> 27.98 fps in all; lava 27.94 -> 29.28; windfall and
+  dragonroost reach the 30 fps cap). Phase 4 (the render thread) got its first win from the runtime half of the PGO profile
+  rather than from a restructuring. Write-up: docs/switch-port.md "Round 49", docs/pgo.md, docs/pgo-progress.md.

docs/perf-test.md

@@ -73,6 +73,11 @@ baseline built before that option: for a like-for-like comparison leave `switchS
 
 ## Rules for agents
 
+- **Check the clocks first.** Read the `[dk] clocks:` line of the session log (debug server `raw logtext`): it must say CPU 1020 MHz
+  and the GPU profile of the build under test (307 MHz before 2026-10-23, 384 MHz default since PR #85). sys-clk or a saved
+  460 MHz GPU profile makes the scenes sit at the 30 fps cap and hides real differences (2026-10-23: the same changes read
+  "inside the noise" at 1224/460 and +2% to +7% at stock). The lava room at stock should be near 27.9 fps (reference below).
+
 - Check first that the console answers and runs the USA game (`tools/switch/wwhd_debug.py info`).
 - Tell the user before the first deploy (the game restarts) and say how long it will take. If the console does not
   answer, ask them to start the game with the debug server on; do not guess.