Feat/perf node separation

apps/perf-sensor/skills/java-on-eks-optimization/SKILL.md

@@ -204,13 +204,16 @@ Read the file and pass the values through. The *why* for each:
   → `measure` (the items must pass on the current Deployment, or the policy rejects its next
   update: say which fail and fix them in the same change). Artifact: `k8s/policy.yaml` from
   `kyverno-policy.yaml`, `<namespace>` filled, expressions verbatim. It covers items 1, 3, 9
-  and 10; name the others as runtime items the sensor keeps scoring. Close with the step-5
-  line. Reference: `kyverno-policy.md`.
+  and 10; name the others as runtime items the sensor keeps scoring. The answer takes the
+  section 5 format like every other: `Root cause` is that a regression reaches the cluster
+  before the next measurement, `Evidence` is `measure`'s values for items 1, 3, 9 and 10 on
+  the current Deployment. Reference: `kyverno-policy.md`.
 - **"how are we doing / score"** → run the `java-on-eks-checklist` skill.
 
 ## 5. Answer format
 
-The answer starts with the first `Root cause` line: no preamble about the work you did.
+Every answer, the policy one included, starts with the first `Root cause` line: no preamble
+about the work you did, and no answer that only names the items it covered.
 
 ```
 Root cause: <one line, with tool + signal>

infra/scripts/deploy/java-performance/ide-layout.sh

@@ -7,7 +7,10 @@
 #
 # Writes ~/.code-editor-server/data/User/tasks.json (task "Shell", runOn folderOpen) and
 # sets "task.allowAutomaticTasks": "on" in the user settings (without it Code Editor asks
-# before running the task). Runs as ec2-user after ide/code-editor.sh. Idempotent.
+# before running the task) and "terminal.integrated.environmentChangesIndicator": "off" (the
+# Claude Code extension asks to relaunch every terminal Claude runs in, which otherwise marks
+# the tab with a warning badge for the rest of the session). Runs as ec2-user after
+# ide/code-editor.sh. Idempotent.
 # =============================================================================
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -18,7 +21,8 @@ SETTINGS="${USER_DIR}/settings.json"
 [ -f "${SETTINGS}" ] || { log_error "no Code Editor user settings at ${SETTINGS} (ide/code-editor.sh)"; exit 1; }
 
 tmp="$(mktemp)"
-jq '. + {"workbench.startupEditor": "terminal", "task.allowAutomaticTasks": "on"}' "${SETTINGS}" > "${tmp}"
+jq '. + {"workbench.startupEditor": "terminal", "task.allowAutomaticTasks": "on",
+         "terminal.integrated.environmentChangesIndicator": "off"}' "${SETTINGS}" > "${tmp}"
 mv "${tmp}" "${SETTINGS}"
 
 cat > "${USER_DIR}/tasks.json" <<'JSON'

infra/scripts/deploy/java-performance/nodepool-pin.sh

@@ -9,6 +9,11 @@
 # (e.g. m8azn at 5 GHz) and startup "improves" from hardware, not from the change.
 # Family, not type, keeps capacity flexible across sizes.
 #
+# Consolidation: WhenEmpty only. The pool carries just the app (platform-placement.sh), and
+# one m6a.xlarge fits every state of the session, rollout surge and boosted start included.
+# WhenEmptyOrUnderutilized (the eks.sh default) would replace the node when the app shrinks
+# and move the pod mid-measurement; WhenEmpty only removes a node nothing runs on.
+#
 # Replaces the three scheduling requirements (cpu-manufacturer, instance-category,
 # instance-generation) with one (instance-family); capacity-type, arch, os, cpu and
 # memory requirements stay. Idempotent. Runs after setup/eks.sh.
@@ -38,7 +43,8 @@ PATCH="$(kubectl get nodepool "${NODEPOOL}" -o json | jq -c --arg fam "${INSTANC
                          "eks.amazonaws.com/instance-generation",
                          "eks.amazonaws.com/instance-family") | not))
   + [{"key":"eks.amazonaws.com/instance-family","operator":"In","values":[$fam]}]
-  | {spec:{template:{spec:{requirements:.}}}}')"
+  | {spec:{template:{spec:{requirements:.}},
+           disruption:{consolidationPolicy:"WhenEmpty",consolidateAfter:"5m"}}}')"
 kubectl patch nodepool "${NODEPOOL}" --type merge -p "${PATCH}" >/dev/null \
-  && log_success "NodePool ${NODEPOOL} pinned to ${INSTANCE_FAMILY} (existing nodes drain on consolidation; new nodes are ${INSTANCE_FAMILY})" \
+  && log_success "NodePool ${NODEPOOL} pinned to ${INSTANCE_FAMILY}, consolidation WhenEmpty (existing nodes drain on consolidation; new nodes are ${INSTANCE_FAMILY})" \
   || { log_error "NodePool patch failed"; exit 1; }

infra/scripts/deploy/java-performance/platform-placement.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+# =============================================================================
+# platform-placement.sh — keep the platform off the application's node (java-performance only).
+#
+# The app selects the `workshop` NodePool (karpenter.sh/nodepool: workshop). The platform
+# workloads have no selector, so Karpenter places them wherever there is room, usually on
+# the app's 4-vCPU node. There the app shares physical cores (SMT) with Prometheus,
+# Pyroscope and Kyverno, and the session's measurements move with that load (startup and
+# steady CPU 10–60 % higher in a test run). This script pins every platform Deployment and
+# StatefulSet to the Auto Mode built-in `general-purpose` NodePool, so the `workshop` node
+# carries only the app and the DaemonSets. The app manifest is unchanged.
+#
+# Namespaces: monitoring (Prometheus, Grafana, Pyroscope, Alloy, sensor), kyverno,
+# kube-startup-cpu-boost-system, and metrics-server in kube-system. Pods move once, at
+# bootstrap; Prometheus, Grafana and Pyroscope keep their data on their PVCs. The pods are
+# marked do-not-disrupt, so consolidation does not move them later.
+# Idempotent. Run after the platform installs (template stage E). A later `helm upgrade`
+# of one of these releases drops the selector: run this script again afterwards.
+# =============================================================================
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/../../lib/common.sh"
+
+POOL="${PLATFORM_NODEPOOL:-general-purpose}"
+kubectl get nodepool "${POOL}" >/dev/null 2>&1 || { log_error "NodePool ${POOL} not found (EKS Auto Mode built-in pools disabled?)"; exit 1; }
+
+# do-not-disrupt: the general-purpose pool consolidates underutilized nodes after 30 s. A
+# consolidation during the session would move the sensor or Kyverno in the middle of a
+# checklist or an admission; the annotation keeps their nodes until the pods go.
+PATCH="{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"karpenter.sh/do-not-disrupt\":\"true\"}},\"spec\":{\"nodeSelector\":{\"karpenter.sh/nodepool\":\"${POOL}\"}}}}}"
+RC=0
+pin() {   # pin NAMESPACE KIND/NAME...
+    local ns="$1"; shift
+    for w in "$@"; do
+        if kubectl -n "${ns}" patch "${w}" --type merge -p "${PATCH}" >/dev/null; then
+            log_info "${ns}/${w} → ${POOL}"
+        else
+            log_error "patch failed: ${ns}/${w}"; RC=1
+        fi
+    done
+}
+
+for ns in monitoring kyverno kube-startup-cpu-boost-system; do
+    kubectl get namespace "${ns}" >/dev/null 2>&1 || { log_warning "namespace ${ns} not found, skipped"; continue; }
+    mapfile -t W < <(kubectl -n "${ns}" get deployment,statefulset -o name)
+    [ "${#W[@]}" -gt 0 ] && pin "${ns}" "${W[@]}"
+done
+kubectl -n kube-system get deployment metrics-server >/dev/null 2>&1 && pin kube-system deployment/metrics-server
+
+for ns in monitoring kyverno kube-startup-cpu-boost-system; do
+    kubectl get namespace "${ns}" >/dev/null 2>&1 || continue
+    for w in $(kubectl -n "${ns}" get deployment,statefulset -o name); do
+        kubectl -n "${ns}" rollout status "${w}" --timeout=600s >/dev/null || { log_error "${ns}/${w} not ready on ${POOL}"; RC=1; }
+    done
+done
+
+[ "${RC}" -eq 0 ] && log_success "Platform workloads on NodePool ${POOL}; the workshop node carries the app only" || exit 1

infra/scripts/setup/monitoring.sh

@@ -174,8 +174,13 @@ pyroscope:
       prometheus.io/scrape: "true"
       prometheus.io/port: "4040"
       prometheus.io/path: /metrics
+  # The metastore (the index of the blocks in S3) lives on the data volume: on an emptyDir
+  # a rescheduled pod starts with an empty index and every earlier profile, still in S3,
+  # becomes unqueryable. A PVC (the default gp3 class from eks.sh) keeps it across moves,
+  # like Prometheus and Grafana.
   persistence:
-    enabled: false
+    enabled: true
+    size: 10Gi
   resources:
     requests:
       cpu: 200m

infra/scripts/templates/java-performance.sh

@@ -15,6 +15,7 @@
 #      --- wait for A ---
 #   D  app          app-deploy (namespace, Pod Identity, manifests)      } in
 #   E  platform     perf-profiler install, startup-cpu-boost, perf-sensor } parallel
+#      placement    platform Deployments/StatefulSets → general-purpose NodePool
 #   F  verify       ALB answers, sensor measures, Claude wiring present
 #
 # Each stage script is idempotent and runnable on its own from the IDE.
@@ -99,6 +100,9 @@ cat "$D_LOG"; cat "$E_LOG"
 [ "$D_RC" -eq 0 ] && log_success "$(stamp)Stage D done (app deployed)" || { log_error "$(stamp)Stage D FAILED"; exit 1; }
 [ "$E_RC" -eq 0 ] && log_success "$(stamp)Stage E done (profiler, boost, sensor)" || { log_error "$(stamp)Stage E FAILED"; exit 1; }
 
+# --- platform placement: platform pods off the app's node ------------------------------
+run "Platform on the general-purpose NodePool" "$DEPLOY/platform-placement.sh"
+
 # --- F: verify --------------------------------------------------------------------------
 stage "F: verify"
 run "Environment verified"          "$DEPLOY/verify.sh"

infra/scripts/ws-test/README.md

@@ -120,7 +120,8 @@ Numbers vary with the measured working set (memory `max(1.4 × peak, 1.5 × floo
 |---|---|---|---|---|
 | Measure the baseline | `checklist` | — | 5/12 | ❌ 2 3 4 6 8 10 11 (6: a 1-core request is several times the steady CPU) |
 | Improve the manifest | `sizeMemory`, `sizeCpu`, `startupLog` | `k8s/deployment.yaml`, `k8s/startup-cpu-boost.yaml` | 9/12 | 2 3 4 6 → ✅; 8 halves (≈ 15 s → ≈ 8 s); `jfr.container` effectiveCpuCount 1 with `cpuQuotaCores` 2.0; `cpu.max` 1 core after the resize |
-| Improve the image | `startupLog`, `measure` | `pom.xml`, one Java file, `Dockerfile.crac`, `k8s/deployment.yaml` | 10/12 | 8 → ✅ (Restored, < 0.1 s); 3 4 5 stay ✅ from the checkpoint; 2 may sit near its bar (the restored pod's working set is lower under a limit sized for the cold one) |
+| Improve the image, AOT | `startupLog`, `profileTop` | `Dockerfile.aot` | (no checklist) | startup line *Started* in about a third of the boosted cold start |
+| Improve the image, CRaC | `startupLog`, `measure` | `pom.xml`, one Java file, `Dockerfile.crac`, `k8s/deployment.yaml` | 10/12 | 8 → ✅ (Restored, < 0.1 s); 3 4 5 stay ✅ from the checkpoint; 2 may sit near its bar (the restored pod's working set is lower under a limit sized for the cold one) |
 | Improve the code | `diagnoseBlocking`, `sizeMemory`, `sizeCpu` | `UnicornService.java`, `k8s/deployment.yaml` (sizes, `terminationGracePeriodSeconds`), `Dockerfile.crac` | 12/12 | 10 11 → ✅ (blocked 0, in-transaction 0; pool waits shown, not scored); memory steps down, new heap baked into the checkpoint, rebuild stated |
 | Enforce the practices | `measure` | `k8s/policy.yaml` | 12/12 | none; the policy matches `references/kyverno-policy.yaml`; the PolicyReport shows the running Deployment passing; an apply with `-Xmx` is denied with the item 3 message |
 

infra/scripts/ws-test/capture/ide.mjs

@@ -1,10 +1,9 @@
 // ide.mjs tools|scm OUT_DIR — Code Editor screenshots (run through ide.sh, which prepares
 // the IDE and sets IDE_URL and WS_ACCOUNT).
 //
-//   tools  the IDE as it opens (deploy/java-performance/ide-layout.sh): Claude Code started
-//          in the editor-area terminal lists the MCP tools of perf-sensor and eks-mcp, the
-//          Shell terminal in the panel -> OUT_DIR/tools.png. Starts an interactive Claude:
-//          never while a ws-test runs.
+//   tools  Claude Code in a fresh editor-area terminal lists the MCP tools of perf-sensor and
+//          eks-mcp, the Shell terminal in the panel -> OUT_DIR/tools.png. Starts interactive
+//          Claude sessions (two, see below): never while a ws-test runs.
 //   open   the IDE as it opens: Explorer, the editor-area terminal, the Shell terminal
 //          -> OUT_DIR/open.png
 //   scm    the Source Control view with SCM_FILE's diff open (default deployment.yaml)
@@ -18,8 +17,24 @@ const { browser, page } = await open(process.env.IDE_URL);
 if (what === 'open') {
   await page.waitForTimeout(6000);
 } else if (what === 'tools') {
+  // The first terminal that runs Claude Code activates its VS Code extension, which then
+  // asks to relaunch that terminal and marks its tab with a warning badge for the rest of
+  // the session. So: run Claude once in a throwaway terminal, close it, and take the shot in
+  // a second one, where the extension's variables are set at creation and no badge appears.
+  const newTerminal = () =>
+    page.locator('.editor-group-container .editor-actions .codicon-plus').first().click();
   await page.locator('.editor-group-container .tab', { hasText: 'zsh' }).first().click();
   await page.waitForTimeout(800);
+  await newTerminal();
+  await page.waitForTimeout(4000);
+  await typeInTerminal(page, 'claude');
+  await page.waitForTimeout(18000);
+  await typeInTerminal(page, '/exit');
+  await page.waitForTimeout(4000);
+  await typeInTerminal(page, 'exit');       // closes the throwaway terminal and its tab
+  await page.waitForTimeout(3000);
+  await newTerminal();
+  await page.waitForTimeout(4000);
   await typeInTerminal(page, 'claude');
   await page.waitForTimeout(15000);
   await typeInTerminal(page, 'List the MCP tools you can see from perf-sensor and eks-mcp, names only.');