feat(observability): add game-performance and relay signals with a complete scrape setup #725

Extend the existing Prometheus integration so operators can distinguish game performance problems, container pressure, failed scraping, and relay connectivity problems. Provide a complete enabled configuration instead of treating monitor creation as proof of successful scraping.

Existing coverage and remaining gap

#220 added the agent's real per-server gauges, and #476 moved agent scraping to a dedicated plain-HTTP metrics listener and added the NetworkPolicy exception. Those fixes are present. The agent collector exports CPU, memory, disk, and player measurements with known/unknown semantics. This issue does not request rebuilding those metrics.

Module-specific readings use a separate on-demand /status handler with a five-second cache and RCON extraction; they do not feed the Prometheus collector. The pinned Minecraft template declares seed and difficulty, not TPS. The shipped PrometheusRule focuses on reconcile errors, queues, failed servers, and backups, rather than tick performance, resource pressure, or relay service quality.

The supported Helm values already document that serviceMonitors.enabled: true needs a nonempty serviceMonitors.scrapeNamespaceSelector when network policies are enabled. The ingress exception and agent PodMonitor exist. The enhancement is to make the complete setup easier to use and validate, not to reopen the resolved mTLS scrape bug.

Static review at commit 04e41075 identified these coverage gaps. No production scrape failure, tick degradation, memory incident, or relay saturation was measured. Per-game tick support depends on the game and must not be inferred from CPU usage or fabricated when unavailable.

Recommended change

• Publish a complete monitoring example covering serviceMonitors.enabled, monitor-selection labels, scrapeNamespaceSelector, Prometheus discovery of the games namespace, prometheusRules.enabled, and grafanaDashboards.enabled.

• Add a clear chart note or validation path for enabled agent monitors with network policies but no scraping namespace. Preserve restrictive network defaults and support externally managed policy exceptions.

• Define bounded numeric per-game performance metrics with explicit units, timestamps/freshness, and unsupported/unknown handling. Reuse collected samples rather than issuing costly RCON calls on every scrape.

• Integrate existing Kubernetes/container/node metrics for throttling, OOM/restarts, resource pressure, and pending pods instead of misrepresenting process RSS as total container memory.

• Add alerts and dashboards for scrape loss and provider connection/forward health; consume the companion tunnel-health work rather than duplicating its implementation.

Acceptance criteria

• A CI monitoring scenario discovers and scrapes agents from a distinct namespace with default-deny policies and the documented values.

• Missing configuration produces actionable guidance; external policy management remains supported.

• Unsupported metrics remain absent, zero stays distinguishable from unknown, and metric labels/cardinality are bounded.

• Rule tests cover scrape loss, pressure, game degradation where supported, relay failure/recovery, and duplicate operator replicas without double-counting.