Skip to content

feat: add bounded game performance and relay monitoring - #771

Open
Baylem wants to merge 25 commits into
GameplanePanel:stack/pr-763from
Baylem:fix/audit-observability
Open

Baylem wants to merge 25 commits into
GameplanePanel:stack/pr-763from
Baylem:fix/audit-observability

Conversation

@Baylem

@Baylem Baylem commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Operators need to distinguish missing measurements, game degradation, resource pressure and relay failures. This adds cached numeric game-performance measurements and relay metrics, with a complete Prometheus configuration and a real policy-enforced scrape scenario in CI.

Modules can explicitly declare tick/frame rates or tick duration, units and an optional degradation threshold. One bounded source feeds both dashboard routes and Prometheus; scrapes never execute RCON. Unknown, invalid and stale readings have no value series, while genuine zero remains visible. Source RCON receives an aggregate response limit and an absolute exchange deadline so a streaming response cannot defeat the collector's resource bounds.

Tunnel metrics consume the same sanitized health evidence as readiness. The chart adds scrape-policy validation, tunnel discovery, pressure/game/relay alerts and a workload dashboard. The documented Prometheus example aligns monitor labels, discovery namespaces and ingress permissions; explicitly managed external policies remain supported. The CI fixture uses Prometheus Operator and Cilium, verifies actual agent/tunnel scraping, revokes/restores ingress and checks denial from an unapproved namespace. Promtool fixtures cover alert firing/recovery and duplicate operator readings.

No game module is automatically opted in. Playit's pinned client still reports unknown live connectivity, and infrastructure rules require kubelet/cAdvisor and kube-state-metrics collectors as documented.

Includes #768 relay health and its #765/#762 prerequisites, #763 network isolation, and #759 shared web build correction. The embedded Tailscale SDK follows upstream v1.104.2 with a Go 1.27.3 minimum. Management-placement, network-isolation and monitoring buckets are registered in change-based test selection. Monitoring bootstrap retries transient bundle-download failures within a fixed time budget and retains checksum verification.

Expected controller retries use explicit polling delays while genuine errors retain backoff. Mod deletion and its HTTP DELETE manifest update use the same opened directory handle, preventing symlink aliases or directory replacement from redirecting removal. Regression sources cover retry pacing, retained lifecycle fences, link-target preservation and atomic metadata cleanup.

Validation: All 107 CI checks passed on 5042ab115f9b1fe1bcd7e433064345f869257ab4 (main CI run), including lint, builds, coverage gates, unit/controller tests and E2E suites. Go production/regression-source and web type compilation passed locally; required actionlint/offline zizmor and whitespace checks passed. Runtime tests ran in CI under the repository's compile-only local policy.

The NuclearOption RCON client also applies its absolute command deadline before writing the request, preventing a peer that stops reading from holding the shared client indefinitely. Both blocked-write stages have regression coverage. Monitoring selector validation follows the operator NetworkPolicy gate, so standalone panel monitoring remains usable without a game scrape selector.

Closes #725.

Signed-off-by: Baylem <zelathis@gmail.com>
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added area: agent in-pod sidecar area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: specs spec-kit feature specs and design docs area: web React dashboard type: feature New capability or user-facing addition labels Oct 21, 2026
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Baylem added 13 commits October 21, 2026 22:49
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem requested a review from ValgulNecron as a code owner October 24, 2026 14:53
@ValgulNecron

Copy link
Copy Markdown
Collaborator

Hey, thanks for this one. The performance readouts and tunnel metrics look solid. A couple of fixes before merge:

  1. A down Tailscale relay never raises the real alert. In tunnel/tailscale.go:100-103 an unhealthy tailnet reports Connection: "unknown", so gameplane_tunnel_connection_known is 0 and GameplaneRelayUnavailable (monitoring-rules.yaml:21-23, needs known == 1) never fires; only the info-level GameplaneRelayHealthUnknown does. Since tsnet gives us a real "down" signal, please report "unavailable" when the tailnet was up and then fails. Playit has the same issue the other way round: it always reports "unknown", so its info alert fires permanently for every playit server. Please fix that too (a real signal, or don't emit the info alert for providers that can never report it).

  2. tailscaleHealth is dead code now: runTailscale serves relayHealthHandler instead (tailscale.go:88), and the only test, TestTailnetHealthNeedsForwardingReadiness (tailscale_test.go:199-201), exercises the unused handler. Please remove tailscaleHealth and point that test at the handler that's actually served.

Optional: several commits here are copies of ones in #765, #763, #758 and #770 (lint bump, CRD regen, operator waits, metrics probe, event-stream e2e, NuclearOption bounds, isolation verdict). If you can drop the ones that already live in those PRs

Thanks!

Signed-off-by: Baylem <zelathis@gmail.com>

Baylem commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

Addressed in cfe8c19:

  • Tailscale failures now remain unavailable while forwarding is down, including retry backoff and the next registration attempt. The supervisor records that local fact; fresh provider observations replace it after recovery and still expire normally. The regression closes a real listener and checks the served readiness and metrics handlers through failure, retry, and recovery.
  • Playit's unsupported live-health capability is exposed separately. Its honest unknown state no longer causes a permanent informational alert; providers that can report live health still get that alert.
  • Removed the unused tailscaleHealth handler and moved its readiness coverage to the relayHealthHandler actually served.

On the optional duplicate-commit cleanup: I have retained the unmerged prerequisite changes so this PR builds independently and includes the reviewed corrections from #763, #765, and #768. Removing them now would either remove required behavior or rewrite the published branch. The shared changes will leave the effective diff as those PRs land; I have kept this update additive.

Operator/tunnel production builds and affected regression-source compilation pass. Runtime tests and lint remain CI-only under the repository policy.

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>

Baylem commented Oct 27, 2026

Copy link
Copy Markdown
Member Author

cb5e73b7 fixes the wipe inventory regression by separating broad inventory queries from the filtered sentinel safety query, without changing production behavior or weakening the one-full-inventory assertion. It also carries over #768's shared production retry observer so the retry regressions exercise the same callback as the running relay. This branch retains its unavailable/down health semantics. Upstream conflicts are resolved; controller/tunnel regression-source compilation and workflow static checks passed. Fresh CI is running.

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>

Baylem commented Oct 31, 2026

Copy link
Copy Markdown
Member Author

Merged upstream’s frpc dependency fix in a80d3c2; the production binary build confirms go-jose 4.1.6. Operator/tunnel source compilation and the required workflow checks passed.

The multicluster failure occurred before the ORAS fixture started: its pod was in ImagePullBackOff for ghcr.io/oras-project/oras:v1.3.3. The logs omit the underlying registry error. The image manifest is available now and includes both runner architectures. The new push starts a full CI run to rebuild the expired image artifacts and retry fixture acquisition; test deadlines and assertions are unchanged.

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem changed the base branch from master to stack/pr-763 November 4, 2026 05:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: agent in-pod sidecar area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: specs spec-kit feature specs and design docs area: web React dashboard type: feature New capability or user-facing addition

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): add game-performance and relay signals with a complete scrape setup

2 participants