docs/ops: define and exercise panel database, Kubernetes state, and key recovery #728

Production recovery needs a tested procedure that distinguishes panel SQL state from Kubernetes workload state. A game-volume backup cannot reconstruct users, role bindings, audit history, or standalone cluster credentials.

Priority: Medium; production-readiness follow-up. Evidence: static inspection of upstream 04e41075cafc4c1a99bcb3723b0a8745ee810957 (2026-10-16). The failure scenarios below have not been reproduced in a running cluster.

Existing behavior to preserve

• docs/architecture.md:67 assigns desired workload reconciliation to Kubernetes CRs and the operator.
• api/cmd/management.go:19 uses Kubernetes management storage in combined mode and SQL-backed management storage in standalone mode.
• api/internal/controlplane/store.go:45 requires the original encryption key for existing standalone credentials and rejects missing/wrong keys.
• docs/standalone-panel.md:67 already documents database/WAL backup, separate key custody, and restoration permissions. This issue requests a broader recovery contract and drill, not replacement of that guidance.
• docs/install.md:871 describes upgrade validation, which is distinct from disaster recovery.

Recommended fix

Document an explicit backup inventory and recovery order for both modes:

1. SQL database: users, role bindings, sessions/shares, configuration, audit state, and standalone management rows. For PostgreSQL, define an external backup/PITR and failover integration; for SQLite, retain a consistent snapshot or stopped-copy procedure rather than copying only the live database file.
2. Kubernetes state: CRDs, GameServer/Module/Backup objects, relevant Secrets, and the cluster's own recovery procedure.
3. Workload data: primary and other persistent volumes, storage-class constraints, and Restic/CSI recovery dependencies.
4. Keys: separately protected standalone encryption key and relevant certificate/credential recovery. Retain the key versions required by retained database backups.

Define operator-selected RPO/RTO targets and demonstrate them; do not promise a universal target in chart defaults. After restoring old SQL state, reconcile authorization and revoked credentials against the intended recovery point. Invalidate restored sessions/share links when the recovery policy requires it. Validate workload identity/UID associations before granting access; do not infer ownership from pod names.

Acceptance criteria

• Runbooks cover database outage, database loss, original-key loss, and restoring SQL while game pods continue running.
• A drill restores users/permissions, registrations, and credential-backed operations with the correct key, and fails closed with a wrong key.
• Recovery does not recreate or delete live games just to reconstruct the panel.
• The documentation identifies which data cannot be recovered from a game backup and records measured RPO/RTO for the drill fixture.
• Recovery steps preserve audit evidence and explicitly handle stale authorization/session state.

Related: #629 introduced standalone storage and #707 covers PostgreSQL operation/migration. This is a recovery follow-up to those capabilities.