Skip to content

api/chart: prevent PostgreSQL rollout overlap while API invariants require one process #726

Closed

Description

@Baylem

The chart documents a single API process for both SQLite and PostgreSQL, but the combined PostgreSQL Deployment uses the Kubernetes default rolling strategy. An upgrade can start a second writer even when api.replicas=1.

Priority: High. Evidence: static inspection of upstream 04e41075cafc4c1a99bcb3723b0a8745ee810957 (2026-10-16). The failure scenarios below have not been reproduced in a running cluster.

Evidence and impact

To exercise the gap, upgrade an active combined PostgreSQL installation while mutating requests reach both the old and new API pods. Both processes can pass invariants protected only by their own mutex. SQL-backed sessions already exist; this report does not request a replacement session store.

Recommended fix

  1. While singleton operation remains the contract, reject api.replicas != 1 for enabled API installations in every mode and render this strategy for every database driver:
    replicas: 1
    strategy:
      type: Recreate
      rollingUpdate: null
  2. Extend the existing Deployment-strategy migration to enabled combined PostgreSQL installations: both the migration Role gate and migration init-container gate currently require SQLite. Remove stale rollingUpdate fields, and verify an upgrade from an existing PostgreSQL RollingUpdate Deployment as well as a fresh install.
  3. Document that Recreate prevents normal rollout overlap but does not provide fencing after forced deletion or a partitioned node. Do not advertise API HA from replica count alone.
  4. Treat real HA as a follow-up requiring DB-level serialization of audit append/prune and user-management transactions, coordinated background work/rate limiting, and process-failure tests. PostgreSQL transaction-scoped advisory locks or locked invariant rows must cover the complete check-and-write transaction.

Acceptance criteria

  • Chart renders reject unsupported replica counts for SQLite and PostgreSQL in combined and standalone modes.
  • A populated combined PostgreSQL installation upgrades without overlapping ready API writers or invalid Deployment strategy fields.
  • Audit verification and last-user-manager invariants survive concurrent requests during the upgrade.
  • SQLite and standalone behavior remain covered.

Related: #519 implemented PostgreSQL dialect support and #707 added PostgreSQL-backed E2E coverage. Neither makes process-local locking safe across simultaneous API processes.

Activity

  1. ValgulNecron commented on Oct 20, 2026

    @ValgulNecron
    Collaborator

    Thanks, confirmed on master: strategy: Recreate is only rendered for standalone or SQLite (charts/gameplane/templates/api.yaml:226-250), so a combined PostgreSQL install gets the default RollingUpdate and briefly runs two API processes during an upgrade, while the audit hash chain and user-management lock are per-process (api/internal/audit/audit.go:58-62, api/internal/db/db.go:83). api.replicas != 1 is also only rejected in standalone mode (validate.yaml:16-17).

    Go ahead with a PR. This matches the existing decision for the standalone panel (one API process, Recreate), extended to every install:

    • Render strategy: Recreate (with rollingUpdate: null) for every enabled API Deployment, PostgreSQL included. We accept the short API outage on upgrade.
    • Reject api.replicas other than 1 in all modes in validate.yaml.
    • Extend the strategy-migration gates in crd-apply-hook.yaml (:111, :204) to PostgreSQL installs, so upgrading an existing RollingUpdate Deployment doesn't fail, and cover it in the e2e upgrade test.
    • Document that Recreate doesn't fence a partitioned node; multi-replica HA stays a separate future feature.
  2. Baylem commented on Oct 25, 2026

    @Baylem
    MemberAuthor

    Implemented in #758: every enabled API uses one replica with Recreate/rollingUpdate:null, PostgreSQL upgrades receive the strategy migration, and the CI availability drill covers a populated upgrade. The install and standalone guides document the short outage and the need to fence a partitioned old process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions