Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
name: Lint

on: # yamllint disable-line rule:truthy
push:
branches: [main]
pull_request:
branches: [main]

permissions: {}

jobs:
naming-lint:
name: Naming Lint
runs-on: ubuntu-latest

permissions:
contents: read

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
persist-credentials: false

- name: Check Multigres name capitalization
run: ./tools/naming_linter.sh
shell: bash
12 changes: 6 additions & 6 deletions content/blog/deploying-the-multigres-operator.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,13 @@ A Multigres cluster is made up of several components that work together across a

**GlobalTopoServer** is a managed etcd cluster that records the topology state: which databases exist, which cells they live in, and where every component is registered. Cells are user-defined groupings that map to availability zones. In a multi-zone deployment, the topology splits into a global server for cluster-wide state and per-cell servers for local discovery, so a partitioned cell keeps operating against its own local view.

**MultiAdmin** is the management plane for the cluster, including a web UI.
**Multiadmin** is the management plane for the cluster, including a web UI.

**MultiGateway** speaks the Postgres wire protocol. Your application connects to a gateway, which forwards queries to the right MultiPooler over gRPC. Adding more gateways scales Multigres’ connection capacity horizontally.
**Multigateway** speaks the Postgres wire protocol. Your application connects to a gateway, which forwards queries to the right Multipooler over gRPC. Adding more gateways scales Multigres’ connection capacity horizontally.

**MultiOrch:** the orchestrator. One set of MultiOrch instances per shard, running across cells. It watches replication health, appoints leaders through a consensus protocol, runs failovers, and coordinates bootstrap, backup, restore, and scaling events. When you apply the manifest, MultiOrch runs bootstrap as a consensus-backed election. This is the same code path as every later failover- so, there is no separate provisioning script that can race with itself.
**Multiorch:** the orchestrator. One set of Multiorch instances per shard, running across cells. It watches replication health, appoints leaders through a consensus protocol, runs failovers, and coordinates bootstrap, backup, restore, and scaling events. When you apply the manifest, Multiorch runs bootstrap as a consensus-backed election. This is the same code path as every later failover- so, there is no separate provisioning script that can race with itself.

**Postgres** pods themselves, managed by pgctld which owns the local Postgres process and MultiPooler. One pooler per Postgres instance.
**Postgres** pods themselves, managed by pgctld which owns the local Postgres process and Multipooler. One pooler per Postgres instance.

## Deploying on EKS

Expand All @@ -45,7 +45,7 @@ The short version of what you need is a kubeconfig pointed at an EKS cluster wit

In the Manifest linked to above, you’ll see three cells, each mapped to a real AWS availability zone. `AT_LEAST_2` durability means every committed write is acknowledged by one standby (one primary and one standby). The operator infers everything else from the templates: resource limits, gateway configuration, topo server sizing — and builds the full topology.

Run `kubectl apply -f demo-multi-az.yaml` and then `kubectl get pods -w`. Within a minute you have a global topo server, MultiAdmin, MultiGateway, MultiOrch across three zones, and Postgres pool pods across three zones.
Run `kubectl apply -f demo-multi-az.yaml` and then `kubectl get pods -w`. Within a minute you have a global topo server, Multiadmin, Multigateway, Multiorch across three zones, and Postgres pool pods across three zones.

You should eventually see pods for global topo, multiadmin, multigateway, multiorch, and three poolers.

Expand Down Expand Up @@ -78,7 +78,7 @@ The operator provisions new standby pods across all three zones simultaneously.

## Connecting

Applications inside the cluster connect through the MultiGateway service:
Applications inside the cluster connect through the Multigateway service:

`postgresql://postgres:<password>@demo-multi-az-multigateway:5432/postgres`

Expand Down
8 changes: 4 additions & 4 deletions content/blog/drop-in-postgres-proven-with-a-real-app.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,16 @@ Most poolers impose a hard limit on client connections. Once that limit is reach

Multigres splits connection pooling into two distinct jobs:

1. `MultiGateway` is the tier that accepts client connections. It scales horizontally: connection capacity grows by adding gateways, not by tuning a single process. That is where the large connection counts come from.
2. `MultiPooler` sits behind MultiGateway and runs next to every Postgres instance. It manages the real Postgres connections - a small, stable set that client connections are multiplexed onto.
1. `Multigateway` is the tier that accepts client connections. It scales horizontally: connection capacity grows by adding gateways, not by tuning a single process. That is where the large connection counts come from.
2. `Multipooler` sits behind Multigateway and runs next to every Postgres instance. It manages the real Postgres connections - a small, stable set that client connections are multiplexed onto.

The gateway gives you the connections and the pooler keeps the backend side small. Both are managed by the same Multigres machinery that is responsible for replication and failover.

## Pooling as a property of the cluster

In Multigres, connection pooling is a property of the cluster, not a service bolted on in front of it. Each Postgres instance has its own `MultiPooler`, co-located with it and registered in the same topology as the rest of the cluster.
In Multigres, connection pooling is a property of the cluster, not a service bolted on in front of it. Each Postgres instance has its own `Multipooler`, co-located with it and registered in the same topology as the rest of the cluster.

This pooler also supervises the Postgres instance. If Postgres goes down, the pooler is the component that notices and brings it back, through `pgctld`. `MultiOrch` handles failover across instances, and the gateway watches the topology - when the primary moves, the gateway routes to whichever pooler now owns it. The application keeps talking to the gateway and is unaware that anything has changed.
This pooler also supervises the Postgres instance. If Postgres goes down, the pooler is the component that notices and brings it back, through `pgctld`. `Multiorch` handles failover across instances, and the gateway watches the topology - when the primary moves, the gateway routes to whichever pooler now owns it. The application keeps talking to the gateway and is unaware that anything has changed.

## Technical Deep Dive

Expand Down
4 changes: 2 additions & 2 deletions content/blog/handling-edge-cases-using-consensus.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,15 @@ The full demo:

Start with the simplest version. A Multigres cluster has one primary and two replicas. An operator wants a third replica, for read capacity, for a multi-zone deployment, or to prepare for a planned shrink elsewhere.

The Kubernetes operator scales the number of replicas from three to four. A new pod comes up. A new MultiPooler registers in the topology of the cluster. A new Postgres process starts as a fresh replica. The orchestrator notices the new node and brings it into the cluster.
The Kubernetes operator scales the number of replicas from three to four. A new pod comes up. A new Multipooler registers in the topology of the cluster. A new Postgres process starts as a fresh replica. The orchestrator notices the new node and brings it into the cluster.

The naive way to bring it in is to point the new replica at the current primary, let it restore from a backup, let it stream WAL forward, and once it has caught up, count it as a member.

The naive approach is wrong because it treats membership as a local orchestration detail. A standby cannot simply join the cohort and start acknowledging writes. Let's unpack in detail an interesting failure mode where this becomes a problem.

## The race

The hard part in distributed systems is that changes take time to propagate, and agents may take action based on stale information. Multiple MultiOrch instances are watching the cluster from different cells. They do not share local state, and for short periods of time they can have different views of the world. The challenge is that those local decisions still have to honor the cluster's consensus rules. That is intentional. It is part of what makes the orchestrator partition-tolerant.
The hard part in distributed systems is that changes take time to propagate, and agents may take action based on stale information. Multiple Multiorch instances are watching the cluster from different cells. They do not share local state, and for short periods of time they can have different views of the world. The challenge is that those local decisions still have to honor the cluster's consensus rules. That is intentional. It is part of what makes the orchestrator partition-tolerant.

The risk appears when cluster membership is implemented naively using only the local view of a coordinator. During a membership change, one orchestrator may already believe a new replica has joined, while another may still be operating with the old view of the cluster.

Expand Down
16 changes: 8 additions & 8 deletions content/blog/high-availability-from-first-principles.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ Multigres provides strict guarantees. The HA implementation is anchored in a bod

Together, these enable the team change the implementation with confidence. New failover scenarios do not require re-proving the safety of the entire system. Instead, each change is evaluated against a well-defined set of invariants that preserve the core guarantees.

## MultiOrch: the coordinator
## Multiorch: the coordinator

The HA coodinator in Multigres is called "MultiOrch". It uses a more efficient approach to HA than most other database implementations. Take a Raft implementation, where every node is a candidate: nodes vote among themselves, elect a leader, and replicate the log. The members do all the work.
The HA coodinator in Multigres is called "Multiorch". It uses a more efficient approach to HA than most other database implementations. Take a Raft implementation, where every node is a candidate: nodes vote among themselves, elect a leader, and replicate the log. The members do all the work.

Generalized consensus separates the roles:

Expand All @@ -42,11 +42,11 @@ Generalized consensus separates the roles:
3. `Observers` replicate completed requests for read scaling.
4. `Coordinators` run health checks, detect failures, and drive rule changes (including leadership changes).

In Multigres, MultiOrch is the coordinator and Postgres processes can be leaders, followers, or observers depending on their state.
In Multigres, Multiorch is the coordinator and Postgres processes can be leaders, followers, or observers depending on their state.

In our deployments, we spread each Postgres cluster across three availability zones to provide cross-AZ redundancy. We also run MultiOrch in multiple AZs so that the system can tolerate a coordinator becoming network-partitioned from part of the cluster.
In our deployments, we spread each Postgres cluster across three availability zones to provide cross-AZ redundancy. We also run Multiorch in multiple AZs so that the system can tolerate a coordinator becoming network-partitioned from part of the cluster.

Typically there are two or three MultiOrch instances per shard, one per cell, each watching the cluster from an independent vantage point. They can all act independently and they do not talk to each other directly. Each instance observes the state of the shard, detects failures independently, and races to repair the cluster when it sees a problem.
Typically there are two or three Multiorch instances per shard, one per cell, each watching the cluster from an independent vantage point. They can all act independently and they do not talk to each other directly. Each instance observes the state of the shard, detects failures independently, and races to repair the cluster when it sees a problem.

The remediation follows the consensus rules. A coordinator that wants to perform a leadership change has to:

Expand All @@ -63,13 +63,13 @@ If two coordinators race, only one can complete the sequence. The other one noti

The demo video above demonstrates the full sequence of a primary failover in a local Kubernetes cluster. This is what happens in that sequence of events from the consensus perspective:

**Detection:** The cluster is healthy: one primary, two replicas, three MultiOrch instances watching. A demo button kills the Postgres process on the primary. Within a configurable timeout, every MultiOrch notices that the primary's pooler has stopped reporting healthy status.
**Detection:** The cluster is healthy: one primary, two replicas, three Multiorch instances watching. A demo button kills the Postgres process on the primary. Within a configurable timeout, every Multiorch notices that the primary's pooler has stopped reporting healthy status.

**Race:** Each MultiOrch starts a failover attempt. Each one picks a new term number, one higher than any term it has observed on the cohort, and asks the replicas to pledge to it. Replicas accept only one coordinator per term, so only one MultiOrch can gather a quorum of pledges for this round. The losers back off and retry later if needed.
**Race:** Each Multiorch starts a failover attempt. Each one picks a new term number, one higher than any term it has observed on the cohort, and asks the replicas to pledge to it. Replicas accept only one coordinator per term, so only one Multiorch can gather a quorum of pledges for this round. The losers back off and retry later if needed.

**Revocation:** The quorum's pledge to a new term is the revocation, and we recruit enough replicas for it to do two things. First, the old primary can no longer satisfy the cluster's durability policy with whatever replicas remain, so it can't durably commit any new writes under the old term. Second, we recruit enough replicas that they can't be part of any other quorum either, preventing a network-partitioned slice of the cluster from forming an independent quorum and electing a competing leader. This gives the coordinator sole authority to establish the next rule (in this demo the new leadership).

**Discovery:** The winning MultiOrch inspects the WAL position reported by each pledged replica and identifies the most-advanced replica that satisfies the durability requirements.
**Discovery:** The winning Multiorch inspects the WAL position reported by each pledged replica and identifies the most-advanced replica that satisfies the durability requirements.

**Candidacy:** The new leader is chosen from the nodes already tied at that most-advanced position. There may be more than one; any of them is safe. Reachability and the cluster's durability policy are checked against the proposed cohort before the choice is finalized.

Expand Down
12 changes: 6 additions & 6 deletions content/blog/multigres-cluster-bootstrap.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ A recording of the same flow:

A Multigres deployment runs four services alongside Postgres, plus a topology server.

- **MultiGateway.** Speaks the Postgres wire protocol, so your application thinks it is talking to a Postgres server. The gateway forwards queries to the right MultiPooler over gRPC.
- **MultiPooler.** Sits next to a single Postgres instance and owns its connection pool. One pooler per Postgres process.
- **Multigateway.** Speaks the Postgres wire protocol, so your application thinks it is talking to a Postgres server. The gateway forwards queries to the right Multipooler over gRPC.
- **Multipooler.** Sits next to a single Postgres instance and owns its connection pool. One pooler per Postgres process.
- **pgctld.** Runs alongside the pooler and manages the local Postgres process: `initdb`, start, stop, restart.
- **MultiOrch.** The orchestrator. Watches replication, appoints leaders, runs failovers, and coordinates bootstrap, backup, restore, and scale.
- **Multiorch.** The orchestrator. Watches replication, appoints leaders, runs failovers, and coordinates bootstrap, backup, restore, and scale.
- **Topology server.** A distributed key-value store, typically etcd. Records which databases exist, which cells they live in, and where every component is registered.

Cells are user-defined groupings that map to availability zones or regions. In a multi-zone deployment, the topology splits into a global server holding cluster-wide state and per-cell servers holding local discovery state. A partitioned cell keeps operating against its own local view.
Expand All @@ -33,7 +33,7 @@ When the manifest applies, each service registers itself with the topology serve

Poolers register next, each bound to a Postgres instance through pgctld. Every data directory is empty. Every pooler advertises itself as REPLICA, the safe default. The cluster still has no primary.

This is the most fragile moment of a cluster's life. If a hand-rolled bootstrap script picked a primary now, two scripts could run in parallel and pick two different primaries. This is a split-brain. Multigres avoids it by coordinating leader appointment through MultiOrch, which runs as a small set of independent instances that agree on the leader through our [consensus protocol](/blog/generalized-consensus-part1).
This is the most fragile moment of a cluster's life. If a hand-rolled bootstrap script picked a primary now, two scripts could run in parallel and pick two different primaries. This is a split-brain. Multigres avoids it by coordinating leader appointment through Multiorch, which runs as a small set of independent instances that agree on the leader through our [consensus protocol](/blog/generalized-consensus-part1).

## Seed backup first, then leader

Expand All @@ -43,9 +43,9 @@ Bootstrap runs in two phases.

Each pooler independently runs `initdb`, starts Postgres through pgctld, and creates the Multigres schema and pgBackRest stanza. The poolers then race for a backup lease. One wins, takes a full backup of its local data directory, and writes it to S3. The others find that a backup already exists and skip the create step. Every pooler then tears down its local data directory and restores from the shared backup. After this phase, every pooler holds the same restored state, in hot-standby mode.

### Phase 2: MultiOrch appoints the initial leader
### Phase 2: Multiorch appoints the initial leader

MultiOrch waits for enough initialized poolers to satisfy the durability policy. It then claims the exclusive right to initialize the shard, picks one of the restored poolers as the leader, and promotes it to primary. The other poolers point at the new primary and start streaming WAL.
Multiorch waits for enough initialized poolers to satisfy the durability policy. It then claims the exclusive right to initialize the shard, picks one of the restored poolers as the leader, and promotes it to primary. The other poolers point at the new primary and start streaming WAL.

Bootstrap and failover share the same election code path. A new cluster is a failover with no prior leader.

Expand Down
Loading
Loading