Stage 7: land P0 cluster-formation substrate + spec-7.2 IC data-plane#30
Merged
Conversation
added 30 commits
July 7, 2026 10:11
A locally-mastered GES request enqueued into the work queue had no wakeup: LMON only drained it when it woke for another reason, worst case a full heartbeat interval (<= 1s). Mirror the outbound-ring enqueue family and SetLatch the drain consumer right after publishing the slot (publish-before-signal). Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
…10ms LAN-scale pacing for the fast-deny / timeout retry path: 10/20/40/80 ms (total 150 ms) instead of 100/200/400/800 ms (total 1500 ms). The per-attempt reply wait is unchanged (cluster.gcs_reply_timeout_ms), so slow-but-successful serves are unaffected; only the sleep after a fast denial (DENIED_DEDUP_FULL / DENIED_EPOCH_STALE) or reply timeout shrinks. Floor lowered 10 -> 1 ms for tuning headroom; raise the GUC on slow or congested interconnects. Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
Both READY publish sites (CR page construct + undo/verdict serve) left the finished result parked until LMON's next natural wakeup (typically 100-250ms under load, worst case one heartbeat interval). Kick the shipper right after the atomic READY store, mirroring the enqueue-side wakeup family. Publish-before-signal preserved by the existing pg_write_barrier + atomic store ordering. Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
Classify every LMON main-loop duty (24 entries, ClusterLmonDuty) into never-lazy correctness families (fence / sweeps / clean-leave / node- remove / reconfig / GRD recovery / BOC / sinval RESET / PI-discard -- order and cadence verbatim, I47-class order writs untouched) and 8 lazy-able queue-consumption families (GES work queue / ship-ready / grd outbound / sinval out x2 / tt-hint / dedup TTL / backup) gated behind a producer-set dirty bitmask plus a >= 1 Hz floor. - producers mark_dirty at enqueue (inside the shared push helpers where one exists); dedup TTL is time-based and rides the floor; the backup protocol self-re-marks while an operation is in flight (its advance steps are fence-driven, not producer-driven) - cluster.ic_duty_lazy (bool, SIGHUP, default on) is the escape hatch; off restores run-every-iteration behavior - classification is a no-default switch (-Wswitch guards new entries) and the unit truth table pins all 24 rows (157/157 unit green) Baseline: duty chain measured ~140us/iteration avg (max 923us) on a 2-node pair; this removes the fixed per-wakeup dilution tax, it is NOT the 0.5-1s ship-latency fix (that is the wakeup family + D2 WES). Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
…kets) Sixteen log-scale buckets (500us .. 30s + inf) in ClusterGcsBlockShared, recorded at the single normal-exit funnel of the block request path (GRANTED / STORAGE_FALLBACK / READ_IMAGE completions; terminal-ereport exits lose the sample, mirroring the xp scopes). Always on -- unlike the GUC-gated xnode-profile scopes -- because this is the ruler for the ship-latency value gate (p99 < 20ms, p50 < 5ms) and the later wait- closure acceptance legs. Exposed as dump category 'gcs' keys ship_hist_us_le_<bound> / ship_hist_us_inf. Spec: spec-7.2-ic-data-plane-decoupling.md (D6 slice)
check_GUC_init asserts the C variable initializer matches the GUC boot_val; the D1 default lowering changed only the registration. Caught by initdb (cassert) on the first post-change cluster boot. Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
…rows) The 16 ship_hist_us_* rows added by the D6 histogram slice land in dump category 'gcs'; t/110 L1 and t/112 L3 pin the category cardinality. Spec: spec-7.2-ic-data-plane-decoupling.md (D6 slice)
Optional [node.N] key, host:port, validated like public_addr. Empty or absent means the node offers no DATA plane; never derived from interconnect_addr by offset (adjacent-port collisions on single-host deployments make an implicit default unsound). The consumer lands with the LMS data-plane loop; startup enforcement for multi-node clusters arrives with the plane flip so intermediate trees keep running without conf changes. ClusterNodeInfo grows by the appended field (layout-append rule), which pushes the 128-slot ClusterConf container past its old 64 KiB anchor -- budget raised to 128 KiB (shmem-only anchor, not a wire contract). Spec: spec-7.2-ic-data-plane-decoupling.md (D2a)
Introduce ClusterICPlane (CONTROL = 0, DATA = 1) and carve the single tier1 shmem region into one instance per plane. Process-local tier1 state (fd arrays, buffers, HELLO machines) is naturally per-process, so a process's plane is implied: the working alias Tier1Shmem follows tier1_my_plane (CONTROL by default; the LMS loop re-aims it via cluster_ic_tier1_set_my_plane). Every existing tier1 function -- the CONNECTED send gate included -- lands on its own plane's peers[] and listener metadata unchanged; LMON, backends and SQL observers stay on the CONTROL instance and contract. HELLO rides plane + conn_epoch in the documented-zero pad (capabilities precedent, V1 stays 64B): plane byte at offset 40, conn_epoch LE at 44-51, spec-7.3 worker bytes (41/42) reserved zero. A CONTROL HELLO with epoch 0 stays byte-identical to pre-7.2 (unit compat pin); verify paths reject cross-plane HELLOs, and a DATA HELLO with epoch 0 is refused fail-closed (INV-7.2-CONN-EPOCH substrate). Plane-selected addresses: the DATA plane binds/dials data_addr only -- no data_addr means no DATA plane (LOG + listener off; never falls back to the CONTROL address). RDMA lanes stay CONTROL-owned (DATA is TCP-only in 7.2). Spec: spec-7.2-ic-data-plane-decoupling.md (D2b)
The D2b nonzero-ness check false-positived on a fresh cluster: the initial epoch is legitimately 0 on both sides, so every initial DATA HELLO was refused in a reconnect storm (caught by the first two-node smoke). The sound INV-7.2-CONN-EPOCH receiver check is equality with this node's current cluster epoch -- a dialer on a stale epoch is refused and reconnects after observing the bump (fail-closed, self-healing); cross-version senders are already excluded by the HELLO version gate. Spec: spec-7.2-ic-data-plane-decoupling.md (D2b fix)
New cluster_lms_data_plane.c: the LMS aux process binds the node's declared data_addr, dials peers by the same mesh role as the CONTROL plane, runs the HELLO state machines over its own fds, and waits on a WaitEventSet (DATA sockets + MyLatch) in place of the historic plain WaitLatch -- the latch-driven park-serve drains are untouched. No heartbeats and no liveness scan on this plane: CONTROL (CSSD/LMON) is the single liveness authority (Q-D2-1); connect-establishment timeouts, TCP-error close + backoff reconnect, and requester timeouts cover the rest. A node without data_addr runs no DATA plane and LMS keeps the historic latch-only loop. No message routes over the plane yet: the GCS block family flips in the D3/D4 atomic switch; until then the pump only carries HELLO. Spec: spec-7.2-ic-data-plane-decoupling.md (D2c)
ClusterPair/Triple/Quad allocate one extra free port per node and declare it as data_addr in the generated pgrac.conf (allocator- provided, never offset-derived -- adjacent-port collisions made offset defaults unsound). data_ports recorded on the harness object. Spec: spec-7.2-ic-data-plane-decoupling.md (D2d)
ClusterICMsgTypeInfo gains an owning-plane byte (zero = CONTROL, so every existing registration is CONTROL without edits) plus the LMS producer macros (_LMS / _LMS_WORKER / _LMS_DATA; the combined mask pre-includes the spec-7.3 worker bit). Two gates enforce the plane: - dispatch: a frame whose msg_type belongs to the other plane is dropped fail-closed and counted (plane_misroute_reject, per-plane tier1 instance); the connection stays up (unlike the peer-level unregistered/forged-broadcast rejections). - physical send (after the dest=self short-circuit, which stays open to every mask-allowed logical producer): cross-plane send ereports -- an LMS-context emitter of a CONTROL message must stage it into the CONTROL outbound ring for LMON (staging rule). No msg_type is DATA yet: the five GCS block types flip in one commit at the end of the D3+D4 sequence (no half-migrated window). Spec: spec-7.2-ic-data-plane-decoupling.md (D3)
…lper cluster_gcs_send_transition_nowait sent GCS_REQUEST raw; post-flip its callers (the remote X->S downgrade inside the FORWARD handler and the BAST-nudge branch) execute in LMS, where a direct CONTROL-plane send trips the router plane gate by design. Route through gcs_send_envelope_or_loopback, whose existing != B_LMON branch stages the frame into the GRD outbound ring for LMON to send (the D0 staging case; fire-and-forget semantics unchanged, reply still HC74-drops). Spec: spec-7.2-ic-data-plane-decoupling.md (D3)
The dormant half of the D3/D4 atomic flip: - cluster_lms_outbound.c: DATA twin of the GRD outbound ring (Q6-B; single-tail FIFO, LMS is the only consumer, WOULD_BLOCK requeues at the HEAD so per-peer byte streams never reorder -- INV-7.2-DATA-FIFO). The GCS block REQUEST pre-send hook (direct-land arm) migrates with the DATA consumer. New shmem region + t/020 baselines. - cluster_lms_wakeup(): SIGUSR1 -> latch (was SIG_IGN in the LMS skeleton); ring enqueue publishes before the kick. - backend staging entry (enqueue_backend_msg) routes by the msg_type's registered plane: DATA -> LMS ring, everything else unchanged. - cluster_gcs_block_family_on_data_plane(): registry probe (REPLY stands in for all five). The LMON ship_ready / pi_discard tick sites skip when the family is DATA; the LMS loop runs both plus the DATA ring drain when it is. Every branch reads CONTROL today -- the flip commit only edits the six registration structs and the whole pipeline pivots at once (no half-migrated window). Spec: spec-7.2-ic-data-plane-decoupling.md (D4)
GetNamedLWLockTranche requires a matching RequestNamedLWLockTranche in the process_shmem_requests window (I15 pattern); without it the first tranche lookup FATALs at startup. Also stub the register hook for the shmem unit harness. Spec: spec-7.2-ic-data-plane-decoupling.md (D4)
Connection-generation fence for the DATA plane (spec-7.2 D5): - conn_epoch recorded per peer at every CONNECTED transition (dialer stamps its HELLO epoch; accept sides record the peer's), giving the connection an epoch identity. - sender gate in tier1_send_bytes: a DATA frame never goes out on a connection bound to a different epoch (HARD_ERROR -> close + re-HELLO rebinds at the current epoch). Cross-epoch interleave on one byte stream would defeat the reconfig-rebuild ordering argument (8.A). CONTROL is exempt -- its single LMON stream carries the epoch events themselves. - epoch watch in the LMS tick: a bump proactively force-closes the whole DATA mesh (reconnect immediately); the sender gate covers the window before the tick notices. - write-fence linkage: while the fence is enforcing and writes are disallowed, LMS sends nothing on the DATA plane (ship_ready / pi-discard / ring drain all held; thaw wakes the latch and the held frames drain). Block images must not leave a fenced node. Spec: spec-7.2-ic-data-plane-decoupling.md (D5)
Two wait events (F12 four-point symmetry: enum / wait_event.c name map / views table / count): ClusterLmsDataRecv is the data-plane WaitEventSet wait; ClusterLmsDataSend is the same wait while any peer holds a backpressured partial frame (waiting for WRITEABLE drainage). CLUSTER_WAIT_EVENTS_COUNT 118 -> 120; every count pin across the TAP + unit surfaces follows. Two injection points: cluster-lms-data-dispatch fires before the inbound envelope pump of a readable DATA peer (hit surface for the flip e2e legs); cluster-lms-conn-reset SKIP forces a one-shot close of the whole DATA mesh (the reset/epoch legs' trigger) and shares the epoch-watch reset path. Registry 158 -> 160 rows (t/015). Spec: spec-7.2-ic-data-plane-decoupling.md (D6)
…2 D6 The D6 observability commit added two injection points (cluster-lms-data-dispatch + cluster-lms-conn-reset, registry 158 -> 160) and two wait events (ClusterLmsDataRecv/Send, 118 -> 120) but refreshed only part of the pinned assertion surface. Left behind and failing on the branch tip: - t/015 test 3: the full name-list string_agg lacked the two new injection names (the row-count leg alone was updated) - t/020 L15: total registry size still pinned 158 - cluster_regress cluster_smoke: both wait-event count pins (regular + gcluster view) still expected 118 Spec: spec-7.2-ic-data-plane-decoupling.md
An ADG standby without an LMS has no DATA-plane listener, so with the GCS block family on the LMS-owned DATA plane every cross-node block request from or to it retransmits to exhaustion. Mirror the MRP/RFS standby spawn: the LMS spawn gate accepts PM_HOT_STANDBY alongside PM_RUN. Grant ownership stays with LMON (HC4) on every node. Spec: spec-7.2-ic-data-plane-decoupling.md
Two-node cold restart deadlocked the DATA mesh: the HELLO verify layer rejected peers whose declared conn_epoch differed from ours, but cross-node epochs legitimately differ during the cold-form / reconfig window, and connection establishment is the very channel the envelope epoch observation converges through. Remove the HELLO-layer conn_epoch rejection (the plane-mismatch rejection stays) and record the LOCAL current epoch at all three CONNECTED sites, so the sender-side epoch gate compares a connection against the same node's view. Stale frames remain dropped per message by the envelope HC100 check -- receiver-side stale-epoch protection lives at the envelope layer, not the HELLO layer. Spec: spec-7.2-ic-data-plane-decoupling.md
The D5 write-fence linkage paused the WHOLE LMS data-plane send side while the fence is enforcing and writes are disallowed. That turns the write fence into a read fence on the fenced node: the outbound ring carries only backend solicitations (REQUEST / FORWARD / CR + undo fetches -- the master's image REPLY is sent from the dispatch context and never staged), and those are exactly the frames a fenced node's own catalog reads and post-crash rejoin depend on. A full-cluster cold restart wedged behind its own reads: ring full, every connection died with a failed-to-enqueue FATAL, and the starved heartbeats fed a fail-stop reconfig storm. The ungated CONTROL ring never had this problem pre-flip. Keep the fence pause on the image-bearing legs (READY serve ship + PI-discard notes): block images must not leave a fenced node. Drain the solicitation ring unconditionally. Spec: spec-7.2-ic-data-plane-decoupling.md
The atomic plane flip (H-5, no half-migrated window): the five block msg_types (REQUEST / REPLY / FORWARD / INVALIDATE / INVALIDATE_ACK) register plane = DATA with the producer mask extended by the LMS family. REDECLARE alone stays on the CONTROL plane: recovery re-declare must survive a DATA-mesh teardown mid-episode, and none of the five migrated handlers emits it. The registry probe pivots the LMON tick sites, the LMS loop and the outbound-ring routing off these six structs, so this one edit moves dispatch, physical sends and drains onto the LMS-owned tier1 instance. Riding the flip, as one commit must: - master-side fail-stop episode fence at the REQUEST wire handler entry (phase_for_tag == RECOVERING -> DENIED_RESOURCE_RECOVERING, 53R9L retry-safe): a stale-view remote requester routes straight to the master, so the requester-side acquire gate alone cannot close the phantom-holder overtake window - pgrac.conf post_validate hard gate: multi-node + cluster enabled + a real interconnect tier requires data_addr on every node (FATAL at startup instead of the first cross-node block request hanging) - pg_cluster_state gcs keys block_family_plane + plane_misroute_reject (83 -> 85) with unit stubs; TAP gcs-key / wait-event baselines and stale label texts refreshed (t/110-118) - per-node data_addr for every handwritten multi-node conf in the TAP fleet (t/078, 288, 289, 334, 335, 336, 337, 339, 346) - t/358 data-plane flip e2e: flip fact + DATA listeners on both nodes, bidirectional full-table X transfer with wire-request growth asserted after the swap (single-block request asserts are master-placement dependent: HC72 master==self short-circuits the wire REQUEST and rides FORWARD), staging survival via zero plane misroutes and zero plane-gate log errors, cross-held read-back; online-join write-gate polling and OID burn-align per the t/347 idiom Spec: spec-7.2-ic-data-plane-decoupling.md
D6 added 2 injection points (cluster-lms-data-dispatch,
cluster-lms-conn-reset), 2 Cluster: Interconnect wait events
(ClusterLmsDataRecv/Send) and 1 shmem region (lms data outbound), but the
D6 baseline patch only synced t/015 / t/020 / cluster_smoke and missed the
same counts asserted elsewhere (L462 grep-all-assertion-surfaces).
Sync the remaining stale baselines:
- injection registry 158 -> 160 (+ 316 -> 320 sub-keys)
t/017, t/018, t/021, t/022, t/023, t/024, t/030
- Cluster: Interconnect wait events 9 -> 11
t/010, t/011
- shmem regions 79 -> 80 (with visibility inject)
t/021, t/022, t/023
Test-only; no source change. The runtime values were already correct;
only the hardcoded expected counts were stale. All 9 files green.
Spec: spec-7.2-ic-data-plane-decoupling.md
Adds t/360 covering the reachable D7 legs and honest-SKIPping the legs
blocked on a pre-existing substrate wall (user ruling 2026-07-07; see
Stage-7 substrate follow-up register):
Reachable (real assertions):
L1 flip fact: block_family_plane=1 both nodes, DATA listeners bound
L2 value gate: ping-pong X, requester ship histogram p50<5ms, p99<20ms
(measured p50/p99 <= 500us on loopback, ~40x headroom)
L3 hygiene: zero plane misroutes, no plane-gate errors
KNOWN-BLOCKED (SKIP with reason; mirrors t/339's pattern):
L4 kill -9 LMS x3 -- F1-8 block_device crash-recovery raw_layout_lock
needs GES, unavailable before the crash-restarting node re-joins
(recovery-vs-membership substrate gap, orthogonal to the DATA plane)
L5 conn-reset injection -- F6-1 persistent GUC storm crashes the passive
LMS with kevent() EBADF; needs one-shot arm + WES fd hardening
L6 data-dispatch injection -- F6-2 point sits off the actual block recv
path (hits stay 0); needs repositioning
23 tests, 6 honest SKIPs. t/348 (3-node) regression green.
Spec: spec-7.2-ic-data-plane-decoupling.md
Publish and seal native-era XID authority after seed shutdown, adopt prehistory from per-node anchors, and max-merge startup nextXid before ShmemVariableCache seeding. Spec: spec-6.15b-xid-authority-native-era.md
A second cluster.enabled=off seed pass on a SEALED authority re-opens the native era: clear SEALED at bootstrap so a crash of that pass leaves joiners fail-closed (53RB5, unsealed) instead of silently adopting the previous pass's stale high-water -- false-invisible for every xid the crashed pass consumed. The pass's own clean shutdown re-publishes and re-seals monotonically. Adds cluster_xid_authority_begin_native_run(), the bootstrap enabled=off arm call, unit coverage (seal kept-hw/cleared-flag/idempotent/re-seal), and t/361 N4 (sealed pass-1 -> unsealing pass-2 boot -> immediate-kill -> joiner refuses adoption). Spec: spec-6.15b-xid-authority-native-era.md
added 28 commits
July 8, 2026 18:24
t/358-361 are reserved by parallel lanes (spec-7.2 / S-xid), t/363 by the S-dead lane; occupancy verified against origin branches. The 4-node formation + kill -9 + convergence run gets its own shard for wall-clock isolation. Spec: spec-4.6a-grd-recovery-liveness.md
…adow unit Same inline suppression + reason as the other cluster_unit main() signatures (test_cluster_grd.c precedent); a newer local cppcheck flags it, the CI baseline version does not.
…y baseline + async pre-bump WAIT_EPOCH wedge The ship-gate nightly on the marker-async head surfaced two lane-introduced regressions that the fast-gate matrix does not cover: 1. Dump-category baseline miss: reconfig telemetry added a pg_cluster_state category (55 -> 56), but three cross-node TAPs that assert the total (t/204/205/206) were not updated — only the locally-run L405 subset was. Bump them to 56 with the spec-2.29a note. 2. Async pre-bump WAIT_EPOCH wedge (behavior regression): the async fence / join-Phase-1 / node-remove marker staging bumps the membership epoch at stage-entry but only publishes the reconfig event once the voting-disk marker ACKs — now several LMON ticks later instead of the pre-async single-tick spin. In that window the coordinator's OWN GRD recovery IDLE tick re-captured the already-bumped epoch as its WAIT_EPOCH baseline, so the P0 accept that followed the publish read old == cur and froze the affected shards forever — the spec-4.6a section-0 shape, here triggered by the coordinator on itself with no IC piggyback (reproduced deterministically as t/293 fail-stop remaster + t/326 node-leave shard reopen). Fix: while any pre-bump stage is live, the GRD IDLE tick holds its last stable pre-reconfig baseline instead of re-capturing (cluster_reconfig_has_pending_prebump_stage guard). This restores the pre-async bump->publish atomicity as seen by GRD without touching the epoch bump timing or any user-visible reconfig contract. Adds the cluster_grd_recovery_event_old_epoch accessor + a baseline-hold unit test. Local gates (cassert): unit 158 binaries (incl. baseline-hold test), t/293 + t/325 + t/326 family all green (were red), t/204/205/206 + observability TAPs, PG regress 219/219, clang-format/headers/scn-cmp clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…sion ②b, others-dead) The spec-2.29a marker-async change turned the clean-leave COMMITTING marker into an async, multi-tick wait. In a slow environment (nightly Linux) the leaving node finishes its drain and stops heart-beating inside that wait window, so CSSD marks it SUSPECTED->DEAD and bumps the global dead_generation — and the coherence gate, which compared that SCALAR dead_generation, wrongly read it as a third-party death intruding mid-drain and escalated an otherwise healthy clean leave (t/310 dormant-member legs, t/331 clean_leave x idle). apply_clean_leave_as_coordinator bumps + publishes atomically, so this is NOT the baseline-hold wedge (②a); it is the coherence predicate counting the leaving node's OWN expected DEAD. Fix: compare an others-dead bitmap (the dead set EXCLUDING the leaving node) instead of the scalar dead_generation, at all four coherence sites (CL_COHERENT macro, drive_drain commit-point, staged-ACK re-check, non-staged pre-check). cl_state snapshots the others-dead set at bind; the pure predicate does a bitmap compare (fail-closed on a missing view) and is unit-tested against the reflexive-case matrix. Every third-party incoherence still escalates (epoch move OR a non-leaving node entering the others-dead set); only the leaving node's own expected transition is tolerated. 8.A / CL-I3 protection is unchanged — argued in spec-2.29a §②b with a case matrix. Local gates (cassert): policy unit reflexive-case matrix, cluster_unit 158 binaries, PG regress 219/219, clang-format/headers/scn-cmp clean. NOTE: t/310/331 are timing-sensitive — the false escalation only reproduces in a slow environment, so the fix is verified by nightly, not locally (a local run passes the assertions). A separate pre-existing cassert teardown SIGABRT flake in t/310 (isolated: reproduces without this change) is unrelated. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…hash) The P6 all-done gate and the WAIT_EPOCH coordinator witness keyed the cross-node DONE on event_id, but event_id folds the sender-local cssd_dead_generation, which drifts with each survivor's private flap observation history and never converges across nodes: any flap asymmetry made the survivors compute different ids for the same episode, drop each other's DONEs, and wedge P6 forever — nondeterministically reintroducing the permanent-freeze shape this branch exists to remove. The quorum-accepted dead SET is what actually converges, so the DONE payload now carries a hash over the dead bitmap alone (same kernel as the event_id hash, riding the same request-id field pair — no envelope change), stamped per episode at P0 accept. Accounting, the P6 gate and the witness compare against the stamp; event_id stays a purely local accept-dedup scope. A late DONE from a previous episode still cannot back a new one: a coordinator re-election changes the dead set (hash mismatch) and a same-node re-death rides a higher epoch through the interposed JOIN bump (epoch conjunct). Pre-accept frames now mismatch the previous stamp and are dropped; senders re-announce every tick, so accounting always lands after the accept snapshot — closing the first-event pre-accept window as a side effect. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R2/R4)
Revert the scope short-circuit that returned NORMAL for a dead static master's blocks wherever online thread recovery cannot run (the default configuration and every >2-node deployment): it skipped both the is_materialized cold-block door and the redo-coverage lost-write door, so a committed write only the dead node saw could be silently read stale from shared storage. No other guard sits on that read path — the GRD freeze ends with the episode, the HW gate covers only extend high-water marks, and the page-SCN checks ride the ship path. Out of scope the posture is now an explicit bounded retryable error (53R9L, hint updated to name the way out: restart the failed node, or enable online thread recovery in a supported scope) for every tag whose static master is dead — including never-written blocks, which the cold-block door cannot distinguish from not-yet-replayed ones — and the door reopens the moment the failed node stops being DEAD. In-scope behavior is byte-identical. The now-orphaned scope helper is removed. t/362 asserts the honest two-outcome contract on fresh connections (success or explicit SQLSTATE — 53R9L or the TT-unknown visibility door — never a hang), rides pre-warmed sessions for convergence observability, and pins the D12 cleanup counter to a positive delta; t/293's post-kill extend leg follows the same posture. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R1/R3)
…grade cleanup - Order the worker's terminal-result store after its backoff-deadline store (pg_memory_barrier) so the relaunch decider never pairs a fresh BLOCKED with a stale deadline and skips one backoff wait. - A structural cause discovered only by the WORKER (SIGHUP race) now leaves the retry deadline unset so the FSM's next tick takes the MARK_STRUCTURAL branch and emits the once-per-episode operator WARNING with the configuration hint. - Wrap the local S->X upgrade in PG_TRY and release the temporary S claim when the ACK wait throws (cancel via CHECK_FOR_INTERRUPTS), not only on the false return; document why the completed upgrade hard-resets the local S refcount (the X grant subsumes all local S declarations). Linker-only exception stubs for the units that link cluster_pcm_lock.o. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R5/R6/R7/R9)
…e-bump shmem bit Two review-r2 items on top of the ②b others-dead coherence fix: P2-1 — the leaving node's barrier-tick own-commit latch (cl_leaving_barrier_tick) inferred 'my leave committed' from epoch>baseline AND the others-dead bitmap unchanged. The bitmap is not monotone: a third-party node that false-fail- stopped (bumping the epoch) then recovered leaves CSSD hysteresis DEAD->ALIVE, so the bitmap rebounds to its bound value while the scalar dead_generation only advances. If the leaver's first epoch>baseline observation lands after that rebound it would mis-latch a leave the survivor coordinator actually refused, suppress the barrier-deadline escalation, and hang forever in BARRIER_WAIT. Restore the scalar conjunct (epoch>baseline AND others_dead==bound AND dead_gen==baseline) via the pure, unit-tested cluster_clean_leave_own_commit_ latched predicate. This does not reintroduce the ②b false positive: the leaver's OWN alive->DEAD never bumps ITS OWN dead_generation (a node does not observe itself dead), so on the leaver side the scalar stays at baseline through its drain. Survivor-side coherence keeps the bitmap-only check. Adds the rebound negative-leg unit (others_dead unchanged + dead_gen advanced -> escalate). t/274 — a backend-context coordinator (the fail-stop inject test) advances the epoch then SYNCHRONOUSLY waits for the fence marker before publishing (MyBackendType != B_LMON path). During that seconds-long wait the epoch is bumped but unpublished, and a concurrent LMON GRD IDLE tick re-captures the post-bump epoch as its WAIT_EPOCH baseline -> old==cur -> the thread replay slot never reaches DONE -> HG#1b/HG#5 done-poll timeout. The ②a baseline-hold guard only reads LMON-process-local staged flags, invisible to a backend's pre-bump window. Publish it in shmem: ReconfigShmem.prebump_sync_active set before the bump, cleared at every return; the guard reads the shmem bit OR the local stage. 8.A (durable-marker-before-publish) unchanged. Reproduced the wedge locally at 87f7930 (HG#1b slot-DONE poll times out forever), then verified the fix: t/274 goes 18/18 green (exit 2 -> 0). This also fixes the long-standing main flake (the wedge is deterministic once LMON is freed from the sync-marker park; it was merely probabilistic before) — the t/274 known-flake / L66 registration should be revised. Local gates (cassert): cluster_unit 158 binaries (incl. own-commit-latched + version-coherent matrices), t/274 18/18, PG regress 219/219, clang-format/ headers/scn-cmp clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
The leaving node's barrier-tick own-commit latch must be backed by
direct evidence (the durable COMMITTED marker confirmation for THIS
leave attempt) and must be immune to third-party transient false-DEAD
flap noise: the scalar dead_generation is monotone, so a flap during
the leave window advances it forever and the r2 P2-1 three-conjunct
inference then refuses a healthy committed leave until the barrier
deadline escalates it (nightly t/331 C1/C4 false-escalation, run
28948167577). The reworked U3b matrix pins:
(a) evidence, no noise -> latch
(d) evidence + flap noise -> still latch (RED on current code)
(c) no evidence, any noise state -> never latch (deadline escalation
stays armed; the r2 P2-1 refused-leave mis-latch wedge stays shut)
Known-red TDD commit: leg (d) fails on the current three-conjunct
predicate; the follow-up commit flips the latch to marker evidence.
Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
A crash between write_header_both's two durable renames leaves the transitioned flag (unseal / CLUSTER_ERA stamp) in the primary copy only, with a stale pre-transition .bak behind it. Nothing on the restart path rewrote the pair -- the bootstrap gates skipped the transition call when the primary already showed the new flag, and the transition functions early-returned on the same observation -- so the stale .bak survived the whole native run. Any later transient primary read failure then fell back to it: a stale SEALED .bak hands joiners the previous pass's high-water (false-invisible + xid reissue), and a stale pre-CLUSTER_ERA .bak re-opens the native-era re-entry guard. Make the two flag transitions idempotent re-asserts: the bootstrap calls them unconditionally on every boot of their respective arm, and they rewrite BOTH copies until both validate with the post-transition flags (a missing or invalid copy counts as unsettled and is restored too). Once both copies are settled the call is a no-write no-op, so the steady-state boot cost is two header reads. All fail-closed semantics (53RB5 refusals, CLUSTER_ERA re-entry FATAL, corrupt-authority PANIC) are unchanged. Red-first unit coverage: torn-window tests for both transitions (complete the transition, re-install the old image as .bak, re-run the boot path, assert the .bak is repaired and that a primary read failure falls back to the repaired image, not the stale one). Spec: spec-6.15b-xid-authority-native-era.md
…osed) The divergent-lineage prefix check broke out of its comparison loop and returned CONSISTENT on the FIRST missing local pg_xact page. That is only sound for a limit that ends where the local tree ends; pg_xact front truncation (SimpleLruTruncate removes whole low segments once every xid they cover is frozen past oldestXid) means a joiner can have segment 0000 legitimately absent while later segments are present AND divergent -- the first-page ENOENT then declared the clone CONSISTENT without ever comparing the surviving pages, and the adopt arm overwrote the joiner's own live-xid outcome pages with the seed's bits (false-visible / false-invisible). Pass the caller's recovery-anchor oldestXid into the check and start the comparison at oldestXid's own 2-bit slot: bits below it are frozen truth that CLOG never consults again and whose on-disk survival is an accident of truncation timing, so they are exempt whether the pages survive or not (neither truncation nor tuple freezing rewrites surviving CLOG bytes, so this can never hide live evidence). Within the comparable range [oldestXid, min(own_next, native_hw)) a missing local page is an anomaly -- pg_xact always covers [oldestXid, nextXid) on a well-formed node -- and now returns UNAVAILABLE (fail-closed 53RB5 at the caller), never CONSISTENT. Sub-byte boundaries at oldestXid are masked at 2-bit precision. Red-first unit coverage: front-truncated divergent clone must DIVERGE (returned CONSISTENT pre-fix), divergence strictly below oldestXid stays CONSISTENT (frozen exemption, mid-byte lead mask), oldestXid at the divergent slot DIVERGES, and a hole inside the comparable range is UNAVAILABLE (the old truth-table leg expecting CONSISTENT for a missing segment encoded exactly the blind spot and now expects UNAVAILABLE). Spec: spec-6.15b-xid-authority-native-era.md
…not inference Nightly t/331 C1 (clean_leave x idle) / C4 (leave_remove x idle) regressed at the r2 P2-1 head (run 28948167577; parent green): the leaver's barrier-tick own-commit latch inferred its commit from epoch>baseline + others-dead bitmap unchanged + scalar dead_generation unchanged. The scalar is monotone, so a third-party transient SUSPECTED->DEAD->ALIVE flap on the leaver's local CSSD view during the leave window advances it forever; the latch then refuses a leave the coordinator ACTUALLY committed and the immediate-escalate arm (or the barrier deadline) aborts it: phase ends ABORTED_ESCALATE -> IDLE, never 'committed', breaking the C1/C4 drains+commits assertions. Reverting to bitmap-only would re-open the r2 P2-1 wedge (rebound mis-latch of a REFUSED leave suppresses escalation forever), so neither inference is usable. Replace inference with direct evidence: latch <=> the durable COMMITTED marker for THIS leave attempt is confirmed. The evidence already reaches the leaver with zero extra IO: the coordinator sends the nonce-bound LEAVE_COMMITTED only after its qvotec ACKs the COMMITTED marker majority-durable (there is no runtime voting-disk read path on the leaver, and none is needed - the LMON tick budget this spec protects is untouched). cl_committed_handler now routes through a pure identity gate (self-addressed + currently leaving + per-attempt nonce + committed epoch past the bound baseline; fail-closed on any mismatch, so a stale confirmation - and through it a stale COMMITTED marker from a previous leave of the same node - can never false-latch) and records the attested committed epoch E before publishing the evidence flag; the barrier tick consumes E instead of re-reading its possibly-stale local epoch view. The latch and the P1-V0.7 exit gate collapse into one step (the evidence IS the durable-truth-source confirmation). Escalation semantics are unchanged: no evidence by the barrier deadline -> the existing cl_escalate path, which is exactly what bounds a refused leave (it never gets a COMMITTED marker). Third-party flaps no longer matter: the coherence observations are now contract inputs the predicate must ignore (pinned by the U3b unit matrix, red-first in the parent commit) and feed only a flap-noise LOG at the latch. Survivor-side coherence sites (drive_drain / staged-ACK / pre-check) are untouched. Unit: U3b matrix flipped green; new U3c pins the evidence identity gate (match / stale-nonce / misrouted / not-leaving / epoch-not-advanced). Local gates: t/331 x4 all green (6/6), t/310 24/24, t/363 18/18, t/274 18/18 (prebump shmem-bit stays green), cluster_unit 158 binaries, cluster_regress 13/13, PG core 219/219, clang-format-18 + comment-headers clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
WAIT_BARRIER / WAIT_CLUSTER carried only epoch-coherence guards, so a
survivor whose CSSD detected a coordinator-folded multi-death late kept
announcing REDECLARE_DONE under its stale dead-bitmap hash: peers that
stamped the full set dropped those frames on the composite key and P6 —
which has no timeout by design — wedged permanently (survivor-behind
detection skew). Add a mid-episode guard that aborts to IDLE on a newer
local event whose dead-bitmap hash differs: the next tick re-consumes
the event, re-stamps recovery_event_bitmap_hash, re-runs recovery
against the fuller dead set and re-announces DONE under the new
composite key. Same-set dead_generation drift is absorbed without
episode churn (the event_id keeps its local ABA-scoping role). The
coordinator-behind direction needs no handling: the coordinator's own
later detection bumps the epoch again and the epoch guards abort.
Correct the stamp-site / accounting / hash-helper comments that
overstated the bitmap as quorum-ratified (each node stamps its OWN
accepted event's bitmap; convergence is eventual via the CSSD deadband
plus this guard), and assert a stamped episode hash is never 0 — 0 is
reserved as unstamped in the mark_peer_done drop test, and a JOIN
episode's all-zero bitmap hashes to a fixed nonzero constant.
Unit: survivor-behind growth leg (H({1}) stamped at the folded epoch,
grown {1,2} event arrives -> abort, re-stamp, full-set remaster, DONE
accounting converges) plus a same-set drift no-churn leg; both go red
with the guard disabled.
Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-2, r3-P3a)
…arrier hw_remaster_record_terminal stores the backoff deadline before the terminal result with a release fence between the stores, but the LMON relaunch decider loaded result-then-deadline with no read-side pairing: on a weakly-ordered CPU it could still pair a fresh BLOCKED with a stale zero deadline and skip one backoff wait. Insert pg_read_barrier() between the result and deadline loads (attempts needs no fence — it is written only by the single-writer LMON FSM) and cross-reference the pairing at the write site. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P3b)
The serve-gate unseal (static master no longer CSSD-DEAD -> NORMAL) is heartbeat liveness, not a direct recovery-completion signal. Document the traced proof of why returning NORMAL is nevertheless safe: CSSD — the only heartbeat sender — is spawned by the phase-4 driver at the PM_RUN transition, after the startup process completed the node's crash recovery against shared storage; and even under a stale-ALIVE view a fetch cannot complete against a still-recovering node, because the master-side handler default-denies until the node is an in-quorum MEMBER and only the (equally post-PM_RUN) QVOTEC process can establish the quorum lease after a restart wiped shmem. Deny replies surface as bounded 53R9L, endpoint unavailability as bounded 53R90, and no path falls back to a silent local storage read. Replaces the previous wording that asserted recovery-completion semantics without grounding. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-1 prove-safe)
… leg The Amendment v1.2 (R1) serve-gate fix had no test lock: the t/362 two-outcome legs stay green under a reverted gate (their success arm accepts a NORMAL serve of a dead-master page) and no unit links the real phase decision. Add L4h: after the kill, with the dead node never restarted, once the observed survivor's own recovery episode reached IDLE, a FRESH backend on a cold survivor runs a wide pg_class/pg_attribute scan that MUST fail with exactly 53R9L — no success arm, bounded return. An out-of-scope formation never publishes the dead node's materialization proof, so every dead-master page stays RECOVERING while the node is down; the scan spans dozens of catalog pages hashed ~1/4 to the dead node and cold-reads them past the 16MB test pool, and it touches no dead-node-written rows (heap INSERTs only, no DDL), so 53R9L is the only legal outcome. Verified red-first: with the gate reverted the suite fails at exactly this leg while every other leg stays green. The episode-IDLE wait polls the pre-warmed session with the exact query shape it ran pre-kill and compares in Perl: a cast or operator added to the SQL performs fresh syscache lookups that themselves cold-read a dead-master catalog page and trip 53R9L (the very mechanism under test). Also accept the 53R51 write-fence rejection in the pre-existing two-outcome write legs: a transient lease/epoch fence right after the reconfig is an explicit, retryable fail-closed rejection inside the same honest contract. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P1-1 TAP hard leg)
…nt_old_epoch after r3 3-way merge
…hards, 361 into crossnode-b)
… P0 substrate
Bring the spec-7.2-ic-data-plane branch (25 commits, D0-D7) on top of
stage7-p0-integration so the D7 fault-injection legs (L4/L5/L6) can be
re-evaluated against the formation/reconfig-liveness/xid-authority P0
substrate (spec-6.15b PR#24 + spec-4.6a PR#26 + spec-2.29a PR#25).
Conflict resolutions (cross-surface grd/lmon/gcs + counter baselines):
cluster_lmon.c -- keep BOTH the stage7-p0 iter_started_at profiling
timestamp AND the spec-7.2 D1 lazy-duty >=1Hz floor;
reuse the single iteration-start timestamp for both
(TIER_1 and stub loops).
cluster_lms.c -- union of includes (cssd + epoch from P0, gcs_block +
write_fence from spec-7.2).
test_cluster_lmon.c -- keep BOTH new unit tests; UT_PLAN 6->7 (both sides
bumped 5->6 independently; the union is 7).
t/015,017,018,020-024,030 -- injection registry 158 base + spec-2.29a
(+1 cluster-qvotec-marker-service-hold) + spec-7.2 D6
(+2 cluster-lms-data-dispatch/conn-reset) = 161 (was
159 on P0 / 160 on spec-7.2 in isolation). Name list
regenerated from the merged cluster_inject.c registry
(ORDER BY name); t/030 M5 sub-keys 161x2 = 322.
Verified: clean build (all targets), cluster_unit 160/160 binaries pass,
zero conflict markers remain. D7 leg re-evaluation follows in later
commits.
The spec-4.6a Amendment v1.2 (R2) composite convergence key gated both
recovery_done_epoch[] and recovery_done_bitmap_hash[] behind one drop gate
in cluster_grd_recovery_mark_peer_done(). A rejoining node never P0-accepts
the JOIN episode as its own FSM episode (JOIN_COMMITTED is published
coordinator-side only), so its local episode-hash stamp stays 0 forever; the
composite gate then dropped every survivor REDECLARE_DONE, the join fence
(cluster_grd_block_view_rebuilt) never lifted, and every joiner-home block
access fail-closed 53R9L permanently. Nightly t/347 L4-iii (rejoin with
xid_striping) and t/353 L5 (fresh insert on reborn node0) pinned the wedge.
Split the two accounting axes:
* done_epoch[] accounts unconditionally (monotonic-max), feeding the
epoch-only join fence. Sound because REDECLARE_DONE(epoch=E) is
broadcast only after grd_block_redeclare_scan_complete(E), so a DONE at
E proves the sender finished its WAIT_BARRIER re-declare at E -- the
fence's safety condition, independent of the sender's episode dead set.
* done_bitmap_hash[] stays composite-gated for the P6 cluster gate and the
WAIT_EPOCH coordinator witness, which still AND both axes and stay
fail-closed on the withheld hash half (r3-P2-2 multi-death skew
protection unchanged).
New unit leg test_recovery_idle_joiner_accounts_done_epoch_for_fence asserts
the fence lifts once survivor DONEs land while the composite half stays
withheld; the stale-bitmap and dead-set-growth legs gain a negative
assertion that the epoch axis alone never fires the composite escape.
Spec: spec-4.6a-grd-recovery-liveness.md
The same S-reconfig nightly that pinned the join-fence wedge also flagged
two stale dump-count baselines that lagged already-shipped counters:
* t/108 pg_cluster_state pcm category 22 -> 23 (spec-4.6a D12
dead_cleanup_entries).
* t/249 grd_recovery dump roster 18 -> 31 keys plus the exact sorted key
list (spec-4.6 2.4 base + spec-5.16 D5 join direction + spec-4.6a
liveness/episode surface).
Both stay exact-count / exact-roster assertions; the code already emits the
31-key grd_recovery set (verified against cluster_debug.c).
Spec: spec-4.6a-grd-recovery-liveness.md
…KIP t/360 L5
The cluster-lms-conn-reset injection modelled a single epoch-bump DATA-mesh
reset, but its only arm path is the process-local injection GUC, which stays
armed and re-set skip_pending on every LMS tick -- a reset storm that starved
the mesh and (on the passive side) tripped the accept/close WES churn into a
crash, then hit the F1-8 block_device recovery wall. That made t/360's L5
leg KNOWN-BLOCKED.
Fix: latch the injected reset so it fires exactly ONCE per LMS process, and
only latch after it has actually torn down a live connection (so an arm-
before-connect race retries instead of wasting the single shot). The real
epoch-bump reset path (cur_epoch != dp_last_epoch) is unaffected.
t/360 L5 is now a real e2e leg (installcheck, 3x stable):
L5.1 the injected reset fires once (reset-log count +1)
L5.2 the mesh reconnects -- both nodes emit a fresh "state CONNECTED"
after the reset (node0 re-dials, node1 re-accepts)
L5.3 one-shot: exactly one reset event, no per-tick storm
L5.2 proves reconnect via log evidence rather than a post-reset SQL probe:
by that point the value-gate ping-pong has burned fault_t's rows' xmax into
recycled TT slots, so ANY access (read or write) fail-closes with "cluster
TT status unknown" -- the pre-existing #119 recycled-slot wall, orthogonal to
the DATA plane and not caused by the reset. Block-shipping capability itself
is quantified by the L2 value gate (394 ships, p99 <= 500us).
L4 (F1-8) and L6 (F6-2) stay honest SKIP; the stage7 P0 substrate was
evaluated and does not unblock either (L4 needs the F1 recovery-vs-membership
fix; L6 needs the data-dispatch injection point repositioned -- observability
polish, block-over-DATA already proven by t/358 + the value gate).
Spec: spec-7.2-ic-data-plane-decoupling.md D7 / F6-1
t/353 L5 crashes node0 and has the reborn node take a FRESH HW space lease, which requires the survivor to complete HW remaster of the dead node's shards. HW remaster reads the dead node's per-thread WAL, so on this shared_data 2-node pair it is only recoverable when cluster.wal_threads_dir is configured (cluster_hw_remaster_recoverable). The pair was created with shared_data but WITHOUT wal_threads_root, so recoverable() returned false, the remaster was marked BLOCKED_STRUCTURAL, and the reborn node's fresh-lease DDL/insert/count fail-closed (L5 subtests 28-31, nightly crossnode-b). Opt the pair into wal_threads_root, matching every other HW-remaster- dependent kill test (t/247/248/274/293). This completes the per-thread WAL harness config that the RACvsRAC/ClusterQuad harness already carries but this ClusterPair test was missing. Spec: spec-4.6a-grd-recovery-liveness.md
…t counter, epoch short-circuit Code review r2 on the F6-1 one-shot conn-reset commit (three findings): - Finding 1 (over-claim + convergence): the L5.2 note claimed the recycled remote-ITL-slot fail-closed boundary blocks ANY access to fault_t. It is actually per-recycled-slot and does not wall a fresh table's new, un-recycled xids. Corrected the wording (t/360 L5.2 comment) and added t/360 L5.4/L5.5: after the reset+reconnect, a FRESH coincidence table (fault2_t) is X-swapped cross-node, proving block transfer converges (both requesters' block_request_count grow, zero plane misroutes) -- the direct convergence proof the reconnect log evidence alone did not give. - Finding 2 (observability): implement the lms_conn_resets counter -- a shmem atomic in ClusterLmsSharedState, bumped once per DATA connection torn down by a reset (epoch bump in production, injection in the test), dumped under pg_cluster_state category='lms'. t/360 L5.4 asserts it recorded the reset. The cluster-lms-data-dispatch injection point stays a deferred follow-up: wiring a counter on its currently-unreachable branch would be a permanently-zero (false) observable. - Finding 3 (nit): restore the epoch-bump short-circuit precedence. inject_reset is armed only when there is no epoch bump, so a real bump is attributed to the epoch (not the F6-1 one-shot) and does not consume the one-shot latch; the injection still fires exactly once on a later tick. Test-only path; production is unaffected. Also complete the D6 wait-event snapshot bump (118 -> 120, +2 LMS DATA-plane wait events) in the stage3/4/5.5/5/beta acceptance unit tests, which the earlier bump left stale (stage2 + views + retransmit were already 120). Added the matching cluster_lms_get_conn_resets stub in test_cluster_debug. Local gates: cluster_unit 160/160 (0 not-ok), cluster_regress 13/13, PG core 219/219, t/358 flip PASS, t/360 PASS (L5 convergence + L4/L6 honest SKIP, value gate p99 <= 500us). Spec: spec-7.2-ic-data-plane-decoupling.md
Advances the reconfig P0 contribution from 0d57fa9 to 04ca161, bringing in the two nightly-surfaced fixes (t/347 join-fence done-epoch axis 152b579 + t/353 wal_threads_root 04ca161) plus the dead-master serve-gate/read-barrier hardening. Integration now carries all three P0 lanes at their tips (xid d8b1d6d, dead ea2cdc9, reconfig 04ca161) fused with spec-7.2 IC data-plane. Spec: spec-4.6a, spec-7.2
The spec-7.2 D3+D4 flip e2e test file was missing the mandatory 'Author: SqlRush <sqlrush@gmail.com>' header (CLAUDE.md rule 11); it never surfaced on the 7.2 lane because that branch was not pushed to a main-based fast-gate. Add IDENTIFICATION + Author + copyright to match the sibling t/360 banner. Spec: spec-7.2
sqlrush
force-pushed
the
stage7-72-integration
branch
from
15f90af to
b62331e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this lands
Fuses the Stage 7 P0 cluster-formation substrate with the spec-7.2
inter-connect data-plane decoupling onto
main.P0 lanes (all at tip):
stage7-p0-xid-authority, tipd8b1d6d414)stage7-p0-dead-diagnosis, tipea2cdc99ee)stage7-p0-reconfig-liveness, tip04ca16191e— folds thet/347 join-fence done-epoch axis + t/353
wal_threads_rootnightly fixes)stage7-p0-integration,bfe00bc2e9)Data plane:
injection + review r2 convergence probe / conn-reset counter / epoch
short-circuit), tip
38236adf49fused.Local verification (macOS, --enable-cluster --enable-cassert)
Multi-node kill/recovery TAP legs (t/249, t/347) are not verifiable in the
local macOS sandbox (initdb has no text-search config for the
zh_CN.UTF-8locale → pg_ctl start fails); they are covered by the Linux fast-gate + nightly.
Known-red note
stage6-crossnode-a(t/347 restart leg:Tests: 44 Failed: 0, then pg_ctlstart) is already red on
main's own nightly (runs 28934920246 /28860681493) and is not introduced by this change. No new failing shard is
added relative to current
main.Spec: spec-7.2, spec-4.6a, spec-7.0