feat(cluster): undo-block shared-storage physical data plane (spec-5.22b D2-2)#28
Open
sqlrush wants to merge 4 commits into
Open
feat(cluster): undo-block shared-storage physical data plane (spec-5.22b D2-2)#28sqlrush wants to merge 4 commits into
sqlrush wants to merge 4 commits into
Conversation
added 4 commits
July 8, 2026 12:37
Name an undo block (including the segment TT header block) as a first-class cluster resource: (owner_node, undo_segment, block_no, generation) encoded into the 16-byte ClusterResId wire format as class 0xF9. Undo is the first owner-as-master resid class: the resource master IS the owning instance (cluster_undo_resid_master), never a GRD shard-hash node. The GRD hash-master lookup now fails closed (Assert + new SQLSTATE 53R9Q) if an undo resid reaches it, and the generation predicate (segment wrap_count) lets callers fail closed on recycled-segment stale references. Pure identity layer only: no grant/PI/serving data plane, no write path change, no on-disk format or wire ABI change (ClusterResId stays 16 bytes; new class byte value only). - new cluster_undo_resid.h: class byte 0xF9 + collision StaticAssert net (9 header-visible classes + LOCKTAG_LAST_TYPE) + field mapping - new cluster_undo_resid.c: encode/decode/is_undo/master/ generation_matches pure layer (standalone-linkable) - cluster_grd.c: fail-closed undo-class guard at cluster_grd_lookup_master entry - errcodes.txt: 53R9Q ERRCODE_CLUSTER_UNDO_RESID_HASH_ROUTED - tests: test_cluster_undo_resid (8 cases: round-trip, class byte, discriminator, 16B wire lock, owner bounds, owner-as-master, generation); guard-fire leg in test_cluster_grd via an ereport trampoline; t/006 SQLSTATE spot-check line Spec: spec-5.22a-undo-block-resource-identity.md
…2-1)
D2-1 wires the D1 undo-block resource identity into the GCS data plane's
routing half: two predicates that keep undo resources off the GRD/GCS
hash-master path. Their authority lives at the owning instance, so the
master IS the encoded owner_node (cluster_undo_resid_master), never a
shard hash -- the D1-5 guard already fails closed (53R9Q) on any undo
resid that reaches cluster_grd_lookup_master / cluster_gcs_lookup_master.
- cluster_undo_gcs.{c,h} (new): cluster_undo_block_lookup_master returns
the owner_node; cluster_undo_block_master_is_self is the local
fast-path gate (owner_node == cluster_node_id). The owner-incarnation
epoch self-check that L364 requires before serving from the local fast
path is deferred to the grant/acquire path (D2-3), where the
co-sampled live-authority triple is available, so this routing layer
stays pure (node-id only) and cluster_unit links it standalone.
- src/backend/cluster/Makefile: cluster_undo_gcs.o added to OBJS.
- test_cluster_undo_gcs (new, U1/U2): lookup_master returns owner (not a
hash); master_is_self true for owner==self, false for a foreign owner.
Links cluster_version.o + cluster_undo_resid.o + cluster_undo_gcs.o
with a test-owned cluster_node_id stub.
Zero behaviour change: no consumer routes undo through these predicates
yet (undo_gcs_coherence data plane is D2-3+), so this mirrors D1's
land-then-wire shape.
Spec: spec-5.22b-undo-block-gcs-integration.md
…22b D2-2)
D2-2 gives an undo segment an optional physical home on the shared
cluster_fs root under GCS, breaking the DataDir-local hardcode in the
undo I/O path. The migration is gated on a new GUC
(cluster.undo_gcs_coherence, default off), so with the default the
whole change is inert and every path resolves to the local DataDir
byte-for-byte as before -- mirroring D1/D2-1's land-then-wire shape.
Path intent (the core contract):
- ClusterUndoPathIntent {RUNTIME_SHARED, MATERIALIZED_LOCAL} names which
home a segment resolves to. cluster_undo_intent_for_owner(owner)
derives it -- an own-instance owner is RUNTIME_SHARED, a foreign owner
(in D2 only ever a dead-origin copy recovery rebuilt in the local
DataDir) is MATERIALIZED_LOCAL. Static inline in cluster_undo_alloc.h
so the ~30 call sites add no link dependency.
- cluster_undo_path_uses_shared_root(intent, peer_mode, coherence) is
the single pure decision function: RUNTIME_SHARED resolves under the
shared root only when peer-mode AND coherence are on;
MATERIALIZED_LOCAL always stays local (P1-3 hard contract -- the
dead-origin by-xid resolve path must never move to shared storage).
Threading + migration:
- intent is a new first parameter on cluster_undo_path_resolve and the
four undo smgr APIs (read/write_block, read/write_header_bytes) and is
carried in the get_segment_fd fd-cache key.
- the redo write surface (cluster_undo_xlog.c build_undo_segment_path /
redo_open_segment / redo_stamp_slot) delegates to the SAME
cluster_undo_path_resolve, so runtime and redo cannot split-brain an
own-instance segment onto different roots.
- the shared home resolves via cluster_shared_fs_undo_path_resolve /
_instance_dir_resolve (owner-partitioned instance_<N> under
cluster.shared_data_dir). Undo is outside the RelFileLocator
namespace, so this is a path resolver, not a shared_fs vtable callback.
- ensure-instance-subdir gains a dual branch (shared root -> pg_mkdir_p
on the shared tree; local -> the existing mkdir) so a coherence-on
tree is physically complete.
Supporting:
- new GUC cluster.undo_gcs_coherence (bool, default off, PGC_SIGHUP);
check hook rejects turning it on while cluster.shared_data_dir is
unset (GUC_check_errdetail).
- D4 tripwire: cluster_undo_path_resolve asserts
(intent == RUNTIME_SHARED) == (owner == self), guarding the future
dead-owner foreign-runtime-shared serve case.
- drop a now-redundant local extern of
cluster_undo_segments_max_per_instance (the new cluster_guc.h include
already declares it) to clear a shadowed-declaration finding.
- cluster_undo_alloc.h now includes cluster/cluster_scn.h (latent).
Default path (coherence off) is byte-for-byte the pre-D2-2 behaviour.
Tests: test_cluster_undo_gcs U3-U11 (shared-root branch per mode,
materialized-never-migrates, intent derivation); intent parameter
threaded through test_cluster_undo_buf / test_cluster_tt_durable stubs.
cluster_unit 159/159, PG 219/219, cluster_regress 13/13, and the
undo/recovery/CR TAP set (018/070/213/215/247/248/253) all green with
coherence off.
Spec: spec-5.22b-undo-block-gcs-integration.md
…c-5.22b D2-3)
Add cluster_undo_block_acquire_shared: a peer acquires a coherent shared
view of an owning instance's undo block through owner-as-master routing.
The owner (authority and holder) ships the current image over the reused
undo-TT fetch wire; the requesting peer consumes the shipped image and
never opens the foreign undo file. Admission is fail-closed on two ANDed
dimensions -- segment-generation anti-ABA and owner-incarnation epoch plus
durable coverage -- so any miss/DENIED/doubt leaves the caller's existing
fail-closed visibility boundary untouched.
Structure follows the pure-policy split so cluster_unit links the decision
core standalone:
- cluster_undo_gcs.c gains the pure predicates cluster_undo_grant_armed
(coherence x peer-mode gate), cluster_undo_grant_admissible (reuses
cluster_vis_live_authority_covers_policy and the D1 generation check),
and cluster_undo_grant_reader_pcm_mode/_transition (S via N->S).
- cluster_undo_gcs_grant.c (new) holds the runtime wrapper with its
heavy fetch/epoch/GUC dependencies, registered in the backend OBJS.
Everything is gated behind cluster.undo_gcs_coherence (default off), so the
default path is byte-for-byte unchanged. The master==self local fast path
is fail-closed in this increment pending its owner-incarnation self-check;
the live-owner peer read path is fully implemented.
test_cluster_undo_gcs gains U4/U6/U7/U9 (reader lock contract, epoch and
generation admissibility, coherence gate); its recipe links the pure
live-authority policy object. Runtime wrapper is covered by the later TAP
legs.
Local gates (coherence off): cluster_unit 159/159 (undo_gcs 9/9), PG
219/219, cluster_regress 13/13, undo/CR/recovery TAP 6 files/160 tests,
clang-format-18 clean, comment-headers clean.
Spec: spec-5.22b-undo-block-gcs-integration.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
D2-2 gives an undo segment an optional physical home on the shared
cluster_fsroot under GCS, breaking the DataDir-local hardcode in the undo I/O path. The migration is gated on a new GUC (cluster.undo_gcs_coherence, default off), so the default path is byte-for-byte the pre-D2-2 behaviour — this mirrors D1 / D2-1's land-then-wire shape.Path intent (core contract):
ClusterUndoPathIntent {RUNTIME_SHARED, MATERIALIZED_LOCAL}names which home a segment resolves to.cluster_undo_intent_for_owner(owner)derives it — own-instance owner →RUNTIME_SHARED; a foreign owner (in D2 only ever a dead-origin copy recovery rebuilt in the local DataDir) →MATERIALIZED_LOCAL. Static inline incluster_undo_alloc.hso the ~30 call sites add no link dependency.cluster_undo_path_uses_shared_root(intent, peer_mode, coherence)is the single pure decision function:RUNTIME_SHAREDresolves under the shared root only when peer-mode and coherence are on;MATERIALIZED_LOCALalways stays local — the dead-origin by-xid resolve path must never move to shared storage.Threading + migration:
intentis a new first parameter oncluster_undo_path_resolveand the four undo smgr APIs (read/write_block,read/write_header_bytes), carried in theget_segment_fdfd-cache key.cluster_undo_xlog.c:build_undo_segment_path/redo_open_segment/redo_stamp_slot) delegates to the samecluster_undo_path_resolveso runtime and redo cannot split-brain an own-instance segment onto different roots.cluster_shared_fs_undo_path_resolve/_instance_dir_resolve(owner-partitionedinstance_<N>undercluster.shared_data_dir). Undo is outside theRelFileLocatornamespace, so this is a path resolver, not ashared_fsvtable callback.ensure-instance-subdirgains a dual branch (shared root →pg_mkdir_p; local → the existingmkdir).Supporting:
cluster.undo_gcs_coherence(bool, default off,PGC_SIGHUP); check hook rejects turning it on whilecluster.shared_data_diris unset.cluster_undo_path_resolveasserts(intent == RUNTIME_SHARED) == (owner == self).extern int cluster_undo_segments_max_per_instance(the newcluster_guc.hinclude already declares it) to clear a shadowed-declaration finding.Stacking / merge order
This PR is stacked on #23 (D1,
spec-5.22a-undo-resid) and includes D1 + D2-1 + D2-2 commits, since D1 is not yet merged. Intended merge order is D1 → D2-1 → D2-2; final merge is deferred to the substrate merge window.Test plan
cluster_unit159/159 (new:test_cluster_undo_gcsU3–U11 — shared-root branch per mode, materialized-never-migrates, intent derivation)cluster_regress13/13linkdb fast-gate CIall 5 jobs green (this PR)Spec:
spec-5.22b-undo-block-gcs-integration.md