feat(cluster): cross-node write-write MVCC coordination (terminal-writer chaining)#27
Merged
Conversation
added 12 commits
July 9, 2026 22:11
A local DML that runs into a TERMINAL remote writer holder previously failed closed unconditionally (SQLSTATE 53R9H). Behind the new GUC cluster.crossnode_write_write (default off, PGC_SUSET) the writer-wait bridge now resolves the holder's outcome from cluster authority (page ITL ref + TT overlay + origin live-IC verdict; never the local CLOG/ProcArray view of a foreign xid) and maps it onto the native TM_Result contract: - remote UPDATE committed -> TM_Updated after the single-hop new version is probed reachable + chain-valid; the callers route the result into their native failure exits, which fill tmfd from the on-page tuple (ctid/xmax real values, cmax = InvalidCommandId per the native other-transaction contract) - remote DELETE committed -> TM_Deleted - remote writer aborted -> TM_Ok (released xmax cleared, caller re-validates and continues) - anything unprovable -> the pre-existing 53R9H floor (fail closed) heap_lock_tuple gains the same treatment for remote WRITER holders, which previously fell through to the native XactLockTableWait on a foreign raw xid (alias/hang); LockWaitSkip/LockWaitError are honored before the bridge blocks. A committed foreign deleter is never stamped HEAP_XMAX_INVALID (shared-page hint poisoning). With the GUC off the fail-closed floor is unchanged. New pure policy file cluster_writer_chain.c (outcome -> TM_Result mapping) with full truth-table unit coverage; cluster_unit 158/158, cluster_regress 13/13 locally green. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
spec-6.15 D7 taught SetHintBits to suppress xmax hint stamping when the xid is another node's class (native CLOG/ProcArray answers are void for foreign xids). That suppression silently also swallowed the spec-5.2 cross-node cleanup that clears a RELEASED remote lock-only xmax after the cluster authority proved the holder terminal -- reopening the window where compute_new_xmax_infomask folds a dead remote xid into a node-local MultiXact that aliases on every peer. Add cluster_heap_stamp_released_xmax_invalid(): a direct, authority- backed HEAP_XMAX_INVALID stamp (no commit-LSN interlock is needed for released/aborted marks) used by the writer-wait bridge callers and the heap_lock_tuple terminal handlers -- the only sanctioned bypass of the foreign-class suppression, because the fact comes from cluster TT / live-IC verdict authority, never from a native reading of the foreign xid. The spec-7.1a D0 aborted-writer paths depend on the stamp landing so the native re-determination sees HEAP_XMAX_INVALID instead of consulting HeapTupleHeaderIsOnlyLocked on a foreign xid. Ordinary SetHintBits suppression for foreign-class xids is unchanged (committed foreign outcomes are never stamped; readers re-resolve via cluster authority). cluster_unit 158/158, cluster_regress 13/13 locally green. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
…w LSN The live-authority covers gate admitted a recycled/remote xid resolution only when the origin's flush LSN was numerically at/after the tuple's page LSN. Under per-thread WAL that raw compare is unsound: a page last written by another WAL thread carries an LSN from a different stream, so live resolutions were refused persistently (measured) and the reverse direction could false-admit (latent). Replace the page-LSN anchor with an SCN total-order conclusiveness demand (AD-008 Lamport): the origin co-samples its SCN clock with the served content (before the content read, same ordering argument as the existing high-water sample) and ships it in the previously must-be-zero authority-trailer bytes (wire size unchanged; a zero trailer from an older peer reads as absent and refuses). The requester demands conclusiveness for its snapshot read_scn -- or, on no-snapshot legs (terminal-writer resolution), for its own clock sampled BEFORE the fetch -- and admits only when the shipped clock is at/after the demand. Every shipped SCN is Lamport-observed before the gate can refuse, so a refusal self-heals on retry. Epoch equality and the downstream tt_generation wrap qualification are unchanged; all refusals keep the pre-existing fail-closed boundary. cluster_unit 158/158 (covers truth table rewritten: false-refuse regression + false-pass guard + absent-sample/absent-demand refusals), cluster_regress 13/13 locally green. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
…dict A LIVE remote ITL ref (slot still bound to the asked xid) whose TT overlay lookup missed stayed UNKNOWN forever when the origin's tt_status_hint propagation was lost: fail-closed (53R97) but never self-healing, so the affected scan could not complete. Behind cluster.crossnode_write_write, escalate such a miss to the origin's complete own-TT verdict on demand -- the same shipped live-IC verdict machinery the recycled-slot resolution uses; no shared-undo data plane is involved and the origin answers only for its own xids with terminal outcomes. Exact terminal outcomes install the exact-key memo; a below-horizon bound is returned snapshot-relative and never memoized. A genuinely in-progress holder still resolves UNKNOWN (no verdict is served) and keeps the fail-closed retry, which converges once the holder terminates. GUC off keeps the UNKNOWN floor byte-identical. cluster_unit 158/158 locally green. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
…wait/chase sibling Sweep of every native XactLockTableWait / MultiXactIdWait / update-chain chase in heapam.c for paths a node-foreign xid or foreign-marker multixact could still reach natively (L349/L352 sibling discipline): - heap_lock_tuple multixact wait: a remote-marker multi reached the native MultiXactIdWait (node-local multixact ids alias across instances); floor it 53R9H before the wait, mirroring the writer-wait bridge's multi floor in heap_update/heap_delete. - heap_lock_tuple KeyShare pre-sleep chain chase: a remote-marker multi updater seeded heap_lock_updated_tuple, whose entry decode (MultiXactIdGetUpdateXid) reads the local pg_multixact; floor at the call site (the marker needs the page). - heap_lock_updated_tuple entry: floor a non-multi node-foreign chain root (defense in depth behind the call-site floors). - heap_lock_updated_tuple_rec: floor a chain member whose xmin or xmax is node-foreign (or a remote-marker multi) before the native DidAbort/DidCommit/member-decode judging -- cross-node update-chain locking is the spec-7.1a Q9 multi-hop forward; fail closed retryably instead of mis-walking. - heap_delete cluster post-wait re-validation: add the native VM all-visible re-check the cluster branch was missing (the lock is dropped across the wait; an unpinned VM page that became all-visible must restart from l1 exactly as the native locker path does). All floors are strictly tightening (fail-closed on paths that were natively unsound); no admit path is widened. cluster_unit 158/158, cluster_regress 13/13 locally green. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
Five new counters in the 'visibility' dump category (same atomic bump/read shape as the spec-3.14 D8 family): - writer_chain_resolved_count / writer_chain_failclosed_count: the terminal-writer bridge chaining outcome split (53R9H keeps bumping the pre-existing vis_conflict_failclosed_count as well) - xmax_resolved_count: positive foreign-xmax verdict consumptions in the MVCC keeps-visible gate and the SatisfiesUpdate fork - overlay_refresh_count: live overlay misses resolved by the D4 origin pull - covers_scn_refuse_count: covers-gate refusals under the D3 SCN regime (both the block leg and the verdict leg) No new wait event: the chaining paths introduce no new blocking primitive (the next-version probe rides the ordinary cluster-coherent buffer read and the writer wait keeps the existing cross-node TX enqueue wait event). t/223 visibility-category baseline 6 -> 11 keys updated in-commit. cluster_unit 158/158; smoke TAP subset (010/030/050/200/226) + t/223 all green locally. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
…riers The trailer byte-carrier helpers take the value as 'v' (the expected_pi_watermark_scn_bytes precedent) and the covers-gate header prose avoids a raw comparator on an _scn name -- both hits were lint-shape only; no behavior change. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
…ence, not a verdict The CP3 single-block fetch fast leg trusted a shipped TT slot's COMMITTED stamp as a terminal commit. Durable COMMITTED stamps land at pre-commit (2PC COMMIT PREPARED stamps before the commit record), so a stamped-then-crashed xid is still in-doubt and concluding from the stamp alone is a false-committed hazard for the read fast leg, the writer chain and the overlay pull alike. Route every COMMITTED proof through the origin verdict leg, whose C1b TransactionIdDidCommit cross-check refuses a stamped-then-crashed xid; only ABORTED proofs keep the fast path (an abort stamp is terminal and irreversible). Routing every stamp to the verdict leg turns the serve's wrap-suspect refusal (a live slot whose stamped scn sits below the retention horizon) from an unreachable corner into a routine outcome, which would leave such xids permanently unresolvable. The serve now answers that shape with the existing COMMITTED_BELOW_HORIZON bound over max(stamped scn, max gated-recycle horizon) -- the same frozen-bound consumer contract as the zero-match arm, with the C1b check unchanged -- and still refuses when no gated recycle bounds a possible recurrence. t/346 evolves with the substrate it now exercises: a live remote ref whose overlay lookup misses is resolved by the origin pull, which is armed by cluster.crossnode_write_write, so the pair boots with it on (as it already does for xid striping), and the L5 warm read retries through the leg-(e) first-read transient. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
…er-chain probe A committed remote cross-partition UPDATE leaves the moved-partitions sentinel in the old tuple's t_ctid. The D0 new-version probe passed it to ReadBuffer unchecked: the sentinel block is InvalidBlockNumber == P_NEW, so every such wake silently extended the old partition by one empty page before the probe failed. Split the chain decision on the sentinel (keeping the CWO_UNRESOLVABLE fail-closed floor) and refuse any forward pointer whose block is invalid or beyond the relation's end at the probe entry. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
…eign predicate The l4 chain-walk and chain-root fail-closed floors used cluster_xid_foreign_class_cheap, whose contract is suppression-only: it over-reports below-floor pre-striping local xids as foreign, turning a purely local update chain into a spurious cross-node write conflict. Switch the three floors to cluster_xid_provably_foreign (false when underivable, preserving native authority over pre-striping history), matching the visibility-side floors. The SetHintBits suppression sites keep the cheap predicate by contract. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
t/365 covers today's write-write boundary (in-flight remote-writer conflicts fail closed, retries converge with no lost update, arming the GUC invents no resolution for an undecided writer), the C1b verdict-routing leg (a fetched COMMITTED stamp is finalized via the origin verdict leg -- rtvis_verdict_wire_count moves), the moved-partitions no-bloat invariant, and the 2PC in-doubt hard boundary (a PREPARED xact's value is never served nor chained; COMMIT PREPARED heals the read and the owner closes the value chain over the deferred-transfer boundary). The writer-wait chain gain legs, the crash-in-window injection and the below-floor chain-walk leg are KNOWN-BLOCKED in the file header with their blockers. Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md
sqlrush
force-pushed
the
spec-7.1a-write-write
branch
from
dbbcb63 to
4d353df
Compare
The merge-window rebase onto main unioned the spec-7.2 cluster_ic_duty_lazy and spec-7.1a cluster_crossnode_write_write externs; the hand-resolved line used a tab before the trailing comment. clang-format-18 collapses it to a single space. Format-only; no semantic change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the cross-instance write-write coordination band (D0-D6):
cluster.crossnode_write_write(default off, PGC_SUSET): a local DML that runs into a TERMINAL remote writer maps the authoritatively-resolved outcome onto the native TM_Result contract (remote UPDATE committed -> TM_Updated with single-hop new-version probe; DELETE -> TM_Deleted; aborted -> TM_Ok) instead of unconditionally failing closed (53R9H). Unprovable outcomes keep the fail-closed floor; GUC off keeps the pre-existing behavior byte-identical.visibilitydump category; t/223 baseline updated in-commit.Test plan
Note: t/209 fails locally on this machine identically on unmodified main (CREATE TABLE GES acquire timeout under its
cluster.pcm_grd_max_entries=0conf) — pre-existing, not introduced here.Spec: spec-7.1a-cross-instance-write-write-mvcc-coordination.md