Current behavior
For the entire duration of a predicate move, Zero aborts every commit that touches the predicate: movePredicate calls blockTablet before streaming starts and releases it only when the move returns, and the oracle rejects blocked predicates at commit time (dgraph/cmd/zero/oracle.go: "Commits on predicate %s are blocked due to predicate move"). With multi-hour moves of large tablets, that is a multi-hour write outage on the predicate. A failed move also discards all streamed work; there is no resumability.
Proposed direction
Two-phase move:
- Bulk phase: stream the tablet at ts1 without blocking commits.
- Delta phase: block commits, take ts2, stream only keys changed since ts1 (badger's stream framework supports
SinceTs), then flip the tablet to the destination group.
The write-abort window shrinks from O(move duration) to O(delta), minutes instead of hours. An alternative or complement is moving the bulk data outside Raft entirely, the way snapshot streaming already transfers group data.
Progress visibility belongs in this redesign too. MovePredicate is a unary RPC (protos/pb.proto), so Zero is blind while a move runs. A pollable move-status RPC on the source Alpha would let Zero log progress and ETA, and eventually replace the wall-clock timeout with stall detection: cancel only when bytes sent stops advancing.
Context
Surfaced while fixing #9784 (size-aware move timeout). Contained throughput work in the receive path is tracked in #9790; this issue covers the design-level availability fix.
Jira: DGR-309 (https://istari.atlassian.net/browse/DGR-309)
Current behavior
For the entire duration of a predicate move, Zero aborts every commit that touches the predicate:
movePredicatecallsblockTabletbefore streaming starts and releases it only when the move returns, and the oracle rejects blocked predicates at commit time (dgraph/cmd/zero/oracle.go: "Commits on predicate %s are blocked due to predicate move"). With multi-hour moves of large tablets, that is a multi-hour write outage on the predicate. A failed move also discards all streamed work; there is no resumability.Proposed direction
Two-phase move:
SinceTs), then flip the tablet to the destination group.The write-abort window shrinks from O(move duration) to O(delta), minutes instead of hours. An alternative or complement is moving the bulk data outside Raft entirely, the way snapshot streaming already transfers group data.
Progress visibility belongs in this redesign too.
MovePredicateis a unary RPC (protos/pb.proto), so Zero is blind while a move runs. A pollable move-status RPC on the source Alpha would let Zero log progress and ETA, and eventually replace the wall-clock timeout with stall detection: cancel only when bytes sent stops advancing.Context
Surfaced while fixing #9784 (size-aware move timeout). Contained throughput work in the receive path is tracked in #9790; this issue covers the design-level availability fix.
Jira: DGR-309 (https://istari.atlassian.net/browse/DGR-309)