Add shard split and merge as a derived lineage by wolpert · Pull Request #18 · codeheadsystems/the-sharder

wolpert · 2026-09-19T16:28:15Z

Split and merge return, as a lineage derived from each strategy's own keyspace geometry rather than
as a reinstated range strategy. range stays withdrawn and the SPLIT prefix admits no further
identifier.

Why this is a repair as much as a feature

Asking for split and merge surfaced a live defect. Two ring snapshots at factor 2 gain a node
carrying one token, which divides the extent its neighbour bounded. The plan the port built named a
source equal to the destination and told two nodes to copy a shard from themselves, naming neither
node that held the data. Removing the token was quieter: the shard that absorbs the folded extent
keeps its replica set, so it appeared in no delta entry and received no handoff at all. That is what
a node joining or leaving a ring topology did, which is the central operation of the storage
walkthrough.

TOPO-211 answers which shards changed owner. Nothing answered which shards changed contents, and
the two questions differ exactly when an epoch changes which shards exist.

What the change is

A lineage relates extents rather than identifiers. It is derived from the topology document, so
formatVersion stays at 1.0, the schema is untouched, every document valid before is valid now
and digests the same, and no new strategy kind is added. The new requirements take the LIN prefix
on the migration surface at the migration level: no conformance surface and no level is added,
and no TOPO-*, FENCE-*, or PLACE-* requirement moves.

Lineage is deliberately kept out of the ownership delta. TOPO-211 is tested at core, which a
port cannot decline, and enriching it would oblige every future port to implement ring interval
containment and directory precedence before declaring anything at all.

Where the two snapshots enumerate the same shards the lineage is the identity, so the regenerated
tree is additions only.

Commits

Commit	What it does
`9248ff6`	repairs `TOPO-213` in the port, which reported no entry for a shard only the earlier snapshot enumerates
`dc531bf`	derives the lineage, classifies it, and plans a division over it
`7747423`	runs the local step a division leaves on one node
`000b4c2`	decides a `directory` extent over the prefix trie
`e58320e`	exposes the lineage as an operation and orients the user guide around it
`f416575`	enforces `minEpoch` and the configured `topologyId`, which were accepted and ignored
`bea4494`	reports the ten events of the `migration` surface, which were declared and never emitted

Decision records

0086 derives the lineage from extents. 0087 settles that a strategy's lineage is derived while the
integrator's storage is declared. 0088 places it in the migration surface. 0089 requires a
division to be undone by a combination, which keeps MOVE-233 unconditional and avoids
reinstating the carve-out SPLIT-171 carried. 0090 records that a handoff is aborted rather than
rebased across a lineage boundary. 0091 states that the library refuses an unaligned change and
reports the rest without enforcing a preference. 0092 settles how an observability sink reaches the
coordinator.

0022 and 0054 are amended rather than superseded, in the form 0053 gives. 0054 was right to
withdraw range; one sentence of its consequences was not.

Four defects found on the way

Each is the same shape: an artefact that looks like coverage, and nothing executing it.

TOPO-213: the port reported one group of delta entries where the specification and the Python
reference give two. Every ownership-delta pair was slot at one slotCount, where the two
shard sets are always equal, so the clause was unreachable.
LIN-031: the lineage had no exported surface. The harness reached it through an internal class
inside the module, so the level passed over a requirement nothing implemented.
TOPO-071 and the first row of TOPO-061: minEpoch and expectedTopologyId were accepted,
validated, reported by router.configuration(), and never consulted.
OBS-020: no driver asserts that a running library emitted any event, for any surface.
30-conformance.md claimed otherwise. It now says plainly that emission is unasserted.

Known gap

The cross-language emission-assertion mechanism is not built. Emission is checked by the port's own
tests, and the inventory is the contract the suite holds a port to. Closing it properly means
modelling event emission in the reference, a general expectation in both drivers, and a decision
record.

Verification

./build.sh reports every port built. The generator's run.sh completes with no failures over 609
vector cases, 26 scenarios, and 28 properties; the withdrawal register is clean at 62 identifiers;
conformance/declarations/java.json was re-declared against each new suite revision only after the
port passed it. verifyDocLinks and verifyDocStyle are both clean.

Two repairs were verified by reverting them and watching the new artefact fail: the TOPO-213
vector, and the acceptance floor scenario.

🤖 Generated with Claude Code

`TOPO-213` fixes the order of an ownership delta in two groups: first the entries for shards the second snapshot enumerates, then the entries for shards only the first snapshot enumerates. The Java port computed the first group alone. `OwnershipDelta.between` iterated `after.placement().shards()`, so a shard that vanished between the two snapshots produced no entry, and the Javadoc stated that omission as though it were intended. `PLACE-075` bounds the cost of a delta over "the shards the two snapshots enumerate between them", and the Python reference has always computed that union. The two implementations therefore disagreed wherever the two snapshots enumerated different shard sets, and nothing detected it: every pair in `vectors/topology/ownership-delta.json` was `slot` at one `slotCount`, where the two shard sets are always equal and the second group is always empty. The one pair that differs, at `slotCount` 6 against 12, is the `TOPO-231` incomparability refusal and computes no delta at all. The second clause of `TOPO-213` had no executable test. A `ring` pair now carries one. Adding the token `0000000000002000` divides the extent the token `0000000000003000` bounded, so the later snapshot enumerates a shard the earlier one does not; removing it folds that extent back, so the earlier snapshot enumerates one the later one does not. The removal case fails against the unrepaired port and passes against the repaired one, and the addition case passes against both, which is exactly where the two implementations diverged. This is a repair to a port rather than a change to the design. No requirement is restated, no identifier is withdrawn, and no document changes. The suite revision changes because the suite gained two cases, so `conformance/declarations/java.json` re-declares against it and `ports/java/conformance/report.txt` is regenerated from the run the declaration names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

An ownership delta joins two snapshots on the shard identifier. Where an epoch changes which shards exist, that join has no answer for the shards that appeared or vanished, and their contents still have to come from somewhere. Nothing in the design answered where. The cost was reachable and silent. Two ring snapshots at factor 2 gain a node carrying one token, which divides the extent its neighbour bounded. The new shard has no entry in the earlier snapshot, so the plan named a source equal to the destination and told two nodes to copy a shard from themselves, naming neither node that held the data. Removing the token was quieter still: the shard that absorbs the folded extent keeps its replica set, so it appeared in no delta entry and received no handoff at all. This is what a node joining or leaving a ring topology did, which is the central operation of the storage walkthrough. A lineage is the second join, over the keys a shard holds rather than over its name. It is derived from the topology document, so `formatVersion` stays at 1.0, the schema is untouched, every document valid before is valid now and digests the same, and no new strategy kind is added. `range` stays withdrawn and the `SPLIT` prefix admits no further identifier. The new requirements take the `LIN` prefix on the `migration` surface, tested at the `migration` level. No conformance surface and no level is added, and no `TOPO-*`, `FENCE-*`, or `PLACE-*` requirement moves. Lineage is deliberately kept out of the ownership delta: `TOPO-211` is tested at `core`, which a port cannot decline, and enriching it would oblige every future port to implement ring interval containment before declaring anything at all. Under `ring` an extent is the token interval of `RING-020` and containment is unsigned interval arithmetic. That code sits in `core.internal.placement` rather than beside the coordinator because `verifyUnsignedComparisons` covers that package and nothing else, and a ring extent wraps past zero, which is where a signed comparison hides. The gate caught one while this was being written. Under `slot` `TOPO-231` holds `slotCount` equal, so every extent equals its counterpart and the lineage is the identity. Under `directory` a pair whose entry sets differ is refused with a named cause until directory extents are defined, which is a correct answer in place of the silent mis-plan it replaces. Under `rendezvous` there is no extent, and `MOVE-251` already refused. The property the change rests on is that an equal shard set is an equal extent set, because the shard identifier renders the geometry under every kind. Every pair the suite carried before this is such a pair, so the regenerated tree is additions only. Records: 0086 derives the lineage from extents, 0087 settles that a strategy's lineage is derived rather than declared while the integrator's storage is declared, 0088 places it in the `migration` surface, and 0091 states that the library refuses an unaligned change and reports the rest without enforcing a preference. 0022 and 0054 are amended rather than superseded, in the form 0053 gives: 0054 was right to withdraw `range`, and one sentence of its consequences was not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A division that moves no data still has work to do. A node that holds the parent under the earlier snapshot and the child under the later one has to divide its own copy so that what it holds matches the extent it now owns, and the plan the lineage produced named no such step. The keys were in the right place and the node's idea of which shard they belonged to was not. `MOVE-001` gains `dividing` and `MOVE-021` gains four transitions, which is the largest single edit in this work because that requirement says "exactly these transitions and no others". It lands after the lineage itself is settled and vector-backed rather than beside it. A local step names one node as both its source and its destination, which is the one place `LIN-042` does not apply, and runs `planned` to `dividing` to `complete`. There is no second party to prepare, quiesce, or cut over to, so the long sequence would be meaningless rather than merely wasteful. `LIN-057` orders it: a division before every handoff that draws from the divided parent, a fold after every handoff that draws into the folded shard, because a destination copying from a parent being divided underneath it would copy an extent that is changing. The step is reversible, and that is the point of 0089. `0022` had a local split that could not be undone, which is where `SPLIT-171` came from, and it was the single carve-out in `MOVE-233` that `0051` argued against and `0054` was glad to remove. Requiring the inverse hook keeps `MOVE-233` unconditional: a division is undone by `combine` and a fold by `divide`, so `dividing` reaches `aborting` like every other state that does and `MOVE-421` names it beside the other three. The cost lands on the integrator where it can be seen. Whether a storage can divide a copy in place is a property of that storage, so `HookDeclaration` gains `supportsLineage` and `LIN-053` refuses a plan that needs a local step without it, in preference to calling a hook that was never implemented. That is the shape 0085 established for `supportsRollback`. `HandoffContext` gains `sourceShardId`. It is the member doing the real work in an ordinary handoff too: a transfer hook for a divided child is told to move a shard the source does not hold, and without the parent in the context it cannot find the bytes. `MOVE-011` gains the failure kind `undivided`, for a copy that matches neither the parent's extent nor the child's. The four kinds it already carried each name a condition of a copy that moved between nodes, and this one never left the node it is on. 0090 records what is deliberately not done: a handoff whose shard a newer snapshot no longer enumerates is aborted rather than rewritten onto the children. That is what `MOVE-096` already says and what the port already does, and a transitive rebase would have to decide which child inherits a partial copy and whether a quiesce lease taken against the parent still binds. Relaxing a refusal later is cheap; withdrawing a clever rebase that proved wrong is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`LIN-013` refused a `directory` pair whose entry sets differ, which was a correct answer standing in for the right one while the ring geometry settled. It is restated here to define the extent, and the refusal goes with it. A directory extent is a matcher narrowed by the precedence of `PLACE-065`: the keys its entry wins once every entry of the same table that outranks it has taken what it claims. An exact matcher outranks every prefix and a longer prefix outranks a shorter one, so every entry that outranks another and overlaps it is contained in it, and an extent is one matcher minus a finite set of matchers inside it. `LIN-016` states how that is decided without evaluating the table over generated keys, which `LIN-001` forbids for the same reason `TOPO-221` forbids it of the ownership delta. Every decoded matcher value of either table is a node of one trie, and each node contributes two regions: the key equal to it, and the keys strictly extending it that no deeper node is a prefix of. Two keys of one region match exactly the same entries of either table, so a region is the finest distinction either table can draw and the regions together admit every key. An extent is the set of regions its entry wins, and equality, containment, and disjointness are set operations over those. An exact matcher wins only the region that is its own key. A region of keys strictly extending a node contains no node of the trie, and every matcher value is one, so no exact matcher can lie inside it. Refining `prefix:ab` into `ab0` and `ab1` while keeping `ab` divides one extent into three, because `ab` still wins the keys neither longer prefix claims. An entry that wins keys the earlier table matched to no shard under `DIR-010` names a shard with no parent, which is the `fresh` class, and `LIN-043` emits no handoff for it. That case exists only under `directory`, because it is the only kind whose `shardOf` answers with no shard, and it is why the requirement had no executable test until now. Every port declaring `migration` and exposing `directory` implements this, which is the scoping the strategy surface axis already gives and 0087 argues for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The user guide was reviewed against the code and three claims did not survive it. Two were mine and one predates this work. `LIN-031` requires the lineage to be an operation the integrator calls, and the port had none. `ShardExtents` sits in `core.internal.placement`, which `module-info.java` does not export, and `HandoffCoordinator` exposed `plan` alone. The conformance harness reached the classification by calling the internal class from inside the module, so the `migration` level passed over a requirement nothing implemented. `HandoffCoordinator` now carries `lineage(from, to)`, `ShardLineage` and `LineageClass` are exported beside it, and the harness goes through that surface, so a port that computes a lineage it cannot expose now fails the level rather than passing it. `30-conformance.md` no longer lists `LIN-031` among the requirements without an executable test, because it has one. The public `topology.OwnershipDelta` Javadoc still described the behaviour `TOPO-213` repaired. The internal copy was corrected with the repair and the exported one, which is the copy an integrator reads, was not. `lineage` refused a differing `topologyId` with the cause `topologyMismatch` while `plan` answers `incomparableShards` for the same input. Two operations disagreeing about one mistake, and the cause was the wrong one: its own documentation describes a plan named against a router that does not hold that topology, which is a different situation. Both now answer `incomparableShards`, and the interface documents all the refusals rather than three of four. The guide gains an orientation section before the first router, covering what a control plane holds and what the library does not do, how a published document reaches a running router and what installation replaces, and the everyday call from a key to the nodes that hold it. The last of those states where routing cost is paid rather than what it is: preparation is per snapshot and not per call, a call consumes the bounded prefix of `CORE-046` rather than the whole ordering, `rendezvous` is the exception because it scores the whole eligible set, and the whole-ordering calls belong off the routing path. It also gains the lineage operation beside the ownership delta, because the two are siblings that answer different questions: the delta joins on the shard identifier and answers who changed owner, the lineage joins on the keys a shard holds and answers where a new shard's contents are. The pair is the thing a reader needs and the guide did not carry it. One correction the review made to this author's own summary is worth recording: a shard that only the later snapshot enumerates is not absent from the ownership delta. It appears with an empty before set and every node in `gained`. What the delta cannot answer for it is where its contents are. The acceptance rule the guide stated was wrong and predates this work. It said the library accepts no document whose epoch is at or below the one in force. `TOPO-061` accepts an equal epoch with an equal digest as a no-op that refreshes freshness and refuses an equal epoch with a differing digest as a conflict rather than as stale, and `TopologyLoader` implements exactly that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`TOPO-071` requires an implementation to reject any document whose epoch is below `minEpoch`, including the first after a restart, and the first row of `TOPO-061` refuses a document whose `topologyId` differs from the one in force or configured. The Java port accepted both settings through `RouterConfig.Builder`, validated them, reported them in `router.configuration()` and then consulted neither. `TopologyLoader` had no field for either and was constructed with the strategy map alone, so a process configured with a floor installed a document below it and a process configured with an identifier adopted the first document it was handed whatever cluster it named. Both rows precede the row that installs a first document, which is the whole point of them: they are what a process has instead of a snapshot in force when it comes back up. An operator who sets `minEpoch` to stop a restart serving an older topology got no protection and no error, which is the worst shape a missing check can take. The Python reference has implemented both since it was written, in the row order the table gives, so this is a port defect rather than an ambiguity. Nothing detected it because nothing exercised it: `coverage.json` recorded `TOPO-071` as covered by `properties/properties.json` alone, through a property that names the identifier, and that property's witness is `topology-rollback`, which runs with neither setting configured and has no step below a floor. The identifier was claimed as covered while no artefact could reach it. `topology-acceptance-floor` reaches it now. It configures both, then refuses a document below the floor with nothing in force, refuses a foreign document that is also below the floor as a conflict rather than as stale, which is what fixes the row order rather than merely the rows, and installs the first document at the floor. It fails against the unenforced port and passes against this one. Two smaller repairs came with it. A scenario whose first document is refused has nothing in force, so both drivers now compare `epochInForce` only where the expectation states one; they previously assumed every rejection had a snapshot behind it. The user guide documents both settings, which it could not honestly do while they did nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`OBS-020` tables ten events under `migration.` and requires an implementation to emit the events of the surfaces it exposes. The Java port declared all ten in the inventory the `migration` vector reads, and emitted none of them. The cause was structural rather than ten missed call sites. A router is built from a `RouterConfig`, which carries the metrics registry and the event sink, so `DefaultRouter` holds a `MetricsHolder` and reports through it. A coordinator is built by `Sharder.coordinator()`, which takes nothing, so there was no sink to report through and nowhere for one to come from. 0092 settles where it comes from: `Sharder.coordinator(config)`, following `Sharder.loader(config)`, which already takes the whole configuration to answer a narrower question. The sinkless factory stays and means a coordinator that reports nothing. All ten now emit at the trigger `OBS-020` names, with the payload members it names. `state_changed` reports from the one chokepoint every transition passes through, and reports nothing for a trigger the machine dropped because the state moved under a hook, since no transition happened. `failed` follows it where the transition reached `failed`, carrying the kind of `MOVE-011`. `quiesce_expired` is raised only where the machine says the lease is spent, because an idle outcome in that position is otherwise a concurrent abort. Nothing in the suite checks any of this, and that is the part worth recording. No driver asserts that a running library emitted a named event, for any surface. `30-conformance.md` claimed otherwise: it said emission is asserted where the suite drives one, naming `sharder.health.ejection_refused` in the health scenarios. Those scenarios do carry an `ejectionsRefused` member, and neither driver reads it, so it is data rather than coverage. A port that declares an inventory it never emits from passes every level, which is exactly the state this port was in for the whole `migration` surface. The section now says that plainly, and a port's own tests are named as where emission is checked until a driver asserts it. `MigrationTest` carries that test for this port: it drives a handoff to completion through a coordinator with a sink and asserts the events, their severities, the common members of `OBS-021`, and the payload members of each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

wolpert and others added 7 commits September 19, 2026 07:48

wolpert merged commit c4f61f7 into main Sep 19, 2026
5 checks passed

wolpert deleted the shard-lineage branch September 19, 2026 16:31

Navigation Menu

Repository navigation

Add shard split and merge as a derived lineage - #18

Add shard split and merge as a derived lineage#18
wolpert merged 7 commits into
mainfrom
shard-lineage

wolpert commented Sep 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

wolpert commented Sep 19, 2026

Why this is a repair as much as a feature

What the change is

Commits

Decision records

Four defects found on the way

Known gap

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant