Skip to content

feat(asof-share): carry frozen graphs; frozen rows carry the freeze tx (s24.1.1) - #831

Merged
scbrown merged 4 commits into
mainfrom
malcolm/s2411-frozen-cut
Oct 25, 2026
Merged

scbrown merged 4 commits into
mainfrom
malcolm/s2411-frozen-cut

Conversation

@scbrown

@scbrown scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator

What

An as-of share now carries frozen graphs. A cut reads a frozen graph's rows from its verified pack, not from main, which no longer holds them. A follower can therefore re-bootstrap across a freeze, which mints a new log_epoch, and end equal to the source in every named graph.

The correctness rule this needed (ruled B1)

freeze_io::export_graph_history re-inserts a frozen graph's transactions into the pack, numbered from 1. It keeps no main ids. So pack transaction ids are file-local; store/freeze.rs already says so, and it is why as_of_tx is refused over attachments. Consequently:

rule why
a watermark before a scoped graph's freeze is refused the pack cannot say which rows existed then
pack rows are selected by state (op = 1 AND retracted_tx IS NULL), not by tx the ids can't be compared with the watermark; a frozen graph refuses writes, so its state at any watermark from the freeze on is its state at the freeze
a frozen row's timestamp, actor and source come from the pack's own transactions main's transaction with the same number is an unrelated one
every frozen record carries F, the main transaction that froze its graph an applier buckets records by tx and keys progress and duplicates on it
urn:quipu:graph:meta is refused in every cut (scope and context) F writes that graph, so a cut naming it would mix two graphs into F's bucket

F is the tx of the graph's live lifecycleState "frozen" fact in the meta graph.

Tests

test shows
a_follower_rebootstraps_across_a_freeze_and_equals_every_named_graph freeze → epoch change on the next page → re-bootstrap → tail → equal; control: no freeze equals first time
a_frozen_graphs_records_keep_their_provenance_and_carry_the_freeze_tx three unrelated transactions first, so the two numberings differ (asserted). Timestamp, actor and source are unchanged across the freeze; all frozen records carry one tx, later than every original and ≤ W; no other graph's record shares it
a_cut_from_before_a_scoped_graphs_freeze_is_refused T before the freeze is refused; the same request at the head is served
the_graph_metadata_graph_is_never_cut the meta graph is refused in scope and as context; control served
a_frozen_scan_never_reuses_main_metadata_for_its_own_tx_numbers arranged collision: main's last scanned tx is 1 and the pack's first is its own 1

Instrument note: the provenance falsifier first PASSED, because the fixture wrote G first, so main tx 1 equalled pack tx 1. The adversarial prefix made it fail on the defect.

Sabotage, each restored from a saved copy with its sha256 checked. Each turned its test red; X2 and X6 were re-run after the final refactor:

  • meta not refused;
  • pre-freeze watermark served;
  • metadata borrowed from main;
  • frozen rows emit the pack's own tx;
  • retracted pack rows included (4 tests red);
  • lookup cache kept across scans.

Not in this PR

  • Scoped stream (s24.2): it must apply the same meta-graph refusal when it exists.
  • Shipping the pack file itself as a share member (option A): recorded for s24.2.
  • The follower keeps a frozen graph as an ordinary data graph; lifecycle stays source-local (ruling (a)).

Gates

  • default lib 2883 / 0 and server 247 / 0, under a HOME-escape guard (clean);
  • clippy -D warnings -A missing-docs for full and shacl,owl;
  • wasm32 lib check;
  • fmt, file size and the repository pendant scripts.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF

scbrown and others added 4 commits October 25, 2026 00:07
Local checkpoint, not for push yet. Known open items: asof_share.rs at 515
code lines (limit 500); clippy type_complexity (scans vec) and
unused_self (Store::verify_frozen_pack); untested fallback to a pack's own
transactions table. Tests: 25/25 as-of incl. three frozen tests; sabotages
F1 (no pack scan) and F2 (no verify) red; shacl,owl 3236/0; server 240/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
The DELIVER 2 sequence end to end: freeze, the epoch change on the
follower's next page, re-bootstrap from the cut at N, the tail resumed
from N, and a follower equal to the source in every named graph, the
frozen one read from its pack through the pack's own terms. Control:
with no freeze the first bootstrap already equals.

The source's freeze metadata (urn:quipu:graph:meta) lies outside every
cut; the test pins it as the only difference outside the named graphs.
Dropping the pack scan from the cut turns the test red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
….1.1)

A pack renumbers its transactions, so a cut that reads pack rows emits file-local tx ids, borrows main metadata for them, and serves a T before the freeze. Both tests fail on purpose until the ruled fix. Also: verify_frozen_pack becomes an associated fn and moves out of sync_frozen_attachments doc block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
…venance (s24.1.1)

A pack renumbers its transactions from 1 and keeps no main ids, so the cut
was filtering pack rows against the main watermark, borrowing main's
metadata for pack-local ids and emitting those ids as `tx`. An applier
buckets records by `tx`, so frozen rows could merge into an unrelated
transaction. Ruled (B1) by wu:

- A watermark before a scoped graph's freeze is refused: its earlier
  state cannot be cut.
- A pack's rows are selected by state (live, unretracted), which equals
  the graph's state at any watermark from the freeze on.
- Their timestamp, actor and source come from the pack's own
  transactions, never from main; the lookup cache never outlives a scan.
- Every frozen record carries F, the main transaction that froze it.
- The graph-metadata graph is refused in every cut, scope or context: F
  writes it, so a cut naming it would mix two graphs into F's bucket.

Falsifiers, each red under its sabotage: provenance kept and tx == F with
no collision; the pre-freeze refusal; the meta-graph refusal; and an
arranged main/pack tx-number collision for the lookup cache.

Also: the scan plan moves into asof_share/support.rs as a named struct
(type_complexity, file-size ratchet), verify_frozen_pack becomes an
associated fn and leaves sync_frozen_attachments' doc block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
@scbrown
scbrown force-pushed the malcolm/s2411-frozen-cut branch from 993ac2b to 98fcde3 Compare October 25, 2026 04:14
@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Non-author Ian: APPROVE at 98fcde36bc9124f66178ccba34f2db5d955662e5, native CONTENT 5c13f3d5a113ea2a9f337e5c2ab5bafb15bc31e828959639618841429ed1378f accepted by the current helper. Full four-path review after completed-green/current native preparation. The helper binds this review to the proposal and explicitly requires the supported merge --diff-bound landing path; this is not merge admission.

Reviewed state selection, pre-freeze refusal, metadata-graph exclusion, per-pack provenance lookup/cache lifetime, pack verification/registry comparison and composed term resolution. No blocking findings.

Validation: 30 supplied as-of conformance tests plus one independent two-frozen-pack control PASS (31/0). The independent control preserves both graphs and their timestamp/actor/source, and checks distinct main-ordered freeze transactions across packs. Inner exit0; outer HOME guard0, 157 watched paths +5 UID resources clean. Temporary test inclusion restored byte-for-byte; checkout clean. Initial wrong filter selected zero tests and is excluded from evidence.

Source/library review only. No deploy, runtime activation, lifecycle or merge action. Normal holds, current merge admission and exact-main watch remain with the lander.

@scbrown
scbrown merged commit 44d11a0 into main Oct 25, 2026
24 checks passed
@scbrown
scbrown deleted the malcolm/s2411-frozen-cut branch October 25, 2026 04:50
scbrown added a commit that referenced this pull request Oct 25, 2026
…to the S3b stream branch

Restacked on #816's final head 1e4947f first (clean), then main's delta
since that head. One real conflict, src/asof_share.rs: this branch
extracted the per-graph checks into resolve_graphs (shared with the
replication stream), and #831 changed the same loop to read frozen graphs
from their packs and refuse the graph-metadata graph. Resolution:

- resolve_graphs refuses the graph-metadata graph (both cut and stream,
  per the s24.1.1 ruling) and attached packs, and no longer refuses
  frozen graphs itself;
- the cut reads frozen graphs from their packs exactly as #831 does
  (pack re-verified inside the transaction, frozen_member);
- the stream keeps this branch's reviewed behaviour and refuses frozen
  graphs explicitly (refuse_frozen), now tested together with the
  metadata refusal.

The derived rest-api.md witness hash is recomputed from the merged file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
scbrown added a commit that referenced this pull request Oct 25, 2026
…ope.rs

The merge with #831 took src/asof_share.rs to 526 code lines (limit 500).
scope_lists, resolve_graphs and refuse_frozen, the checks the cut and the
replication stream share, now live in their own submodule and are
re-exported unchanged, so their callers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
scbrown added a commit that referenced this pull request Oct 25, 2026
…lenge (#817)

* fix(changes): carry valid time and byte bodies so a feed replica equals its source

A replica built only from /changes records diverged from its source in two
ways (local-sync-mode.md F3, F4):

- Records carried no valid time, so a fact valid since 2025 replayed as valid
  from its commit time. Records now carry valid_from and valid_to. A later
  retract closes the asserted row in place, so a row a later transaction
  closed reports valid_to null and the closing arrives as the retract's own
  record. Otherwise a served record would change after the fact, and a
  replica would assert an already-closed fact its retract could not find.
- Byte values were reduced to bytes_len. They now also carry the body as
  base64.

Both are additive fields. New replay tests rebuild a store from the feed and
compare a canonical live-fact digest: a plain control, valid time (open and
bounded), bytes, and an assert record that reads the same before and after a
later retract. Removing each change turns its test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* fix(owl): write a retract row when a functional value is superseded

A functional-property supersede closed the prior value with an UPDATE and
wrote no op=0 row, unlike an explicit retract. The change feed derives its
records from rows, so it showed the new value and never the end of the old
one, and a replica built from the feed kept both (local-sync-mode.md F1).

The supersede now writes the same immutable op=0 row an explicit retract
writes, at the superseding transaction. The resident read model reads op=1
rows and their retracted_tx stamp, so it is unaffected.

The replay test asserts the replica holds exactly one current value;
removing the new row turns it red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* fix(stream): send an oversized transaction as parts instead of ending the stream

A /changes/stream page larger than the 1 MiB frame budget sent quipu.error
without advancing the cursor and ended the stream. A consumer reconnecting at
that cursor hit the same transaction and the same dead end forever
(local-sync-mode.md F6).

Such a transaction now arrives as quipu.changes.begin, numbered
quipu.changes.part frames and a quipu.changes.commit barrier (design §6).
Only the barrier carries the SSE id, so Last-Event-ID never lands inside a
transaction and a reconnect replays it from the first part. The barrier digest
is SHA-256 over the parts' data exactly as sent; each part carries the hash of
the parts before it. A single record larger than a frame is refused with
quipu.changes.tx_too_large and no id, leaving the cursor unchanged.

Tests stream a 2500-record transaction: every part fits, the digests chain,
the reassembled records equal the page, the next transaction follows, and a
reconnect replays from part 1. Disabling the split turns the test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* refactor(stream): build oversized-transaction frames independent of SSE

chunk_transaction now returns transport-neutral frames (event name,
optional cursor id, data string), and the SSE stream turns them into
events only at the edge. Another transport can send the same frames and
the same digests unchanged. A new test checks that the frames SSE sends
equal the ones chunk_transaction returns, byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF

* feat(changes): name the source store and log epoch on every page

A change page did not say which store produced it, and nothing marked a
history boundary, so a follower could not tell a restored or cloned primary,
or one that had removed served records, from the one it was following
(local-sync-mode.md §5, F5).

Every page now carries source.store_id and source.log_epoch. The epoch lives
in a new one-row log_epoch table, minted when the store is created (or first
opened by this version) and minted again by Store::mint_log_epoch for an
operation that removes served history. Share reconstruction excludes the
table, so a restored copy mints its own epoch and its followers re-bootstrap.
The explicit restore/adopt ownership step is deferred to aegis-s24.2, per the
xdvg.1 ruling.

Tests cover the page source, an epoch stable across reopen, a new epoch after
minting, and a store that predates the table getting one on first open.
Removing the field turns both tests red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* fix(stream): name the source on a chunked transaction's begin and commit

An oversized transaction's begin and commit frames carried no source,
so a follower could not epoch-check a chunked transaction the way it
checks a whole page, although a changed epoch must mean re-bootstrap,
never splice. Both frames now carry the page's {store_id, log_epoch}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF

* feat(applier): the shared applier, slice 1: a replica from the shipped feed

One component turns the committed feed into a read-only replica, so the HA
follower, a scoped follower and the in-page browser store never write their
own projection logic (docs/design/shared-applier-api.md). This slice applies
the frames that ship today, given as (event, data) text exactly as SSE or a
WebSocket delivers them: /changes pages and the begin/part/commit frames of
an oversized transaction. It does no I/O, so it builds for wasm32 too.

- The first frame binds the replica to its source (store_id, log_epoch); a
  frame from another source returns Rebootstrap, never a splice.
- Each source transaction applies in one savepoint with the persisted
  cursor; a chunked one is buffered, its running digest checked part by
  part, and nothing is visible until the commit verifies (or the caller is
  told to resume from the cursor).
- A record the replica cannot reproduce exactly (a byte value sent as its
  length only, an unknown value shape) refuses the whole transaction.
- Freshness is (last applied tx, time since the last frame), never bounded.

The feed-replay conformance tests now use the same record->datum
conversion (record_to_datum) and share live_digest, so there is one mapping.
The cursor table is excluded from shares, like the log epoch it is bound to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* test(applier): a digest-sensitivity control and the bridge's JSON contract

The replica equality oracle now sits beside the code it judges, so prove
it sees valid time, byte content and graph (review ask). Outcome and
Freshness serialize tagged by kind, the contract a browser bridge returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): refuse a gap; a page names the cursor it answers

A /changes page covers every transaction in its scope after the cursor it
was asked for, but did not say which cursor that was. Transaction ids are
global and a per-graph page skips other graphs' ids, so the applier could
not tell a lost page from an ordinary skip and would splice past it.

- Every page now carries `since`, and the chunked `begin` and `commit`
  frames repeat it.
- The applier resumes (nothing applied) when a page or a chunked
  transaction answers a cursor above its own. A new replica starts at 0
  until a bootstrap exists. A page that names no `since` is refused.
- A page arriving while a chunked transaction is open discards the chunk
  and resumes. Before, the page could apply the chunk's transaction, move
  the cursor past it, and the commit was then dropped as a duplicate.

Seven tests:
- a dropped page;
- a first page not at 0;
- a page without `since`;
- a page inside an open chunk;
- an interrupted chunk restarted from a new begin;
- a re-opened applier holding no staged parts;
- a chunk that skips a transaction.
With the two checks disabled, four of them fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(knowledge): refresh the src/lib.rs source witness after adding the applier

The applier commit added `pub mod applier` to src/lib.rs, so its
repository-knowledge witness hash went stale (the pendant source controls
job). The record's summary still holds; only the hash changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(applier): slice 2, a replica bootstrapped from an as-of share

The shared applier's bootstrap (docs/design/as-of-share.md). A share carries
the source's live facts at its watermark as ordinary change records, so
bootstrap and tail use ONE record_to_datum and prove equality with the same
live digest. It arrives as one `quipu.asof` frame, or as
`quipu.asof.begin` / `.part` / `.commit` with the same digest chain as a
chunked transaction (the part and commit checks are now shared `Chunk`
methods).

All or nothing. These refuse with nothing written:
- a replica that is not empty;
- a manifest that is not quipu-asof-share/v1, or does not claim valid time
  and lossless bytes;
- a record that is not an assert, or is in a graph the manifest does not
  list;
- a graph whose records do not match the manifest's count and digest
  (SHA-256 over sorted, key-sorted compact JSON lines, `share_graph_digest`).
On success the facts and the cursor (store_id, log_epoch, watermark_tx) commit
in one savepoint, and slice 1 tails from /changes?since=watermark.

Seven tests:
- bootstrap then tail to equality with the source (valid time, bytes, named
  graph, a later retraction), and the cursor survives reopening;
- chunked equals single-frame, and nothing is visible before the commit;
- a dropped part resumes with nothing written, and a restarted cut converges;
- every untrustworthy share refused, with nothing written and no cursor;
- only an empty replica;
- a new epoch after bootstrap means rebootstrap;
- the digest ignores key and record order but not content.
With the count/digest and emptiness checks disabled, two tests fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(bindings): a replica through the binding facade and the browser Database

The shared applier joins the binding facade, so the in-page store and any
other binding reach it through one engine: Engine::replica_apply_frame,
replica_progress and replica_freshness, exported to JavaScript on the wasm
Database as replicaApplyFrame / replicaProgress / replicaFreshness (JSON text
in the contract Outcome and Freshness serialize to).

- The first replica call makes the store a read-only replica: local writes
  through invoke (knot, episode, set, retract, shapes load/remove) are
  refused before dispatch with read_only_replica / not_started. Reads work.
- The applier writes what the source already accepted, so local write
  validation is suspended for the applier's own writes and restored after.
  Re-judging source data against local shapes could refuse it and fork the
  replica silently. A test proves both halves: the adopted shape refuses the
  data when written locally, and the replica applies it from the feed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(knowledge): refresh the rest-api.md source witness

15c8128 changed docs/book/src/reference/rest-api.md without moving its
witness, so the repository-knowledge check refused the registry. Only the
source_sha256 moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(applier): a duplicate is checked against the retained transaction digest

Rule 6 of the shared applier API (docs/design/shared-applier-api.md): a
transaction at or below the cursor was skipped as a duplicate without
looking at it, so a redelivered tx 5 with changed records answered exactly
like an exact replay.

The applier now computes each transaction's content digest itself from the
decoded records (domain tag, store id, log epoch, source tx, then each
record's length-prefixed exact encoding, sorted), independent of framing,
and retains it in replica_applied_tx in the same savepoint as the records
and the cursor. A redelivery is Duplicate only when its digest equals the
retained one; a different digest is Stopped { DigestMismatch }, no
retained row is Stopped { DigestUnverifiable }, nothing is written, and the
stop is sticky. A frame from a new source still answers Rebootstrap.

replica_applied_tx is excluded from shares with the cursor it belongs to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(tests): the bindings timestamp constant is not shacl-only

record_source uses TS in every build, so the no-default-features test
build failed to compile (E0425).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): stream an as-of share into the replica instead of holding it

Slice 2 buffered every part of a chunked share in memory until the commit
(found on a 9.16M-record share). The begin frame now opens ONE local
savepoint; each part's facts are written as it arrives, in batches the
size of a part, and its canonical lines go to a temporary table so the
per-graph digests are computed in sorted order at the commit. Any failed
check rolls the savepoint back with nothing written, as before; the facts
and the cursor still commit together.

Same-connection reads now see a share's parts before its commit. The
replica has no cursor until Bootstrapped, and a reader must not use it
before then. A tail frame mid-share aborts the share, and an applier
opened on a store left mid-share rolls it back.

The digest chain moves to src/applier_chunk.rs and counts records without
keeping them when the caller streams. examples/applier_bootstrap_rss
measures peak RSS for a synthetic share at a chosen size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): a stop persists across reopening, and an unknown op is refused

Review findings on the shared applier:

- P1: the sticky Stopped outcome lived only in memory, so a fresh Applier
  on the same replica tailed on past a recorded digest mismatch. The stop
  is now kept in replica_stop (first stop wins) and restored by open; a
  replica whose history disagreed with its source stays stopped until it
  is rebuilt.
- P2: any op other than "retract" was applied as an assert. Only
  "assert" and "retract" are accepted, in the record conversion and the
  transaction digest alike; anything else refuses the transaction with
  nothing written.

replica_stop is excluded from shares with the cursor it belongs to.
Controls: a stop survives reopening (fails with open ignoring the stop),
and an unknown op is refused (fails with unknown ops read as assert).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(applier): slice 3a, stream sessions, signed coverage and the freshness challenge

The applier's side of the scoped replication stream (local-sync-mode.md
§5, shared-applier-api.md rules 1-8):

- A signed session frame binds a session id to the source, the pinned
  scope digest and its since; a since above the cursor resumes.
- frame_seq must be exactly last + 1, and each transaction newer than
  the last the session sent; anything else drops the session and resumes.
- Coverage is verified against a caller-pinned stream key (never learned
  from the stream), persisted beside the cursor, and never moves back.
- Freshness is bounded only by a coverage frame answering an outstanding
  challenge of the open session, consumed on use. A heartbeat, a replayed
  or unsolicited nonce, a new session, a reopen, end_session or a clock
  discontinuity never yields a bound.
- Control frames: revoked (persisted, sticky across reopening),
  epoch_end (rebootstrap).

replicate_wire.rs holds the signed bytes, shared with the producer that
follows: domain tag quipu-replicate/v1, then length-prefixed fields.

22 stream tests; each of 12 mutations of the checks fails at least one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): a store holding local facts never becomes a replica

Review 6387176506 (CHANGES): replica_apply_frame promoted any Engine to a
read-only replica on its first call, and the applier tailed the feed
into a store that already held local facts. The replica would then equal
no source while its cursor and digests claim it does.

- Applier::new_replica_refusal: a store may start a replica only if it
  already follows a source or holds no facts (shapes are not facts).
- The applier refuses a tail into such a store with nothing written.
- The binding checks before promoting: the frame is refused and the
  store stays an ordinary writable store.

Paired controls: with either gate removed, its test fails.
record_to_datum and value_of move unchanged to applier_record.rs, and the
gate lives in applier_start.rs, to keep applier.rs under the size gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(bindings): carry the stream freshness fields on a store that is not a replica

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* test(applier): the as-of share producer and the applier agree at the seam

Now that #805 is on main: a real asof_share::cut of a file-backed source
(300 items, unicode and escaped titles, bytes, a superseded value), framed
as the server sends it (one quipu.asof frame, or begin/parts/commit),
bootstraps an empty replica to the source's live facts; one tampered part
with the chain left intact is refused with nothing written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(replicate): the scoped replication stream and its freshness challenge

The source side of local-sync-mode.md §5, slice 3b of the shared applier:

- POST /replicate/stream: one SSE session per follower over the scope an
  as-of share would serve (same rules via asof_share::scope_lists and
  resolve_graphs, now shared, and the same scope digest). Frames: a signed
  session frame, one tx frame per source transaction (all its in-scope
  graphs; begin/parts/commit when too large), signed coverage after each
  batch and as a 15 s heartbeat, and signed control (epoch_end,
  tx_too_large).
- Coverage W is read in ONE read transaction with the page: the head when
  the page reached it, else the page's last transaction.
- POST /replicate/challenge delivers a nonce to its session's producer,
  which answers it from a read made after receipt that reached the head.
- Signed with the server's existing ed25519 identity under the
  quipu-replicate/v1 domain; a server without a key serves no stream.
- Both routes are write-class (fail-closed like /share/asof) until
  per-graph read authorization has its ruling.

Supporting changes: Store::changes_in_graphs (a graph-set page),
PartBuilder::in_session (parts carry session and frame_seq inside the
digested text), and the applier maps tx_too_large to Rebootstrap.

Tests: 8 library tests drive sessions into the applier (scoped equality,
tail and filtered no-op, challenge only at the head, heartbeat, parts,
epoch end, refusals, scope digest); 6 server tests (stream-only follower
follows a new commit within 5 s, challenge through the route bounds
freshness, 404/400/503 refusals, disconnect frees the session); 5 producer
mutations each fail a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* style(replicate): borrow a transaction's records when framing them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* style(replicate): let-else in the producer loop; read SSE chunks as str directly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): keep coverage-only anchors and end the session on chunk recovery

Review findings on #816 (non-author review):

P1. A scoped stream can verify coverage before its first matching
transaction. That coverage was persisted, but on reopen Stream::load
read it only for the source of applied progress, and there is none, so
the cursor was lost. Coverage now loads with the source it was verified
from. check_source anchors on applied progress or, before any, on that
covered source, so another source or epoch still asks for a rebootstrap.

P2. A chunk part or commit that broke returned Resume directly. That
cleared the chunk but kept the stream session and any earlier proof.
Every in-session Resume now goes through fault(), as a sequence gap
already did.

Controls, each failing with its repair reverted:
- coverage before any transaction anchors across reopen: W kept, a
  session at W opens, one past W resumes from W;
- coverage from one source refuses another, before and after reopen;
- a broken chunk ends the session and its proofs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* refactor(asof-share): move the shared scope checks into asof_share/scope.rs

The merge with #831 took src/asof_share.rs to 526 code lines (limit 500).
scope_lists, resolve_graphs and refuse_frozen, the checks the cut and the
replication stream share, now live in their own submodule and are
re-exported unchanged, so their callers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(replicate-stream): open in one snapshot; bound a transaction before it is built

Review findings on the stream producer:

- P1: Session::open checked the scope and frozen graphs, then read the
  log epoch, as separate reads. A freeze committed between them opened
  the session on the NEW epoch with a scope checked before the freeze,
  and the first poll signed coverage over zero records. Open now reads
  the epoch first and every check in one read snapshot. A change during
  the open leaves the session on the old epoch, and its first poll ends
  it with epoch_end.
- P2: only a single record larger than a frame was refused, so a
  transaction of many small records was built whole in memory. A
  bounded pre-pass (Store::tx_sizes_in_graphs) counts in-scope rows and
  value bytes in SQL. A transaction over Limits::tx_max_rows or
  tx_max_bytes ends the session with tx_too_large and the cursor
  unchanged. A page stops at page_max_rows, and its coverage reaches
  only what it sent.

Tests: a mid-open freeze through a second handle; a refused oversized
transaction with the follower cursor held; a page cut at the row budget
that still converges; an oversized transaction later in a page. Each
fails under its mutant (epoch-last order, no snapshot, no refusal, no
cut, coverage ignoring the cut).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(replicate-stream): bound a transaction's encoded size, not its value bytes

Review residual on the stream pre-pass: SUM(length(v)) counted only the
value blobs. A Ref's blob is its 9-byte id, and every record also carries
its entity, attribute and graph IRIs plus the transaction's timestamp,
actor and source, so a transaction with small values could still build
far more than the limit.

Store::tx_encoded_bound is now an upper bound on what changes_in_graphs
builds, read without building it:
- IRI lengths (entity, attribute, graph, Ref target) come from SQL
  length() on the terms table, so no term is loaded to measure it;
- the transaction's metadata counts once per record;
- every string counts at JSON's worst-case escaping (6 bytes per byte),
  plus a fixed per-record overhead;
- counting stops once the bound passes the cap.
Limits gains page_max_bytes. The pre-pass refuses on rows first, so a
transaction over the row limit is never read.

The limit tests move to replicate_source_limits_tests.rs. New tests
check the bound against the real encoded records for each shape
(escaped strings and retracts, a Ref to a long IRI, bytes, metadata
repeated across records) and cover long entity IRIs and the page byte
cut. Each fails under its mutant.

Known limit: a term held only in an attached layer is measured by
resolving it once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(replicate-stream): count a string's bytes past an embedded NUL

SQLite's length() on a text value counts characters up to the first NUL,
so an IRI, actor, source or valid time holding a NUL undercounted the
transaction bound (706 against an encoded 6232 in the reviewer's
control). Every text length in tx_encoded_bound and term_len now reads
length(CAST(x AS BLOB)), which counts every byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant