Repository navigation
feat(applier): slice 3a, stream sessions, signed coverage and the freshness challenge - #816
Conversation
…ls its source A replica built only from /changes records diverged from its source in two ways (local-sync-mode.md F3, F4): - Records carried no valid time, so a fact valid since 2025 replayed as valid from its commit time. Records now carry valid_from and valid_to. A later retract closes the asserted row in place, so a row a later transaction closed reports valid_to null and the closing arrives as the retract's own record. Otherwise a served record would change after the fact, and a replica would assert an already-closed fact its retract could not find. - Byte values were reduced to bytes_len. They now also carry the body as base64. Both are additive fields. New replay tests rebuild a store from the feed and compare a canonical live-fact digest: a plain control, valid time (open and bounded), bytes, and an assert record that reads the same before and after a later retract. Removing each change turns its test red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
A functional-property supersede closed the prior value with an UPDATE and wrote no op=0 row, unlike an explicit retract. The change feed derives its records from rows, so it showed the new value and never the end of the old one, and a replica built from the feed kept both (local-sync-mode.md F1). The supersede now writes the same immutable op=0 row an explicit retract writes, at the superseding transaction. The resident read model reads op=1 rows and their retracted_tx stamp, so it is unaffected. The replay test asserts the replica holds exactly one current value; removing the new row turns it red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
… the stream A /changes/stream page larger than the 1 MiB frame budget sent quipu.error without advancing the cursor and ended the stream. A consumer reconnecting at that cursor hit the same transaction and the same dead end forever (local-sync-mode.md F6). Such a transaction now arrives as quipu.changes.begin, numbered quipu.changes.part frames and a quipu.changes.commit barrier (design §6). Only the barrier carries the SSE id, so Last-Event-ID never lands inside a transaction and a reconnect replays it from the first part. The barrier digest is SHA-256 over the parts' data exactly as sent; each part carries the hash of the parts before it. A single record larger than a frame is refused with quipu.changes.tx_too_large and no id, leaving the cursor unchanged. Tests stream a 2500-record transaction: every part fits, the digests chain, the reassembled records equal the page, the next transaction follows, and a reconnect replays from part 1. Disabling the split turns the test red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
chunk_transaction now returns transport-neutral frames (event name, optional cursor id, data string), and the SSE stream turns them into events only at the edge. Another transport can send the same frames and the same digests unchanged. A new test checks that the frames SSE sends equal the ones chunk_transaction returns, byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
A change page did not say which store produced it, and nothing marked a history boundary, so a follower could not tell a restored or cloned primary, or one that had removed served records, from the one it was following (local-sync-mode.md §5, F5). Every page now carries source.store_id and source.log_epoch. The epoch lives in a new one-row log_epoch table, minted when the store is created (or first opened by this version) and minted again by Store::mint_log_epoch for an operation that removes served history. Share reconstruction excludes the table, so a restored copy mints its own epoch and its followers re-bootstrap. The explicit restore/adopt ownership step is deferred to aegis-s24.2, per the xdvg.1 ruling. Tests cover the page source, an epoch stable across reopen, a new epoch after minting, and a store that predates the table getting one on first open. Removing the field turns both tests red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
An oversized transaction's begin and commit frames carried no source,
so a follower could not epoch-check a chunked transaction the way it
checks a whole page, although a changed epoch must mean re-bootstrap,
never splice. Both frames now carry the page's {store_id, log_epoch}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
…d feed One component turns the committed feed into a read-only replica, so the HA follower, a scoped follower and the in-page browser store never write their own projection logic (docs/design/shared-applier-api.md). This slice applies the frames that ship today, given as (event, data) text exactly as SSE or a WebSocket delivers them: /changes pages and the begin/part/commit frames of an oversized transaction. It does no I/O, so it builds for wasm32 too. - The first frame binds the replica to its source (store_id, log_epoch); a frame from another source returns Rebootstrap, never a splice. - Each source transaction applies in one savepoint with the persisted cursor; a chunked one is buffered, its running digest checked part by part, and nothing is visible until the commit verifies (or the caller is told to resume from the cursor). - A record the replica cannot reproduce exactly (a byte value sent as its length only, an unknown value shape) refuses the whole transaction. - Freshness is (last applied tx, time since the last frame), never bounded. The feed-replay conformance tests now use the same record->datum conversion (record_to_datum) and share live_digest, so there is one mapping. The cursor table is excluded from shares, like the log epoch it is bound to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…tract The replica equality oracle now sits beside the code it judges, so prove it sees valid time, byte content and graph (review ask). Outcome and Freshness serialize tagged by kind, the contract a browser bridge returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
A /changes page covers every transaction in its scope after the cursor it was asked for, but did not say which cursor that was. Transaction ids are global and a per-graph page skips other graphs' ids, so the applier could not tell a lost page from an ordinary skip and would splice past it. - Every page now carries `since`, and the chunked `begin` and `commit` frames repeat it. - The applier resumes (nothing applied) when a page or a chunked transaction answers a cursor above its own. A new replica starts at 0 until a bootstrap exists. A page that names no `since` is refused. - A page arriving while a chunked transaction is open discards the chunk and resumes. Before, the page could apply the chunk's transaction, move the cursor past it, and the commit was then dropped as a duplicate. Seven tests: - a dropped page; - a first page not at 0; - a page without `since`; - a page inside an open chunk; - an interrupted chunk restarted from a new begin; - a re-opened applier holding no staged parts; - a chunk that skips a transaction. With the two checks disabled, four of them fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…e applier The applier commit added `pub mod applier` to src/lib.rs, so its repository-knowledge witness hash went stale (the pendant source controls job). The record's summary still holds; only the hash changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
The shared applier's bootstrap (docs/design/as-of-share.md). A share carries the source's live facts at its watermark as ordinary change records, so bootstrap and tail use ONE record_to_datum and prove equality with the same live digest. It arrives as one `quipu.asof` frame, or as `quipu.asof.begin` / `.part` / `.commit` with the same digest chain as a chunked transaction (the part and commit checks are now shared `Chunk` methods). All or nothing. These refuse with nothing written: - a replica that is not empty; - a manifest that is not quipu-asof-share/v1, or does not claim valid time and lossless bytes; - a record that is not an assert, or is in a graph the manifest does not list; - a graph whose records do not match the manifest's count and digest (SHA-256 over sorted, key-sorted compact JSON lines, `share_graph_digest`). On success the facts and the cursor (store_id, log_epoch, watermark_tx) commit in one savepoint, and slice 1 tails from /changes?since=watermark. Seven tests: - bootstrap then tail to equality with the source (valid time, bytes, named graph, a later retraction), and the cursor survives reopening; - chunked equals single-frame, and nothing is visible before the commit; - a dropped part resumes with nothing written, and a restarted cut converges; - every untrustworthy share refused, with nothing written and no cursor; - only an empty replica; - a new epoch after bootstrap means rebootstrap; - the digest ignores key and record order but not content. With the count/digest and emptiness checks disabled, two tests fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…Database The shared applier joins the binding facade, so the in-page store and any other binding reach it through one engine: Engine::replica_apply_frame, replica_progress and replica_freshness, exported to JavaScript on the wasm Database as replicaApplyFrame / replicaProgress / replicaFreshness (JSON text in the contract Outcome and Freshness serialize to). - The first replica call makes the store a read-only replica: local writes through invoke (knot, episode, set, retract, shapes load/remove) are refused before dispatch with read_only_replica / not_started. Reads work. - The applier writes what the source already accepted, so local write validation is suspended for the applier's own writes and restored after. Re-judging source data against local shapes could refuse it and fork the replica silently. A test proves both halves: the adopted shape refuses the data when written locally, and the replica applies it from the feed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
15c8128 changed docs/book/src/reference/rest-api.md without moving its witness, so the repository-knowledge check refused the registry. Only the source_sha256 moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…n digest
Rule 6 of the shared applier API (docs/design/shared-applier-api.md): a
transaction at or below the cursor was skipped as a duplicate without
looking at it, so a redelivered tx 5 with changed records answered exactly
like an exact replay.
The applier now computes each transaction's content digest itself from the
decoded records (domain tag, store id, log epoch, source tx, then each
record's length-prefixed exact encoding, sorted), independent of framing,
and retains it in replica_applied_tx in the same savepoint as the records
and the cursor. A redelivery is Duplicate only when its digest equals the
retained one; a different digest is Stopped { DigestMismatch }, no
retained row is Stopped { DigestUnverifiable }, nothing is written, and the
stop is sticky. A frame from a new source still answers Rebootstrap.
replica_applied_tx is excluded from shares with the cursor it belongs to.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
record_source uses TS in every build, so the no-default-features test build failed to compile (E0425). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…bootstrap-20261024
…ng it Slice 2 buffered every part of a chunked share in memory until the commit (found on a 9.16M-record share). The begin frame now opens ONE local savepoint; each part's facts are written as it arrives, in batches the size of a part, and its canonical lines go to a temporary table so the per-graph digests are computed in sorted order at the commit. Any failed check rolls the savepoint back with nothing written, as before; the facts and the cursor still commit together. Same-connection reads now see a share's parts before its commit. The replica has no cursor until Bootstrapped, and a reader must not use it before then. A tail frame mid-share aborts the share, and an applier opened on a store left mid-share rolls it back. The digest chain moves to src/applier_chunk.rs and counts records without keeping them when the caller streams. examples/applier_bootstrap_rss measures peak RSS for a synthetic share at a chosen size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…0261024 # Conflicts: # docs/knowledge/repository-knowledge.json # src/server/feed_stream_tests.rs # src/store/changes_replay_tests.rs
…bootstrap-20261024
…0261024 # Conflicts: # src/server/feed_stream.rs # src/share_completeness.rs
…bootstrap-20261024
…refused Review findings on the shared applier: - P1: the sticky Stopped outcome lived only in memory, so a fresh Applier on the same replica tailed on past a recorded digest mismatch. The stop is now kept in replica_stop (first stop wins) and restored by open; a replica whose history disagreed with its source stays stopped until it is rebuilt. - P2: any op other than "retract" was applied as an assert. Only "assert" and "retract" are accepted, in the record conversion and the transaction digest alike; anything else refuses the transaction with nothing written. replica_stop is excluded from shares with the cursor it belongs to. Controls: a stop survives reopening (fails with open ignoring the stop), and an unknown op is refused (fails with unknown ops read as assert). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…bootstrap-20261024 # Conflicts: # src/applier.rs
…shness challenge The applier's side of the scoped replication stream (local-sync-mode.md §5, shared-applier-api.md rules 1-8): - A signed session frame binds a session id to the source, the pinned scope digest and its since; a since above the cursor resumes. - frame_seq must be exactly last + 1, and each transaction newer than the last the session sent; anything else drops the session and resumes. - Coverage is verified against a caller-pinned stream key (never learned from the stream), persisted beside the cursor, and never moves back. - Freshness is bounded only by a coverage frame answering an outstanding challenge of the open session, consumed on use. A heartbeat, a replayed or unsolicited nonce, a new session, a reopen, end_session or a clock discontinuity never yields a bound. - Control frames: revoked (persisted, sticky across reopening), epoch_end (rebootstrap). replicate_wire.rs holds the signed bytes, shared with the producer that follows: domain tag quipu-replicate/v1, then length-prefixed fields. 22 stream tests; each of 12 mutations of the checks fails at least one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
Review 6387176506 (CHANGES): replica_apply_frame promoted any Engine to a read-only replica on its first call, and the applier tailed the feed into a store that already held local facts. The replica would then equal no source while its cursor and digests claim it does. - Applier::new_replica_refusal: a store may start a replica only if it already follows a source or holds no facts (shapes are not facts). - The applier refuses a tail into such a store with nothing written. - The binding checks before promoting: the frame is refused and the store stays an ordinary writable store. Paired controls: with either gate removed, its test fails. record_to_datum and value_of move unchanged to applier_record.rs, and the gate lives in applier_start.rs, to keep applier.rs under the size gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…bridge-20261024 # Conflicts: # src/applier.rs # src/applier_tests.rs
…tream-s3a-20261024 # Conflicts: # docs/knowledge/repository-knowledge.json
|
Restacked onto #803 head 4a6864a (which is on main d68a89a); new head b8193e9. The merge is conflict-free except one knowledge-witness hash, which was recomputed. Review scope is unchanged: 5a2b5af and 311075d. Local results: default suite 3247 passed, 0 failed; server 240 passed, 0 failed; clippy on 3 feature sets, fmt, the size gate and wasm32 are clean. Stays DRAFT until #803 lands. 🤖 Generated with Claude Code |
…0261024 # Conflicts: # src/applier.rs # src/applier_bootstrap.rs # src/applier_bootstrap_tests.rs
|
Ready for review. #803 has landed (bb9254c), and this PR now merges main at 43a88e6. The merge kept main's Review scope: 5a2b5af and 311075d (S3a: stream sessions, signed coverage, the freshness challenge). Local results:
Earlier in this run, 🤖 Generated with Claude Code |
|
Non-author Ian — CHANGES at
Validation: 22 supplied stream tests plus the independent nonempty persistence control pass; the independent coverage-only reopen control fails ( |
… recovery Review findings on #816 (non-author review): P1. A scoped stream can verify coverage before its first matching transaction. That coverage was persisted, but on reopen Stream::load read it only for the source of applied progress, and there is none, so the cursor was lost. Coverage now loads with the source it was verified from. check_source anchors on applied progress or, before any, on that covered source, so another source or epoch still asks for a rebootstrap. P2. A chunk part or commit that broke returned Resume directly. That cleared the chunk but kept the stream session and any earlier proof. Every in-session Resume now goes through fault(), as a sequence gap already did. Controls, each failing with its repair reverted: - coverage before any transaction anchors across reopen: W kept, a session at W opens, one past W resumes from W; - coverage from one source refuses another, before and after reopen; - a broken chunk ends the session and its proofs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
Repair for the review CHANGES (comment 6391197995). The new head is 0718837, one commit on top of 43a88e6. The head is frozen for the same reviewer.
New controls, each failing with its repair reverted (3/3):
Local results:
|
|
Same-branch refresh onto current main after the strict classification refused 0718837 because 045f1fa was missing.
Head frozen for the same reviewer (ian). The original CHANGES review stands until that review. |
|
Second same-branch refresh: main moved to 7c9a80d (#819, attested writes) while 58b9da7's CI ran.
Head frozen for the same reviewer (ian). Fresh hosted CI is running. |
…0261025 # Conflicts: # docs/knowledge/repository-knowledge.json
|
Third same-branch refresh: main 9fc5294 (#820). New head 2e21333.
Head frozen for the same reviewer (ian). Fresh CI is running. |
|
Refreshed onto main 07f1a77 (#825 and #799, docs only). New head f4eb009, a merge with no conflicts. Proposal hunks are identical: with |
|
Refreshed onto main d6c403c (#823, tests only: 🤖 Generated with Claude Code |
|
Refreshed onto main 00e0972 (#786, #821, #830); ian's native admission required it. New head 1e4947f, a merge with no conflicts. Proposal hunks are identical: with 🤖 Generated with Claude Code |
|
Non-author Ian: SAME repair source review COMPLETE, no remaining blocking source findings at
Repair delta0718837c inspected; seven core stream/wire/applier files are byte-identical071->current1e494. Current native PREPARED/complete-green admission preceded SAME source inspection. Validation: 28 stream tests PASS (25 supplied + original2 reviewer controls +1 new bad-part control), 3 wire tests PASS. Both inner0 and both outer0; each HOME guard157 watched paths+5 UID resources unchanged. Temporary tests restored byte-for-byte; own checkout clean. No runtime/deploy/activation action. Native approval NOT ACCEPTED yet. Wu supplied current diff-bound record3c05b0ba on maind63e3f07 after main advanced during the review. Current helperc6e318 refused final review binding: main Test(default) and Test(shacl) in progress, Release correctness missing. No exception/retry/merge. Retain Ian sole binding-only follow-up when exact-main gate is green; no further broad audit needed. Normal diff-bound merge admission/holds/rollback/exact-main watch remain with the lander. |
|
Ian SAME binding END: native CONTENT |
…to the S3b stream branch Restacked on #816's final head 1e4947f first (clean), then main's delta since that head. One real conflict, src/asof_share.rs: this branch extracted the per-graph checks into resolve_graphs (shared with the replication stream), and #831 changed the same loop to read frozen graphs from their packs and refuse the graph-metadata graph. Resolution: - resolve_graphs refuses the graph-metadata graph (both cut and stream, per the s24.1.1 ruling) and attached packs, and no longer refuses frozen graphs itself; - the cut reads frozen graphs from their packs exactly as #831 does (pack re-verified inside the transaction, frozen_member); - the stream keeps this branch's reviewed behaviour and refuses frozen graphs explicitly (refuse_frozen), now tested together with the metadata refusal. The derived rest-api.md witness hash is recomputed from the merged file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…lenge (#817) * fix(changes): carry valid time and byte bodies so a feed replica equals its source A replica built only from /changes records diverged from its source in two ways (local-sync-mode.md F3, F4): - Records carried no valid time, so a fact valid since 2025 replayed as valid from its commit time. Records now carry valid_from and valid_to. A later retract closes the asserted row in place, so a row a later transaction closed reports valid_to null and the closing arrives as the retract's own record. Otherwise a served record would change after the fact, and a replica would assert an already-closed fact its retract could not find. - Byte values were reduced to bytes_len. They now also carry the body as base64. Both are additive fields. New replay tests rebuild a store from the feed and compare a canonical live-fact digest: a plain control, valid time (open and bounded), bytes, and an assert record that reads the same before and after a later retract. Removing each change turns its test red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac * fix(owl): write a retract row when a functional value is superseded A functional-property supersede closed the prior value with an UPDATE and wrote no op=0 row, unlike an explicit retract. The change feed derives its records from rows, so it showed the new value and never the end of the old one, and a replica built from the feed kept both (local-sync-mode.md F1). The supersede now writes the same immutable op=0 row an explicit retract writes, at the superseding transaction. The resident read model reads op=1 rows and their retracted_tx stamp, so it is unaffected. The replay test asserts the replica holds exactly one current value; removing the new row turns it red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac * fix(stream): send an oversized transaction as parts instead of ending the stream A /changes/stream page larger than the 1 MiB frame budget sent quipu.error without advancing the cursor and ended the stream. A consumer reconnecting at that cursor hit the same transaction and the same dead end forever (local-sync-mode.md F6). Such a transaction now arrives as quipu.changes.begin, numbered quipu.changes.part frames and a quipu.changes.commit barrier (design §6). Only the barrier carries the SSE id, so Last-Event-ID never lands inside a transaction and a reconnect replays it from the first part. The barrier digest is SHA-256 over the parts' data exactly as sent; each part carries the hash of the parts before it. A single record larger than a frame is refused with quipu.changes.tx_too_large and no id, leaving the cursor unchanged. Tests stream a 2500-record transaction: every part fits, the digests chain, the reassembled records equal the page, the next transaction follows, and a reconnect replays from part 1. Disabling the split turns the test red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac * refactor(stream): build oversized-transaction frames independent of SSE chunk_transaction now returns transport-neutral frames (event name, optional cursor id, data string), and the SSE stream turns them into events only at the edge. Another transport can send the same frames and the same digests unchanged. A new test checks that the frames SSE sends equal the ones chunk_transaction returns, byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF * feat(changes): name the source store and log epoch on every page A change page did not say which store produced it, and nothing marked a history boundary, so a follower could not tell a restored or cloned primary, or one that had removed served records, from the one it was following (local-sync-mode.md §5, F5). Every page now carries source.store_id and source.log_epoch. The epoch lives in a new one-row log_epoch table, minted when the store is created (or first opened by this version) and minted again by Store::mint_log_epoch for an operation that removes served history. Share reconstruction excludes the table, so a restored copy mints its own epoch and its followers re-bootstrap. The explicit restore/adopt ownership step is deferred to aegis-s24.2, per the xdvg.1 ruling. Tests cover the page source, an epoch stable across reopen, a new epoch after minting, and a store that predates the table getting one on first open. Removing the field turns both tests red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac * fix(stream): name the source on a chunked transaction's begin and commit An oversized transaction's begin and commit frames carried no source, so a follower could not epoch-check a chunked transaction the way it checks a whole page, although a changed epoch must mean re-bootstrap, never splice. Both frames now carry the page's {store_id, log_epoch}. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF * feat(applier): the shared applier, slice 1: a replica from the shipped feed One component turns the committed feed into a read-only replica, so the HA follower, a scoped follower and the in-page browser store never write their own projection logic (docs/design/shared-applier-api.md). This slice applies the frames that ship today, given as (event, data) text exactly as SSE or a WebSocket delivers them: /changes pages and the begin/part/commit frames of an oversized transaction. It does no I/O, so it builds for wasm32 too. - The first frame binds the replica to its source (store_id, log_epoch); a frame from another source returns Rebootstrap, never a splice. - Each source transaction applies in one savepoint with the persisted cursor; a chunked one is buffered, its running digest checked part by part, and nothing is visible until the commit verifies (or the caller is told to resume from the cursor). - A record the replica cannot reproduce exactly (a byte value sent as its length only, an unknown value shape) refuses the whole transaction. - Freshness is (last applied tx, time since the last frame), never bounded. The feed-replay conformance tests now use the same record->datum conversion (record_to_datum) and share live_digest, so there is one mapping. The cursor table is excluded from shares, like the log epoch it is bound to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * test(applier): a digest-sensitivity control and the bridge's JSON contract The replica equality oracle now sits beside the code it judges, so prove it sees valid time, byte content and graph (review ask). Outcome and Freshness serialize tagged by kind, the contract a browser bridge returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(applier): refuse a gap; a page names the cursor it answers A /changes page covers every transaction in its scope after the cursor it was asked for, but did not say which cursor that was. Transaction ids are global and a per-graph page skips other graphs' ids, so the applier could not tell a lost page from an ordinary skip and would splice past it. - Every page now carries `since`, and the chunked `begin` and `commit` frames repeat it. - The applier resumes (nothing applied) when a page or a chunked transaction answers a cursor above its own. A new replica starts at 0 until a bootstrap exists. A page that names no `since` is refused. - A page arriving while a chunked transaction is open discards the chunk and resumes. Before, the page could apply the chunk's transaction, move the cursor past it, and the commit was then dropped as a duplicate. Seven tests: - a dropped page; - a first page not at 0; - a page without `since`; - a page inside an open chunk; - an interrupted chunk restarted from a new begin; - a re-opened applier holding no staged parts; - a chunk that skips a transaction. With the two checks disabled, four of them fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(knowledge): refresh the src/lib.rs source witness after adding the applier The applier commit added `pub mod applier` to src/lib.rs, so its repository-knowledge witness hash went stale (the pendant source controls job). The record's summary still holds; only the hash changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * feat(applier): slice 2, a replica bootstrapped from an as-of share The shared applier's bootstrap (docs/design/as-of-share.md). A share carries the source's live facts at its watermark as ordinary change records, so bootstrap and tail use ONE record_to_datum and prove equality with the same live digest. It arrives as one `quipu.asof` frame, or as `quipu.asof.begin` / `.part` / `.commit` with the same digest chain as a chunked transaction (the part and commit checks are now shared `Chunk` methods). All or nothing. These refuse with nothing written: - a replica that is not empty; - a manifest that is not quipu-asof-share/v1, or does not claim valid time and lossless bytes; - a record that is not an assert, or is in a graph the manifest does not list; - a graph whose records do not match the manifest's count and digest (SHA-256 over sorted, key-sorted compact JSON lines, `share_graph_digest`). On success the facts and the cursor (store_id, log_epoch, watermark_tx) commit in one savepoint, and slice 1 tails from /changes?since=watermark. Seven tests: - bootstrap then tail to equality with the source (valid time, bytes, named graph, a later retraction), and the cursor survives reopening; - chunked equals single-frame, and nothing is visible before the commit; - a dropped part resumes with nothing written, and a restarted cut converges; - every untrustworthy share refused, with nothing written and no cursor; - only an empty replica; - a new epoch after bootstrap means rebootstrap; - the digest ignores key and record order but not content. With the count/digest and emptiness checks disabled, two tests fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * feat(bindings): a replica through the binding facade and the browser Database The shared applier joins the binding facade, so the in-page store and any other binding reach it through one engine: Engine::replica_apply_frame, replica_progress and replica_freshness, exported to JavaScript on the wasm Database as replicaApplyFrame / replicaProgress / replicaFreshness (JSON text in the contract Outcome and Freshness serialize to). - The first replica call makes the store a read-only replica: local writes through invoke (knot, episode, set, retract, shapes load/remove) are refused before dispatch with read_only_replica / not_started. Reads work. - The applier writes what the source already accepted, so local write validation is suspended for the applier's own writes and restored after. Re-judging source data against local shapes could refuse it and fork the replica silently. A test proves both halves: the adopted shape refuses the data when written locally, and the replica applies it from the feed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(knowledge): refresh the rest-api.md source witness 15c8128 changed docs/book/src/reference/rest-api.md without moving its witness, so the repository-knowledge check refused the registry. Only the source_sha256 moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * feat(applier): a duplicate is checked against the retained transaction digest Rule 6 of the shared applier API (docs/design/shared-applier-api.md): a transaction at or below the cursor was skipped as a duplicate without looking at it, so a redelivered tx 5 with changed records answered exactly like an exact replay. The applier now computes each transaction's content digest itself from the decoded records (domain tag, store id, log epoch, source tx, then each record's length-prefixed exact encoding, sorted), independent of framing, and retains it in replica_applied_tx in the same savepoint as the records and the cursor. A redelivery is Duplicate only when its digest equals the retained one; a different digest is Stopped { DigestMismatch }, no retained row is Stopped { DigestUnverifiable }, nothing is written, and the stop is sticky. A frame from a new source still answers Rebootstrap. replica_applied_tx is excluded from shares with the cursor it belongs to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(tests): the bindings timestamp constant is not shacl-only record_source uses TS in every build, so the no-default-features test build failed to compile (E0425). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(applier): stream an as-of share into the replica instead of holding it Slice 2 buffered every part of a chunked share in memory until the commit (found on a 9.16M-record share). The begin frame now opens ONE local savepoint; each part's facts are written as it arrives, in batches the size of a part, and its canonical lines go to a temporary table so the per-graph digests are computed in sorted order at the commit. Any failed check rolls the savepoint back with nothing written, as before; the facts and the cursor still commit together. Same-connection reads now see a share's parts before its commit. The replica has no cursor until Bootstrapped, and a reader must not use it before then. A tail frame mid-share aborts the share, and an applier opened on a store left mid-share rolls it back. The digest chain moves to src/applier_chunk.rs and counts records without keeping them when the caller streams. examples/applier_bootstrap_rss measures peak RSS for a synthetic share at a chosen size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(applier): a stop persists across reopening, and an unknown op is refused Review findings on the shared applier: - P1: the sticky Stopped outcome lived only in memory, so a fresh Applier on the same replica tailed on past a recorded digest mismatch. The stop is now kept in replica_stop (first stop wins) and restored by open; a replica whose history disagreed with its source stays stopped until it is rebuilt. - P2: any op other than "retract" was applied as an assert. Only "assert" and "retract" are accepted, in the record conversion and the transaction digest alike; anything else refuses the transaction with nothing written. replica_stop is excluded from shares with the cursor it belongs to. Controls: a stop survives reopening (fails with open ignoring the stop), and an unknown op is refused (fails with unknown ops read as assert). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * feat(applier): slice 3a, stream sessions, signed coverage and the freshness challenge The applier's side of the scoped replication stream (local-sync-mode.md §5, shared-applier-api.md rules 1-8): - A signed session frame binds a session id to the source, the pinned scope digest and its since; a since above the cursor resumes. - frame_seq must be exactly last + 1, and each transaction newer than the last the session sent; anything else drops the session and resumes. - Coverage is verified against a caller-pinned stream key (never learned from the stream), persisted beside the cursor, and never moves back. - Freshness is bounded only by a coverage frame answering an outstanding challenge of the open session, consumed on use. A heartbeat, a replayed or unsolicited nonce, a new session, a reopen, end_session or a clock discontinuity never yields a bound. - Control frames: revoked (persisted, sticky across reopening), epoch_end (rebootstrap). replicate_wire.rs holds the signed bytes, shared with the producer that follows: domain tag quipu-replicate/v1, then length-prefixed fields. 22 stream tests; each of 12 mutations of the checks fails at least one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(applier): a store holding local facts never becomes a replica Review 6387176506 (CHANGES): replica_apply_frame promoted any Engine to a read-only replica on its first call, and the applier tailed the feed into a store that already held local facts. The replica would then equal no source while its cursor and digests claim it does. - Applier::new_replica_refusal: a store may start a replica only if it already follows a source or holds no facts (shapes are not facts). - The applier refuses a tail into such a store with nothing written. - The binding checks before promoting: the frame is refused and the store stays an ordinary writable store. Paired controls: with either gate removed, its test fails. record_to_datum and value_of move unchanged to applier_record.rs, and the gate lives in applier_start.rs, to keep applier.rs under the size gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(bindings): carry the stream freshness fields on a store that is not a replica Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * test(applier): the as-of share producer and the applier agree at the seam Now that #805 is on main: a real asof_share::cut of a file-backed source (300 items, unicode and escaped titles, bytes, a superseded value), framed as the server sends it (one quipu.asof frame, or begin/parts/commit), bootstraps an empty replica to the source's live facts; one tampered part with the chain left intact is refused with nothing written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * feat(replicate): the scoped replication stream and its freshness challenge The source side of local-sync-mode.md §5, slice 3b of the shared applier: - POST /replicate/stream: one SSE session per follower over the scope an as-of share would serve (same rules via asof_share::scope_lists and resolve_graphs, now shared, and the same scope digest). Frames: a signed session frame, one tx frame per source transaction (all its in-scope graphs; begin/parts/commit when too large), signed coverage after each batch and as a 15 s heartbeat, and signed control (epoch_end, tx_too_large). - Coverage W is read in ONE read transaction with the page: the head when the page reached it, else the page's last transaction. - POST /replicate/challenge delivers a nonce to its session's producer, which answers it from a read made after receipt that reached the head. - Signed with the server's existing ed25519 identity under the quipu-replicate/v1 domain; a server without a key serves no stream. - Both routes are write-class (fail-closed like /share/asof) until per-graph read authorization has its ruling. Supporting changes: Store::changes_in_graphs (a graph-set page), PartBuilder::in_session (parts carry session and frame_seq inside the digested text), and the applier maps tx_too_large to Rebootstrap. Tests: 8 library tests drive sessions into the applier (scoped equality, tail and filtered no-op, challenge only at the head, heartbeat, parts, epoch end, refusals, scope digest); 6 server tests (stream-only follower follows a new commit within 5 s, challenge through the route bounds freshness, 404/400/503 refusals, disconnect frees the session); 5 producer mutations each fail a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * style(replicate): borrow a transaction's records when framing them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * style(replicate): let-else in the producer loop; read SSE chunks as str directly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(applier): keep coverage-only anchors and end the session on chunk recovery Review findings on #816 (non-author review): P1. A scoped stream can verify coverage before its first matching transaction. That coverage was persisted, but on reopen Stream::load read it only for the source of applied progress, and there is none, so the cursor was lost. Coverage now loads with the source it was verified from. check_source anchors on applied progress or, before any, on that covered source, so another source or epoch still asks for a rebootstrap. P2. A chunk part or commit that broke returned Resume directly. That cleared the chunk but kept the stream session and any earlier proof. Every in-session Resume now goes through fault(), as a sequence gap already did. Controls, each failing with its repair reverted: - coverage before any transaction anchors across reopen: W kept, a session at W opens, one past W resumes from W; - coverage from one source refuses another, before and after reopen; - a broken chunk ends the session and its proofs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * refactor(asof-share): move the shared scope checks into asof_share/scope.rs The merge with #831 took src/asof_share.rs to 526 code lines (limit 500). scope_lists, resolve_graphs and refuse_frozen, the checks the cut and the replication stream share, now live in their own submodule and are re-exported unchanged, so their callers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(replicate-stream): open in one snapshot; bound a transaction before it is built Review findings on the stream producer: - P1: Session::open checked the scope and frozen graphs, then read the log epoch, as separate reads. A freeze committed between them opened the session on the NEW epoch with a scope checked before the freeze, and the first poll signed coverage over zero records. Open now reads the epoch first and every check in one read snapshot. A change during the open leaves the session on the old epoch, and its first poll ends it with epoch_end. - P2: only a single record larger than a frame was refused, so a transaction of many small records was built whole in memory. A bounded pre-pass (Store::tx_sizes_in_graphs) counts in-scope rows and value bytes in SQL. A transaction over Limits::tx_max_rows or tx_max_bytes ends the session with tx_too_large and the cursor unchanged. A page stops at page_max_rows, and its coverage reaches only what it sent. Tests: a mid-open freeze through a second handle; a refused oversized transaction with the follower cursor held; a page cut at the row budget that still converges; an oversized transaction later in a page. Each fails under its mutant (epoch-last order, no snapshot, no refusal, no cut, coverage ignoring the cut). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(replicate-stream): bound a transaction's encoded size, not its value bytes Review residual on the stream pre-pass: SUM(length(v)) counted only the value blobs. A Ref's blob is its 9-byte id, and every record also carries its entity, attribute and graph IRIs plus the transaction's timestamp, actor and source, so a transaction with small values could still build far more than the limit. Store::tx_encoded_bound is now an upper bound on what changes_in_graphs builds, read without building it: - IRI lengths (entity, attribute, graph, Ref target) come from SQL length() on the terms table, so no term is loaded to measure it; - the transaction's metadata counts once per record; - every string counts at JSON's worst-case escaping (6 bytes per byte), plus a fixed per-record overhead; - counting stops once the bound passes the cap. Limits gains page_max_bytes. The pre-pass refuses on rows first, so a transaction over the row limit is never read. The limit tests move to replicate_source_limits_tests.rs. New tests check the bound against the real encoded records for each shape (escaped strings and retracts, a Ref to a long IRI, bytes, metadata repeated across records) and cover long entity IRIs and the page byte cut. Each fails under its mutant. Known limit: a term held only in an attached layer is measured by resolving it once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb * fix(replicate-stream): count a string's bytes past an embedded NUL SQLite's length() on a text value counts characters up to the first NUL, so an IRI, actor, source or valid time holding a NUL undercounted the transaction bound (706 against an encoded 6232 in the reviewer's control). Every text length in tx_encoded_bound and term_len now reads length(CAST(x AS BLOB)), which counts every byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Slice 3a of the shared applier, stacked on #803 (S2), which is stacked on #798 (S1b). Review scope: the commits beyond #803, which are 5a2b5af and 311075d, plus the merge 82ce0a3.
This is the applier's side of the scoped replication stream (
docs/design/local-sync-mode.md§5;docs/design/shared-applier-api.mdrules 1-8). The producer (GET /replicate/streamandPOST /replicate/challenge) is the next slice.What it adds
quipu.replicate.sessionframe binds a session id to the source, the scope digest and itssince.Applier::trust_stream. Neither is ever learned from the stream.sinceabove the cursor answersResume.frame_seqmust be exactly last + 1.txmust be newer than the last one the session sent.Resume. Gaps in tx ids are normal and are never read as loss.replica_coverage, share-Excluded) and never moves back. A reconnect resumes frommax(applied, coverage), so a filtered no-op advances the cursor. Coverage proves completeness only.staleness_bound_ms = now - t_sendholds only from a coverage frame that answers an outstanding challenge of the open session, and the nonce is consumed. None of these yield a bound:end_sessionorclock_discontinuity.Freshness::within(ms)gates abounded=read.revokedis persisted asStop::Revokedand stays sticky across reopening.epoch_endanswersRebootstrap.src/replicate_wire.rs. The signed bytes, shared with the producer: domain tagquipu-replicate/v1, then length-prefixed fields.Decision (quipu lead). Coverage is signed with the existing ed25519 identity (
src/signing.rs), domain-separated. The S0 crate is a P3 gate, not a P2 one (local-sync §12). The bytes do not change when S0 is extracted.Controls (local-sync §12, P2)
22 stream tests cover:
within(100)is false;within(100)true; the same proof read 500 ms later is false;Mutations: 12 mutations of the checks, each failing at least one test.
Local results:
--features shacl,onnx,server --bin quipu-server: 236 passed, 0 failed;🤖 Generated with Claude Code
https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb