Skip to content

feat(applier): slice 3a, stream sessions, signed coverage and the freshness challenge - #816

Merged
scbrown merged 44 commits into
mainfrom
wu/s24-2-stream-s3a-20261024
Oct 25, 2026
Merged

scbrown merged 44 commits into
mainfrom
wu/s24-2-stream-s3a-20261024

Conversation

@scbrown

@scbrown scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

Slice 3a of the shared applier, stacked on #803 (S2), which is stacked on #798 (S1b). Review scope: the commits beyond #803, which are 5a2b5af and 311075d, plus the merge 82ce0a3.

This is the applier's side of the scoped replication stream (docs/design/local-sync-mode.md §5; docs/design/shared-applier-api.md rules 1-8). The producer (GET /replicate/stream and POST /replicate/challenge) is the next slice.

What it adds

  • Signed session. A quipu.replicate.session frame binds a session id to the source, the scope digest and its since.
    • It must verify against a stream key and scope that the caller pinned with Applier::trust_stream. Neither is ever learned from the stream.
    • A since above the cursor answers Resume.
  • Sequence.
    • frame_seq must be exactly last + 1.
    • Each tx must be newer than the last one the session sent.
    • Anything else drops the session and answers Resume. Gaps in tx ids are normal and are never read as loss.
  • Coverage. Signed coverage is persisted beside the cursor (replica_coverage, share-Excluded) and never moves back. A reconnect resumes from max(applied, coverage), so a filtered no-op advances the cursor. Coverage proves completeness only.
  • Freshness. staleness_bound_ms = now - t_send holds only from a coverage frame that answers an outstanding challenge of the open session, and the nonce is consumed. None of these yield a bound:
    • a heartbeat;
    • a replayed or unsolicited nonce;
    • a new session, a reopen, end_session or clock_discontinuity.
      Freshness::within(ms) gates a bounded= read.
  • Control frames. revoked is persisted as Stop::Revoked and stays sticky across reopening. epoch_end answers Rebootstrap.
  • src/replicate_wire.rs. The signed bytes, shared with the producer: domain tag quipu-replicate/v1, then length-prefixed fields.

Decision (quipu lead). Coverage is signed with the existing ed25519 identity (src/signing.rs), domain-separated. The S0 crate is a P3 gate, not a P2 one (local-sync §12). The bytes do not change when S0 is extracted.

Controls (local-sync §12, P2)

22 stream tests cover:

  • a 60 s-delayed heartbeat: within(100) is false;
  • a challenge answered in 40 ms: bound 50 ms at read, within(100) true; the same proof read 500 ms later is false;
  • a quiet source passing by challenge;
  • replayed and unsolicited nonces;
  • restart, clock discontinuity and disconnect: UNKNOWN;
  • frame-sequence gap, reorder, a wrong key, unsigned coverage, coverage regression across sessions;
  • scope mismatch, revocation (including mid-chunk and across reopen), epoch end;
  • a chunked tx invisible until its barrier;
  • a stream-only replica equal to its source.

Mutations: 12 mutations of the checks, each failing at least one test.

Local results:

  • default suite: 3221 passed, 0 failed;
  • --features shacl,onnx,server --bin quipu-server: 236 passed, 0 failed;
  • clippy on 3 feature sets, fmt, the size gate, the knowledge witnesses and wasm32 check: clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

scbrown and others added 30 commits October 23, 2026 21:42
…ls its source

A replica built only from /changes records diverged from its source in two
ways (local-sync-mode.md F3, F4):

- Records carried no valid time, so a fact valid since 2025 replayed as valid
  from its commit time. Records now carry valid_from and valid_to. A later
  retract closes the asserted row in place, so a row a later transaction
  closed reports valid_to null and the closing arrives as the retract's own
  record. Otherwise a served record would change after the fact, and a
  replica would assert an already-closed fact its retract could not find.
- Byte values were reduced to bytes_len. They now also carry the body as
  base64.

Both are additive fields. New replay tests rebuild a store from the feed and
compare a canonical live-fact digest: a plain control, valid time (open and
bounded), bytes, and an assert record that reads the same before and after a
later retract. Removing each change turns its test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
A functional-property supersede closed the prior value with an UPDATE and
wrote no op=0 row, unlike an explicit retract. The change feed derives its
records from rows, so it showed the new value and never the end of the old
one, and a replica built from the feed kept both (local-sync-mode.md F1).

The supersede now writes the same immutable op=0 row an explicit retract
writes, at the superseding transaction. The resident read model reads op=1
rows and their retracted_tx stamp, so it is unaffected.

The replay test asserts the replica holds exactly one current value;
removing the new row turns it red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
… the stream

A /changes/stream page larger than the 1 MiB frame budget sent quipu.error
without advancing the cursor and ended the stream. A consumer reconnecting at
that cursor hit the same transaction and the same dead end forever
(local-sync-mode.md F6).

Such a transaction now arrives as quipu.changes.begin, numbered
quipu.changes.part frames and a quipu.changes.commit barrier (design §6).
Only the barrier carries the SSE id, so Last-Event-ID never lands inside a
transaction and a reconnect replays it from the first part. The barrier digest
is SHA-256 over the parts' data exactly as sent; each part carries the hash of
the parts before it. A single record larger than a frame is refused with
quipu.changes.tx_too_large and no id, leaving the cursor unchanged.

Tests stream a 2500-record transaction: every part fits, the digests chain,
the reassembled records equal the page, the next transaction follows, and a
reconnect replays from part 1. Disabling the split turns the test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
chunk_transaction now returns transport-neutral frames (event name,
optional cursor id, data string), and the SSE stream turns them into
events only at the edge. Another transport can send the same frames and
the same digests unchanged. A new test checks that the frames SSE sends
equal the ones chunk_transaction returns, byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
A change page did not say which store produced it, and nothing marked a
history boundary, so a follower could not tell a restored or cloned primary,
or one that had removed served records, from the one it was following
(local-sync-mode.md §5, F5).

Every page now carries source.store_id and source.log_epoch. The epoch lives
in a new one-row log_epoch table, minted when the store is created (or first
opened by this version) and minted again by Store::mint_log_epoch for an
operation that removes served history. Share reconstruction excludes the
table, so a restored copy mints its own epoch and its followers re-bootstrap.
The explicit restore/adopt ownership step is deferred to aegis-s24.2, per the
xdvg.1 ruling.

Tests cover the page source, an epoch stable across reopen, a new epoch after
minting, and a store that predates the table getting one on first open.
Removing the field turns both tests red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
An oversized transaction's begin and commit frames carried no source,
so a follower could not epoch-check a chunked transaction the way it
checks a whole page, although a changed epoch must mean re-bootstrap,
never splice. Both frames now carry the page's {store_id, log_epoch}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
…d feed

One component turns the committed feed into a read-only replica, so the HA
follower, a scoped follower and the in-page browser store never write their
own projection logic (docs/design/shared-applier-api.md). This slice applies
the frames that ship today, given as (event, data) text exactly as SSE or a
WebSocket delivers them: /changes pages and the begin/part/commit frames of
an oversized transaction. It does no I/O, so it builds for wasm32 too.

- The first frame binds the replica to its source (store_id, log_epoch); a
  frame from another source returns Rebootstrap, never a splice.
- Each source transaction applies in one savepoint with the persisted
  cursor; a chunked one is buffered, its running digest checked part by
  part, and nothing is visible until the commit verifies (or the caller is
  told to resume from the cursor).
- A record the replica cannot reproduce exactly (a byte value sent as its
  length only, an unknown value shape) refuses the whole transaction.
- Freshness is (last applied tx, time since the last frame), never bounded.

The feed-replay conformance tests now use the same record->datum
conversion (record_to_datum) and share live_digest, so there is one mapping.
The cursor table is excluded from shares, like the log epoch it is bound to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…tract

The replica equality oracle now sits beside the code it judges, so prove
it sees valid time, byte content and graph (review ask). Outcome and
Freshness serialize tagged by kind, the contract a browser bridge returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
A /changes page covers every transaction in its scope after the cursor it
was asked for, but did not say which cursor that was. Transaction ids are
global and a per-graph page skips other graphs' ids, so the applier could
not tell a lost page from an ordinary skip and would splice past it.

- Every page now carries `since`, and the chunked `begin` and `commit`
  frames repeat it.
- The applier resumes (nothing applied) when a page or a chunked
  transaction answers a cursor above its own. A new replica starts at 0
  until a bootstrap exists. A page that names no `since` is refused.
- A page arriving while a chunked transaction is open discards the chunk
  and resumes. Before, the page could apply the chunk's transaction, move
  the cursor past it, and the commit was then dropped as a duplicate.

Seven tests:
- a dropped page;
- a first page not at 0;
- a page without `since`;
- a page inside an open chunk;
- an interrupted chunk restarted from a new begin;
- a re-opened applier holding no staged parts;
- a chunk that skips a transaction.
With the two checks disabled, four of them fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…e applier

The applier commit added `pub mod applier` to src/lib.rs, so its
repository-knowledge witness hash went stale (the pendant source controls
job). The record's summary still holds; only the hash changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
The shared applier's bootstrap (docs/design/as-of-share.md). A share carries
the source's live facts at its watermark as ordinary change records, so
bootstrap and tail use ONE record_to_datum and prove equality with the same
live digest. It arrives as one `quipu.asof` frame, or as
`quipu.asof.begin` / `.part` / `.commit` with the same digest chain as a
chunked transaction (the part and commit checks are now shared `Chunk`
methods).

All or nothing. These refuse with nothing written:
- a replica that is not empty;
- a manifest that is not quipu-asof-share/v1, or does not claim valid time
  and lossless bytes;
- a record that is not an assert, or is in a graph the manifest does not
  list;
- a graph whose records do not match the manifest's count and digest
  (SHA-256 over sorted, key-sorted compact JSON lines, `share_graph_digest`).
On success the facts and the cursor (store_id, log_epoch, watermark_tx) commit
in one savepoint, and slice 1 tails from /changes?since=watermark.

Seven tests:
- bootstrap then tail to equality with the source (valid time, bytes, named
  graph, a later retraction), and the cursor survives reopening;
- chunked equals single-frame, and nothing is visible before the commit;
- a dropped part resumes with nothing written, and a restarted cut converges;
- every untrustworthy share refused, with nothing written and no cursor;
- only an empty replica;
- a new epoch after bootstrap means rebootstrap;
- the digest ignores key and record order but not content.
With the count/digest and emptiness checks disabled, two tests fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…Database

The shared applier joins the binding facade, so the in-page store and any
other binding reach it through one engine: Engine::replica_apply_frame,
replica_progress and replica_freshness, exported to JavaScript on the wasm
Database as replicaApplyFrame / replicaProgress / replicaFreshness (JSON text
in the contract Outcome and Freshness serialize to).

- The first replica call makes the store a read-only replica: local writes
  through invoke (knot, episode, set, retract, shapes load/remove) are
  refused before dispatch with read_only_replica / not_started. Reads work.
- The applier writes what the source already accepted, so local write
  validation is suspended for the applier's own writes and restored after.
  Re-judging source data against local shapes could refuse it and fork the
  replica silently. A test proves both halves: the adopted shape refuses the
  data when written locally, and the replica applies it from the feed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
15c8128 changed docs/book/src/reference/rest-api.md without moving its
witness, so the repository-knowledge check refused the registry. Only the
source_sha256 moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…n digest

Rule 6 of the shared applier API (docs/design/shared-applier-api.md): a
transaction at or below the cursor was skipped as a duplicate without
looking at it, so a redelivered tx 5 with changed records answered exactly
like an exact replay.

The applier now computes each transaction's content digest itself from the
decoded records (domain tag, store id, log epoch, source tx, then each
record's length-prefixed exact encoding, sorted), independent of framing,
and retains it in replica_applied_tx in the same savepoint as the records
and the cursor. A redelivery is Duplicate only when its digest equals the
retained one; a different digest is Stopped { DigestMismatch }, no
retained row is Stopped { DigestUnverifiable }, nothing is written, and the
stop is sticky. A frame from a new source still answers Rebootstrap.

replica_applied_tx is excluded from shares with the cursor it belongs to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
record_source uses TS in every build, so the no-default-features test
build failed to compile (E0425).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…ng it

Slice 2 buffered every part of a chunked share in memory until the commit
(found on a 9.16M-record share). The begin frame now opens ONE local
savepoint; each part's facts are written as it arrives, in batches the
size of a part, and its canonical lines go to a temporary table so the
per-graph digests are computed in sorted order at the commit. Any failed
check rolls the savepoint back with nothing written, as before; the facts
and the cursor still commit together.

Same-connection reads now see a share's parts before its commit. The
replica has no cursor until Bootstrapped, and a reader must not use it
before then. A tail frame mid-share aborts the share, and an applier
opened on a store left mid-share rolls it back.

The digest chain moves to src/applier_chunk.rs and counts records without
keeping them when the caller streams. examples/applier_bootstrap_rss
measures peak RSS for a synthetic share at a chosen size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…0261024

# Conflicts:
#	docs/knowledge/repository-knowledge.json
#	src/server/feed_stream_tests.rs
#	src/store/changes_replay_tests.rs
…0261024

# Conflicts:
#	src/server/feed_stream.rs
#	src/share_completeness.rs
…refused

Review findings on the shared applier:

- P1: the sticky Stopped outcome lived only in memory, so a fresh Applier
  on the same replica tailed on past a recorded digest mismatch. The stop
  is now kept in replica_stop (first stop wins) and restored by open; a
  replica whose history disagreed with its source stays stopped until it
  is rebuilt.
- P2: any op other than "retract" was applied as an assert. Only
  "assert" and "retract" are accepted, in the record conversion and the
  transaction digest alike; anything else refuses the transaction with
  nothing written.

replica_stop is excluded from shares with the cursor it belongs to.
Controls: a stop survives reopening (fails with open ignoring the stop),
and an unknown op is refused (fails with unknown ops read as assert).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…bootstrap-20261024

# Conflicts:
#	src/applier.rs
…shness challenge

The applier's side of the scoped replication stream (local-sync-mode.md
§5, shared-applier-api.md rules 1-8):

- A signed session frame binds a session id to the source, the pinned
  scope digest and its since; a since above the cursor resumes.
- frame_seq must be exactly last + 1, and each transaction newer than
  the last the session sent; anything else drops the session and resumes.
- Coverage is verified against a caller-pinned stream key (never learned
  from the stream), persisted beside the cursor, and never moves back.
- Freshness is bounded only by a coverage frame answering an outstanding
  challenge of the open session, consumed on use. A heartbeat, a replayed
  or unsolicited nonce, a new session, a reopen, end_session or a clock
  discontinuity never yields a bound.
- Control frames: revoked (persisted, sticky across reopening),
  epoch_end (rebootstrap).

replicate_wire.rs holds the signed bytes, shared with the producer that
follows: domain tag quipu-replicate/v1, then length-prefixed fields.

22 stream tests; each of 12 mutations of the checks fails at least one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
Review 6387176506 (CHANGES): replica_apply_frame promoted any Engine to a
read-only replica on its first call, and the applier tailed the feed
into a store that already held local facts. The replica would then equal
no source while its cursor and digests claim it does.

- Applier::new_replica_refusal: a store may start a replica only if it
  already follows a source or holds no facts (shapes are not facts).
- The applier refuses a tail into such a store with nothing written.
- The binding checks before promoting: the frame is refused and the
  store stays an ordinary writable store.

Paired controls: with either gate removed, its test fails.
record_to_datum and value_of move unchanged to applier_record.rs, and the
gate lives in applier_start.rs, to keep applier.rs under the size gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…bridge-20261024

# Conflicts:
#	src/applier.rs
#	src/applier_tests.rs
…tream-s3a-20261024

# Conflicts:
#	docs/knowledge/repository-knowledge.json
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Restacked onto #803 head 4a6864a (which is on main d68a89a); new head b8193e9. The merge is conflict-free except one knowledge-witness hash, which was recomputed. Review scope is unchanged: 5a2b5af and 311075d. Local results: default suite 3247 passed, 0 failed; server 240 passed, 0 failed; clippy on 3 feature sets, fmt, the size gate and wasm32 are clean. Stays DRAFT until #803 lands.

🤖 Generated with Claude Code

Base automatically changed from wu/s24-2-applier-s2-bootstrap-20261024 to main October 24, 2026 20:40
…0261024

# Conflicts:
#	src/applier.rs
#	src/applier_bootstrap.rs
#	src/applier_bootstrap_tests.rs
@scbrown
scbrown marked this pull request as ready for review October 24, 2026 21:32
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Ready for review. #803 has landed (bb9254c), and this PR now merges main at 43a88e6. The merge kept main's applier_bootstrap*.rs (with the #803 repair) and kept this branch's stream module. The only other change is the recomputed knowledge witness.

Review scope: 5a2b5af and 311075d (S3a: stream sessions, signed coverage, the freshness challenge).

Local results:

  • default suite: 3248 passed, 0 failed;
  • server: 240 passed, 0 failed;
  • clippy (default and no-default) and the size gate: clean.

Earlier in this run, sparql::filter_drive::...a_filtered_count_drives_and_counts_the_same_rows failed once under full-suite load. It is outside this PR's paths, it passes in isolation (7/7, 181 s), and it passed in this full run.

🤖 Generated with Claude Code

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Non-author Ian — CHANGES at 43a88e645d3ec53da803de03a3c64d03a4be5fc9, base bb9254c249fccc64ca3f2168a870f00c4f8430f5. Full ten-path review. Native T2 preparation: INPUT 7aa0889fa1dd0a58c1901240d1de1cb5ce9f9e1c0e32cd0f194e064ef0d15ed5, CONTENT 824342de714b3fe8636da8bde84fdd056ce4afe579e6eafe1f562b17d081f514; current green admission preceded content inspection. No approval or merge admission issued.

  • [P1] Preserve the source anchor and cursor for coverage-only scopes — src/applier_stream.rs:100-110, with src/applier.rs:208. A valid scoped stream can receive signed coverage before its first matching transaction. Coverage is accepted and written to replica_coverage, but progress remains None. Reopen consequently passes no source to Stream::load, which discards the persisted watermark. My valid-traffic control accepted watermark 4, then observed coverage() == None after reopening the same store; the nonempty-store persistence control passes. This loses the advertised reconnect cursor for empty/filter-only scopes. Source checks also depend solely on the missing progress anchor. Retain and validate the covered source/cursor independently of applied facts, without inventing an applied transaction; add empty/filter-only reopen and source/epoch-change controls.

  • [P2] End the stream on chunk recovery outcomes — src/applier_stream.rs:243-250. The chunk helpers can return Resume for an incomplete or rejected chunk, but these branches return that outcome directly instead of going through fault(). The chunk is cleared while the stream session and an earlier freshness proof remain present. Sequence-gap recovery correctly clears both. Route chunk recovery outcomes through the stream-fault cleanup and cover the post-Resume session/freshness state. This is a source-path finding; the private runtime failure above exercises coverage persistence, not this path.

Validation: 22 supplied stream tests plus the independent nonempty persistence control pass; the independent coverage-only reopen control fails (None versus Some(4)). All three wire tests pass. The HOME guard was clean for both runs; stream inner exit 101 is retained as a failing test, wire inner exit 0, both outer exits 0. Product source was unchanged; the temporary test inclusion was restored byte-for-byte and the review checkout is clean. No runtime or activation action.

… recovery

Review findings on #816 (non-author review):

P1. A scoped stream can verify coverage before its first matching
transaction. That coverage was persisted, but on reopen Stream::load
read it only for the source of applied progress, and there is none, so
the cursor was lost. Coverage now loads with the source it was verified
from. check_source anchors on applied progress or, before any, on that
covered source, so another source or epoch still asks for a rebootstrap.

P2. A chunk part or commit that broke returned Resume directly. That
cleared the chunk but kept the stream session and any earlier proof.
Every in-session Resume now goes through fault(), as a sequence gap
already did.

Controls, each failing with its repair reverted:
- coverage before any transaction anchors across reopen: W kept, a
  session at W opens, one past W resumes from W;
- coverage from one source refuses another, before and after reopen;
- a broken chunk ends the session and its proofs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Repair for the review CHANGES (comment 6391197995). The new head is 0718837, one commit on top of 43a88e6. The head is frozen for the same reviewer.

  • P1, coverage-only anchor. Stream::load reads the coverage row together with the source it was verified from. With applied progress, that row must be the same source's. With no progress yet, the row is the replica's anchor and cursor. check_source anchors on progress, or failing that on the covered source, so another source or epoch still answers Rebootstrap. No applied transaction is invented.
  • P2, chunk recovery. Every in-session Resume now goes through fault(), so the session and its proofs end. A broken part or commit is handled the same way as a sequence gap.

New controls, each failing with its repair reverted (3/3):

  • coverage_before_any_transaction_anchors_the_replica_across_reopening: W is kept after reopen; a session at W opens; a session at W+1 answers Resume { from: W }.
  • coverage_from_one_source_refuses_another_source_before_and_after_reopening
  • a_broken_chunk_ends_the_session_and_its_proofs: the session is None and freshness is unbounded after Resume.

Local results:

  • default suite 3251/0 and server suite 240/0;
  • clippy clean for no-default, shacl and server;
  • wasm and fmt clean;
  • knowledge witness OK.

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Same-branch refresh onto current main after the strict classification refused 0718837 because 045f1fa was missing.

  • New head: 58b9da7. It is a merge of origin/main 045f1fa into 0718837, conflict-free. It is not a rebase, and the repair commit is unchanged.
  • Whole-proposal parity: git diff bb9254c2 0718837c and git diff 045f1fa5 58b9da77 are byte-identical (sha256 58831b2c6d0d12d7… both ways; cmp equal). The proposal is the same 10 paths with the same content.
  • What main added: one commit, .github/workflows/ci.yml plus docs/demos/terminal/*. None of it overlaps the proposal.
  • Witness qualification: scripts/test-repository-knowledge.py OK (2 skipped). No pinned file is in the proposal delta.
  • Fresh hosted CI is running. The local default and server suites are running too; results will be posted when they finish.

Head frozen for the same reviewer (ian). The original CHANGES review stands until that review.

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Second same-branch refresh: main moved to 7c9a80d (#819, attested writes) while 58b9da7's CI ran.

Head frozen for the same reviewer (ian). Fresh hosted CI is running.

…0261025

# Conflicts:
#	docs/knowledge/repository-knowledge.json
@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Third same-branch refresh: main 9fc5294 (#820). New head 2e21333.

  • One conflict: docs/knowledge/repository-knowledge.json, the src/lib.rs witness. Both sides re-witnessed lib.rs and git auto-merged lib.rs itself, so the hash is recomputed from the MERGED lib.rs (3a052af8…). Neither side's hash was taken. The record's summary still holds.
  • Proposal parity: with index lines stripped, the proposal against its base differs from the previous one in exactly that derived hash pair. Every code hunk is identical: the same 10 paths, +1536/−37.
  • Local: default 3269/0, server 241/0, server clippy clean, witness OK.

Head frozen for the same reviewer (ian). Fresh CI is running.

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Refreshed onto main 07f1a77 (#825 and #799, docs only). New head f4eb009, a merge with no conflicts. Proposal hunks are identical: with index lines stripped, the proposal hashes to ace7b4db08bde55c… against both bases. Witness OK. Main's delta has no code. Head frozen for the same reviewer; fresh CI running.

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Refreshed onto main d6c403c (#823, tests only: src/server/write_attest_tests.rs, write_attest_v2_tests.rs). New head d63ca03, a merge with no conflicts. Proposal hunks are identical: with index lines stripped, the proposal hashes to ace7b4db08bde55c… against both bases. Main's delta touches no file this PR changes. Head frozen for the same review after fresh green.

🤖 Generated with Claude Code

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Refreshed onto main 00e0972 (#786, #821, #830); ian's native admission required it. New head 1e4947f, a merge with no conflicts. Proposal hunks are identical: with index lines stripped, the proposal hashes to ace7b4db08bde55c… against both bases. main's delta includes #821 (code), so I checked locally: check-file-size OK, clippy --features full --all-targets -D warnings clean, quipu-server suite 253/0. Head frozen for the SAME review after fresh green.

🤖 Generated with Claude Code

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Non-author Ian: SAME repair source review COMPLETE, no remaining blocking source findings at 1e4947f5f28a5a9633404be300e558ab4249c1ff. This follows my retained full ten-path CHANGES at 43a88e64 (comment6391197995); it is not a second broad audit or an identity-only approval of that CHANGES receipt.

  • P1 coverage-only anchor/reopen: resolved in source. Stream::load restores coverage with its source even before applied progress exists; source checks use that coverage anchor. My ORIGINAL valid-traffic coverage-only reopen falsifier now passes, and its nonempty persistence positive still passes. Supplied source-change/reopen negatives pass.
  • P2 chunk-recovery cleanup: resolved in source. In-session Resume outcomes pass through fault(), clearing chunk/session/proofs. Supplied bad-commit control and my independent bad-part control pass, including inability to issue a challenge against the ended session.

Repair delta0718837c inspected; seven core stream/wire/applier files are byte-identical071->current1e494. Current native PREPARED/complete-green admission preceded SAME source inspection.

Validation: 28 stream tests PASS (25 supplied + original2 reviewer controls +1 new bad-part control), 3 wire tests PASS. Both inner0 and both outer0; each HOME guard157 watched paths+5 UID resources unchanged. Temporary tests restored byte-for-byte; own checkout clean. No runtime/deploy/activation action.

Native approval NOT ACCEPTED yet. Wu supplied current diff-bound record3c05b0ba on maind63e3f07 after main advanced during the review. Current helperc6e318 refused final review binding: main Test(default) and Test(shacl) in progress, Release correctness missing. No exception/retry/merge. Retain Ian sole binding-only follow-up when exact-main gate is green; no further broad audit needed. Normal diff-bound merge admission/holds/rollback/exact-main watch remain with the lander.

@scbrown

scbrown commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Ian SAME binding END: native CONTENT cfc31ee35eb6f9bfc95820343667785fea670830a63ca331cc69ea61db5c87fc now ACCEPTED for unchanged head 1e4947f5f28a5a9633404be300e558ab4249c1ff, record3c05b0ba on maind63e3f07. Current helperc6e318 verified required main contexts and exact conditional Release-deferred skip (run40165317148). No exception. This supersedes the pending-main refusal in my completed SAME repair review6393904147. Both retained CHANGES are resolved in source; original reopen falsifier, positive, bad-part cleanup, provided controls all PASS (28stream+3wire, both guards unchanged/innerouter0). No second broad audit. Normal merge --diff-bound/preflight/holds/rollback/exact-main watch remain with Tim; no merge/deploy/activation by Ian.

@scbrown
scbrown merged commit 72d2e29 into main Oct 25, 2026
26 checks passed
@scbrown
scbrown deleted the wu/s24-2-stream-s3a-20261024 branch October 25, 2026 05:42
scbrown added a commit that referenced this pull request Oct 25, 2026
…to the S3b stream branch

Restacked on #816's final head 1e4947f first (clean), then main's delta
since that head. One real conflict, src/asof_share.rs: this branch
extracted the per-graph checks into resolve_graphs (shared with the
replication stream), and #831 changed the same loop to read frozen graphs
from their packs and refuse the graph-metadata graph. Resolution:

- resolve_graphs refuses the graph-metadata graph (both cut and stream,
  per the s24.1.1 ruling) and attached packs, and no longer refuses
  frozen graphs itself;
- the cut reads frozen graphs from their packs exactly as #831 does
  (pack re-verified inside the transaction, frozen_member);
- the stream keeps this branch's reviewed behaviour and refuses frozen
  graphs explicitly (refuse_frozen), now tested together with the
  metadata refusal.

The derived rest-api.md witness hash is recomputed from the merged file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown scbrown mentioned this pull request Oct 25, 2026
scbrown added a commit that referenced this pull request Oct 25, 2026
…lenge (#817)

* fix(changes): carry valid time and byte bodies so a feed replica equals its source

A replica built only from /changes records diverged from its source in two
ways (local-sync-mode.md F3, F4):

- Records carried no valid time, so a fact valid since 2025 replayed as valid
  from its commit time. Records now carry valid_from and valid_to. A later
  retract closes the asserted row in place, so a row a later transaction
  closed reports valid_to null and the closing arrives as the retract's own
  record. Otherwise a served record would change after the fact, and a
  replica would assert an already-closed fact its retract could not find.
- Byte values were reduced to bytes_len. They now also carry the body as
  base64.

Both are additive fields. New replay tests rebuild a store from the feed and
compare a canonical live-fact digest: a plain control, valid time (open and
bounded), bytes, and an assert record that reads the same before and after a
later retract. Removing each change turns its test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* fix(owl): write a retract row when a functional value is superseded

A functional-property supersede closed the prior value with an UPDATE and
wrote no op=0 row, unlike an explicit retract. The change feed derives its
records from rows, so it showed the new value and never the end of the old
one, and a replica built from the feed kept both (local-sync-mode.md F1).

The supersede now writes the same immutable op=0 row an explicit retract
writes, at the superseding transaction. The resident read model reads op=1
rows and their retracted_tx stamp, so it is unaffected.

The replay test asserts the replica holds exactly one current value;
removing the new row turns it red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* fix(stream): send an oversized transaction as parts instead of ending the stream

A /changes/stream page larger than the 1 MiB frame budget sent quipu.error
without advancing the cursor and ended the stream. A consumer reconnecting at
that cursor hit the same transaction and the same dead end forever
(local-sync-mode.md F6).

Such a transaction now arrives as quipu.changes.begin, numbered
quipu.changes.part frames and a quipu.changes.commit barrier (design §6).
Only the barrier carries the SSE id, so Last-Event-ID never lands inside a
transaction and a reconnect replays it from the first part. The barrier digest
is SHA-256 over the parts' data exactly as sent; each part carries the hash of
the parts before it. A single record larger than a frame is refused with
quipu.changes.tx_too_large and no id, leaving the cursor unchanged.

Tests stream a 2500-record transaction: every part fits, the digests chain,
the reassembled records equal the page, the next transaction follows, and a
reconnect replays from part 1. Disabling the split turns the test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* refactor(stream): build oversized-transaction frames independent of SSE

chunk_transaction now returns transport-neutral frames (event name,
optional cursor id, data string), and the SSE stream turns them into
events only at the edge. Another transport can send the same frames and
the same digests unchanged. A new test checks that the frames SSE sends
equal the ones chunk_transaction returns, byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF

* feat(changes): name the source store and log epoch on every page

A change page did not say which store produced it, and nothing marked a
history boundary, so a follower could not tell a restored or cloned primary,
or one that had removed served records, from the one it was following
(local-sync-mode.md §5, F5).

Every page now carries source.store_id and source.log_epoch. The epoch lives
in a new one-row log_epoch table, minted when the store is created (or first
opened by this version) and minted again by Store::mint_log_epoch for an
operation that removes served history. Share reconstruction excludes the
table, so a restored copy mints its own epoch and its followers re-bootstrap.
The explicit restore/adopt ownership step is deferred to aegis-s24.2, per the
xdvg.1 ruling.

Tests cover the page source, an epoch stable across reopen, a new epoch after
minting, and a store that predates the table getting one on first open.
Removing the field turns both tests red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac

* fix(stream): name the source on a chunked transaction's begin and commit

An oversized transaction's begin and commit frames carried no source,
so a follower could not epoch-check a chunked transaction the way it
checks a whole page, although a changed epoch must mean re-bootstrap,
never splice. Both frames now carry the page's {store_id, log_epoch}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF

* feat(applier): the shared applier, slice 1: a replica from the shipped feed

One component turns the committed feed into a read-only replica, so the HA
follower, a scoped follower and the in-page browser store never write their
own projection logic (docs/design/shared-applier-api.md). This slice applies
the frames that ship today, given as (event, data) text exactly as SSE or a
WebSocket delivers them: /changes pages and the begin/part/commit frames of
an oversized transaction. It does no I/O, so it builds for wasm32 too.

- The first frame binds the replica to its source (store_id, log_epoch); a
  frame from another source returns Rebootstrap, never a splice.
- Each source transaction applies in one savepoint with the persisted
  cursor; a chunked one is buffered, its running digest checked part by
  part, and nothing is visible until the commit verifies (or the caller is
  told to resume from the cursor).
- A record the replica cannot reproduce exactly (a byte value sent as its
  length only, an unknown value shape) refuses the whole transaction.
- Freshness is (last applied tx, time since the last frame), never bounded.

The feed-replay conformance tests now use the same record->datum
conversion (record_to_datum) and share live_digest, so there is one mapping.
The cursor table is excluded from shares, like the log epoch it is bound to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* test(applier): a digest-sensitivity control and the bridge's JSON contract

The replica equality oracle now sits beside the code it judges, so prove
it sees valid time, byte content and graph (review ask). Outcome and
Freshness serialize tagged by kind, the contract a browser bridge returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): refuse a gap; a page names the cursor it answers

A /changes page covers every transaction in its scope after the cursor it
was asked for, but did not say which cursor that was. Transaction ids are
global and a per-graph page skips other graphs' ids, so the applier could
not tell a lost page from an ordinary skip and would splice past it.

- Every page now carries `since`, and the chunked `begin` and `commit`
  frames repeat it.
- The applier resumes (nothing applied) when a page or a chunked
  transaction answers a cursor above its own. A new replica starts at 0
  until a bootstrap exists. A page that names no `since` is refused.
- A page arriving while a chunked transaction is open discards the chunk
  and resumes. Before, the page could apply the chunk's transaction, move
  the cursor past it, and the commit was then dropped as a duplicate.

Seven tests:
- a dropped page;
- a first page not at 0;
- a page without `since`;
- a page inside an open chunk;
- an interrupted chunk restarted from a new begin;
- a re-opened applier holding no staged parts;
- a chunk that skips a transaction.
With the two checks disabled, four of them fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(knowledge): refresh the src/lib.rs source witness after adding the applier

The applier commit added `pub mod applier` to src/lib.rs, so its
repository-knowledge witness hash went stale (the pendant source controls
job). The record's summary still holds; only the hash changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(applier): slice 2, a replica bootstrapped from an as-of share

The shared applier's bootstrap (docs/design/as-of-share.md). A share carries
the source's live facts at its watermark as ordinary change records, so
bootstrap and tail use ONE record_to_datum and prove equality with the same
live digest. It arrives as one `quipu.asof` frame, or as
`quipu.asof.begin` / `.part` / `.commit` with the same digest chain as a
chunked transaction (the part and commit checks are now shared `Chunk`
methods).

All or nothing. These refuse with nothing written:
- a replica that is not empty;
- a manifest that is not quipu-asof-share/v1, or does not claim valid time
  and lossless bytes;
- a record that is not an assert, or is in a graph the manifest does not
  list;
- a graph whose records do not match the manifest's count and digest
  (SHA-256 over sorted, key-sorted compact JSON lines, `share_graph_digest`).
On success the facts and the cursor (store_id, log_epoch, watermark_tx) commit
in one savepoint, and slice 1 tails from /changes?since=watermark.

Seven tests:
- bootstrap then tail to equality with the source (valid time, bytes, named
  graph, a later retraction), and the cursor survives reopening;
- chunked equals single-frame, and nothing is visible before the commit;
- a dropped part resumes with nothing written, and a restarted cut converges;
- every untrustworthy share refused, with nothing written and no cursor;
- only an empty replica;
- a new epoch after bootstrap means rebootstrap;
- the digest ignores key and record order but not content.
With the count/digest and emptiness checks disabled, two tests fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(bindings): a replica through the binding facade and the browser Database

The shared applier joins the binding facade, so the in-page store and any
other binding reach it through one engine: Engine::replica_apply_frame,
replica_progress and replica_freshness, exported to JavaScript on the wasm
Database as replicaApplyFrame / replicaProgress / replicaFreshness (JSON text
in the contract Outcome and Freshness serialize to).

- The first replica call makes the store a read-only replica: local writes
  through invoke (knot, episode, set, retract, shapes load/remove) are
  refused before dispatch with read_only_replica / not_started. Reads work.
- The applier writes what the source already accepted, so local write
  validation is suspended for the applier's own writes and restored after.
  Re-judging source data against local shapes could refuse it and fork the
  replica silently. A test proves both halves: the adopted shape refuses the
  data when written locally, and the replica applies it from the feed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(knowledge): refresh the rest-api.md source witness

15c8128 changed docs/book/src/reference/rest-api.md without moving its
witness, so the repository-knowledge check refused the registry. Only the
source_sha256 moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(applier): a duplicate is checked against the retained transaction digest

Rule 6 of the shared applier API (docs/design/shared-applier-api.md): a
transaction at or below the cursor was skipped as a duplicate without
looking at it, so a redelivered tx 5 with changed records answered exactly
like an exact replay.

The applier now computes each transaction's content digest itself from the
decoded records (domain tag, store id, log epoch, source tx, then each
record's length-prefixed exact encoding, sorted), independent of framing,
and retains it in replica_applied_tx in the same savepoint as the records
and the cursor. A redelivery is Duplicate only when its digest equals the
retained one; a different digest is Stopped { DigestMismatch }, no
retained row is Stopped { DigestUnverifiable }, nothing is written, and the
stop is sticky. A frame from a new source still answers Rebootstrap.

replica_applied_tx is excluded from shares with the cursor it belongs to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(tests): the bindings timestamp constant is not shacl-only

record_source uses TS in every build, so the no-default-features test
build failed to compile (E0425).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): stream an as-of share into the replica instead of holding it

Slice 2 buffered every part of a chunked share in memory until the commit
(found on a 9.16M-record share). The begin frame now opens ONE local
savepoint; each part's facts are written as it arrives, in batches the
size of a part, and its canonical lines go to a temporary table so the
per-graph digests are computed in sorted order at the commit. Any failed
check rolls the savepoint back with nothing written, as before; the facts
and the cursor still commit together.

Same-connection reads now see a share's parts before its commit. The
replica has no cursor until Bootstrapped, and a reader must not use it
before then. A tail frame mid-share aborts the share, and an applier
opened on a store left mid-share rolls it back.

The digest chain moves to src/applier_chunk.rs and counts records without
keeping them when the caller streams. examples/applier_bootstrap_rss
measures peak RSS for a synthetic share at a chosen size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): a stop persists across reopening, and an unknown op is refused

Review findings on the shared applier:

- P1: the sticky Stopped outcome lived only in memory, so a fresh Applier
  on the same replica tailed on past a recorded digest mismatch. The stop
  is now kept in replica_stop (first stop wins) and restored by open; a
  replica whose history disagreed with its source stays stopped until it
  is rebuilt.
- P2: any op other than "retract" was applied as an assert. Only
  "assert" and "retract" are accepted, in the record conversion and the
  transaction digest alike; anything else refuses the transaction with
  nothing written.

replica_stop is excluded from shares with the cursor it belongs to.
Controls: a stop survives reopening (fails with open ignoring the stop),
and an unknown op is refused (fails with unknown ops read as assert).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(applier): slice 3a, stream sessions, signed coverage and the freshness challenge

The applier's side of the scoped replication stream (local-sync-mode.md
§5, shared-applier-api.md rules 1-8):

- A signed session frame binds a session id to the source, the pinned
  scope digest and its since; a since above the cursor resumes.
- frame_seq must be exactly last + 1, and each transaction newer than
  the last the session sent; anything else drops the session and resumes.
- Coverage is verified against a caller-pinned stream key (never learned
  from the stream), persisted beside the cursor, and never moves back.
- Freshness is bounded only by a coverage frame answering an outstanding
  challenge of the open session, consumed on use. A heartbeat, a replayed
  or unsolicited nonce, a new session, a reopen, end_session or a clock
  discontinuity never yields a bound.
- Control frames: revoked (persisted, sticky across reopening),
  epoch_end (rebootstrap).

replicate_wire.rs holds the signed bytes, shared with the producer that
follows: domain tag quipu-replicate/v1, then length-prefixed fields.

22 stream tests; each of 12 mutations of the checks fails at least one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): a store holding local facts never becomes a replica

Review 6387176506 (CHANGES): replica_apply_frame promoted any Engine to a
read-only replica on its first call, and the applier tailed the feed
into a store that already held local facts. The replica would then equal
no source while its cursor and digests claim it does.

- Applier::new_replica_refusal: a store may start a replica only if it
  already follows a source or holds no facts (shapes are not facts).
- The applier refuses a tail into such a store with nothing written.
- The binding checks before promoting: the frame is refused and the
  store stays an ordinary writable store.

Paired controls: with either gate removed, its test fails.
record_to_datum and value_of move unchanged to applier_record.rs, and the
gate lives in applier_start.rs, to keep applier.rs under the size gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(bindings): carry the stream freshness fields on a store that is not a replica

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* test(applier): the as-of share producer and the applier agree at the seam

Now that #805 is on main: a real asof_share::cut of a file-backed source
(300 items, unicode and escaped titles, bytes, a superseded value), framed
as the server sends it (one quipu.asof frame, or begin/parts/commit),
bootstraps an empty replica to the source's live facts; one tampered part
with the chain left intact is refused with nothing written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* feat(replicate): the scoped replication stream and its freshness challenge

The source side of local-sync-mode.md §5, slice 3b of the shared applier:

- POST /replicate/stream: one SSE session per follower over the scope an
  as-of share would serve (same rules via asof_share::scope_lists and
  resolve_graphs, now shared, and the same scope digest). Frames: a signed
  session frame, one tx frame per source transaction (all its in-scope
  graphs; begin/parts/commit when too large), signed coverage after each
  batch and as a 15 s heartbeat, and signed control (epoch_end,
  tx_too_large).
- Coverage W is read in ONE read transaction with the page: the head when
  the page reached it, else the page's last transaction.
- POST /replicate/challenge delivers a nonce to its session's producer,
  which answers it from a read made after receipt that reached the head.
- Signed with the server's existing ed25519 identity under the
  quipu-replicate/v1 domain; a server without a key serves no stream.
- Both routes are write-class (fail-closed like /share/asof) until
  per-graph read authorization has its ruling.

Supporting changes: Store::changes_in_graphs (a graph-set page),
PartBuilder::in_session (parts carry session and frame_seq inside the
digested text), and the applier maps tx_too_large to Rebootstrap.

Tests: 8 library tests drive sessions into the applier (scoped equality,
tail and filtered no-op, challenge only at the head, heartbeat, parts,
epoch end, refusals, scope digest); 6 server tests (stream-only follower
follows a new commit within 5 s, challenge through the route bounds
freshness, 404/400/503 refusals, disconnect frees the session); 5 producer
mutations each fail a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* style(replicate): borrow a transaction's records when framing them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* style(replicate): let-else in the producer loop; read SSE chunks as str directly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(applier): keep coverage-only anchors and end the session on chunk recovery

Review findings on #816 (non-author review):

P1. A scoped stream can verify coverage before its first matching
transaction. That coverage was persisted, but on reopen Stream::load
read it only for the source of applied progress, and there is none, so
the cursor was lost. Coverage now loads with the source it was verified
from. check_source anchors on applied progress or, before any, on that
covered source, so another source or epoch still asks for a rebootstrap.

P2. A chunk part or commit that broke returned Resume directly. That
cleared the chunk but kept the stream session and any earlier proof.
Every in-session Resume now goes through fault(), as a sequence gap
already did.

Controls, each failing with its repair reverted:
- coverage before any transaction anchors across reopen: W kept, a
  session at W opens, one past W resumes from W;
- coverage from one source refuses another, before and after reopen;
- a broken chunk ends the session and its proofs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* refactor(asof-share): move the shared scope checks into asof_share/scope.rs

The merge with #831 took src/asof_share.rs to 526 code lines (limit 500).
scope_lists, resolve_graphs and refuse_frozen, the checks the cut and the
replication stream share, now live in their own submodule and are
re-exported unchanged, so their callers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(replicate-stream): open in one snapshot; bound a transaction before it is built

Review findings on the stream producer:

- P1: Session::open checked the scope and frozen graphs, then read the
  log epoch, as separate reads. A freeze committed between them opened
  the session on the NEW epoch with a scope checked before the freeze,
  and the first poll signed coverage over zero records. Open now reads
  the epoch first and every check in one read snapshot. A change during
  the open leaves the session on the old epoch, and its first poll ends
  it with epoch_end.
- P2: only a single record larger than a frame was refused, so a
  transaction of many small records was built whole in memory. A
  bounded pre-pass (Store::tx_sizes_in_graphs) counts in-scope rows and
  value bytes in SQL. A transaction over Limits::tx_max_rows or
  tx_max_bytes ends the session with tx_too_large and the cursor
  unchanged. A page stops at page_max_rows, and its coverage reaches
  only what it sent.

Tests: a mid-open freeze through a second handle; a refused oversized
transaction with the follower cursor held; a page cut at the row budget
that still converges; an oversized transaction later in a page. Each
fails under its mutant (epoch-last order, no snapshot, no refusal, no
cut, coverage ignoring the cut).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(replicate-stream): bound a transaction's encoded size, not its value bytes

Review residual on the stream pre-pass: SUM(length(v)) counted only the
value blobs. A Ref's blob is its 9-byte id, and every record also carries
its entity, attribute and graph IRIs plus the transaction's timestamp,
actor and source, so a transaction with small values could still build
far more than the limit.

Store::tx_encoded_bound is now an upper bound on what changes_in_graphs
builds, read without building it:
- IRI lengths (entity, attribute, graph, Ref target) come from SQL
  length() on the terms table, so no term is loaded to measure it;
- the transaction's metadata counts once per record;
- every string counts at JSON's worst-case escaping (6 bytes per byte),
  plus a fixed per-record overhead;
- counting stops once the bound passes the cap.
Limits gains page_max_bytes. The pre-pass refuses on rows first, so a
transaction over the row limit is never read.

The limit tests move to replicate_source_limits_tests.rs. New tests
check the bound against the real encoded records for each shape
(escaped strings and retracts, a Ref to a long IRI, bytes, metadata
repeated across records) and cover long entity IRIs and the page byte
cut. Each fails under its mutant.

Known limit: a term held only in an attached layer is measured by
resolving it once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* fix(replicate-stream): count a string's bytes past an embedded NUL

SQLite's length() on a text value counts characters up to the first NUL,
so an IRI, actor, source or valid time holding a NUL undercounted the
transaction bound (706 against an encoded 6232 in the reviewer's
control). Every text length in tx_encoded_bound and term_len now reads
length(CAST(x AS BLOB)), which counts every byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant