Repository navigation
feat(applier): the shared applier, slice 1: a replica from the shipped feed - #796
Conversation
…ls its source A replica built only from /changes records diverged from its source in two ways (local-sync-mode.md F3, F4): - Records carried no valid time, so a fact valid since 2025 replayed as valid from its commit time. Records now carry valid_from and valid_to. A later retract closes the asserted row in place, so a row a later transaction closed reports valid_to null and the closing arrives as the retract's own record. Otherwise a served record would change after the fact, and a replica would assert an already-closed fact its retract could not find. - Byte values were reduced to bytes_len. They now also carry the body as base64. Both are additive fields. New replay tests rebuild a store from the feed and compare a canonical live-fact digest: a plain control, valid time (open and bounded), bytes, and an assert record that reads the same before and after a later retract. Removing each change turns its test red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
A functional-property supersede closed the prior value with an UPDATE and wrote no op=0 row, unlike an explicit retract. The change feed derives its records from rows, so it showed the new value and never the end of the old one, and a replica built from the feed kept both (local-sync-mode.md F1). The supersede now writes the same immutable op=0 row an explicit retract writes, at the superseding transaction. The resident read model reads op=1 rows and their retracted_tx stamp, so it is unaffected. The replay test asserts the replica holds exactly one current value; removing the new row turns it red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
… the stream A /changes/stream page larger than the 1 MiB frame budget sent quipu.error without advancing the cursor and ended the stream. A consumer reconnecting at that cursor hit the same transaction and the same dead end forever (local-sync-mode.md F6). Such a transaction now arrives as quipu.changes.begin, numbered quipu.changes.part frames and a quipu.changes.commit barrier (design §6). Only the barrier carries the SSE id, so Last-Event-ID never lands inside a transaction and a reconnect replays it from the first part. The barrier digest is SHA-256 over the parts' data exactly as sent; each part carries the hash of the parts before it. A single record larger than a frame is refused with quipu.changes.tx_too_large and no id, leaving the cursor unchanged. Tests stream a 2500-record transaction: every part fits, the digests chain, the reassembled records equal the page, the next transaction follows, and a reconnect replays from part 1. Disabling the split turns the test red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
chunk_transaction now returns transport-neutral frames (event name, optional cursor id, data string), and the SSE stream turns them into events only at the edge. Another transport can send the same frames and the same digests unchanged. A new test checks that the frames SSE sends equal the ones chunk_transaction returns, byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
A change page did not say which store produced it, and nothing marked a history boundary, so a follower could not tell a restored or cloned primary, or one that had removed served records, from the one it was following (local-sync-mode.md §5, F5). Every page now carries source.store_id and source.log_epoch. The epoch lives in a new one-row log_epoch table, minted when the store is created (or first opened by this version) and minted again by Store::mint_log_epoch for an operation that removes served history. Share reconstruction excludes the table, so a restored copy mints its own epoch and its followers re-bootstrap. The explicit restore/adopt ownership step is deferred to aegis-s24.2, per the xdvg.1 ruling. Tests cover the page source, an epoch stable across reopen, a new epoch after minting, and a store that predates the table getting one on first open. Removing the field turns both tests red. Refs aegis-xdvg.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
An oversized transaction's begin and commit frames carried no source,
so a follower could not epoch-check a chunked transaction the way it
checks a whole page, although a changed epoch must mean re-bootstrap,
never splice. Both frames now carry the page's {store_id, log_epoch}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
…d feed One component turns the committed feed into a read-only replica, so the HA follower, a scoped follower and the in-page browser store never write their own projection logic (docs/design/shared-applier-api.md). This slice applies the frames that ship today, given as (event, data) text exactly as SSE or a WebSocket delivers them: /changes pages and the begin/part/commit frames of an oversized transaction. It does no I/O, so it builds for wasm32 too. - The first frame binds the replica to its source (store_id, log_epoch); a frame from another source returns Rebootstrap, never a splice. - Each source transaction applies in one savepoint with the persisted cursor; a chunked one is buffered, its running digest checked part by part, and nothing is visible until the commit verifies (or the caller is told to resume from the cursor). - A record the replica cannot reproduce exactly (a byte value sent as its length only, an unknown value shape) refuses the whole transaction. - Freshness is (last applied tx, time since the last frame), never bounded. The feed-replay conformance tests now use the same record->datum conversion (record_to_datum) and share live_digest, so there is one mapping. The cursor table is excluded from shares, like the log epoch it is bound to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…tract The replica equality oracle now sits beside the code it judges, so prove it sees valid time, byte content and graph (review ask). Outcome and Freshness serialize tagged by kind, the contract a browser bridge returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
A /changes page covers every transaction in its scope after the cursor it was asked for, but did not say which cursor that was. Transaction ids are global and a per-graph page skips other graphs' ids, so the applier could not tell a lost page from an ordinary skip and would splice past it. - Every page now carries `since`, and the chunked `begin` and `commit` frames repeat it. - The applier resumes (nothing applied) when a page or a chunked transaction answers a cursor above its own. A new replica starts at 0 until a bootstrap exists. A page that names no `since` is refused. - A page arriving while a chunked transaction is open discards the chunk and resumes. Before, the page could apply the chunk's transaction, move the cursor past it, and the commit was then dropped as a duplicate. Seven tests: - a dropped page; - a first page not at 0; - a page without `since`; - a page inside an open chunk; - an interrupted chunk restarted from a new begin; - a re-opened applier holding no staged parts; - a chunk that skips a transaction. With the two checks disabled, four of them fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
New commit 569130d: refuse a gap; a page names the cursor it answers. Review scope is now the last 3 commits (ff89873, 18781af, 569130d). The applier could not detect a lost page. Transaction ids are global, and a per-graph page skips other graphs' ids, so "tx > cursor" cannot tell a gap from a normal skip. The fix:
7 new tests (18 applier tests in total). With the two checks disabled, 4 of them fail. Full default suite: 3153 passed, 0 failed. clippy on 3 feature sets, fmt, the size gate and the wasm32 Compatibility: a follower on this applier refuses pages from a server that predates |
…e applier The applier commit added `pub mod applier` to src/lib.rs, so its repository-knowledge witness hash went stale (the pendant source controls job). The record's summary still holds; only the hash changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
15c8128 changed docs/book/src/reference/rest-api.md without moving its witness, so the repository-knowledge check refused the registry. Only the source_sha256 moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…n digest
Rule 6 of the shared applier API (docs/design/shared-applier-api.md): a
transaction at or below the cursor was skipped as a duplicate without
looking at it, so a redelivered tx 5 with changed records answered exactly
like an exact replay.
The applier now computes each transaction's content digest itself from the
decoded records (domain tag, store id, log epoch, source tx, then each
record's length-prefixed exact encoding, sorted), independent of framing,
and retains it in replica_applied_tx in the same savepoint as the records
and the cursor. A redelivery is Duplicate only when its digest equals the
retained one; a different digest is Stopped { DigestMismatch }, no
retained row is Stopped { DigestUnverifiable }, nothing is written, and the
stop is sticky. A frame from a new source still answers Rebootstrap.
replica_applied_tx is excluded from shares with the cursor it belongs to.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
Follow-up: rule 6 of #793 (merged as d28140d), now at 3442c9d. Before, a transaction at or below the cursor was skipped as Now:
New controls in
Also 0572529: the Local checks with a private target:
Still a draft, stacked on the F-series. #798 needs a restack onto 3442c9d. |
…0261024 # Conflicts: # docs/knowledge/repository-knowledge.json # src/server/feed_stream_tests.rs # src/store/changes_replay_tests.rs
…0261024 # Conflicts: # src/server/feed_stream.rs # src/share_completeness.rs
|
CHANGES at d7c590e R1 P1 — terminal stop is lost when reopening the applier. Applier::open unconditionally initializes stopped=None; only cursor and applied digests are retained. Independent real-library control: a digest mismatch stops the instance and blocks a later transaction, but reconstructing Applier over the same unchanged replica then applies that later transaction. This is not a rebuild and violates the terminal/sticky contract. Retain the stop across open/restart, bound to the replica/source, and clear it only through the explicit recovery/rebuild contract. Add a stopped -> reopen -> still stopped control and a normal reopen positive control. R2 P2 — unknown operation is silently normalized to assertion. record_to_datum treats every op other than retract as assert, and digest encoding makes the same normalization. An independent unknown-operation frame is reported Applied and advances progress. Refuse unrecognized operation discriminants before any source-transaction write or cursor advance; use the same validated interpretation for digest and application. Retain assert/retract positive controls. Full eleven-path source audit, including since/gap handling, retained digest, share exclusions and feed changes. Exact public library private fixture: normal replay/reopen positive PASS; both above safety assertions FAIL with the observed Applied outcomes. HOME157 watched paths+5 UID resources CLEAN. Initial private Cargo package-name mismatch ran no tests and is preserved; corrected exact-head fixture compiled and ran. No production replica, replay, model, source edit or merge. Native973b preparation only/contentc0816bb0; no approval recorded. Tim retains sole SAME repair-delta review after author repair, fresh green and native admission; stack order796 ->798 ->803 stays. |
…refused Review findings on the shared applier: - P1: the sticky Stopped outcome lived only in memory, so a fresh Applier on the same replica tailed on past a recorded digest mismatch. The stop is now kept in replica_stop (first stop wins) and restored by open; a replica whose history disagreed with its source stays stopped until it is rebuilt. - P2: any op other than "retract" was applied as an assert. Only "assert" and "retract" are accepted, in the record conversion and the transaction digest alike; anything else refuses the transaction with nothing written. replica_stop is excluded from shares with the cursor it belongs to. Controls: a stop survives reopening (fails with open ignoring the stop), and an unknown op is refused (fails with unknown ops read as assert). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
|
Repair for review 6386381515, new head 258cba7. One commit on the same branch.
Paired controls, each shown to FAIL without its fix:
Local, private target:
#798 and #803 stay behind #796, and I will refresh them onto this head. |
|
APPROVE at 258cba7 SAME repair-delta review against my original full eleven-path source review/CHANGES atd7c590e5. Exactly four paths changed: applier open/record decoding, digest/stop persistence, two author controls, and local replica_stop share exclusion. Other original proposal paths are unchanged. R1 P1 resolved: the first terminal stop is retained in replica_stop, restored by open, and never cleared by the applier. This state is excluded alongside local cursor/digest state from share export. R2 P2 resolved: one assert/retract validator is used by application and digest encoding; unknown operations refuse before transaction/cursor commit. Five independent exact-public-library controls PASS: normal reopen/exact replay; original stopped-applier reopen falsifier; full disk Store close/reopen still stopped; original unknown-operation falsifier; valid retract positive. HOME157 paths+5 UID resources CLEAN. First local retry ran no tests due guarded-HOME rustup configuration, retained; corrected previously verified explicit toolchain/cache setup ran all five. No source edits or production replica/replay. Current native T2 f2ac prepared after exact-head complete green; accepted38c59c12 binds DIFF/normal merge --diff-bound only. Native base-green proof is6666 ancestor;8255 tip CI is still pending, not claimed green. Clean merge tree and whole inverse restoring entire8255 prepared. Previous old-head CHANGES remain historical; this exact repair resolves both. No wasm/browser/adoption or runtime acceptance inferred. |
Three conflicts, resolved once: - src/lib.rs: keep both `pub mod applier` (#796) and `pub mod asof_share`. - src/store/changes_replay_tests.rs: main's version (#796 moved live_digest and the record -> datum mapping into the applier). The as-of tests' replay_onto/apply_tx now sit on that shared mapping, and main's replay() is replay_onto from 0; datum is pub(super) again. - docs/knowledge/repository-knowledge.json: re-hash the src/lib.rs witness. changes_asof_tests imports live_digest from crate::applier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
Stacked on the change-feed series (#780 → F1 → F6 → F5 #789, base
614ce73d). Review scope: the last commit only (feat(applier)). It stays a draft until its base merges.Slice 1 of the shared applier (
docs/design/shared-applier-api.md, #793): ONE component that turns the committed feed into a read-only replica, so the HA follower, a scoped follower and the in-page browser store share one projection path.(event, data)text, exactly as SSE or a WebSocket delivers them:/changespages and the chunkedbegin/part/commitframes.wasm32-unknown-unknown.(store_id, log_epoch). A frame from another source →Rebootstrap, never a splice.Resume { from: cursor }.(last applied tx, time since the last frame)withbounded: false.record_to_datumand sharelive_digest, so there is one conversion.replica_progressis excluded from shares, likelog_epoch.Tests (9 new): equal replica from the feed; idempotent replay; epoch change → Rebootstrap with nothing spliced; page without a source refused; a length-only byte value refused with nothing written; a chunked transaction invisible until a verified commit; an out-of-order part → Resume; a chunked commit without a source refused; cursor survives reopen, freshness never bounded.
Local: full
cargo test --release --no-fail-fastpassed every group except share-completeness, which this commit then fixes (26/0 re-run, incl. the changes conformance tests); clippy default andshacl,onnx,serverclean;cargo check --target wasm32-unknown-unknown --no-default-features --libclean for this module.🤖 Generated with Claude Code
https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb