Skip to content

feat(applier): the shared applier, slice 1: a replica from the shipped feed - #796

Merged
scbrown merged 15 commits into
mainfrom
wu/s24-2-applier-s1-20261024
Oct 24, 2026
Merged

scbrown merged 15 commits into
mainfrom
wu/s24-2-applier-s1-20261024

Conversation

@scbrown

@scbrown scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

Stacked on the change-feed series (#780 → F1 → F6 → F5 #789, base 614ce73d). Review scope: the last commit only (feat(applier)). It stays a draft until its base merges.

Slice 1 of the shared applier (docs/design/shared-applier-api.md, #793): ONE component that turns the committed feed into a read-only replica, so the HA follower, a scoped follower and the in-page browser store share one projection path.

  • It applies today's frames as (event, data) text, exactly as SSE or a WebSocket delivers them: /changes pages and the chunked begin/part/commit frames.
  • It does no I/O, so it builds for wasm32-unknown-unknown.
  • The first frame binds the replica to its source (store_id, log_epoch). A frame from another source → Rebootstrap, never a splice.
  • Each source transaction applies in one savepoint together with the persisted cursor.
  • A chunked transaction is buffered and its running digest checked part by part. Nothing is visible until the commit verifies; a broken sequence returns Resume { from: cursor }.
  • A record the replica cannot reproduce exactly (a byte value sent as its length only, an unknown value shape) refuses the whole transaction with nothing written.
  • Freshness is (last applied tx, time since the last frame) with bounded: false.
  • The feed-replay conformance tests now use the same record_to_datum and share live_digest, so there is one conversion.
  • replica_progress is excluded from shares, like log_epoch.

Tests (9 new): equal replica from the feed; idempotent replay; epoch change → Rebootstrap with nothing spliced; page without a source refused; a length-only byte value refused with nothing written; a chunked transaction invisible until a verified commit; an out-of-order part → Resume; a chunked commit without a source refused; cursor survives reopen, freshness never bounded.

Local: full cargo test --release --no-fail-fast passed every group except share-completeness, which this commit then fixes (26/0 re-run, incl. the changes conformance tests); clippy default and shacl,onnx,server clean; cargo check --target wasm32-unknown-unknown --no-default-features --lib clean for this module.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

scbrown and others added 8 commits October 23, 2026 21:42
…ls its source

A replica built only from /changes records diverged from its source in two
ways (local-sync-mode.md F3, F4):

- Records carried no valid time, so a fact valid since 2025 replayed as valid
  from its commit time. Records now carry valid_from and valid_to. A later
  retract closes the asserted row in place, so a row a later transaction
  closed reports valid_to null and the closing arrives as the retract's own
  record. Otherwise a served record would change after the fact, and a
  replica would assert an already-closed fact its retract could not find.
- Byte values were reduced to bytes_len. They now also carry the body as
  base64.

Both are additive fields. New replay tests rebuild a store from the feed and
compare a canonical live-fact digest: a plain control, valid time (open and
bounded), bytes, and an assert record that reads the same before and after a
later retract. Removing each change turns its test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
A functional-property supersede closed the prior value with an UPDATE and
wrote no op=0 row, unlike an explicit retract. The change feed derives its
records from rows, so it showed the new value and never the end of the old
one, and a replica built from the feed kept both (local-sync-mode.md F1).

The supersede now writes the same immutable op=0 row an explicit retract
writes, at the superseding transaction. The resident read model reads op=1
rows and their retracted_tx stamp, so it is unaffected.

The replay test asserts the replica holds exactly one current value;
removing the new row turns it red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
… the stream

A /changes/stream page larger than the 1 MiB frame budget sent quipu.error
without advancing the cursor and ended the stream. A consumer reconnecting at
that cursor hit the same transaction and the same dead end forever
(local-sync-mode.md F6).

Such a transaction now arrives as quipu.changes.begin, numbered
quipu.changes.part frames and a quipu.changes.commit barrier (design §6).
Only the barrier carries the SSE id, so Last-Event-ID never lands inside a
transaction and a reconnect replays it from the first part. The barrier digest
is SHA-256 over the parts' data exactly as sent; each part carries the hash of
the parts before it. A single record larger than a frame is refused with
quipu.changes.tx_too_large and no id, leaving the cursor unchanged.

Tests stream a 2500-record transaction: every part fits, the digests chain,
the reassembled records equal the page, the next transaction follows, and a
reconnect replays from part 1. Disabling the split turns the test red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
chunk_transaction now returns transport-neutral frames (event name,
optional cursor id, data string), and the SSE stream turns them into
events only at the edge. Another transport can send the same frames and
the same digests unchanged. A new test checks that the frames SSE sends
equal the ones chunk_transaction returns, byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
A change page did not say which store produced it, and nothing marked a
history boundary, so a follower could not tell a restored or cloned primary,
or one that had removed served records, from the one it was following
(local-sync-mode.md §5, F5).

Every page now carries source.store_id and source.log_epoch. The epoch lives
in a new one-row log_epoch table, minted when the store is created (or first
opened by this version) and minted again by Store::mint_log_epoch for an
operation that removes served history. Share reconstruction excludes the
table, so a restored copy mints its own epoch and its followers re-bootstrap.
The explicit restore/adopt ownership step is deferred to aegis-s24.2, per the
xdvg.1 ruling.

Tests cover the page source, an epoch stable across reopen, a new epoch after
minting, and a store that predates the table getting one on first open.
Removing the field turns both tests red.

Refs aegis-xdvg.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185RxTpLsPCQFB9ni72Nzac
An oversized transaction's begin and commit frames carried no source,
so a follower could not epoch-check a chunked transaction the way it
checks a whole page, although a changed epoch must mean re-bootstrap,
never splice. Both frames now carry the page's {store_id, log_epoch}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
…d feed

One component turns the committed feed into a read-only replica, so the HA
follower, a scoped follower and the in-page browser store never write their
own projection logic (docs/design/shared-applier-api.md). This slice applies
the frames that ship today, given as (event, data) text exactly as SSE or a
WebSocket delivers them: /changes pages and the begin/part/commit frames of
an oversized transaction. It does no I/O, so it builds for wasm32 too.

- The first frame binds the replica to its source (store_id, log_epoch); a
  frame from another source returns Rebootstrap, never a splice.
- Each source transaction applies in one savepoint with the persisted
  cursor; a chunked one is buffered, its running digest checked part by
  part, and nothing is visible until the commit verifies (or the caller is
  told to resume from the cursor).
- A record the replica cannot reproduce exactly (a byte value sent as its
  length only, an unknown value shape) refuses the whole transaction.
- Freshness is (last applied tx, time since the last frame), never bounded.

The feed-replay conformance tests now use the same record->datum
conversion (record_to_datum) and share live_digest, so there is one mapping.
The cursor table is excluded from shares, like the log epoch it is bound to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…tract

The replica equality oracle now sits beside the code it judges, so prove
it sees valid time, byte content and graph (review ask). Outcome and
Freshness serialize tagged by kind, the contract a browser bridge returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
A /changes page covers every transaction in its scope after the cursor it
was asked for, but did not say which cursor that was. Transaction ids are
global and a per-graph page skips other graphs' ids, so the applier could
not tell a lost page from an ordinary skip and would splice past it.

- Every page now carries `since`, and the chunked `begin` and `commit`
  frames repeat it.
- The applier resumes (nothing applied) when a page or a chunked
  transaction answers a cursor above its own. A new replica starts at 0
  until a bootstrap exists. A page that names no `since` is refused.
- A page arriving while a chunked transaction is open discards the chunk
  and resumes. Before, the page could apply the chunk's transaction, move
  the cursor past it, and the commit was then dropped as a duplicate.

Seven tests:
- a dropped page;
- a first page not at 0;
- a page without `since`;
- a page inside an open chunk;
- an interrupted chunk restarted from a new begin;
- a re-opened applier holding no staged parts;
- a chunk that skips a transaction.
With the two checks disabled, four of them fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

New commit 569130d: refuse a gap; a page names the cursor it answers. Review scope is now the last 3 commits (ff89873, 18781af, 569130d).

The applier could not detect a lost page. Transaction ids are global, and a per-graph page skips other graphs' ids, so "tx > cursor" cannot tell a gap from a normal skip. The fix:

  • /changes pages carry since, the cursor they answer; the chunked begin/commit frames repeat it.
  • The applier answers resume when since is above its cursor.
  • A page arriving inside an open chunk discards the chunk, which closes a silent-loss path: the page could move the cursor past the chunk's transaction, and the commit was then dropped as a duplicate.

7 new tests (18 applier tests in total). With the two checks disabled, 4 of them fail. Full default suite: 3153 passed, 0 failed. clippy on 3 feature sets, fmt, the size gate and the wasm32 cargo check are clean.

Compatibility: a follower on this applier refuses pages from a server that predates since, so the consumer needs this server change deployed first. Both are in this PR.

…e applier

The applier commit added `pub mod applier` to src/lib.rs, so its
repository-knowledge witness hash went stale (the pendant source controls
job). The record's summary still holds; only the hash changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
scbrown and others added 2 commits October 24, 2026 07:54
15c8128 changed docs/book/src/reference/rest-api.md without moving its
witness, so the repository-knowledge check refused the registry. Only the
source_sha256 moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…n digest

Rule 6 of the shared applier API (docs/design/shared-applier-api.md): a
transaction at or below the cursor was skipped as a duplicate without
looking at it, so a redelivered tx 5 with changed records answered exactly
like an exact replay.

The applier now computes each transaction's content digest itself from the
decoded records (domain tag, store id, log epoch, source tx, then each
record's length-prefixed exact encoding, sorted), independent of framing,
and retains it in replica_applied_tx in the same savepoint as the records
and the cursor. A redelivery is Duplicate only when its digest equals the
retained one; a different digest is Stopped { DigestMismatch }, no
retained row is Stopped { DigestUnverifiable }, nothing is written, and the
stop is sticky. A frame from a new source still answers Rebootstrap.

replica_applied_tx is excluded from shares with the cursor it belongs to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: rule 6 of #793 (merged as d28140d), now at 3442c9d.

Before, a transaction at or below the cursor was skipped as Duplicate without being inspected. An exact replay of tx 5 and a tx 5 with changed records therefore answered alike.

Now:

  • The applier computes tx_digest itself from the decoded records, as the doc's "Transaction digest" defines it. Records are sorted, so the digest does not depend on framing.
  • The digest is retained in replica_applied_tx, in the same savepoint as the records and the cursor.
  • On redelivery:
    • equal digest → Duplicate;
    • different digest → Stopped { DigestMismatch { source_tx, applied, received } };
    • no retained row → Stopped { DigestUnverifiable }.
  • Nothing is written on a stop, and the stop is sticky. A frame from a new source still answers Rebootstrap (rule 1 before rule 4).
  • replica_applied_tx is excluded from shares, together with replica_progress.
  • The code lives in a new src/applier_digest.rs because applier.rs sits at the size gate (494 code lines).

New controls in src/applier_digest_tests.rs:

  • an exact redelivery is Duplicate, as a page and as a chunk;
  • a changed record stops the replica, writes nothing, and the stop stays sticky;
  • a missing retained row is unverifiable;
  • a new source after a stop answers Rebootstrap;
  • retained digests survive a reopen;
  • the digest ignores record order but changes with tx and epoch;
  • serialization.

Also 0572529: the rest-api.md witness was stale since 15c8128.

Local checks with a private target:

  • full default suite: 3160 passed, 0 failed;
  • clippy clean on no-default, default and shacl;
  • fmt, the size gate and the knowledge test are OK.

Still a draft, stacked on the F-series. #798 needs a restack onto 3442c9d.

…0261024

# Conflicts:
#	docs/knowledge/repository-knowledge.json
#	src/server/feed_stream_tests.rs
#	src/store/changes_replay_tests.rs
…0261024

# Conflicts:
#	src/server/feed_stream.rs
#	src/share_completeness.rs
@scbrown
scbrown marked this pull request as ready for review October 24, 2026 15:30
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGES at d7c590e

R1 P1 — terminal stop is lost when reopening the applier. Applier::open unconditionally initializes stopped=None; only cursor and applied digests are retained. Independent real-library control: a digest mismatch stops the instance and blocks a later transaction, but reconstructing Applier over the same unchanged replica then applies that later transaction. This is not a rebuild and violates the terminal/sticky contract. Retain the stop across open/restart, bound to the replica/source, and clear it only through the explicit recovery/rebuild contract. Add a stopped -> reopen -> still stopped control and a normal reopen positive control.

R2 P2 — unknown operation is silently normalized to assertion. record_to_datum treats every op other than retract as assert, and digest encoding makes the same normalization. An independent unknown-operation frame is reported Applied and advances progress. Refuse unrecognized operation discriminants before any source-transaction write or cursor advance; use the same validated interpretation for digest and application. Retain assert/retract positive controls.

Full eleven-path source audit, including since/gap handling, retained digest, share exclusions and feed changes. Exact public library private fixture: normal replay/reopen positive PASS; both above safety assertions FAIL with the observed Applied outcomes. HOME157 watched paths+5 UID resources CLEAN. Initial private Cargo package-name mismatch ran no tests and is preserved; corrected exact-head fixture compiled and ran. No production replica, replay, model, source edit or merge. Native973b preparation only/contentc0816bb0; no approval recorded. Tim retains sole SAME repair-delta review after author repair, fresh green and native admission; stack order796 ->798 ->803 stays.

…refused

Review findings on the shared applier:

- P1: the sticky Stopped outcome lived only in memory, so a fresh Applier
  on the same replica tailed on past a recorded digest mismatch. The stop
  is now kept in replica_stop (first stop wins) and restored by open; a
  replica whose history disagreed with its source stays stopped until it
  is rebuilt.
- P2: any op other than "retract" was applied as an assert. Only
  "assert" and "retract" are accepted, in the record conversion and the
  transaction digest alike; anything else refuses the transaction with
  nothing written.

replica_stop is excluded from shares with the cursor it belongs to.
Controls: a stop survives reopening (fails with open ignoring the stop),
and an unknown op is refused (fails with unknown ops read as assert).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Repair for review 6386381515, new head 258cba7. One commit on the same branch.

  • P1, persistent stop: a Stopped outcome is now persisted in replica_stop (one row; the first stop wins) and restored by Applier::open. A replica whose history disagreed with its source stays stopped across reopening, until it is rebuilt. A new source still answers Rebootstrap (rule 1 before rule 4).
  • P2, unknown op: only "assert" and "retract" are accepted, through one helper, op_is_retract, used by both the record conversion and the transaction digest. Any other op refuses the transaction with nothing written; it is never applied as an assert.
  • replica_stop is classified Excluded in share_completeness, next to replica_progress and replica_applied_tx.

Paired controls, each shown to FAIL without its fix:

  • a_stop_survives_reopening_so_a_diverged_replica_never_resumes: FAILS when open ignores the persisted stop.
  • an_unknown_op_is_refused_never_read_as_an_assert: FAILS when unknown ops map to assert.

Local, private target:

  • full default suite 3182 passed, 0 failed;
  • server binary 236 passed, 0 failed;
  • clippy clean on no-default, default and shacl;
  • fmt, size gate (applier.rs 498 code lines) and knowledge test OK.

#798 and #803 stay behind #796, and I will refresh them onto this head.

@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

APPROVE at 258cba7

SAME repair-delta review against my original full eleven-path source review/CHANGES atd7c590e5. Exactly four paths changed: applier open/record decoding, digest/stop persistence, two author controls, and local replica_stop share exclusion. Other original proposal paths are unchanged.

R1 P1 resolved: the first terminal stop is retained in replica_stop, restored by open, and never cleared by the applier. This state is excluded alongside local cursor/digest state from share export. R2 P2 resolved: one assert/retract validator is used by application and digest encoding; unknown operations refuse before transaction/cursor commit.

Five independent exact-public-library controls PASS: normal reopen/exact replay; original stopped-applier reopen falsifier; full disk Store close/reopen still stopped; original unknown-operation falsifier; valid retract positive. HOME157 paths+5 UID resources CLEAN. First local retry ran no tests due guarded-HOME rustup configuration, retained; corrected previously verified explicit toolchain/cache setup ran all five. No source edits or production replica/replay.

Current native T2 f2ac prepared after exact-head complete green; accepted38c59c12 binds DIFF/normal merge --diff-bound only. Native base-green proof is6666 ancestor;8255 tip CI is still pending, not claimed green. Clean merge tree and whole inverse restoring entire8255 prepared. Previous old-head CHANGES remain historical; this exact repair resolves both. No wasm/browser/adoption or runtime acceptance inferred.

@scbrown
scbrown merged commit 4b2372a into main Oct 24, 2026
26 checks passed
@scbrown
scbrown deleted the wu/s24-2-applier-s1-20261024 branch October 24, 2026 16:55
scbrown added a commit that referenced this pull request Oct 24, 2026
Three conflicts, resolved once:
- src/lib.rs: keep both `pub mod applier` (#796) and `pub mod asof_share`.
- src/store/changes_replay_tests.rs: main's version (#796 moved
  live_digest and the record -> datum mapping into the applier). The
  as-of tests' replay_onto/apply_tx now sit on that shared mapping, and
  main's replay() is replay_onto from 0; datum is pub(super) again.
- docs/knowledge/repository-knowledge.json: re-hash the src/lib.rs witness.
changes_asof_tests imports live_digest from crate::applier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant