[scbrown]

Slice 2 of the shared applier: bootstrap from an as-of share (docs/design/as-of-share.md, #799). Stacked on #796 (S1, including its witness refresh c34da2b). The review scope is the last commit, b81ab48. DRAFT until the F-series and #796 land.

The frame contract (for the s24.1a producer)

• One frame: event quipu.asof, data {"manifest": {...}, "records": [...]}.
• Past the frame budget: quipu.asof.begin {manifest, parts, records, digest}; then quipu.asof.part {source_tx: watermark_tx, index, digest_before, records}; then quipu.asof.commit {source_tx, digest}. This is the same chain as F6's chunked transaction: digest is SHA-256 over the parts' data exactly as sent, and digest_before is the hash of the parts before it.
• Manifest:
  • schema: "quipu-asof-share/v1";
  • source {store_id, log_epoch};
  • watermark_tx;
  • completeness {valid_time: true, bytes: "lossless", graphs: {<graph IRI or "ROOT">: {records, digest, frozen}}}.
• Each graph's digest is share_graph_digest: SHA-256 over that graph's records as compact JSON with keys sorted at every level, lines sorted, joined by \n. It may carry a sha256: prefix.
• Records are the /changes record form, op: "assert" only, with typed values, base64 bytes and valid_from/valid_to.

Behaviour

All or nothing, with nothing written on any refusal:

• the replica is not empty;
• the schema is wrong, or the manifest does not claim valid time and lossless bytes;
• a record is not an assert, or sits in a graph outside the manifest;
• a graph's count or digest does not match;
• a byte value is sent as its length only.

On success the facts and the cursor (store_id, log_epoch, watermark_tx) commit in one savepoint, with outcome {"kind": "bootstrapped", "source_tx": N}. S1 then tails from /changes?since=watermark_tx.

Tests (7 new)

• bootstrap then tail to equality with the source (valid time, bytes, named graph, a later retraction), and the cursor survives reopening;
• chunked equals single-frame, and nothing is visible before the commit;
• a dropped part resumes with nothing written, and a restarted cut converges;
• every untrustworthy share is refused (8 cases) with nothing written and no cursor;
• only an empty replica is bootstrapped;
• a new epoch after bootstrap means rebootstrap;
• the digest ignores key and record order, not content.

With the count/digest and emptiness checks disabled, 2 of them fail.

Local checks:

• full default suite: 3160 passed, 0 failed;
• clippy on --no-default-features, shacl and shacl,onnx,server: clean;
• fmt and the size gate: clean;
• wasm32 cargo check: clean.

Known inherited red: the docs/book/src/reference/rest-api.md knowledge witness, which went stale in F6 (#788) and is fixed there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

[scbrown]

Fix for the buffering defect, now at 4cf73df. It is restacked onto #796 at 3442c9d.

BEFORE: the applier held every part of a chunked share in memory until the commit. Found on a 9.16M-record share.

AFTER:

• quipu.asof.begin (or the single frame) opens ONE local savepoint.
• Each part's facts are written as the part arrives, one batch per part.
• Canonical lines spill to a TEMP table. At the commit, each graph's share_graph_digest is computed from them with ORDER BY line (BINARY order is the same bytewise order as sorting the strings). The producer's digest contract is unchanged.
• Any failed check rolls back with nothing written. The facts and the cursor commit together.

Behaviour changes, stated for review:

• A same-connection read can see a share's parts before its commit. The replica has no cursor until Bootstrapped (freshness().source_tx == None), and a reader must not use it before then.
• A tail frame mid-share aborts the share.
• Applier::open on a store left mid-share rolls the open share back.
• A part pushing a graph past its manifest count refuses at once.

New control: a_share_is_streamed_never_held_and_a_reopen_mid_share_discards_it. The applier holds 0 records while parts are already in the store, and a reopen leaves no facts.

PEAK-RSS CONTROL (synthetic one-graph share, 1000 records/part, file-backed replica, release build; examples/applier_bootstrap_rss):

records	outcome	peak RSS	anon RSS	time
100,000	Bootstrapped	30.3 MiB	23.5 MiB	7.9 s
1,000,000	Bootstrapped	124.8 MiB	105.8 MiB	60.0 s
3,000,000	Bootstrapped	125.0 MiB	106.8 MiB	193.6 s

The peak is flat from 1M to 3M (+0.2 MiB for 3x the records), so it no longer grows with the share. A measured run on the same 9.16M-record fixture as the producer is still OPEN: it needs that fixture and resource admission on the host. It is not claimed here.

Local checks with a private target:

• full default suite: 3168 passed, 0 failed;
• clippy clean on no-default, default and shacl;
• fmt, the size gate (applier.rs at 479 code lines) and the knowledge test are OK.

Still a draft on the F-series.

[scbrown]

Restacked onto main d68a89a (#798 landed); new head 4a6864a.

Review scope (own commits beyond main), unchanged from before plus one test-only commit:

• the S2 bootstrap commits;
• 4a6864a, test only: the feat(asof-share): cut a replica bootstrap at one transaction #805 x feat(applier): slice 2, a replica bootstrapped from an as-of share #803 seam (src/applier_seam_tests.rs). It was proven locally earlier and is added now that feat(asof-share): cut a replica bootstrap at one transaction #805 is on main.
• The merge beeb6dd resolves one use line. Main's src/applier*.rs equal the feat(bindings): a replica through the binding facade and the browser Database #798 branch byte for byte.

Seam test:

• A real asof_share::cut of a file-backed source (300 items; unicode and escaped titles; bytes; a superseded value) is framed exactly as src/server/asof.rs sends it.
• Single frame and chunked: both bootstrap an empty replica, and live_digest(replica) == live_digest(source).
• One tampered part, with the chain left intact: not bootstrapped, replica empty.

Local results:

• default suite: 3222 passed, 0 failed;
• --features shacl,onnx,server --bin quipu-server: 240 passed, 0 failed;
• clippy on default, no-default and shacl: nothing new;
• fmt, the size gate and the knowledge witnesses: clean;
• wasm32 check: clean.

MERGEABLE against main. Ready for one non-author codex review after fresh green. #816 restacks onto this head next.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

[scbrown]

CHANGES at 4a6864a

R1 P2 — bootstrap leaves its savepoint and partial facts live after a progress-write error. In src/applier_bootstrap.rs:238, finish_share takes self.share before INSERT INTO replica_progress. If that INSERT fails, ? returns the SQLite error with self.share already None, so neither abort_share nor the next non-share frame rolls back the bootstrap. The same Store can read the imported facts despite apply_frame's documented error contract that the frame was not applied. Ordinary retries also fail the empty-store gate until the caller explicitly reopens the applier.

Independent bounded file-backed counterexample: install a private BEFORE INSERT replica_progress trigger raising ABORT, then apply a valid two-fact single bootstrap. apply_frame returns Err and progress is None, but the same Store reads two facts. A second SQLite connection reads zero: this is an uncommitted transaction leak, NOT a durable partial commit. Dropping/reopening Applier clears the facts, and removing the trigger permits the exact share to bootstrap successfully. Require immediate cleanup on write/verification/commit errors, retain rollback ownership until commit succeeds, and pair error cleanup with a same-instance retry positive. Do not change source data or silently delete prior data.

ONE non-author audit of all six proposal paths, after exact-head complete green and native T2 b873 PREPARED. Six private test groups: 5 PASS / 1 FAIL (the error-cleanup counterexample); positive groups cover exact integer/byte/interval values, independent-reader commit atomicity, six damaged chunk variants, seven malformed manifest/record variants, and interruption/reopen recovery. HOME157 watched paths +5 UID resources CLEAN, inner test exit101 retained. No source edits, production import, approval, merge or runtime acceptance. Tim retains SAME repair-delta custody after author repair, fresh complete green and current native admission.

[scbrown]

Repair for review 6388941162 (CHANGES P2), new head 988e647. One commit.

Finding. A failed progress write after share.take() returned Err with the bootstrap savepoint still open, so the share's facts were visible on the same connection until a reopen.

Fix. bootstrap_frame now wraps all share handling once. Any store error while a share is open discards the share and rolls back its savepoint before the error is returned. That covers the verify, write and finish paths, not only the progress write.

Paired control. a_store_error_mid_share_leaves_no_fact_visible_even_on_the_same_connection drops replica_progress, so the final write fails, then asserts three things on the SAME connection: the call is Err, is_autocommit holds, and live_digest is empty. With the rollback removed it FAILS ("the share's savepoint is still open"); restored, it passes.

Local results:

• default suite: 3223 passed, 0 failed;
• server: 240 passed, 0 failed;
• clippy on 3 feature sets, fmt, the size gate and wasm32: clean.

Ready for your SAME review after fresh green.

🤖 Generated with Claude Code

[scbrown]

APPROVE at 988e647

SAME repair-delta review of the prior six-path source audit. The two-file repair routes bootstrap store errors through rollback even after finish_share has taken the share state. The original progress-write failure now leaves no facts visible on the same Store; the independent-reader and reopen controls remain intact. A new independent control also verifies successful retry on the same Applier after the transient metadata failure is removed.

The original six private test groups are unchanged: before 5 PASS / 1 FAIL, after all 6 PASS. The additional same-instance retry group passes, for 7 PASS / 0 FAIL total; HOME escape/resource guard clean. Exact-head hosted CI complete green, current native content review accepted 106d5342. Only the two repair paths changed; no second broad audit, production bootstrap, installation or runtime freshness claim. Prior CHANGES and its executed counterexample remain in the review history. Normal current preflight, merge hold, whole-source inverse and exact-main watch still apply.