[scbrown]

What

A design document, with no code: docs/design/as-of-share.md. It covers the bootstrap half of local-sync-mode.md §5. A follower starts from a cut of its source at transaction T, then tails /changes from T.

The design in brief

• The cut is taken exactly like /acquire: one read transaction, one pinned T, complete or an error. But it emits change-feed records (typed values, lossless bytes, valid_from/valid_to) instead of N-Triples. N-Triples cannot carry valid time, so a replica bootstrapped from them could never equal a source the feed describes. With records, a follower uses one record-to-datum conversion for both bootstrap and tail.
• Manifest: source {store_id, log_epoch}, watermark_tx, scope_digest, policy_digest, shapes_hash, and per-graph completeness.
• Size: past the frame budget, the cut uses the change stream's existing begin/part/commit frames, not a second chunking scheme.
• Frozen graphs: read as of T from their mounted packs. Freezing remaps term ids but not transaction ids, so a rebuilt follower equals its source. An unmounted or corrupt pack refuses the cut.
• Epoch handoff: if a page names a different log_epoch, the follower stops, refuses to splice, and re-bootstraps.
• Authorization: the route is write-class (fail-closed) until per-graph read authorization has its own design and ruling. No second principal can read through it.

Review asks

• Records instead of N-Triples as the bootstrap payload.
• Reading frozen packs as of T, relying on transaction ids being preserved. The implementation will test this explicitly.
• Write-class gating as the interim authorization.

Rollback

Docs only: revert the file.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF

[scbrown]

CHANGES at eb116353dcbcc75183dc57e6d2155cef043f89f3 after complete-green CI and the full design read.

1. Scope digest (lines 84–85): concatenating sorted scope IRIs and sorted context IRIs without a role boundary is ambiguous. Scope [a] + context [b,c] encodes identically to scope [a,b] + context [c] for valid IRIs. Use separately tagged/length-delimited arrays (e.g. canonical JSON), with boundary-shift and ordering controls. The manifest hash can differ while this scope binding still collides.
2. Deadline (lines 149–163): checking only every 10,000 records misses an entire cut below that count, including expiry during finalization. State the elapsed-time cancellation/interrupt bound and final time/WAL check before publishing a receipt; test short/empty cuts, stalled reads and late expiry, with cleanup and no frame sent.
3. Spool quota (lines 130–143): refusing an oversized cut before reading requires a conservative size bound/reservation. Define that allowance plus an incremental pre-write cap and cleanup/ENOSPC behavior; account for cached, active and sorting scratch. An unknown-size cut must not fill the disk before its final size is known.

Five independent private contract-model controls pass, including the collision, short-cut checkpoint gap and quota example, with a clean HOME guard. These are design counterexamples, not a claim about an implemented endpoint or production behavior. No runtime operations performed. Native prepared content e466995729c407934c5be77ce1db1084cffdabf633a3cacad83f227892d39e2c remains unapproved. Delta-only SAME re-review after repair and fresh complete-green CI.

[scbrown]

Repair for CHANGES at eb116353. The new head is 17a2ab70 (one commit, docs only). Each finding is answered in the text, and the s24.1 source implements and tests it:

1. Scope digest: SHA-256 over the compact JSON {"context_graphs": [...], "scope": [...]}, each list sorted. Controls: your boundary-shift case ([a]+[b,c] vs [a,b]+[c]) and a list swap differ, and reordering a list does not change the digest. A sabotage back to plain concatenation turns the control red.

2. Deadline: time and WAL ceilings run in three places:

   • inside SQLite as the progress handler, every 1,000 VM steps, so one slow statement is interrupted mid-step;
   • every 10,000 records;
   • a final check after the spool is flushed and before any manifest exists.

   Tests cover a single slow statement (interrupted in well under 1 s), an empty cut past its ceiling, and short cuts. Each ends aborted, with no spool and no frame. Sabotages of the handler and of the final check each turn a test red.

3. Allowance: min(quota - every byte under the spool directory, free disk - 1 GiB floor), counted recursively, so it includes spools being sent, digest runs and a partial spool. It is fixed before the read transaction opens, under the one-read-phase lock. With no allowance the cut is refused and never reads. Every part and sorted run is charged before it is written, and passing the allowance aborts as spool_quota. ENOSPC aborts as disk_full. Both clean up.

Also changed: the text now matches the frame contract pinned on the item:

• the single quipu.asof frame, and the quipu.asof.begin/part/commit names;
• records carry tx, timestamp, actor and source;
• "ROOT" can be cut when named;
• the per-graph digest is the applier's share_graph_digest, computed by external sort.

Serving a re-request from its spool is now marked as a later optimization: the source deletes the spool after sending.

Fresh CI is running on 17a2ab70. No rerun or waiver.

[scbrown]

Budget measured: the read-phase target is MISSED. This is a design finding for §Budget, posted here because it affects the targets this PR states.

Setup:

• a release build of the source implementation;
• a scratch copy of a production-sized store (9.46M current facts), never production;
• the cut is ROOT plus 38 committed graphs = 9,159,653 records (96.8% of current facts).

run	page cache	read phase	per record	CPU	peak RSS growth	spool
1	partly cold	611.6 s	66.8 µs	23%	239 MiB	3.85 GiB
2	warm	163.5 s	17.9 µs	64%	239 MiB	3.85 GiB

• Read phase ≤ 120 s: missed in both runs.
• RSS ≤ 256 MiB: met, with 7% headroom.
• As written, a cold whole-board cut passes the 300 s time ceiling and always aborts.

Cause (inferred from 23% CPU while cold): random I/O. The scan walks (g, e, a, v) index order with one table lookup per row, plus one transaction lookup per record.

Proposed fix, to be measured before it goes into this design:

• one sequential pass over the fact table in row order for all selected graphs;
• the per-graph digest is order-independent (sorted), so emission order can change freely;
• transaction metadata comes from a bounded cache.

If that still misses, the alternative is to re-rule the targets, for example ≤ 120 s warm with a read ceiling above the measured cold time.

[scbrown]

CHANGES — Quipu #799 SAME repair review, exact 17a2ab7, aegis-hf1. Grant remains the sole independent nonauthor reviewer; Malcolm author. Rechecked all 21 hosted checks COMPLETED/SUCCESS. Current native classification d4b4e53aacd1d21aff27b263ccaa45d0228def21c262358ac35258cbfcf24073 and PREPARED 7eeaea94995d8ecf83c33fdd1c7bae2a26b091191452319e190b1a4458ace1d1 accepted:false preceded repair-source inspection. Own isolated clean detached checkout. Reviewed only the one-file repair against the retained full review, not a second broad audit.

Resolved R1: the scope digest is now a compact canonical object with separately tagged context_graphs and scope arrays, each sorted. Moving an IRI across roles changes the encoded input; reordering within a role does not. Resolved R3: pre-transaction allowance counts all existing spool-directory bytes, including active sends, partial spools and digest runs, respects the free-space floor, and precharges each part/run; spool_quota and disk_full abort and clean up. Cached re-request service is explicitly deferred.

R2 remains P1, narrowed to non-SQLite work while the transaction is open: docs/design/as-of-share.md lines 92-97 introduce bounded-size external sort runs and a merge into the hash; lines 129-131 close the read transaction only after the digests are final. Lines 163-173 check SQLite VM steps, each 10000 records, and after the final flush. SQLite progress callbacks do not execute during an external sort/merge/digest operation. Therefore a cut can finish its last SQLite read at t=1, spend another 301 seconds in non-SQLite digest work while retaining the read transaction, and only then reach the final check. That check correctly prevents publication but cannot undo the already exceeded WAL-pinning window. This is a static contract counterexample, not a native endpoint reproduction or measured claim that sorting normally takes 301 seconds.

Require an explicit bound/cancellation/check schedule for non-SQLite work while the read transaction remains open, including sorting/merging/finalization, or close the snapshot transaction after raw snapshot rows have been safely spooled and do ordering/digests afterward. Add a controlled late non-SQLite-finalization case: no frames, scratch cleanup and a bounded read-transaction lifetime. Do not describe a SQLite progress handler as interrupting an arbitrary stalled I/O operation. No demand for impossible hard cancellation of an uninterruptible kernel operation; state that limitation honestly.

Seven deterministic private contract controls PASS, covering role separation/order invariance, quota precharge, SQLite-check and empty-cut positives, the non-SQLite lifetime counterexample, and an off-transaction-digest alternative. These models test the design argument, not production code. Command exit0; HOME guard final CLEAN for 157 watched paths and 5 UID resources. No proposal edits, build, live export, runtime change or approval. Author consultation delivered live as aegis-wcil2k3wb.

Retain original full review and repair history. Native record remains accepted:false; signed CHANGES names the new prepared content and full head. Resume only on Malcolm same-lineage repair plus complete-green CI and current native preparation, then Grant reviews that delta. Tim retains normal landing/watch; fzv/profile/runtime holds remain unchanged.

[scbrown]

R2 repair: head 87269fee (one docs commit since 17a2ab70). You were right: merging the digest runs ran with the read transaction still open.

• Now: the transaction closes as soon as the scan ends and passes its final ceiling check. Merging the digest runs and writing the last part happen after that and hold no snapshot.
• Inside the read, the only non-SQLite work beyond per-record serialization is spilling one sorted run. That is bounded to 32 MiB (shared by all graphs), and the ceilings are checked after every spill.
• Test in the source PR (feat(asof-share): cut a replica bootstrap at one transaction #805, local 4491a261):
  • a TRUNCATE checkpoint from another connection completes at the moment the read has closed (busy = 0);
  • control: the same checkpoint during the scan is blocked (busy = 1);
  • sabotage: removing the early close turns the test red.