From 6efadb30a5eef8ce598b65e7f2b27c79386d15d8 Mon Sep 17 00:00:00 2001 From: Steve Brown Date: Sat, 24 Oct 2026 01:46:35 -0400 Subject: [PATCH 1/5] docs(design): the as-of scoped share as the replica bootstrap Design only. It specifies the bootstrap half of local-sync-mode section 5. - A cut taken like an acquisition (one read transaction, one pinned tx) that emits change-feed records with valid time and lossless bytes, so a follower needs one conversion for bootstrap and tail. - A manifest with source identity and log epoch, watermark, scope and policy digests, shapes hash and per-graph completeness. - The feed's chunk frames past the frame budget. - Frozen graphs read from their mounted packs, so a rebuilt follower equals its source. - An epoch handoff that refuses to splice. Until per-graph read authorization has its own ruling, the route is write-class and fails closed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF --- docs/design/as-of-share.md | 189 +++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 docs/design/as-of-share.md diff --git a/docs/design/as-of-share.md b/docs/design/as-of-share.md new file mode 100644 index 000000000..27d6d3181 --- /dev/null +++ b/docs/design/as-of-share.md @@ -0,0 +1,189 @@ +# As-of scoped share: the replica bootstrap + +Status: design, not implemented. This document fills in the bootstrap half of +`local-sync-mode.md` §5. Per-graph read authorization (that design's P1, second +half) is a separate design and a separate ruling. Nothing here grants a second +principal any read. + +## Problem + +A follower must start from a state equal to its source at some transaction +`T`, then tail `/changes` from `T`. Today it cannot do either well: + +- **Replaying the feed from zero** is the only complete path. It is expensive: + a single-record `/changes?since=0` page takes seconds on a large store. It + also cannot reproduce a frozen graph, because a deep freeze removes that + graph's rows from the feed. +- **`POST /acquire`** (`quipu-acquisition/v1`) cuts a consistent, pinned, + hashed read, but cannot serve as the bootstrap: + - Its canonical N-Triples cannot carry a fact's valid interval, so the + bootstrapped replica cannot equal a source that the feed describes with + valid time. + - It has no `log_epoch`, so a follower cannot tie the cut to the history it + will tail. + - It has an 8 MiB cap with no chunking. + - It refuses frozen graphs. + +## Design + +### The cut reuses acquisition; the payload is change records + +The as-of share is produced exactly as an acquisition is: +- one SQLite read transaction; +- the store identity, head and epoch read inside it; +- every selected graph read at one validated transaction `T`; +- complete, or an error with no partial receipt. + +What changes is the payload. Each live fact at `T` is emitted as an `assert` +record in **the same JSON record form as `/changes`**: +- `entity`, `attribute`, `graph`; +- the typed `value`, with bytes as base64; +- `valid_from` and `valid_to`. + +A follower converts bootstrap records and tail records with **one** +record-to-datum conversion, the shared applier's. A fact is live at `T` when: +- it was asserted at or before `T`; +- it was not retracted at or before `T`; +- its valid interval is whatever the source stores for it at `T`. + +The replica proves equality with the same canonical live-fact digest the +feed-replay conformance tests use. A text form that cannot carry valid time +or bytes is refused, not offered as a degraded bootstrap. + +### Manifest + +```json +{ + "schema": "quipu-asof-share/v1", + "source": {"store_id": "...", "log_epoch": "..."}, + "watermark_tx": 742611, + "source_head_tx": 742630, + "scope": ["", "..."], + "context_graphs": ["", "..."], + "scope_digest": "sha256:...", + "policy_digest": "sha256:...", + "shapes_hash": "sha256:...", + "completeness": { + "valid_time": true, + "bytes": "lossless", + "graphs": { + "": {"records": 123, "digest": "sha256:...", "frozen": null}, + "": {"records": 45, "digest": "sha256:...", + "frozen": {"pack_content_hash": "...", "frozen_at_tx": 700001}} + } + }, + "producer": {"version": "...", "git_sha": "...", "git_dirty": false} +} +``` + +- `watermark_tx` is the last source transaction included. The follower tails + from it. +- `scope_digest` is SHA-256 over the sorted scope IRIs, then the sorted + context IRIs, newline-separated. +- `policy_digest` binds the read authorization the cut was served under. + Until per-graph read authorization exists, it names the single rule in + force: an authenticated write-class principal is required. +- Each graph's `digest` is SHA-256 over its records' canonical JSON lines, + sorted. Following the share-manifest encoding, that is compact UTF-8 JSON + with recursively sorted keys. + +The manifest's own hash is computed the same way as the acquisition receipt +hash. + +### Size: the feed's chunk frames, not a second scheme + +A cut that fits one frame is one `quipu.asof` frame. A larger cut uses the +change stream's oversized-transaction frames: +- `begin` with `source_tx = watermark_tx`, the part count, totals, the digest + and the source; +- numbered `part` frames with a running `digest_before`; +- a `commit` barrier repeating the digest and the source. + +These are the same transport-neutral `FeedFrame`s, so the hub and any +transport carry them unchanged. A follower stages every part and applies +nothing until the barrier digest matches. A disconnect mid-cut discards the +stage and restarts the cut; it never resumes half-way. + +### Frozen graphs (the bootstrap must equal the source) + +A frozen graph's rows live in its mounted pack. Freezing respaces term ids +but not transaction ids: `facts.tx`, `facts.retracted_tx` and +`transactions.id` are not remapped. So the pack can be read as of `T` exactly +like main: +- The cut reads the frozen graph's live facts at `T` from the pack. +- It emits them as ordinary records. +- It names the pack's `content_hash` and freeze transaction in `completeness`. + +A follower rebuilt from the share therefore equals the source, including +frozen graphs. Its feed tail then correctly carries nothing more for them. + +If a selected frozen graph's pack is not mounted, or fails its hash +verification, the cut is refused. It is never emitted without that graph. + +### Handoff to the tail, and epoch changes + +The follower records `(store_id, log_epoch, watermark_tx)` and tails +`/changes?since=watermark_tx`. Every page and every chunk frame names its +source. On any page whose `log_epoch` differs from the recorded one, the +follower: +1. stops; +2. refuses to splice; +3. discards its state; +4. bootstraps again from a fresh as-of share. + +A freeze or thaw mints a new epoch; a freeze or thaw that moves no rows does +not. + +### Route + +`POST /share/asof` with `{graphs, context_graphs?, tx?, expected_store_id}`. +- `tx` absent means "now": the cut is taken at the head read inside the + transaction. +- The same validation as `/acquire` applies: 1–64 registered committed graphs + (frozen allowed), no dataset expansion, no implicit ROOT, unknown fields + refused. + +**Authorization.** Until per-graph read authorization lands, the route +requires an authenticated **write-class** principal: it is listed with the +write endpoints for bearer authorization and read-only mode. That is +fail-closed. No unauthenticated or read-only caller can pull a cut, so no +second principal is served before the read model's ruling exists. Making the +route read-class is a change for the read-authorization work, not for this +one. + +## Verification plan + +- **Equality:** + 1. Bootstrap at `T` under concurrent writes, then tail from `T`. + 2. Compare: the follower's live digest equals the source's live digest at + the tail's end, and also at `T` itself. + 3. Control: a replica built by full feed replay from zero equals it too. +- **No gap or overlap:** a write committed during the cut appears exactly + once, in the tail, never in the cut. +- **Valid time and bytes:** fixtures with bounded and open intervals and a + byte value replay exactly. Sabotage: dropping `valid_to`, or emitting + `bytes_len`, turns them red. +- **Frozen graph (s24.1.1):** + 1. Freeze a graph and take a cut. + 2. The follower's digest equals the source's, including the frozen graph. + 3. Control: the same flow without the frozen-pack read fails equality. + 4. An unmounted pack refuses the cut. +- **Epoch conformance (s24.1.1):** + 1. The follower is at `watermark_tx`; the source freezes another graph, + which bumps the epoch. + 2. The follower's next page names the new epoch. + 3. It refuses to splice, re-bootstraps, and equals the source again. + 4. Control: with no epoch check, splicing would leave the follower unequal. + 5. A no-op freeze does not bump, and the follower keeps tailing. +- **Chunking:** a cut over the frame budget arrives as begin, parts and + commit. A dropped part refuses the commit and records nothing. A + disconnect restarts the cut. +- **Authorization:** the route refuses no bearer, refuses read-only mode, and + refuses an unknown or dataset IRI. Control: an authenticated write-class + principal succeeds. + +## Rollback + +- Remove the route. Nothing else changes, and no stored state is added. +- Followers fall back to full feed replay, which still cannot reproduce frozen + graphs. From aa457d4692577ca34e33afb061263cc715b1e81c Mon Sep 17 00:00:00 2001 From: Steve Brown Date: Sat, 24 Oct 2026 02:24:16 -0400 Subject: [PATCH 2/5] docs(design): bound the as-of cut's cost on a large store A whole-board cut is one long read transaction, which pins the WAL. The read now spools to disk and closes its transaction before anything is sent, so a slow follower never holds the source's snapshot. The doc also states the targets (wall time and RSS at production size, measured on a scratch copy in the source PR) and the time and WAL-growth ceilings that abort a cut, which is refused and never sent partially. Only one read phase runs per source at a time. The route checklist (endpoint lists, share completeness, REST reference) is now named too. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF --- docs/design/as-of-share.md | 77 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/docs/design/as-of-share.md b/docs/design/as-of-share.md index 27d6d3181..8a6212f00 100644 --- a/docs/design/as-of-share.md +++ b/docs/design/as-of-share.md @@ -104,6 +104,66 @@ transport carry them unchanged. A follower stages every part and applies nothing until the barrier digest matches. A disconnect mid-cut discards the stage and restarts the cut; it never resumes half-way. +### Budget + +A whole-board cut is one SQLite read transaction over the production store +(about 9.3M current facts). While it is open it pins the WAL: checkpoints +cannot complete past it, and the WAL grows with every write that lands during +the cut. The cut is therefore bounded in four ways. + +**(b) The transaction is open only while reading, never while sending.** +- Under the read transaction, the producer streams records into an on-disk + **spool**: one file of canonical JSON lines per cut, in the scratch + directory. +- It computes the per-graph and whole-cut digests as it writes. +- It closes the transaction as soon as the spool is complete and its digests + are final. +- Frames are then sent from the spool with no transaction open. +- So a slow or stalled follower holds a file on disk, never the source's + snapshot. The WAL is pinned only for the read phase, which is bounded by the + source's own disk and CPU, not by any network. +- The spool is named by the manifest hash. A follower that disconnects and + re-requests the same cut (same scope and `T`) within the spool's lifetime is + served from it, with no new read transaction. +- Spools expire after one hour. They count against a disk quota (default + 8 GiB); a cut that would exceed it is refused before the transaction opens. + +**(a) Targets at production size**, to be measured on a scratch copy of a +production-sized store in the source PR, never against production: +- Read phase: at most **120 s** wall time for a whole-board cut. +- Peak RSS growth: at most **256 MiB** over the idle process, independent of + board size. Records are streamed to the spool, never collected in memory. +- Spool size: reported, expected to be a few GiB at production size. The PR + states the measured figure and the per-fact cost. + +The PR records the build profile (release), the store size, the fact count, +and all three numbers. A target that is missed is a design defect to fix or a +ceiling to re-rule; the budget is never loosened silently. + +**(c) The WAL ceiling: refuse or abort, never partial.** During the read +phase the producer checks its elapsed time and the WAL file's growth since the +cut began, every 10,000 records. Either ceiling aborts the cut: +- wall time over `asof_read_timeout_secs` (default 300); +- WAL growth over `asof_wal_growth_max_mib` (default 1024). + +An abort: +- rolls back the read transaction, so checkpoints resume; +- deletes the partial spool; +- answers `503` with `{"error":"cut_aborted","reason":"wal_ceiling|time_ceiling","retry_after_secs":...}`. + +No manifest and no frame of an aborted cut is ever sent. A follower treats it +as "no bootstrap yet" and retries later; it never applies part of a cut. + +**(d) Admission.** +- At most **one** read phase runs per source at a time. A second request + while one is reading gets `503` with `Retry-After`. It is not queued behind + the open transaction, because queued cuts would extend the pinned window + back-to-back. +- Sending from finished spools is limited by the existing stream slots, not + by this lock. +- The route is write-class, so only authenticated writers can start a cut at + all. + ### Frozen graphs (the bootstrap must equal the source) A frozen graph's rows live in its mounted pack. Freezing respaces term ids @@ -151,6 +211,13 @@ second principal is served before the read model's ruling exists. Making the route read-class is a change for the read-authorization work, not for this one. +The new route also needs the same classification every route gets: +- an entry in `http_auth`'s endpoint lists (write-class here), which the + route-completeness test checks; +- a `share_completeness` classification for anything it persists (the spool + is scratch, not store state); +- a `rest-api.md` reference entry. + ## Verification plan - **Equality:** @@ -175,6 +242,16 @@ one. 3. It refuses to splice, re-bootstraps, and equals the source again. 4. Control: with no epoch check, splicing would leave the follower unequal. 5. A no-op freeze does not bump, and the follower keeps tailing. +- **Budget, on a scratch copy of a production-sized store:** + - Measure and report the wall time and peak RSS of the read phase against + the targets. + - Abort controls: + 1. a write load during the cut that pushes the WAL past a lowered ceiling + aborts the cut, sends no manifest, and lets checkpoints resume; + 2. a lowered time ceiling aborts the same way; + 3. a slow follower reading from the spool does not hold a read transaction + open (checked via the WAL checkpointing during the send). + - A second concurrent cut is refused with `Retry-After`. - **Chunking:** a cut over the frame budget arrives as begin, parts and commit. A dropped part refuses the commit and records nothing. A disconnect restarts the cut. From eb116353dcbcc75183dc57e6d2155cef043f89f3 Mon Sep 17 00:00:00 2001 From: Steve Brown Date: Sat, 24 Oct 2026 03:41:22 -0400 Subject: [PATCH 3/5] docs(as-of-share): blank lines around lists for markdownlint MD032 Pre-commit failed on 13 MD032 blanks-around-lists findings. Whitespace only: one blank line before each list; no wording changes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF --- docs/design/as-of-share.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/design/as-of-share.md b/docs/design/as-of-share.md index 8a6212f00..235267cb8 100644 --- a/docs/design/as-of-share.md +++ b/docs/design/as-of-share.md @@ -29,6 +29,7 @@ A follower must start from a state equal to its source at some transaction ### The cut reuses acquisition; the payload is change records The as-of share is produced exactly as an acquisition is: + - one SQLite read transaction; - the store identity, head and epoch read inside it; - every selected graph read at one validated transaction `T`; @@ -36,12 +37,14 @@ The as-of share is produced exactly as an acquisition is: What changes is the payload. Each live fact at `T` is emitted as an `assert` record in **the same JSON record form as `/changes`**: + - `entity`, `attribute`, `graph`; - the typed `value`, with bytes as base64; - `valid_from` and `valid_to`. A follower converts bootstrap records and tail records with **one** record-to-datum conversion, the shared applier's. A fact is live at `T` when: + - it was asserted at or before `T`; - it was not retracted at or before `T`; - its valid interval is whatever the source stores for it at `T`. @@ -94,6 +97,7 @@ hash. A cut that fits one frame is one `quipu.asof` frame. A larger cut uses the change stream's oversized-transaction frames: + - `begin` with `source_tx = watermark_tx`, the part count, totals, the digest and the source; - numbered `part` frames with a running `digest_before`; @@ -112,6 +116,7 @@ cannot complete past it, and the WAL grows with every write that lands during the cut. The cut is therefore bounded in four ways. **(b) The transaction is open only while reading, never while sending.** + - Under the read transaction, the producer streams records into an on-disk **spool**: one file of canonical JSON lines per cut, in the scratch directory. @@ -130,6 +135,7 @@ the cut. The cut is therefore bounded in four ways. **(a) Targets at production size**, to be measured on a scratch copy of a production-sized store in the source PR, never against production: + - Read phase: at most **120 s** wall time for a whole-board cut. - Peak RSS growth: at most **256 MiB** over the idle process, independent of board size. Records are streamed to the spool, never collected in memory. @@ -143,10 +149,12 @@ ceiling to re-rule; the budget is never loosened silently. **(c) The WAL ceiling: refuse or abort, never partial.** During the read phase the producer checks its elapsed time and the WAL file's growth since the cut began, every 10,000 records. Either ceiling aborts the cut: + - wall time over `asof_read_timeout_secs` (default 300); - WAL growth over `asof_wal_growth_max_mib` (default 1024). An abort: + - rolls back the read transaction, so checkpoints resume; - deletes the partial spool; - answers `503` with `{"error":"cut_aborted","reason":"wal_ceiling|time_ceiling","retry_after_secs":...}`. @@ -155,6 +163,7 @@ No manifest and no frame of an aborted cut is ever sent. A follower treats it as "no bootstrap yet" and retries later; it never applies part of a cut. **(d) Admission.** + - At most **one** read phase runs per source at a time. A second request while one is reading gets `503` with `Retry-After`. It is not queued behind the open transaction, because queued cuts would extend the pinned window @@ -170,6 +179,7 @@ A frozen graph's rows live in its mounted pack. Freezing respaces term ids but not transaction ids: `facts.tx`, `facts.retracted_tx` and `transactions.id` are not remapped. So the pack can be read as of `T` exactly like main: + - The cut reads the frozen graph's live facts at `T` from the pack. - It emits them as ordinary records. - It names the pack's `content_hash` and freeze transaction in `completeness`. @@ -186,6 +196,7 @@ The follower records `(store_id, log_epoch, watermark_tx)` and tails `/changes?since=watermark_tx`. Every page and every chunk frame names its source. On any page whose `log_epoch` differs from the recorded one, the follower: + 1. stops; 2. refuses to splice; 3. discards its state; @@ -197,6 +208,7 @@ not. ### Route `POST /share/asof` with `{graphs, context_graphs?, tx?, expected_store_id}`. + - `tx` absent means "now": the cut is taken at the head read inside the transaction. - The same validation as `/acquire` applies: 1–64 registered committed graphs @@ -212,6 +224,7 @@ route read-class is a change for the read-authorization work, not for this one. The new route also needs the same classification every route gets: + - an entry in `http_auth`'s endpoint lists (write-class here), which the route-completeness test checks; - a `share_completeness` classification for anything it persists (the spool From 17a2ab70a90a0824cee3477f1d0a573c36c999a5 Mon Sep 17 00:00:00 2001 From: Steve Brown Date: Sat, 24 Oct 2026 06:27:27 -0400 Subject: [PATCH 4/5] docs(as-of-share): answer the design review grant's three findings (CHANGES at eb116353): 1. scope_digest is SHA-256 over tagged compact JSON {"context_graphs","scope"}. Moving an IRI between the lists changes it. 2. The ceilings also run inside SQLite every 1,000 VM steps, so one slow statement is interrupted. A final check after the spool is flushed and before any manifest exists covers short and empty cuts. 3. The disk allowance is concrete: min(quota - bytes already under the spool dir, free disk - 1 GiB) It is fixed before the transaction opens. Every part and sorted run is charged before it is written, and ENOSPC aborts as disk_full, with cleanup. Also aligned with the frame contract wu pinned on the item: - the single quipu.asof frame and the quipu.asof.begin/part/commit names; - records carry tx, timestamp, actor and source; - "ROOT" can be cut when named; - per-graph digest = the applier's share_graph_digest, computed by external sort. Serving a re-request from its spool is marked as a later optimization. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF --- docs/design/as-of-share.md | 70 +++++++++++++++++++++++++++----------- 1 file changed, 50 insertions(+), 20 deletions(-) diff --git a/docs/design/as-of-share.md b/docs/design/as-of-share.md index 235267cb8..0ede45fa2 100644 --- a/docs/design/as-of-share.md +++ b/docs/design/as-of-share.md @@ -38,9 +38,11 @@ The as-of share is produced exactly as an acquisition is: What changes is the payload. Each live fact at `T` is emitted as an `assert` record in **the same JSON record form as `/changes`**: -- `entity`, `attribute`, `graph`; +- `entity`, `attribute`, `graph` (`"ROOT"` for the default graph, which is cut + only when named); - the typed `value`, with bytes as base64; -- `valid_from` and `valid_to`. +- `valid_from` and `valid_to`; +- the asserting transaction's `tx`, `timestamp`, `actor` and `source`. A follower converts bootstrap records and tail records with **one** record-to-datum conversion, the shared applier's. A fact is live at `T` when: @@ -81,27 +83,31 @@ or bytes is refused, not offered as a degraded bootstrap. - `watermark_tx` is the last source transaction included. The follower tails from it. -- `scope_digest` is SHA-256 over the sorted scope IRIs, then the sorted - context IRIs, newline-separated. +- `scope_digest` is SHA-256 over the compact JSON + `{"context_graphs": [...], "scope": [...]}`, each list sorted. The lists are + tagged, so moving an IRI from one to the other always changes the digest. - `policy_digest` binds the read authorization the cut was served under. Until per-graph read authorization exists, it names the single rule in force: an authenticated write-class principal is required. -- Each graph's `digest` is SHA-256 over its records' canonical JSON lines, - sorted. Following the share-manifest encoding, that is compact UTF-8 JSON - with recursively sorted keys. +- Each graph's `digest` is the shared applier's `share_graph_digest`: SHA-256 + over its records as compact UTF-8 JSON with keys sorted at every level, + the lines sorted and joined by `\n`. A `sha256:` prefix is allowed. The + producer cannot sort millions of lines in memory, so it sorts runs of at + most 32 MiB, spills them beside the spool and merges them into the hash; + the bytes hashed are the same. The manifest's own hash is computed the same way as the acquisition receipt hash. ### Size: the feed's chunk frames, not a second scheme -A cut that fits one frame is one `quipu.asof` frame. A larger cut uses the -change stream's oversized-transaction frames: +A cut that fits one frame is one `quipu.asof` frame, `{manifest, records}`. A +larger cut uses the change stream's oversized-transaction frames: -- `begin` with `source_tx = watermark_tx`, the part count, totals, the digest - and the source; -- numbered `part` frames with a running `digest_before`; -- a `commit` barrier repeating the digest and the source. +- `quipu.asof.begin` with the manifest, `source_tx = watermark_tx`, the part + count, totals, the digest and the source; +- numbered `quipu.asof.part` frames with a running `digest_before`; +- a `quipu.asof.commit` barrier repeating the digest and the source. These are the same transport-neutral `FeedFrame`s, so the hub and any transport carry them unchanged. A follower stages every part and applies @@ -127,11 +133,19 @@ the cut. The cut is therefore bounded in four ways. - So a slow or stalled follower holds a file on disk, never the source's snapshot. The WAL is pinned only for the read phase, which is bounded by the source's own disk and CPU, not by any network. -- The spool is named by the manifest hash. A follower that disconnects and - re-requests the same cut (same scope and `T`) within the spool's lifetime is - served from it, with no new read transaction. -- Spools expire after one hour. They count against a disk quota (default - 8 GiB); a cut that would exceed it is refused before the transaction opens. +- The spool is named by the manifest hash and deleted when its stream ends. + Serving a re-request of the same cut from its spool is a later + optimization; until then a re-request takes a new cut. Spools a crashed + send left behind are swept after one hour. +- **Disk allowance.** Before the transaction opens, the cut fixes its + allowance: the quota (default 8 GiB) minus every byte already under the + spool directory (spools being sent, digest runs, a partial spool), and + never more than the free disk above a floor (default 1 GiB). One read + phase runs at a time, so no other cut draws on it. With no allowance the + cut is refused and never pins the WAL. Each part and each sorted run is + charged against the allowance before it is written; passing it aborts as + `spool_quota`. A write that still fails for lack of disk (ENOSPC) aborts as + `disk_full`. Either way the partial spool and runs are deleted. **(a) Targets at production size**, to be measured on a scratch copy of a production-sized store in the source PR, never against production: @@ -148,7 +162,16 @@ ceiling to re-rule; the budget is never loosened silently. **(c) The WAL ceiling: refuse or abort, never partial.** During the read phase the producer checks its elapsed time and the WAL file's growth since the -cut began, every 10,000 records. Either ceiling aborts the cut: +cut began: + +- inside SQLite, as the connection's progress handler, every 1,000 VM steps + (the WAL file at most once a second), so even one slow statement is + interrupted mid-step; +- every 10,000 records; +- once more after the spool is flushed and before any manifest exists, so a + short or empty cut cannot finish past a ceiling. + +Either ceiling aborts the cut: - wall time over `asof_read_timeout_secs` (default 300); - WAL growth over `asof_wal_growth_max_mib` (default 1024). @@ -157,7 +180,7 @@ An abort: - rolls back the read transaction, so checkpoints resume; - deletes the partial spool; -- answers `503` with `{"error":"cut_aborted","reason":"wal_ceiling|time_ceiling","retry_after_secs":...}`. +- answers `503` with `{"error":"cut_aborted","reason":"wal_ceiling|time_ceiling|spool_quota|disk_full","retry_after_secs":...}`. No manifest and no frame of an aborted cut is ever sent. A follower treats it as "no bootstrap yet" and retries later; it never applies part of a cut. @@ -265,6 +288,13 @@ The new route also needs the same classification every route gets: 3. a slow follower reading from the spool does not hold a read transaction open (checked via the WAL checkpointing during the send). - A second concurrent cut is refused with `Retry-After`. + - One slow statement is interrupted by the time ceiling mid-step; an empty + cut past its ceiling is aborted by the final check; neither leaves a + spool or sends a frame. + - No disk allowance refuses the cut before it reads. Bytes in nested run + directories count against the quota. +- **Scope digest:** moving an IRI between scope and context changes the + digest; reordering a list does not. - **Chunking:** a cut over the frame budget arrives as begin, parts and commit. A dropped part refuses the commit and records nothing. A disconnect restarts the cut. From 87269fee5cc71431a3078cf36f7186887b96bbcd Mon Sep 17 00:00:00 2001 From: Steve Brown Date: Sat, 24 Oct 2026 08:51:19 -0400 Subject: [PATCH 5/5] docs(as-of-share): close the read after the scan; bound spills (review R2) grant's R2 on 17a2ab70: merging digest runs ran with the read transaction still open, so slow non-SQLite work could pin the WAL past the ceiling. Now the transaction closes as soon as the scan ends and passes its final ceiling check. Merging the runs and writing the last part hold no snapshot. Inside the read, a spill is bounded to one run (32 MiB, shared by all graphs), and the ceilings are checked after every spill. The verification plan adds the checkpoint test (completes after the scan, blocked during it). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BzgSY2LpdGhBhb7D1nHbiF --- docs/design/as-of-share.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/design/as-of-share.md b/docs/design/as-of-share.md index 0ede45fa2..8fdada4ac 100644 --- a/docs/design/as-of-share.md +++ b/docs/design/as-of-share.md @@ -126,9 +126,14 @@ the cut. The cut is therefore bounded in four ways. - Under the read transaction, the producer streams records into an on-disk **spool**: one file of canonical JSON lines per cut, in the scratch directory. -- It computes the per-graph and whole-cut digests as it writes. -- It closes the transaction as soon as the spool is complete and its digests - are final. +- It feeds each record to the whole-cut part digest and to its graph's + digest runs as it writes. Spilling a sorted run holds at most one run's + memory (32 MiB, shared by all graphs), and the ceilings are checked after + every spill. +- It closes the transaction as soon as the scan ends and passes its final + ceiling check. Merging the digest runs and writing the last part happen + after that and hold no snapshot, so no non-SQLite work can pin the WAL + past the read. - Frames are then sent from the spool with no transaction open. - So a slow or stalled follower holds a file on disk, never the source's snapshot. The WAL is pinned only for the read phase, which is bounded by the @@ -168,8 +173,9 @@ cut began: (the WAL file at most once a second), so even one slow statement is interrupted mid-step; - every 10,000 records; -- once more after the spool is flushed and before any manifest exists, so a - short or empty cut cannot finish past a ceiling. +- after every spilled digest run; +- once more when the scan ends, just before the transaction closes and before + any manifest exists, so a short or empty cut cannot finish past a ceiling. Either ceiling aborts the cut: @@ -291,6 +297,9 @@ The new route also needs the same classification every route gets: - One slow statement is interrupted by the time ceiling mid-step; an empty cut past its ceiling is aborted by the final check; neither leaves a spool or sends a frame. + - After the scan, a TRUNCATE checkpoint from another connection completes + (the snapshot is released before digest runs are merged). While the + scan runs, the same checkpoint is blocked (the control). - No disk allowance refuses the cut before it reads. Bytes in nested run directories count against the quota. - **Scope digest:** moving an IRI between scope and context changes the