Skip to content

Share verified Cell publication and pipeline native replication - #67

Merged
forhappy merged 111 commits into
mainfrom
codex/packed-write-publication
Oct 10, 2026
Merged

forhappy merged 111 commits into
mainfrom
codex/packed-write-publication

Conversation

@forhappy

@forhappy forhappy commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Publication backlog previously blocked replication admission under a global ordering lock. Capacity waits and local validation now precede ordered issuance; eight original append rounds use independent ordered follower queues with exact ordered credit and joined shutdown. Ready root checkpoints and complete native captures share one bounded catalog upload and fenced selection. Signed grants and exact coverage retain complete issued-range closure, including prior Fleet ACKs.

Selected-capture cleanup now proceeds independently of the publisher held by an original root task. Original root proof/debt obligations remain joined; later selected suffixes survive root completion. Completed root I/O releases working credit before its queued checkpoint callback. Existing memory, materializer, locator, proof and drain bounds remain unchanged. The managed-actor regression covers held root preparation and callbacks, exact cold outcomes/retries and zero final credit.

This branch incorporates main 36e1f3f6 (PR #64), including bounded SQL admission, the owner reader, schema caching, query/slot telemetry and lane-scoped follower accounting. Explicit external durability sets both writer and reader to SQLite NORMAL; partial configuration failure fences the session. Standalone defaults remain FULL.

Not ready to merge: the fresh integrated candidate regresses write TPS and successful-write latency and fails drain.

The latest serialized comparison measures production e151e779 against 6e2ba162 and pinned celld f2bf6486. Each case uses 2,000 uniform Cells, 96-byte SQL-ledger values, 128 clients/queue slots, 2,000 offered Fleet writes/s, 30 seconds warmup and 60 measured seconds, fresh roles/provider volume and unchanged Cellule ceilings.

System Successful writes/s Successful scheduled p99 ms Errors Dropped offers Warm/cold audit
Cellule baseline 566.45 373.10 26,335 59,520 Both pass; 63,875 ACKs
Integrated candidate 276.62 1,010.32 46,954 56,317 Warm passes; 42,210 ACKs; cold not reached
celld 1,985.03 151.61 0 891 Both pass; 181,087 ACKs

The candidate's observed TPS is 51.17% lower and successful p99 is 2.71 times the baseline. This short combined-change pair does not isolate the root-cleanup effect. Every original output, complete ACK inventory, offer/error/drop counter and latency is independently reconciled. Candidate drain exceeds the unchanged 120-second deadline; logs show fenced materialization and node-log drain, and both followers exit 1. This fails draining and leaves cold recovery unverified; it does not demonstrate acknowledged-data loss. All original tool handles are joined and failed attempts are retained outside Git.

Candidate retained memory is lower, but unpublished log debt grows 29.55→51.54 MiB and oldest debt grows 54.20→101.44 seconds. Mean native submission is 0.30 ms, with publication-slot wait below 0.001 ms; separately sampled Fleet proof wait averages 83.83 ms. Followers group about 9.44 frames per data sync with zero measured append failures. These different populations are not a per-command latency sum, and tmpfs does not qualify physical power-loss durability.

All 16 isolated verification routes pass on the integrated code: format, features/targets, workspace tests, local LTX, Rust 1.97/1.99 Clippy, API docs, boundaries/layout, document fences/links, SQL/peer contracts, script tests and three root-cut regressions. Failed merge/configuration/link attempts and passing repairs are retained. The delivered head differs from measured production only in Markdown. Local checks do not imply remote CI or passing capacity qualification.

The concise measurement report retains earlier failed/interrupted attempts and the complete fresh scope. Raw performance journals and build evidence remain outside the repository.

These are HTTP/SQL application measurements: native Rust/Axum versus celld's JavaScript Worker/Durable Object in Rust/V8, using the same Rust HTTP client. The shared eight-CPU/~8-GiB VM does not qualify a dedicated 8-CPU/16-GiB node. Internal budgets and core paths remain unequal: celld defaults to four ordered one-shot HTTP rounds and a zero window; Cellule uses eight rounds and a general four-millisecond assembly interval. Persistent streaming is not the measured celld default.

Remaining blockers are the integrated regression, overload availability, fenced complete-range drain/recovery, repeated catalog/history verification, node-wide complete-range capture, independent bounded compaction, the real common-core driver, sustained zero-error/drop repetitions and read/mixed guardrails. The acceptance gates are unchanged. Architectural and performance parity remain unmet.

Keep per-Cell fenced root selection while uploading bounded application-scoped
capture cohorts. Issue signed append windows through fresh enrollment and
serialize native proof release with durable closure.

Document the remaining bundle-coverage protocol and preserve qualification
failures. Bundle-based bucket acknowledgments remain disabled.
@forhappy
forhappy force-pushed the codex/packed-write-publication branch from fc8147b to c590664 Compare October 7, 2026 09:04
@forhappy forhappy changed the title Share Cell publication and use signed follower append grants Share Cell publication and add verified bundle coverage APIs Oct 7, 2026
@forhappy
forhappy marked this pull request as ready for review October 8, 2026 01:11
@forhappy
forhappy marked this pull request as ready for review October 10, 2026 06:13
@forhappy
forhappy merged commit 25a76c0 into main Oct 10, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant