Repository navigation
Cluster workerd processes over a shared bucket - #7994
Merged
Merged
Conversation
This is the beginning of merging celld's functionality into workerd. It lets several workerd processes network into a cluster by giving each of them access to the same S3-compatible bucket (`Config.cluster`, `--experimental`). Each Durable Object has one owner at a time, requests reach it from any process, and when its owner dies another process picks it up from the bucket. Objects use the bucket as cold storage with Litestream's algorithm and file formats. For now an object acknowledges a write only once it is fully synced to the bucket; that is much too slow and expensive for most uses. Next we will acknowledge writes once peers in the cluster hold them and buffer the bucket writes. Alarms are also not supported yet.
capture.rs read the WAL with the Unix-only read_exact_at; it seeks and reads instead. The ltx test VFS copied 160 bytes out of SQLite's 152-byte io_methods into a struct whose tail is a non-null fn pointer; each file now keeps its patched table in its own allocation, so nothing is overread or leaked. Drop a word clippy's doc_markdown flags.
Windows SQLite uses the KJ VFS, which cannot report WAL writes, so a cluster object there acknowledged commits it never uploaded: the takeover test on Windows recovered nothing. Refuse walWrite on Windows and skip the WAL-report and cluster tests there.
The capture tests register a Rust VFS with SQLite. A downstream release build compiles SQLite with -fsanitize=cfi-icall, so its first call into the Rust xOpen traps (SIGILL). Production calls Rust from C++ directly through cxx and is unaffected.
Clustered actors' SQLite opens files through a kj::Directory wrapper, as edgeworker's storage relay does: opening checks the lease, and a WAL file reports each successful write. It has no fd, so SQLite uses the KJ VFS on every platform. This drops VfsOptions::afterOpen and walWrite and their native-VFS plumbing, leaving sqlite.c++'s VFS as on main, and the cluster test runs on Windows again.
An object's log is a stream of frames (frames.capnp), each a change to one lane, written by an ownership epoch in sequence. The database is a sqlitePages lane: a commit is a frame of the pages it wrote, so merging a run of commits keeps one copy of each page, and a snapshot is a header and the database. Restore and compaction replay frames into a file, which replaces LTX's k-way merge; a frame a wider file already applied is skipped by its sequence. Each file is an LZ4 frame with a content checksum. Lanes leave room for other kinds of state beside the database, and for commits that span them.
|
The generated output of |
A file says whether it holds frames or a snapshot, so replay no longer needs the level from its key, and a later field or kind of file has a place to go.
A process that restarts at the same address before its predecessor's lease runs out finds the predecessor's record live and routed to it, which is itself: the call came back in and routed again, with no hop limit. Refuse instead, until the lease ends and the object can be claimed.
Every failure returned silently, and a gap in the chain stranded the files past it while the prefix kept compacting. On a soak, one epoch grew 1,454 L0 files after its last snapshot with nothing in the log.
Main now checks in crate_universe's lockfile, so a Cargo.toml that adds crates must repin it (CARGO_BAZEL_REPIN=1). The repin also catches up main's Cargo.lock bumps that the lockfile had not followed.
…ords Nothing set or read `facets`, so own.json always carried `"facets":[]`. Facets come in their own change, which can add the field back. Each record's fields now say what they hold, and NodeRecord's `node` and `address` say what they are: a random id and `host:port`, not a public key and sockaddr bytes.
The cluster test runs on Windows since WAL capture moved to a directory,
but Node's child.kill('SIGTERM') is TerminateProcess there: the node dies
without draining, so the two drain tests read ECONNRESET.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #7994 +/- ##
==========================================
- Coverage 69.38% 68.30% -1.08%
==========================================
Files 696 710 +14
Lines 181896 185140 +3244
Branches 29221 29252 +31
==========================================
+ Hits 126212 126466 +254
- Misses 38610 41592 +2982
- Partials 17074 17082 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The log's frames are the storage relay's, field for field: each commit
is a write of the bytes SQLite wrote to the WAL, at an offset that keeps
growing across checkpoints, each marked by a consume, plus newLane,
initialize and deleteLane. A batch starts and ends at a commit. A
snapshot is a SnapshotHeader and then the database file, never a WAL:
replay checkpoints committed WAL frames into the file first. Members
workerd leaves unset are kept for wire compatibility and marked unused.
Compaction follows SRS's rule: snapshot a session once the log since its
last snapshot reaches the database's size, from that snapshot and the
batches past it, then delete what the new one covers. Batches are never
merged, so the page merge and the level-numbered chain are gone.
Keys match SRS's layout, and none depends on a name's length:
do/<namespace>/<actor>/log/<session>/<end frame>
/snapshot/<session>/<frame>
/own.json
<namespace> is base64url of the first 16 bytes of a labelled SHA-256 of
the tenant (empty for now) and the namespace's uniqueKey; <actor> is the
actor ID's hex. Numbers are SRS's length-prefixed hex, sessions and
snapshot frames descending, so a listing comes newest first. The
FileHeader is dropped: the key says what a file is.
A draining node kept serving its own objects: the owner check came before the drain check, so it reopened them for any request that reached it. It also stayed listed until its last connection finished, so peers routed its objects to a node that no longer accepted them, for up to a lease. Now a draining node refuses its own objects as it refuses claims, and deregisters when the drain starts: peers drop it at their next listing and take its objects over. A renewal in flight at that moment may find the record gone; once draining, its outcome is ignored rather than read as a fence.
Drop the unused accessor that handed out the raw key.
Logging the WAL's bytes put a page a hot object rewrites in the log once per commit: a batch of grouped commits carried every copy, and the log reached the database's size, and so a snapshot, as fast as commits came. The log is a `sqlitePages` lane instead, workerd's own lane type at new ordinals of SRS's schema, which is otherwise unchanged. A commit is a frame of the pages it wrote; a batch merges each run of them to one frame with the newest version of each page, covering the commits `first..=sequence`. A hundred updates of one row log one page, not a hundred: 4 KB instead of 418 KB, 199 bytes instead of 3.3 KB after LZ4. A new database is still SRS's `initialize`, snapshots still SRS's header and the database file, and compaction still SRS's rule.
ry
marked this pull request as ready for review
October 28, 2026 14:53
|
💬 Approved with comments — The changes since the last review (gzip, the alarm refusal revert, the scheduler VFS) have no defects. Only INFO notes remain: one new hardening note on uncapped gzip output, and earlier notes in unchanged code. Findings: 10 info The decision is advisory: this comment never approves, blocks or requests changes on the pull request. 📋 Code Quality: 💬 9 infoNo new defects. Earlier INFO notes in unchanged code are still open. Recommendation: 💬 Optional Nits
View full reviewChanged since the last review
Still open from earlier reviews (INFO, code unchanged): listed in the items below. 🔒 Security: 💬 1 infoOne hardening note: gzip output from the bucket is not capped. Recommendation: 💬 Optional Nits
View full review
✅ All other checks passed📖 AGENTS.md: ✅ LGTMNo issues found. ⚡ Performance: ✅ LGTM
📝 Documentation: ✅ LGTMNo new documentation issues. The stale comments in Reviewed commit: |
Contributor
|
Since last review: 0 resolved, 1 still open, 0 new. The added tests cover shared cluster-token keys and missing-log-batch compaction. Not re-run: api-compat, docs, design-simplicity, jsg-gc (no author changes in their files since the last review) Specialist findings not posted (2)
Reviewed commit: 9d48e083 · github run |
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
workerd exits without dropping its tokio runtime, leaving idle blocking-pool threads unjoined, which TSan reports as leaked: the drain handoff test's owner then exits 66, not 0. Set KJ_CLEAN_SHUTDOWN for the nodes, as the asan and coverage configs do.
Every namespace is clustered, but drain aborted only evictable ones, so an object with preventEviction kept serving its in-flight requests after its node left the cluster, while a peer took it over.
Concurrent first routes to a peer each saw no session, dialed their own, and overwrote each other's in the map, while the stubs they returned kept every one of those sessions alive. The map now holds the dial itself, as a shared future, so they wait on the same one.
Once an epoch opens, own.json names it and its session starts with a snapshot of everything the recovered one held, so no restore reads an earlier session again. Nothing deleted them, leaving another copy of the object in the bucket at every handoff. The next tier pass after an epoch opens deletes every earlier session, including what a superseded owner uploaded after the takeover.
jasnell
reviewed
Oct 28, 2026
jasnell
left a comment
jasnell
left a comment
Collaborator
There was a problem hiding this comment.
These review comments were generated by an AI assistant and may not be perfect.
Not repeating findings already raised by bonk, clanky or @piscisaureus. Summary:
- Restore does not verify snapshot integrity, treats a truncated LZ4 file as complete, and lets a corrupt bucket object trigger an unbounded capnp allocation (
log.rs). Combined with the existing gap finding, a bad restore can be compacted into a new snapshot and made permanent. - Enabling
clusteron an existinglocalDiskdeletes each object's existing data at first access (owner.rs). setAlarm()is refused at commit time, which rolls back the implicit transaction and breaks the actor rather than throwing to the caller; the test passes only via the outer worker's catch.- Drain aborts every namespace, and
abort()always closes storage, so in-flight requests fail at drain start, including non-clustered namespaces. - An authenticated peer receives the full debug-port capability, though the cluster only ever uses
getActor. - Peer links: no liveness detection on cached sessions, no minimum key strength, and only the dialer is authenticated.
jasnell
reviewed
Oct 28, 2026
jasnell
approved these changes
Oct 28, 2026
jasnell
left a comment
jasnell
left a comment
Collaborator
There was a problem hiding this comment.
No blocking concerns... handful of things flagged by the AI reviewer.
A namespace's storage decides at link whether its database writes go to the cluster. Workflow namespaces linked before the cluster existed, so they routed through the cluster but replicated nothing, and a takeover restored stale state. The cluster is now made first and set on each workflow namespace before its storage opens.
Drain aborted every namespace, so in-memory ones, which have nothing to hand off, failed their in-flight requests too.
A takeover replaces an object's local database with the bucket's copy, so adding cluster to a config whose localDisk already held Durable Objects deleted each one on first access. A namespace directory is now marked the first time cluster mode uses it, and one that holds databases without the mark is refused at startup.
A restore read less than a damaged file held without failing: - a snapshot's database was copied without checking its length, and the frame was never read to its end, where the LZ4 checksum is checked; - the LZ4 decoder takes a frame cut at a block boundary, or an empty file, for a whole one, so a batch's last commits were dropped; - with no traversal limit, a corrupt message's segment table was allocated at whatever size it claimed, aborting the process. Every file is now checked to be one whole LZ4 frame before it is read, a snapshot must hold exactly its database, and a message is bounded at 1 GiB.
The refusal was thrown from scheduleRun(), which runs when the write commits, after setAlarm() had resolved: the implicit transaction failed, rolling back the writes since the last await and breaking the actor, and the caller could not catch it. ActorSqlite::Hooks gains checkAlarm(), which setAlarm() calls before it changes anything; the default allows every alarm.
The MAC's other inputs cross the wire, so a short or guessable key can be found offline from one captured CONNECT, and the key grants the whole debug port. The check is a config error, so server-test now covers it and the --experimental refusal.
Records, listings and errors are read into memory, so an endpoint that kept sending could exhaust the process before the body was parsed.
The shared dial moves into Sessions, apart from the factory it needs, so its guarantees are tested on their own: concurrent callers share one dial, and a failed dial or a disconnected session is forgotten, so the next route dials again. A failure kept instead would leave the peer unreachable from this node until it restarts.
A Worker whose service name began with workerd: could use experimental compatibility flags without --experimental, and the name is the config's to choose. No built-in service uses the exemption.
A batch that starts past the frame the log has reached is skipped, for a later, wider batch covers it when uploads race. When none does, a batch between is missing, and the restore succeeded with the state before it, which compaction could then make permanent. The replay now remembers the furthest skipped batch and fails unless it got past it.
Handlers given the same cluster key decode each other's channel tokens, and one given another key rejects them.
This reverts 0067094, which added ActorSqlite::Hooks::checkAlarm() to core io/ only to move a temporary error message in experimental code. Alarm support in cluster mode replaces the refusal.
A namespace opened its scheduler's database through a second VFS, to keep it outside the cluster's wrapper. The wrapper needs no exception: the cluster ignores WAL writes of files no object owns, and the scheduler opens its database at startup, under the lease. So a namespace has one VFS again, as outside cluster mode.
flate2 is already a dependency, so lz4_flex and twox-hash go. Gzip's trailer holds the content's CRC-32 and length, which the decoder checks, so a corrupt or truncated file fails on its own, without walking the frame first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the beginning of merging celld's functionality into workerd. It lets several workerd processes network into a cluster by giving each of them access to the same S3-compatible bucket (
Config.cluster,--experimental). Each Durable Object has one owner at a time, requests reach it from any process, and when its owner dies another process picks it up from the bucket. Objects log their commits to the bucket as Cap'n Proto frames of SQLite pages (frames.capnp) and compact the log into snapshots. For now an object acknowledges a write only once it is fully synced to the bucket; that is much too slow and expensive for most uses. Next PRs will ack writes with the cluster and add support for alarms.To try it, add a
clusterblock to the config naming the bucket and a key the nodes share, then start each node with its own address:A request for any Durable Object can go to any node.