Repository navigation
Maintenance: CI on Node 24, SQLite 3.53 in the browser, Freedom Browser support - #1
Merged
Merged
Conversation
Every run warned that setup-node@v4 targets the deprecated Node 20 runtime and was being forced onto Node 24. All five actions move to their current majors (checkout v7, setup-python v7, setup-node v7, upload-artifact v7, download-artifact v8). None of the breaking changes along the way apply: GitHub-hosted runners, python-version is always given, and the dist artifact is downloaded by name. publish.yml keeps its filename and the pypi environment, which is all the trusted-publisher entry pins. The Python matrix stays at 3.11/3.12 by decision. CLAUDE.md still said the swarmfs floor was 0.4.0; it has been 0.7.1 since 0.3.1. serve.mjs pointed at docs/roadmap.md, now ROADMAP.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The browser reader and the JS publisher were running SQLite 3.44.0 from
Nov 2023. The npm package wa-sqlite never moved past 1.0.0, but the
repository did: the v1.1.2 tag carries a dist built on SQLite 3.53.0.
The six vendored files come from that tag, checked against its git blob
hashes; PROVENANCE.md records how, since `npm pack` no longer works.
Not a drop-in. wa-sqlite replaced its VFS base class: the old VFS.Base
(handleAsync, JS-friendly arguments) is now a raw pointer-level Base,
with FacadeVFS as the convenience layer. SwarmVFS and MemVFS now extend
FacadeVFS: xFoo -> jFoo, an `async` jOpen/jRead is what marks a method
asynchronous for the Asyncify relay, mxPathName is spelled mxPathname,
and the constructor takes (name, module). Only open() and prepare()
construct them, so open() callers see no change.
One trap: write buffers now arrive as a proxy that re-acquires the wasm
heap after a resize, not as a Uint8Array. MemVFS.jWrite passing it
straight to Uint8Array.set() copied zeros; the publish test caught it
("file is not a database"), and it now copies from pData.subarray().
Same page economy as before: the offline smoke test reads 5 pages (20 KB)
for a cold lookup and 25 pages over the session on a 140 MB database,
identical to 1.0.0. npm test passes. The build still has no FTS5, so
the User Guide's notes on that stand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live demo root in ROADMAP.md (d1d54686…, 2026-07-25) had expired with its 28-day batch and returned 404. It is republished as 8b08943b… with the wa-sqlite 1.1.2 reader, on a batch that lasts to about 2026-11-09. js/demo/publish_site.py had not followed v2.1: it copied only vendor/wa-sqlite, while src/feeds.js and src/verify.js import js-sha3 and noble-secp256k1, so a site built with it would fail to load its modules in a browser. It now copies the same four vendor dirs as the WordPress example. Checked before uploading: every import reachable from app.js resolves inside the assembled site. Verified after publishing, through the local Bee 2.8.3 node: - the Node smoke test: 5 reads / 20 KB cold and 23 pages / 92 KB per session, identical to July's measurement; - `swarmlite query --verify` returns the right row via the BMT walk; - headless Chromium loads the page and runs a search (9 and 12 pages fetched), with Bee serving text/javascript and application/wasm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Freedom Browser blocks web content from the node's HTTP API; its injected window.swarm provider is the only route a page has. open() now detects it (window.swarm.isFreedomBrowser) and reads the database chunk by chunk with swarm_readChunk: the same manifest lookup and tree walk as verify: true, with VerifiedChunkStore taking a provider in place of a gateway URL. Freedom verifies each chunk against its address; we re-check the BMT hash anyway, rebuilding the exact chunk from the base64 payload and the span, so the provider doesn't have to be trusted either. It accepts the page's own relative URL, which resolves to bzz://<root>/<path> inside Freedom. `swarm: null` opts out, any window.swarm-style provider can be passed explicitly, and `transport` on the result says which path was taken. The demo shows "via window.swarm" in its status line when it applies. Read budget: Freedom allows 120 public reads / 512 KiB per origin per minute (600 / 5 MiB once connected). Measured against the published demo with a provider stand-in: open plus the first screen is 18 chunk reads (65 KB), a search 16 more. A rate_limited error names the budget. Tests (offline, swarmfs fixtures): auto-detected provider over a bzz:// URL returns the right row with every read a swarm_readChunk and none over HTTP; swarm: null opts out; a tampered chunk from the provider is caught; a bzz:// URL without a provider fails loudly. Feeds still resolve over HTTP only (resolveFeed); a window.swarm path for them would go through swarm_readSingleOwnerChunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 8b08943b… root predates the window.swarm transport. 98863bc7… carries it; demo.db is the same file reference (470308af…), so only the JS and manifest were new to the network. Checked in Freedom Browser (source tree, Playwright through its own live fixtures) on its bundled Ant node at :1635, not the Bee on :1633: the page loads from bzz://, reports "via window.swarm", renders the latest posts (9 pages, 52 KB, 5.9 s) and a search (12 pages, 64 KB, 8.0 s). Per-query time is dominated by one-at-a-time chunk reads; noted as the next step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Asked to make the window.swarm reads parallel, I traced them first with a stand-in provider adding 300 ms per read (roughly Freedom on a cold Ant node). Every read was serial (peak 1 in flight), and SQLite asked for the session's 22 pages in 22 single-page reads, so no read ever needed two chunks at once. The 12 reads beyond one per page were intermediate chunks plus readTree's first-child probe, a full round trip spent only to learn a subtree's child span. - readTree memoizes that span per intermediate span (raw 8 bytes, which carry size and redundancy level; a Bee tree's shape is fixed by both), so each full subtree size is probed once, not once per subtree. The existing per-child span checks still guard every step. - The children a range needs are fetched and walked concurrently, each writing its own slice of the output. - VerifiedChunkStore caps downloads at maxConcurrency (default 8; a done download hands its slot straight to the next waiter) and merges concurrent requests for the same chunk. - SwarmVFS `readahead: n` (open() passes it through, default 0) widens a page miss by up to n uncached neighbours on each side in the same read, which the verified path then fetches in parallel. Measured at 300 ms per read (open / latest / search): before 5+13+16 = 34 reads 1.74 / 4.14 / 4.93 s after 5+11+14 = 30 reads 1.71 / 3.53 / 4.32 s readahead: 2 82 reads, peak 5 2.05 / 2.34 / 4.10 s readahead: 8 130 reads, peak 8 2.06 / 2.68 / 3.84 s Readahead nearly triples the reads for a 5-34% gain, and that would use up Freedom's unconnected budget (120 reads a minute), so it stays opt-in. Most of the time is SQLite descending B-trees across distant pages, one page per round trip; docs say so instead of promising more. Tests: a full read through a slow provider runs chunks in parallel and never above the cap (peak exactly 2 of 2); readahead returns the right row, fetches more pages than reads, with chunks in flight together. The HTTP path is unchanged at readahead 0: smoke test offline and live gives the same 5 reads / 20 KB cold, 23 pages per session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reported from Freedom: in the demo, searches for "query", "root" and "stamp" worked, "page" and "feed" did nothing. Not the data (every word returns 10 rows in Node, over HTTP and window.swarm alike) and not retrieval (all 51 chunks those searches touch come back from the public gateway). It was the search box: it fires 250 ms after each keystroke and never waited for the previous search, and through window.swarm a search takes seconds. Reproduced with a 300 ms provider stand-in: "pag" then "page" 400 ms later gave "database disk image is malformed" for "page". Overlapping opens crash outright, with a wasm trap in Asyncify's rewind. Which word failed only depended on typing speed. wa-sqlite's Asyncify build can only suspend one call at a time, and every open() shares one module, so the lock is module-wide: open_v2 (with vfs_register), query() and close() all chain on one promise queue. The raw sqlite3/db handles bypass it, as the docstring now says. Over HTTP the same race existed, just with a ~150 ms window. The demo also runs one query at a time, newest term wins (terms typed past while one runs are skipped instead of spending Freedom's per-origin read budget), and a failed query now shows in #err instead of vanishing as an unhandled rejection. Test: two opens and three queries started together through a slow provider all succeed with the right rows. Before the fix the same test crashed the process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f54bac5b… replaces 98863bc7…, whose search box could start a query while another was running and fail with "database disk image is malformed". Typing query/page/feed in Freedom at one key per 400 ms (slower than the 250 ms debounce) now returns 10 posts for each, with no errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Maintenance pass from 2026-10-20 (Bee 2.8.3), plus Freedom Browser support for the JS reader. It grew out of the dependency check, so there are several parts, each its own commit.
What changed
CI (
8ef9a12): the actions move off the deprecated Node 20 runtime (checkout v7, setup-python v7, setup-node v7, upload-artifact v7, download-artifact v8). No breaking change applies: hosted runners,python-versionalways set, artifact downloaded by name.publish.ymlkeeps its filename and thepypienvironment, which is all the trusted-publisher entry pins. The Python matrix stays at 3.11/3.12. Also fixes two stale doc pointers (the swarmfs floor in CLAUDE.md, a roadmap path inserve.mjs).Browser SQLite 3.44 → 3.53 (
19ea4bf): vendored wa-sqlite 1.0.0 → 1.1.2. npm still stops at 1.0.0 (Jan 2024), so the files come from the GitHub tag, checked against its git blob hashes. Not a drop-in: wa-sqlite replaced its VFS base class, soSwarmVFSandMemVFSnow extendFacadeVFS. Breaking for code that constructsSwarmVFSdirectly: it is nownew SwarmVFS(name, module, options).open()callers are unaffected.Freedom Browser (
d7a58af): there,open()reads through the injectedwindow.swarmprovider (swarm_readChunk), since that is the only route a page has to the node. Every chunk is still checked against its hash client-side. It accepts the page's ownbzz://<root>/<path>URL,swarm: nullopts out, andtransportreports the path taken.Parallel verified reads (
805e980): the chunks a read needs are fetched concurrently (at most 8 at a time, duplicate requests merged), and the tree walk remembers each subtree's child span instead of fetching an extra chunk to learn it. Newreadaheadoption, off by default. Measured honestly: SQLite asks for one page per read, so the default gain is about 15% (34 → 30 chunk reads for the demo session). Readahead nearly triples the reads for a 5–34% gain.Overlapping queries (
9ab39fd): fixes a bug reported from Freedom. A query started while another was waiting on a page read made SQLite read the wrong pages ("database disk image is malformed", or a crash inside wasm). wa-sqlite's async build can only pause one call at a time, across every open database, soopen,queryandclosenow queue. The demo search box made this happen easily in Freedom, where a search takes seconds. The demo now runs only the newest search term and shows query errors.Demo (
5473529,95fda11,143e99d): the old demo root (d1d54686…) had expired with its batch. Republished asf54bac5b74d06e0480c297174224dc86c34e2a41042bf1828cc78d56c92c1f10, which expires with batchc931c8a5…around 2026-11-09 unless topped up.js/demo/publish_site.pynow copies every vendor directory the reader imports; it had missed js-sha3 and noble-secp256k1 since v2.1.Testing
npm testpasses, with new offline tests forwindow.swarmreads (none over HTTP, tampered chunk caught, opt-out), the concurrency cap, readahead, and overlapping opens and queries. The overlap test crashed the process before the fix.Not in this PR
SwarmVFSconstructor change) call for an npm release when you choose; nothing Python-facing changed.resolveFeed) still goes over HTTP; awindow.swarmpath would useswarm_readSingleOwnerChunk.swarm_readDatain Freedom itself.🤖 Generated with Claude Code