Repository navigation
Write a point-in-time Cyd Bluesky archive v2 export - #112
Merged
Merged
Conversation
Mobile had no way to turn Bluesky saved data into a Cyd Bluesky archive, which made every reader issue depend on synthetic fixtures. ADR 0004 already says building the writer is ungated and only shipping it waits on #100, so this builds it: staging, translation, packaging, and a development-only affordance. The order of the four steps is the design. Staging pauses account-mutating work just long enough to VACUUM INTO a copy of the account database and stat every preserved file, which is the moment the archive describes (ADR 0010). Hashing runs with saving going again, so a file that disagrees with the inventory is caught rather than trusted: gone becomes `unavailable`, never downloaded becomes `missing`, and either makes the archive honestly incomplete instead of quietly short a payload. Translation turns Mobile's private rows into the Bluesky interchange model (ADR 0002); packaging streams the result into a ZIP. Translation invents nothing. Mobile does not store when a like was created, or a bookmark's own record URI, or the reply parent it never saved, so those come out as observation times, selections that name the post, and context carrying only the author DID the AT URI already names. The fixtures doc and check_account_readiness.py list all of it, because a gap nobody wrote down reads as a bug later. Where a link preview's thumbnail was preserved, it is packaged as a `preview` asset rather than left behind as a URL. Two rules keep the archive safe to hand to someone. Nothing writes a foreign key it cannot satisfy — a repost of a post Cyd never saved loses that one link instead of failing the export — and no unavailability reason repeats a storage error's own message, because those name the device path they failed on. Media is stored uncompressed — JPEG and MP4 do not compress, and a streaming deflate cannot know its size before it finishes, which on a phone would mean buffering a whole video. `data.db` is deflated into a second staged file first, since it is mostly empty SQLite pages: 144 KiB becomes 4 KiB, and committed fixture bytes live in repository history forever. The writer carries the pinned schema.sql verbatim, because a phone cannot fetch the contract mid-export; test:archive-contract compares the two so the copy cannot quietly fork. conformance.py passes on archives this produces, and Cyd's own intake reads them back. Export is behind `__DEV__` until #100 (ADR 0004). Resumability, staging cleanup, and the plaintext warning are #99's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#98 asks for committed `complete` and `incomplete` Cyd Bluesky archives made by exporting a real Bluesky test account through Mobile's own version 2 writer (ADR 0016), so that every reader issue has archives carrying real facets, CIDs, image bytes, and identifiers rather than only synthetic ones. These come from glittertop-cyd.bsky.social, curated first. The account held 48 MiB, 45 MiB of which was two videos — one of them a stranger's, pulled in by a like. Both videos and every post written by somebody else are gone, which costs the fixtures video, reply, and link-preview coverage and buys a 3.3 MiB pair that is entirely the maintainer's own accounts. Likes and reposts survived because the account had liked and reposted its own posts. The conversation was already between two accounts the maintainer controls. The pair differs by exactly one asset — the largest image, failed for the length of one export — and by the completeness that follows from it. Getting there needed `--created-at`: an export stamps anything it captured that has no time of its own, such as a profile known only by its DID, with the moment it ran, so the two otherwise drifted apart by the clock. fixtures.test.ts keeps both claims true as the code moves: Mobile's intake still reads them, the pair still differs by one asset, no handle but the maintainer's own appears, and the pair stays inside the 5 MiB budget. A budget in prose does not survive a regeneration; committed media persists in history forever. Making these also caught a documentation bug. Nothing writes Mobile's `follow` table — the unfollow-everyone job fetches follows live — so `relationships` is always empty, and the readiness check was reporting "no follows saved" as a problem to fix rather than as a gap nobody can close. It says so now, and the fixtures doc no longer advises pointing a follow at an account you control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #98.
Mobile could not turn Bluesky saved data into a Cyd Bluesky archive at all, so
every reader issue had only synthetic fixtures to work against. This builds the
version 2 writer and uses it to produce the first real-data fixtures.
How an export works
Four steps, and the order is the design:
VACUUM INTOa copy of the accountdatabase, stat every preserved file. This is the moment the archive
describes (ADR 0010). The pause covers only this, never hashing or packaging.
that disagrees with the inventory becomes an honest asset state rather than a
silent omission: gone is
unavailable, never downloaded ismissing, andeither one makes the archive
incomplete.(ADR 0002), in a pure function over rows and asset results.
data.db, metadata, and manifest into the ZIP.Nothing reads the live account after step 1, so however long the rest takes, the
archive stays internally consistent.
Decisions worth a look
created, a bookmark's own record URI, or a reply parent it never saved. Those
become observation times, selections naming the post, and context carrying
only the author DID the AT URI already names. The full gap list is in
docs/bluesky-archive-fixtures.mdand reprinted by the readiness script.data.dbis deflated. JPEG and MP4 do not compress, anda streaming deflate cannot know its size before it finishes — on a phone that
would mean buffering a whole video. The database is compressed into a second
staged file first, so its empty SQLite pages cost 4 KiB instead of 144 KiB.
schema.sqlis embedded verbatim, because a phone cannot fetchthe contract mid-export.
npm run test:archive-contractcompares the copyagainst the pinned bundle character-for-character so it cannot quietly fork.
__DEV__until Pass bidirectional Cyd Bluesky archive v2 conformance and enable the writer #100 proves conformance both ways(ADR 0004), with a test that fails if the affordance ever renders in a
release build.
Fixtures
testUtils/fixtures/bluesky-archive/{complete,incomplete}.cyd, produced by thiswriter from a curated test account (ADR 0016). The pair differs by exactly one
asset — the largest image, failed for one export — and by the completeness that
follows.
fixtures.test.tsenforces that, plus the 5 MiB budget and the setof handles in
profiles; a budget in prose does not survive a regeneration.Provenance and regeneration steps are in the fixtures README.
Three caveats
like removed at source whose URI Cyd never saw, have no record in the
interchange model to carry that state. Documented rather than silently lost.
account held 45 MiB of video across two posts, one of them a stranger's; that
and every post written by somebody else were removed from the copy before
export. This curates the export's input rather than trimming the archive, so
it honours ADR 0016's reasoning, but not its letter.
inside the account's own. Both fixtures keep one quote post of the
account's whose
payload_jsoncarries the hydrated view it quoted:bsky.app's post, and nested within that a second post bybarmas.bsky.social— display names, avatar URLs, engagement counts, amention of
@brimcrob.bsky.social, and reply URIs naming a fourth DID. It isall public Bluesky content and it stays, because removing it would mean
removing the quote. Note that
fixtures.test.tsreadsprofiles.handleonly, so it does not check payloads for this.
The fixtures therefore cover posts, reposts, likes, chats, and images, but not
video, replies, link previews, or bookmarks. The canonical bundle still covers
those, and it remains the semantic oracle either way (ADR 0008).
Verification
npm test(785),npm run lint,npm run test:archive-contract,npm run test:archive-conformance,check_latest.py— all pass.conformance.pyaccepts archives this writer produces, checked against thepinned contract rather than against Mobile's own idea of the format.
database.
Not in scope
Resumability, staging cleanup, the plaintext warning, and the shippable export
flow are #99. Enabling export for people is #100.
(This was written by an LLM.)
🤖 Generated with Claude Code