Skip to content

Write a point-in-time Cyd Bluesky archive v2 export - #112

Merged
micahflee merged 2 commits into
mainfrom
bluesky-archive-v2-export
Sep 11, 2026
Merged

micahflee merged 2 commits into
mainfrom
bluesky-archive-v2-export

Conversation

@micahflee

@micahflee micahflee commented Sep 11, 2026 •

Copy link
Copy Markdown
Member

Closes #98.

Mobile could not turn Bluesky saved data into a Cyd Bluesky archive at all, so
every reader issue had only synthetic fixtures to work against. This builds the
version 2 writer and uses it to produce the first real-data fixtures.

How an export works

Four steps, and the order is the design:

  1. Stage — pause account-mutating work, VACUUM INTO a copy of the account
    database, stat every preserved file. This is the moment the archive
    describes (ADR 0010). The pause covers only this, never hashing or packaging.
  2. Hash — with saving running again, read every inventoried file. Anything
    that disagrees with the inventory becomes an honest asset state rather than a
    silent omission: gone is unavailable, never downloaded is missing, and
    either one makes the archive incomplete.
  3. Translate — Mobile's private rows into the Bluesky interchange model
    (ADR 0002), in a pure function over rows and asset results.
  4. Package — stream media, data.db, metadata, and manifest into the ZIP.

Nothing reads the live account after step 1, so however long the rest takes, the
archive stays internally consistent.

Decisions worth a look

  • Translation invents nothing. Mobile does not store when a like was
    created, a bookmark's own record URI, or a reply parent it never saved. Those
    become observation times, selections naming the post, and context carrying
    only the author DID the AT URI already names. The full gap list is in
    docs/bluesky-archive-fixtures.md and reprinted by the readiness script.
  • Media is stored, data.db is deflated. JPEG and MP4 do not compress, and
    a streaming deflate cannot know its size before it finishes — on a phone that
    would mean buffering a whole video. The database is compressed into a second
    staged file first, so its empty SQLite pages cost 4 KiB instead of 144 KiB.
  • The pinned schema.sql is embedded verbatim, because a phone cannot fetch
    the contract mid-export. npm run test:archive-contract compares the copy
    against the pinned bundle character-for-character so it cannot quietly fork.
  • Export is behind __DEV__ until Pass bidirectional Cyd Bluesky archive v2 conformance and enable the writer #100 proves conformance both ways
    (ADR 0004), with a test that fails if the affordance ever renders in a
    release build.

Fixtures

testUtils/fixtures/bluesky-archive/{complete,incomplete}.cyd, produced by this
writer from a curated test account (ADR 0016). The pair differs by exactly one
asset — the largest image, failed for one export — and by the completeness that
follows. fixtures.test.ts enforces that, plus the 5 MiB budget and the set
of handles in profiles; a budget in prose does not survive a regeneration.
Provenance and regeneration steps are in the fixtures README.

Three caveats

  • Not every deletion state survives. A bookmark removed at source, and a
    like removed at source whose URI Cyd never saw, have no record in the
    interchange model to carry that state. Documented rather than silently lost.
  • The fixture account was curated on a pulled copy, not on the device. The
    account held 45 MiB of video across two posts, one of them a stranger's; that
    and every post written by somebody else were removed from the copy before
    export. This curates the export's input rather than trimming the archive, so
    it honours ADR 0016's reasoning, but not its letter.
  • That curation removed third-party records, not third-party content nested
    inside the account's own.
    Both fixtures keep one quote post of the
    account's whose payload_json carries the hydrated view it quoted:
    bsky.app's post, and nested within that a second post by
    barmas.bsky.social — display names, avatar URLs, engagement counts, a
    mention of @brimcrob.bsky.social, and reply URIs naming a fourth DID. It is
    all public Bluesky content and it stays, because removing it would mean
    removing the quote. Note that fixtures.test.ts reads profiles.handle
    only, so it does not check payloads for this.

The fixtures therefore cover posts, reposts, likes, chats, and images, but not
video, replies, link previews, or bookmarks. The canonical bundle still covers
those, and it remains the semantic oracle either way (ADR 0008).

Verification

  • npm test (785), npm run lint, npm run test:archive-contract,
    npm run test:archive-conformance, check_latest.py — all pass.
  • conformance.py accepts archives this writer produces, checked against the
    pinned contract rather than against Mobile's own idea of the format.
  • Mobile's own intake takes its own output back in, including the deflated
    database.

Not in scope

Resumability, staging cleanup, the plaintext warning, and the shippable export
flow are #99. Enabling export for people is #100.

(This was written by an LLM.)

🤖 Generated with Claude Code

micahflee and others added 2 commits September 10, 2026 16:29
Mobile had no way to turn Bluesky saved data into a Cyd Bluesky archive, which
made every reader issue depend on synthetic fixtures. ADR 0004 already says
building the writer is ungated and only shipping it waits on #100, so this
builds it: staging, translation, packaging, and a development-only affordance.

The order of the four steps is the design. Staging pauses account-mutating work
just long enough to VACUUM INTO a copy of the account database and stat every
preserved file, which is the moment the archive describes (ADR 0010). Hashing
runs with saving going again, so a file that disagrees with the inventory is
caught rather than trusted: gone becomes `unavailable`, never downloaded
becomes `missing`, and either makes the archive honestly incomplete instead of
quietly short a payload. Translation turns Mobile's private rows into the
Bluesky interchange model (ADR 0002); packaging streams the result into a ZIP.

Translation invents nothing. Mobile does not store when a like was created, or
a bookmark's own record URI, or the reply parent it never saved, so those come
out as observation times, selections that name the post, and context carrying
only the author DID the AT URI already names. The fixtures doc and
check_account_readiness.py list all of it, because a gap nobody wrote down
reads as a bug later. Where a link preview's thumbnail was preserved, it is
packaged as a `preview` asset rather than left behind as a URL.

Two rules keep the archive safe to hand to someone. Nothing writes a foreign
key it cannot satisfy — a repost of a post Cyd never saved loses that one link
instead of failing the export — and no unavailability reason repeats a storage
error's own message, because those name the device path they failed on.

Media is stored uncompressed — JPEG and MP4 do not compress, and a streaming
deflate cannot know its size before it finishes, which on a phone would mean
buffering a whole video. `data.db` is deflated into a second staged file first,
since it is mostly empty SQLite pages: 144 KiB becomes 4 KiB, and committed
fixture bytes live in repository history forever.

The writer carries the pinned schema.sql verbatim, because a phone cannot fetch
the contract mid-export; test:archive-contract compares the two so the copy
cannot quietly fork. conformance.py passes on archives this produces, and
Cyd's own intake reads them back.

Export is behind `__DEV__` until #100 (ADR 0004). Resumability, staging
cleanup, and the plaintext warning are #99's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#98 asks for committed `complete` and `incomplete` Cyd Bluesky archives made by
exporting a real Bluesky test account through Mobile's own version 2 writer
(ADR 0016), so that every reader issue has archives carrying real facets, CIDs,
image bytes, and identifiers rather than only synthetic ones.

These come from glittertop-cyd.bsky.social, curated first. The account held
48 MiB, 45 MiB of which was two videos — one of them a stranger's, pulled in by
a like. Both videos and every post written by somebody else are gone, which
costs the fixtures video, reply, and link-preview coverage and buys a 3.3 MiB
pair that is entirely the maintainer's own accounts. Likes and reposts survived
because the account had liked and reposted its own posts. The conversation was
already between two accounts the maintainer controls.

The pair differs by exactly one asset — the largest image, failed for the
length of one export — and by the completeness that follows from it. Getting
there needed `--created-at`: an export stamps anything it captured that has no
time of its own, such as a profile known only by its DID, with the moment it
ran, so the two otherwise drifted apart by the clock.

fixtures.test.ts keeps both claims true as the code moves: Mobile's intake
still reads them, the pair still differs by one asset, no handle but the
maintainer's own appears, and the pair stays inside the 5 MiB budget. A budget
in prose does not survive a regeneration; committed media persists in history
forever.

Making these also caught a documentation bug. Nothing writes Mobile's `follow`
table — the unfollow-everyone job fetches follows live — so `relationships` is
always empty, and the readiness check was reporting "no follows saved" as a
problem to fix rather than as a gap nobody can close. It says so now, and the
fixtures doc no longer advises pointing a follow at an account you control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@micahflee
micahflee merged commit 9830981 into main Sep 11, 2026
1 check passed
@micahflee
micahflee deleted the bluesky-archive-v2-export branch September 11, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a point-in-time Cyd Bluesky archive v2 export

1 participant