Skip to content

Check v2 archives against the contract before a reader can - #110

Merged
micahflee merged 1 commit into
mainfrom
prereqs-for-v2-export
Sep 10, 2026
Merged

micahflee merged 1 commit into
mainfrom
prereqs-for-v2-export

Conversation

@micahflee

Copy link
Copy Markdown
Member

Prerequisites for #98, so the first real-data fixtures can be trusted when they land. No product code — Python tooling, a runbook, three npm scripts, one CI step.

Why

Mobile builds its v2 writer before the reader that consumes it (ADR 0004), which leaves a gap: until #96 lands, nothing in the app can tell a correct archive from a plausible one. #98 would produce the first real-data fixtures with no way to know they were right, and a wrong fixture committed early is one every later reader issue is then built against.

What's here

File Role
conformance.py Checks any .cyd against the pinned contract — layout, entry types, manifest both directions, metadata vs the archive row, exact schema.sql match, foreign keys, asset digests and byte counts, orphan payloads, completeness iff every asset is available
self_test.py Mutates the canonical fixture twelve ways; each must be rejected
check_account_readiness.py Reports whether a pulled account database is ready to export
docs/bluesky-archive-fixtures.md The runbook for steps only a person with the device can do
bundle.py Pin-loading and download code test_bundle.py already had
npm run check:archive-conformance -- path/to/x.cyd   # ad-hoc, on your export
npm run check:account-readiness -- /tmp/data.db      # before exporting
npm run test:archive-conformance                     # now runs in CI

The self-test earned its place immediately

It caught a bug in the checker on first run: the entry-type check rejected any entry whose mode carried no file-type bits, which is what zipfile.writestr and plenty of real writers emit. Every mutation "failed" for that one wrong reason, including ones it wasn't testing — and it would have rejected valid archives, quite possibly the exporter's own output.

Cases are resealed with a recomputed manifest, because that's how a broken writer actually fails: internally consistent, and wrong. Without resealing, the manifest digest catches everything first and the schema and completeness checks never run at all.

Worth a second look

  • check_account_readiness.py is not verified against a real device. It's smoke-tested against a database built from the CREATE TABLE statements extracted from bluesky-migrations.ts, in both a ready and a failed-download state. The device paths in the runbook are what BaseAccountController.buildAccountPaths and device-storage.ts imply; I couldn't confirm them on hardware.
  • The known-gaps table appears twice, in the runbook and reprinted by the readiness script. Deliberate — it's the thing most likely to be forgotten at the moment a fixture is being made, and it decides how much of Restore a complete Bluesky local account for offline browsing #96 these fixtures can actually cover.
  • Python has no test convention in this repo, so self_test.py is a script with an exit code rather than a pytest suite, matching test_bundle.py.
  • The 5 MiB budget in the runbook is a judgement call, not derived from anything. Canonical complete.cyd is 144 KiB, but it carries 1×1 PNGs where these will carry real JPEGs and an MP4.

npm run lint is clean; 725 tests pass; test:archive-contract and test:archive-conformance both pass.

🤖 Generated with Claude Code

Mobile now builds its version 2 writer before the reader that consumes it
(ADR 0004), which leaves a gap: until #96 lands, nothing in the app can tell a
correct archive from a plausible one. #98 would produce the first real-data
fixtures with no way to know they were right, and a wrong fixture committed
early is one every later reader issue is then built against.

conformance.py closes that gap. It checks any .cyd against the pinned canonical
schema rather than against Mobile's own idea of the format: ZIP layout and entry
types, manifest reconciliation in both directions, metadata shape and its
agreement with the archive row, an exact match against schema.sql, foreign keys,
asset digests and byte counts, orphan payloads, and completeness being true if
and only if every asset is available.

A checker that passes everything would be worse than none, so self_test.py
mutates the canonical fixture twelve ways and requires each to be rejected. It
earned its place immediately by catching a bug here: the entry-type check
rejected any entry whose mode carried no file-type bits, which is what
zipfile.writestr and plenty of real writers emit. Every case "failed" for that
one wrong reason. Cases are resealed with a recomputed manifest, because a
broken writer fails while internally consistent, and without resealing the
manifest digest catches everything before the deeper checks ever run.

check_account_readiness.py covers the other half of the prerequisite, the part
only a person with the device can do. Media downloads finish after records
land, and a pending download becomes an unavailable asset — so an account that
looks saved exports as `incomplete` while claiming to be the `complete`
fixture. It also reprints the interchange groups Mobile cannot populate at all,
so the fixture's real coverage is visible at the moment it is being made rather
than only in the docs.

bundle.py is the pin-loading and download code test_bundle.py already had.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@micahflee
micahflee merged commit 4548e58 into main Sep 10, 2026
1 check passed
@micahflee
micahflee deleted the prereqs-for-v2-export branch September 10, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant