Repository navigation
PE-9205: The drive state artifact as a SQLite file (D12 counter-proposal) - #2197
Merged
Merged
Conversation
A published, encrypted, parsed-state blob a client imports in bulk rather than replaying a drive's history entity by entity. Additive: snapshots stay the ArFS interchange format and the fallback, and a client that does not understand an artifact syncs exactly as it does today. The case is measured rather than assumed. Consuming a snapshot costs one AES decryption, one JSON parse and one insert *per entity* (file_entity.dart:117) - ~42,000 of each on the drive we have been testing, which is the ~80s of chunk processing in its sync log. An artifact costs one decryption and a bulk insert. It is also smaller: a 42k-file database built and weighed is 31.31 MiB, 6.41 MiB gzipped, against 43.92 MiB and ~11.37 MiB for the snapshot covering the same drive. And producing it is cheap in the way snapshots are not - it is already in the local database, so there is no chain re-read. The security section leads because the obvious implementation does permanent harm. The local database holds profiles.encryptedWallet with its keySalt, and drives.encryptedKey with its IV; serialising the database would publish a complete offline attack package on the user's password, permanently, with no delete. Hence export from an allowlist rather than dump, guarded by a test that fails when the schema grows a new table or column. Also: authenticated encryption is mandatory and the codebase's GCM boundary is 100 MiB, which a ~130k-file drive would cross into unauthenticated CTR; and the artifact should be serialised rows rather than a .db file, both to avoid handing an untrusted database to SQLite's parser and to avoid welding the wire format to schemaVersion. Covers how it composes with sync (one more obscuring range - HeightRange needs no change), what ardrive-core-js and the CLI would need, and ArNS as the discovery mechanism, which is already-built plumbing: ArnsRepository points an undername at an arbitrary txId today. No code. Open questions are listed rather than answered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
Reworked so it reads as a standard addition rather than an app feature. - The export is now guarded by schema views rather than an allowlist plus a test. A view names its columns, so a later migration widening `drives` cannot reach the export; the exporter queries views only and is structurally unable to see `encryptedKey` or `keySalt`. Separating key material into its own store was considered and rejected as a dependency: it migrates wallet material out of a live database whose fixtures are stale at v19, and still leaves one drive exported from a database holding all of them. Worth doing on its own security merits, separately. - States the governing principle up front: this is a cache and every failure is a fallback. Unknown version, failed decryption, failed integrity, unexpected row - all mean sync normally, never fail the drive. That is what makes it foolproof; the worst outcome is today's speed. - The entity is specified in ArFS's own conventions, matching how entity-types.mdx documents Snapshot: a `Drive-State-Id` beside `Snapshot-Id`, the same `Block-Start`/`Block-End` meaning, and `Cipher`/`Cipher-IV` per privacy.mdx. - Adds what ar-io-docs needs: a Drive State section in entity-types.mdx, plus data-model, reading-data and privacy updates. It also records a bug found while checking the spec against reality. The published Snapshot format says the metadata field is `dataJson`, in both prose and example; the implementation reads and writes `jsonMetadata`, and a real snapshot contains 189 `jsonMetadata` and zero `dataJson`. Anyone implementing a reader from the docs would get no metadata from any snapshot and silently fall back to fetching every transaction one at a time - the exact failure this work exists to remove. Flagged for its own fix rather than bundled here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
…posal PE-9205 Rewritten after working through discovery, encryption and metadata. The substantive changes: - Corrected sizes. The first model left `bundledIn` null when 188 of 189 real nodes carry it, and assumed a uniform entity mix. Remodelled on the snapshot's actual composition the artifact is 34.63 MiB, 6.65 MiB gzipped, against 43.92 and 11.37 - 1.27x and 1.71x, not the 1.40x and 1.77x first claimed. The size argument is correspondingly demoted: it is the weakest of the three reasons, not the headline. - Privacy is a gain, not a cost, which the first draft had backwards. A snapshot of a private drive is not encrypted: Content-Type is application/json, there is no Cipher tag, and only the per-entity metadata values inside are ciphertext. Every file id, parent folder id and timestamp is public today. A fully encrypted artifact leaks none of it. - The payload is signed by the drive owner. This is what lets discovery be cheap without being credulous: a name is a mutable pointer and proves nothing, so verification cannot live in the discovery path. It also fixes a practical gap - a bundled data item has no L1 header, so `GET /tx/<id>` 404s and its owner is otherwise knowable only through the GraphQL indexer. - ArNS becomes the primary discovery path rather than an accelerator, with GraphQL as fallback. The indexer is the least reliable component in the stack - it is what rate limits and what truncated a drive list under an open circuit breaker - and an artifact should not need it to be found. - Both transports are required, neither may become one. Turbo is a hosted service, and a protocol-level feature that can only be produced through it would depend on one operator staying available. A wallet-only user must be able to publish and read with no Turbo involvement. The signature is what makes L1 and bundled items verify identically. - AES-GCM, never CTR: CTR's advantages are streaming and random access, neither of which applies to a buffered bulk import, and its lack of authentication is not theoretical when gateways have been observed serving truncated data. - Completed the tag set against the ArFS standard - ArFS, Content-Type, Unix-Time and Data-Start/Data-End were all missing - and added Content-Encoding and Entity-Count, the latter as an integrity check rather than a statistic. - Added extensibility (named sections; State-Version bumps only when an older reader would misinterpret, never for additions) and observability, whose rule is that "no artifact used" and "artifact rejected because X" must never look the same in the logs. That distinction is exactly what this month's diagnosis cost days for the want of. - Stated why a new Entity-Type is required rather than a snapshot variant: reusing it would have old clients claim the artifact's block range from its tags, yield nothing from a body they cannot parse, and silently skip those entities while the watermark advances. Every code and spec citation re-verified; corrected one line number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
The document implied it and never said it, which is a real gap when the same file recommends *incremental* snapshots two sections later - a reader could reasonably assume artifacts work the same way. They deliberately do not, and the reasoning inverts. A snapshot is expensive to produce because it re-reads the chain, so increments are what keep production affordable. An artifact is a local database export, so production was never the expensive part and the only cost is the upload. That buys back the property worth more than size: it is self-contained. Any single artifact suffices - no ancestor to locate, none that has to still be retrievable, and no way for one missing link to invalidate a chain, which is exactly the residual risk hanging over snapshot chains. Recorded as policy rather than limitation: `Block-Start` is a tag, not a constant, and the range composition already handles sources covering different spans, so an incremental artifact is just one with a non-zero start. Revisitable without a format break when a drive is large enough that republishing whole state costs real money - which also makes the open question on revision depth load-bearing, since that is the lever keeping a full copy affordable as a drive grows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
Eight decisions closing docs/DRIVE_STATE_ARTIFACT.md section 10, so parallel lanes share one answer instead of each inventing their own. Recorded as the coordinator's calls for review rather than settled protocol - all are reversible before anything is published to chain. Two are deferrals rather than answers, and are marked as such: database views for export safety (a schema migration on a database whose fixtures are stale at v19 is the wrong thing to land unattended - a typed projection plus a drift test is functionally equivalent today) and ArNS discovery (needs an undername convention checked against the ARIO spec). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
The export layer of the drive state artifact (docs/DRIVE_STATE_ARTIFACT.md): a drive's current state as an object graph, the D1 section format, and the exact inverse. - read file_entries, folder_entries and the drive row for one drive only, current state and not revisions (decision D2) - project every table column-wise so encryptedKey, driveKeyGenerated and keyEncryptionIv have no path out of the database, and profiles is never referenced (decision D7, proposal 2.1) - assert the projection against the live Drift schema, so a migration that adds a column to an exported table fails the suite until someone says whether it may be published - order rows by id, because the payload is signed and a signature over a non-deterministic encoding cannot be reproduced - skip unknown sections and fields on read, and reject only a version an older reader would misread (proposal 6), with enumerated failure reasons so a rejection never looks like an absence (proposal 7) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
…E-9205 The envelope and the entity only. No export, no upload, nothing published. - `lib/drive_state/domain/drive_state_envelope.dart` seals a payload in the order the proposal fixes (3.3): sign the plaintext with the owner's wallet, gzip, then AES256-GCM under the drive key. Signing the plaintext rather than the encoding is what lets a bundled artifact, which has no L1 header and therefore no knowable owner outside a GraphQL indexer, verify identically to a top level one (2.2). Opening runs the exact inverse and checks the signature against the drive's known owner. - Neither direction throws. Every way either can fail is an enumerated `DriveStateEnvelopeFailure`, so a caller logs a reason and syncs normally rather than failing a drive over a bad artifact (DECISIONS.md, non-negotiable 2). A bad signature and an artifact signed by somebody else are separate reasons, because they mean different things. - D5 is enforced at the same boundary the uploader uses to choose GCM over CTR: a payload at or above `maxSizeSupportedByGCMEncryption` is refused with a reason, never encrypted with CTR and never split. Reading refuses a `Cipher` tag that is not AES256-GCM for the same reason. - The signature has to travel with the payload it covers, so the compressed body is a small self-describing frame carrying the owner key, the signature and the payload. All big-endian, nothing 64 bit, because `ByteData` has no `getUint64` on the web. - `lib/drive_state/domain/drive_state_entity.dart` carries the 3.2 tags, modelled on `SnapshotEntity` down to its conventions. `Block-Start` is 0: every artifact is a full copy that supersedes every earlier one (3.4). It is a distinct `Entity-Type`, not a snapshot variant, because reusing "snapshot" would have an older client claim this entity's block range from its tags and then sync nothing for it (3.1). - The tag names the entity introduces are added to `EntityTag` and friends the way the existing ones are. Tests are behavioural and were mutation checked: removing the size guard, loosening it to strictly greater, ignoring the verification result, ignoring the owner check, accepting any cipher, dropping the magic bytes check and dropping the Block-Start tag each fail exactly the tests that name them. Test wallets are generated in the test; no key material is committed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
…sed PE-9205 Discovery and the observability vocabulary for the drive state artifact (docs/DRIVE_STATE_ARTIFACT.md §4 and §7). No import, no sync integration. - add a DriveStateEntityHistory GraphQL query, modelled on SnapshotEntityHistory: owners + Entity-Type "drive-state" + Drive-Id, sorted HEIGHT_DESC - add DriveStateDiscovery, an interface an ArNS resolver drops into later (decision D6), and a GraphQL implementation returning candidates newest first by Block-End, re-checked client-side against the drive owner because the indexer is the least trustworthy component in the stack - read a bundled data item and an L1 transaction alike (§4.4); nothing filters on bundledIn - add the §7 outcome vocabulary - used, none-found, unknown-version, signature-failed, decrypt-failed, integrity-failed, count-mismatch, range-already-covered - and a [drive-state] reporter beside the existing [snapshot] instrumentation. The three kinds open with three phrases that are not substrings of one another, so "no artifact was used" and "artifact rejected because X" can never read the same - a lookup that could not complete returns empty and is reported apart from "none found", so a drive is never logged as having no artifact when the gateway simply did not answer Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
Neither is key material, so neither was caught by the leak test - they are withheld for correctness, and the export lane flagged the first itself. `syncCursor` is an opaque cursor issued by one gateway's indexer and means nothing to another, so an importer that adopted it would resume pagination from an arbitrary position. `lastBlockHeight` is worse, and was not flagged. It is the *producer's* watermark, which is a different claim from the artifact's coverage. An importer adopting one that sits above the artifact's Block-End would believe it had synced a range the artifact never contained and skip it - the drive watermark advancing past unsynced entities, which is exactly the silent drop in SYNC_SKIPPED_ENTITY_PERSISTENCE.md, and the failure this whole line of work started from. Coverage comes from the Block-End tag and from nothing else. Both are removed from `ExportedDrive` entirely rather than left nullable, so no later code can put them back by accident, and a test asserts neither name nor value reaches the encoded payload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
The discovery lane branched before the envelope lane's tags merged, so both declared the same four tag names and the same Entity-Type value with identical strings. The discovery lane flagged it as a merge-time edit and described exactly what to do; this is that edit. References now point at `EntityTag.driveStateId` / `stateVersion` / `contentEncoding` / `entityCount` and `EntityTypeTag.driveState`, so there is one definition of each tag name rather than two that agree today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
The reading half of the artifact: open the envelope, check what the tags promised against what the payload holds, and merge one drive's rows into a database that holds every drive. Validation runs to completion before the first write, and the writes are one transaction. A count mismatch, a Drive-Id that disagrees with the tag, or a payload carrying another drive's rows leaves the database exactly as it was - a partial import is worse than none, because it would leave rows nobody trusted behind a watermark saying not to look again. The merge reconciles rather than replaces. Rows are matched by primary key and written only when the artifact's copy is at least as recent as the local one by lastUpdated; a local row the artifact never heard of is kept, since it is far likelier to be a recent upload than a deletion. The drive's key material and sync cursor are never named by the update statement, so they cannot be nulled out by an import. The watermark comes from the Block-End tag and from nothing else. The export withholds the producer's own lastBlockHeight for exactly this reason. The same argument applies once more to Block-Start: an artifact whose coverage starts above what this client has synced merges its rows but does not move the watermark, because doing so would jump a gap nobody walked. Every DriveStateEnvelopeFailure maps onto the coarser vocabulary the discovery lane owns, as a switch expression so a new codec failure is a compile error here rather than a silent default. Nothing throws: the outer catch turns even an unforeseen failure into an outcome, and the caller falls back to an ordinary sync. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
The envelope signed its payload with `wallet.sign` and wrapped the result in a hand-rolled binary frame (magic `ARDRVSTA`, frame version, signature type, three uint32 lengths, owner then signature then payload). Both halves of that were wrong. ArConnect and Wander have deprecated arbitrary-data signing, so an extension user could never have produced an artifact at all. Every ArDrive upload already signs an ANS-104 data item instead, and that path those wallets still grant. The artifact body is now a data item signed through that same seam — `DataItem.sign` with an `ArweaveSigner`, the shape of `ArweaveService.prepareBundledDataItem` and `BDISigner` — serialised with `asBinary()` and encrypted. Opening reverses it with the package's own reader rather than a parser of ours: decrypt, `processDataItem` to parse and verify in one pass, confirm the signer resolves to the drive's owner, then gunzip. - the item's data is the gzipped payload, so the signature covers the bytes a reader holds before it decompresses anything, and unsigned bytes never reach the decompressor - the item carries no tags, target or anchor; the ArFS tags that make an artifact discoverable stay on the enclosing transaction - sign still precedes encrypt, AES-GCM is still the only cipher, and the 100 MiB refusal still has no CTR fallback - `signatureInvalid` and `ownerMismatch` stay distinguishable, which is what lets the fallback log say why a drive skipped its artifact - `unsupportedFrameVersion` is gone with the frame it described; ANS-104 has no version field, so nothing could ever have produced it Tests port to the new shape and keep their coverage — byte-identical round trip, tampered ciphertext, another wallet's artifact, over-size, non-GCM cipher — and gain a case that breaks the signature while leaving the signed bytes alone. Neutering the verifier fails that case and the tampered-payload case, which is the check that they are testing verification and not parsing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
The import lane mapped every `DriveStateEnvelopeFailure` onto an outcome, including `unsupportedFrameVersion`. The signing rewrite deleted that value along with the hand-rolled frame it described — ANS-104 has no frame version field, so nothing could produce it any more. Both branches were green on their own; only the combination was wrong, which is the case a branch's own gate cannot cover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
Extends the plan past ardrive-web to ar-io-docs, ardrive-core-js and ardrive-cli, with status per item and a sequencing that puts the parts needing a human decision on their own line. Two things checked in the sibling repos change what the proposal assumed: ardrive-core-js has no snapshot support at all - its entityTypeValues is drive/file/folder/drive-signature and parseSnapshotData does not exist anywhere in the source. Adding drive-state there would be the first rollup artifact core-js can read, which is more work than the proposal's section 8 implied and also the higher-value half, since core-js is what non-web clients build drive state through. The CLI creates snapshots through its own utility, and a comment there attributes the body shape to a core-js function that does not exist. So the CLI can already produce a rollup nothing in core-js can consume. The drive-state reader belongs in core-js so that split is not repeated. The acceptance test for the whole feature is named explicitly: an artifact written by one implementation, read by another, yielding identical entities. Until that passes, "cross-client format" is an assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
The previous revision claimed ardrive-core-js had no snapshot support and that a CLI comment referencing `parseSnapshotData` was stale. Both wrong. The local core-js checkout was sitting on an old feature branch rather than origin/master, which has a complete 16-file `src/snapshots/` module, a real `parseSnapshotData` used by `arfsdao_anonymous`, and concepts deliberately mirroring this app's `lib/utils/snapshots/`. The error is recorded in the document rather than edited out. It is the stale-checkout failure the agentic framework warns about, made by the coordinator on a sibling repo, and it made the downstream work look larger and lonelier than it is. Corrected consequences, all favourable: the range-composition machinery already exists in core-js, so drive-state mirrors a proven structure; the `parseSnapshotData` → `arfsdao_anonymous` read path is the seam to slot beside; and section 3 is now scoped as `src/drive_state/` alongside `src/snapshots/` rather than as a first-of-its-kind reader. One real constraint surfaced in its place: `src/snapshots/index.ts` says the module is not yet wired into the live listing path. Drive-state should not overtake that integration, or it lands as a reader nothing calls - flagged to coordinate with the desktop work rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
…-9205 D2 said the payload should carry current state only, not revision history, on the reasoning that entries "cover everything the explorer renders". That was false and I did not check it before binding it as a decision three lanes built on. `filesInFolderWithLicenseAndRevisionTransactions` - the query the file explorer actually uses - INNER JOINs `network_transactions` twice, each through a subquery over `file_revisions`. A file row with no revision produces NULL on the ON condition and is dropped from the result. So an artifact carrying entries alone would import 41,000 files and display zero of them, in every folder, while global search and the folder-tree paths (which do not join revisions) would still show them - an internally inconsistent drive. It would not heal either: the import advances the watermark to Block-End, so the next sync queries only beyond it and never fetches the revisions that would satisfy the join. The import lane's own tests could not catch this. Every assertion about landed rows reads `db.select(db.fileEntries)` directly, which is the one read path that does not join revisions. The payload must therefore carry file_revisions, folder_revisions and the network_transactions rows they reference. This costs nothing against the published size figures: the 34.63 MiB measurement already modelled entries plus revisions plus network transactions. It was D2 that departed from what had been measured. Found by the independent import audit; verified against the query. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnYLXFocWgTt9M2CbGYSUP
Composes the three merged drive-state lanes - discovery, the envelope, the importer - into the source `docs/DRIVE_STATE_ARTIFACT.md` §5 puts above snapshots and GraphQL, and wires it into `_syncDrive` ahead of the snapshot pass. - `DriveStateSyncSource` discovers, fetches through the existing `ArweaveService`/`DataGatewayFallback` seam, imports, and reports exactly one §7 outcome per drive per sync. It is the only layer that logs one: every layer below it returns its verdict because none of them knows the sync reached its end. - The imported range becomes an obscuring range through the mechanism that already composes snapshots - it raises the `lastBlockHeight` seed `SnapshotItem.instantiateAll` turns into its `obscuredBy` accumulator, and the start of the total range `HeightRange.difference` subtracts from. No second mechanism, and nothing else in `_syncDrive` moves. - Gated behind `enableSyncFromDriveState`, which defaults to false, and behind a drive key, because v1 is private-drive only (§2.6). With the flag off no discovery query is issued and nothing is constructed. - Every failure is a fallback: discovery, fetch, import and any unexpected throw all leave the sync starting exactly where it would have. The artifact's coverage is clamped to the current block height so a lying `Block-End` cannot build a backwards `Range` and fail the drive. - A transport failure - an unanswered discovery query, an unfetchable body - is logged as a warning note rather than an outcome: §7's vocabulary describes what happened to an artifact, and neither of those learned anything about one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
…ed entities PE-9205 Adds the creation half of the drive state artifact: export -> seal -> ArFS entity, stopping short of upload. - drive_state_creation_service.dart builds a prepared, unsent DriveStateEntity with Entity-Count from the export, Block-End from the drive's own watermark, Block-Start 0, and Data-Start/Data-End derived from the rows it carries. It has no upload collaborator to call. - The D3 precondition runs first, before anything is read or sealed: a drive whose last sync skipped entities is refused, and so is one whose skip state cannot be established. Publishing a gap into an immutable artifact is the failure SYNC_SKIPPED_ENTITY_PERSISTENCE.md describes. - drive_state_sync_skip_status.dart turns SyncCubit's skip report into a three-valued answer, so "cannot tell" is refusable rather than rounded down to clean. SyncCubit gains two read-only getters - when a sync last completed and which drives it covered - because an empty skip map and a clean one are otherwise the same value, and a scoped sync replaces the map wholesale. - A cubit and confirmation modal show what would be published (drive, entity count, size, block range) behind an explicit Publish button. The upload seam is wired and its shipped implementation publishes nothing (D8). - User-facing strings are hardcoded English, matching the statusMessage strings this surface already carries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
`folder_entries` and `file_entries` do not hold only what the chain said. Sync writes a stand-in row for a ghost folder - named after its own uuid, parented to the drive root - and DriveDao writes a placeholder for a discovered drive's root folder so the drive can be opened before its metadata arrives. Both are stamped with *now*; a synced row carries its revision's chain commit time. The importer keeps a local row only when it is strictly newer than the artifact's, so a fabricated row always wins: publishing a drive that held one would replace a correctly synced folder with a uuid-named ghost in every client that imported it, permanently, because an artifact cannot be recalled. Export only chain-derived rows. The discriminator is revision existence - a revision is written only when real metadata is read - which is the same signal DriveDetailCubit uses to tell a genuinely empty drive from one that has never synced. `isGhost` alone would not do: the root folder placeholder deliberately carries no marker, since the upsert that lands real metadata leaves absent columns alone and the flag would stick forever. Entity-Count follows the exported rows, so it stays honest. Three guard weaknesses went with it: * The drift guard had a fourth-table blind spot. The exported and withheld column lists were separate `if (table is X)` chains and the test named the same three tables literally, so a section added for a fourth table passed a guard that never looked at it. That matters because the obvious candidates - network_transactions, arns_records, ant_records - have no driveId at all, and exporting one would publish rows about the user's other drives inside a blob encrypted with one drive's key. Both classifications are now decided together in one place, driven by a single list of exported tables that the test iterates, and the test asserts every other table in the schema is refused. * Nothing asserted that `profiles` is refused - it threw only by falling off the end of a dispatcher. Refusal is now the dispatcher's first check, with explicit tests for `profiles` and the revision tables. * The leak test asserted one 4-byte base64 string, and never value-checked keyEncryptionIv. Drift's default serializer passes a Uint8List through untransformed, so a leak through a generated toJson() would render as a JSON int array and slip past a base64 assertion. The queries are now built by named functions so the test can assert the generated SQL names no withheld column and selects no star - the mechanism rather than one rendering - and the value assertions use high-entropy bytes checked in both forms. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
…ressor PE-9205 Four security fixes to the drive state artifact, on a format that becomes immutable once anything is published. - The coverage claim moves inside the signed payload. `Block-End` lived only as a transaction tag, and the importer wrote it straight to `drives.lastBlockHeight` - so a third party could copy an owner's genuine artifact bytes verbatim, re-publish them under `Block-End: 9999999`, and pass every check: the signature verifies (they are the owner's bytes), the drive decrypts, `Drive-Id` and `Entity-Count` match. The watermark would jump to a height nothing was ever synced to and that range would never be queried again - the silent drop in SYNC_SKIPPED_ENTITY_PERSISTENCE.md. `DriveStateExport` now carries a required `coverage` claim, the importer refuses an artifact whose `Block-Start`/`Block-End` tags disagree with it (`coverage-mismatch`), and the watermark is set from the signed value. The tags stay: discovery needs them to order candidates without downloading. - Decompression is bounded at `maxSizeSupportedByGCMEncryption`, the same 100 MiB the seal side refuses to cross. gzip reaches ~1032:1, so a ~6 MiB artifact expanded to gigabytes, and verification could not help - the bytes are genuinely signed, so the bomb reached the decompressor by design. The bound is enforced during inflation, because a gzip trailer's recorded size is chosen by whoever wrote the stream. Returns `decompressedTooLarge`. - `unsupportedSignatureType` maps to `signatureFailed`, not `unknownVersion`. It is an authorship failure, not "your client is old". - The `Content-Encoding: gzip` tag is gone. The transaction's data is GCM ciphertext and gzip is two layers inside it; an AR.IO gateway echoes the tag onto the HTTP response (`ar-io-node`, src/routes/data/handlers.ts), so a browser would gunzip ciphertext and fail permanently. No other ArDrive entity sets it. Every guard is mutation-checked: removing the cross-check, making `coverage` optional, dropping the bound, reverting the signature mapping and re-adding the tag each fail a test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
The export now carries only rows a revision backs. Two fixtures written before that landed built folders through `addTestFilesToDb`, which writes file revisions but no folder ones, so their folders read as local stand-ins and dropped out of the export. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
Conflict resolution and two integration fixes the merge exposed: - exportDriveState reads the coverage claim through its own named query rather than folding lastBlockHeight into the drive-row projection, whose generated SQL is what the leak test asserts. All four reads now run in one transaction, so the claim and the rows it describes are the same snapshot. - DriveStateEntity.fromEnvelope takes the coverage object instead of a blockEnd integer. The creation service was reading the watermark before the export and tagging that value, so a sync batch landing in between would have tagged a Block-End the signed payload contradicts - an artifact that rejects itself, paid for and permanent. - The sync fixture now moves the producer's watermark before exporting, so its artifacts are self-consistent the way a real one is. The clamp test covers a genuine over-large claim; a new test covers a real body re-tagged with someone else's range. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LFV6xYmFz2meXf5EW2M1FB
…' into PE-9205-drive-state-impl
…wner PE-9205 claimWriterLock answered true when navigator.locks.request rejected or threw, so the tab took the shared IndexedDB database. If another tab already owned it, both wrote blocks with no lock between them, which is the corruption the lock exists to prevent. A failed request now answers false, and the tab works on a private copy: the user still sees their data, only that session's changes are not kept, and the next start asks again. A browser with no Web Locks at all still answers true, since there is nothing to coordinate with and that is how every tab behaved before. The test makes the browser reject for real: the Web Locks spec reserves names starting with '-', and Chrome refuses them with NotSupportedError. Restoring the old answer fails it. Raised by CodeRabbit on #2197. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
Collaborator
|
@coderabbitai On the outside-diff finding in your last review (record a drive as examined only when its walk starts, The up-front announcement is intended and documented on The change would move that announcement into the concurrent closures of |
Collaborator
|
@coderabbitai review |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
ast-grep outline lib/sync/domain/sync_progress.dart --items all
ast-grep outline lib/sync/domain/repositories/sync_repository.dart --match 'syncAllDrives|syncSingleDrive' --items all
sed -n '1,160p' lib/sync/domain/sync_progress.dart
sed -n '740,835p' lib/sync/domain/repositories/sync_repository.dart
rg -n -C 5 'examinedDriveIds|Future\.wait' lib/sync/domain/repositories/sync_repository.dartLength of output: 14514
I checked The conservative invalidation is intentional. Under-reporting could preserve a stale clean verdict after a drive advances. Over-invalidating requires another sync before publishing. My suggestion did not account for that safety trade-off. No change is needed. ✏️ Learnings added
You are interacting with an AI system. |
|
… PE-9205 - the spec section now lists everything a second implementation needs: the payload schema and the envelope written normatively, the encoding rule, snapshot retirement and the ArFS version question. 2.1 is marked merged (ar-io-docs #137) - core-js and the CLI become Stream B, with design, develop and test phases and exit criteria for each. core-js both reads artifacts during drive sync and produces them; the CLI is a thin surface over it. Cross tests run in both directions and gate the stream - D13 recommendation, pending a decision: readers accept both encodings, told apart by the first bytes of the signed payload, since a meta flag sits inside the compressed bytes it would describe. Producers gzip until Library pages exist, then pick the identity threshold by measurement Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
Collaborator
|
@coderabbitai review |
|
…9205 D13, decided: every reader accepts both encodings, and tells them apart by the first bytes of the signed payload. 1f 8b is a gzip stream and is inflated under the existing bound; "SQLite format 3" and a NUL is an uncompressed file and is bounded by the same limit; anything else is refused as before. The bytes are inside the signature, so only the owner chooses, and no tag or meta flag is needed (a meta flag could not work: meta is inside the bytes it would describe). seal still always compresses. This lands before any artifact is published, so no reader in the wild can meet an uncompressed artifact it does not understand, and a producer can later publish a large public artifact uncompressed for Library pages to read by range. - six envelope tests: an uncompressed file opens, gzip and identity of the same bytes open alike, the size bound at exactly and one byte past the limit, another wallet's uncompressed file is refused, a gzip header that does not inflate is refused - mutants: four of five killed. Removing the gzip magic check is equivalent, since the inflater refuses the same bytes; it only makes the log say why - D13 recorded as decided in DECISIONS.md, the proposal, LIBRARY_PAGES.md and the delivery plan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
Every transaction an artifact import regenerated landed confirmed, and the confirmation pass only revisits pending rows. So an upload that had failed on the producer showed as a good file in every client that imported the drive, permanently. D15. - the artifact carries transaction_statuses: the transactions the drive's own rows reference that the producer had not confirmed, with the status it held. Scoped through the rows, so nothing about other drives travels (the reason D2 gave for not carrying statuses) - a reader makes a listed data transaction pending, so its own confirmation pass checks it, as it does for every data transaction a sync reads. That pass finds the owner through the file's drive, so it is right on attached drives. Any other listed transaction stays failed - the producer refuses to publish while a metadata or licence transaction is pending: the confirmation pass would look those up under the signed-in wallet, which on an attached drive is not the owner, and fail a good upload. It clears in minutes, with a message - rows are never dropped for their status, since a gateway can report failed for a transaction that is on chain - a reader refuses an artifact listing a status it cannot apply, a pending metadata transaction, or the same id twice - format 0.2, since the payload changed shape and nothing is published - addFolderRevisionsToDb now writes confirmed metadata, as a sync does; every caller models a synced drive Mutation-checked: removing the refusal, the export, the licence check, the data rule, the metadata rule, the importer's use of statuses, the duplicate check, or the drive scoping each fails a test. Docs: D15 in DECISIONS.md and the proposal; the delivery plan gains the end-to-end review (R1-R8), the open question of which signature proves authorship, with the live gateway evidence, and a rollout sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
D16. The payload was signed a second time, as an ANS-104 data item inside the upload, so the bytes would prove their author however they were found. Readers could not use it, and it cost more than it gave: - the upload's own signature is unreachable for readers today: ardrive.net and turbo-gateway.com serve no X-Arweave-Signature or X-Arweave-Owner headers for a Turbo-bundled item, and GraphQL returns "<not-found>" for its signature and owner key - it put a wallet prompt on publishing, sent Wander and ArConnect through the deprecated arweaveWallet.signature() with saltLength 0, made core-js and the CLI implement data item verification, and made range reads of a public artifact unverifiable Now authorship is the upload's owner as the indexer reports it, the same attribution every ArFS entity gets. Discovery queries and filters by the drive owner, and the importer checks it again before opening anything (new outcome owner-mismatch, replacing signature-failed). A private payload stays authenticated by AES-GCM under the drive key. A public payload is as trustworthy as the gateway, like that drive's own entities; the read gate and the decompression bound face it in full. - the codec seals gzip then (private) GCM, and opens the inverse with the encoding detected from the first bytes; no wallet, no parser - drive_state_artifact_seal.dart, an unused second implementation of the old envelope, is deleted with its test - every comment and doc that argued "the payload is signed, the tags are not" now argues what is true: the payload is what is imported and, privately, authenticated; the tags are what an indexer reported - the proposal's §2.2 is rewritten with the gateway evidence, §4.2 says a name-resolved artifact needs one owner lookup, and the core-js note no longer claims node:sqlite on Node 18 - format stays 0.2: nothing has been published Mutation-checked: removing the importer's owner check, either half of it, discovery's owner filter, or the private path's encryption each fails a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
… PE-9205 Every column in a published format is one every reader has to keep for ever, so the two that carry nothing come out before the first publish: - path: every write in the app stores '', and nothing reads it. Paths are built from parentFolderId, which any reader can do in one pass, and a stored path would have to be rewritten under every descendant on each rename or move - isGhost: only rows a revision vouches for travel, and those are not ghosts, so it was always false A reader supplies them (artifactImportConstants): path '' and isGhost false, which is what the app would hold for the same row, and which turns a folder this client held as a ghost into a real one. The rest of the file columns were checked against the ArFS metadata: thumbnail, fallbackTxId, originalOwner, importSource, assignedNames and pinnedDataOwner are chain facts and stay. action stays as derived and informational (0.3% of the artifact). Artifacts are now about 640 B a file instead of 705: the 100 MiB bound is crossed at about 164,000 files, and a 41,767-file drive is about 26 MiB, 3.9x under it. The spec and the test that pins those figures are updated. Mutation-checked: a sync-path import that marked folders as ghosts, and a SQL import that did, each fail a test (the second needed a new one). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
…PE-9205
D17, both added before the first publish because neither can be added
afterwards without a version bump:
- the SQLite header carries application_id 0x41724453 ("ArDS"), so a
reader or any SQLite tool recognises the file from its bytes, and a
reader refuses a database without it
- meta.producer names the building client as <App-Name>/<App-Version>
(ArDrive-App/<version> here, from package info, injectable in tests).
On a private drive it is authenticated under the drive key, where the
App-Version tag is only what an indexer reports, so it is what a
reader would refuse a build with a known defect by (plan R4)
Dropping the entry tables was measured (42% smaller) and deliberately
not done; the trade-off is in the delivery plan for a decision.
Mutation-checked: a reader that skips the application_id check fails a
test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
The importer read every row of an artifact into a Dart object and wrote it back: 5.4 s at 41,767 files on the VM. Only the drive row, the folders and transaction_statuses come into Dart now, where the merge's guards are decided; the revision tables, licences and file entries are copied by INSERT ... SELECT, and network_transactions is derived in one statement. - every copied value is checked for the type the format declares before anything is copied, since SQLite stores whatever a column is given and the Dart read used to refuse it; a file listed twice is refused; every row must belong to the drive - each import attaches under its own name: concurrent drive imports collided on "artifact" and one fell back to a full sync - in a browser the received artifact is kept in memory only (delete-on-close) instead of being written to IndexedDB and deleted again, and it is detached without waiting on IndexedDB, as before - the artifact a drive last imported is recorded on its drive row (drives.importedArtifactTxId, schema 30) in the import's own transaction; a sync skips it before downloading while the watermark is still at or past its Block-End, so a restart no longer re-downloads and re-imports it Behaviour is unchanged: a harness ran the old and new importer over twelve scenarios and every table came out identical. All new guards are mutation-checked. VM, 41,767 files: 5.4 s to 2.1 s. Release browser build: 26.8 s to 19.2 s (3,000 files: 0.93 s to 0.43 s); most of what remains is the web database's storage layer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
D18 (bulk tables copied by SQLite) and D19 (the imported artifact recorded on the drive row). D12's 584 ms import figure was importDriveState, a bulk copy the app never called; the app's importer took 5.4 s. Plan items 1.17, 1.18 and 1.19, the last for the web database's storage layer, which is most of the remaining import time and every sync's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
The bulk copies ran as bare customStatement calls, which tell drift nothing about the tables they write, so live queries on those tables did not refresh. Folders still went through drift's batch API and appeared; an open folder listed its folders and none of its files until the user navigated away and back. Found in the browser on the demo wallet. Each copy now runs through customUpdate naming its table, and a test keeps the folder open across the import (mutation-checked: with no table named, it fails). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
Where enableDriveStatePublishing is on, the prompt a drive's owner gets when a drive is slow to sync offers to publish its drive state and opens the publish modal, instead of offering a snapshot (D14). Same trigger, same owner-only rule, same "don't ask again" setting. Where it is off, the prompt is unchanged. Hardcoded English like the rest of the feature until the .arb pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
…e modes PE-9205 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
…rements PE-9205 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
docs/WEB_DATABASE.md has the measurements, the design and what building it found. Release builds, the real importer, 41,767 files: Chromium 16.0 s to 1.3 s, Firefox 18.0 s to 7.8 s, saved at commit instead of seconds later; reopening the database 3.5 s to 0.08 s. - web/db_worker.dart holds SQLite on OPFS (SimpleOpfsFileSystem) behind drift's remote protocol; the page gets a DatabaseConnection and every DAO, stream and migration runs unchanged - one tab writes, under the existing Web Lock; a writer whose files a departed worker still holds (a reload) waits for them rather than settling for a session copy; other tabs get an integrity-checked in-memory copy - with no worker script, no OPFS, or any failure, the #2197 IndexedDB path opens instead, unless OPFS already holds this user's database - the first open fills OPFS from #2197's IndexedDB store or the sql.js store, refusing unsound bytes, and marks it done only after the bytes are flushed - drive state artifacts live in the worker's memory for export and import - getFolderTree reads the whole drive in two queries instead of two per folder, held to the old version by a parity test - the password modal no longer touches its context after it is gone, a race the slower database read exposed - the slow-drive nudge goes through driveStatePublishOffer, the gate the publishing-rail test requires - tool/build_db_worker.sh compiles the worker; scr setup and the browser tests run it Browser tests cover open, reopen, every column type, live queries, artifacts, migration, the second tab, a killed worker, a reload, the missing script and the file guard; every new guard is mutation-checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
It was committed without the executable bit (written on a Windows filesystem), so scr setup failed with Permission denied. Marked executable, and called through bash so the file mode never decides it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
… race PE-9205
- The password modal asked for the wallet address in didChangeDependencies,
which runs again on every dependency change, and read its context after
the wait. The dialog closes the moment a login succeeds, so a read still
in flight touched a disposed widget ("Null check operator used on a null
value" in release builds; seen after a reload once the database ran in a
worker, where the read takes a round trip). It now asks once and takes
the bloc before the wait, so the profile name still loads.
- The new-wallet modal had the same read-after-wait; same fix.
- TurboBalanceCubit emitted after the card that asked had closed it
("Cannot emit new states after calling close"), on success after the
request and on error after its half-second wait.
- The profile card counted drives, files and size once, at its first build,
and showed that figure all session; partway through a sync it said 397
files and 3.7 MB for a drive the drives list showed at 773 and 9.58 GiB.
It counts again each time it opens.
Regression tests for the modal and the cubit fail on the old code and pass
on the new.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
An artifact's Block-End is its producer's watermark, the node's /info height at that sync, not a height the GraphQL index had reached. A file mined just below it and indexed late is in no artifact; the producer recovers it on its next sync through the 240-block look-back, but a reader started GraphQL exactly at Block-End and, once its own watermark moved more than 240 blocks past it, never read those blocks again. - After an import, snapshots and GraphQL resume kBlockHeightLookBack below Block-End, the margin ordinary sync applies below its own watermark. Re-reading the band records nothing twice: revisions no newer than the latest held are skipped and inserts ignore duplicates. - New composition test: a file at 1450, absent from an artifact ending at 1500, now lands; it fails on the old start point. - Composition and drive-state tests moved to the handover point (1260), keeping each scenario's meaning; no-gap checks expect the look-back band. - Spec section 5 states the rule for every reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
… PE-9205 The watermark says every row below it is stored: later syncs re-read only the look-back below it, and a drive state artifact publishes it as Block-End. Sync raised it inside the per-page parse, on any page shorter than the batch, and before that page's own rows were written: - the last page of every 1000-transaction chunk whenever the batch size does not divide 1000 (it is 200 shared between the drives left to sync, so 66 with three drives left), or any page holding a skipped entity; - so a sync that stopped after the first chunk (a closed tab, a cancel, a failed fetch or write) left a large drive claiming the tip without the rest of its history, until a deep sync. This is on dev and production. It is now raised once, after the stream, when every chunk has committed, and only when transactions were received, as before. isHidden is no longer rewritten with the value from the start of the sync, which could only undo a hide or unhide applied since. The count uses totalTransactionsReceived: totalTransactionsProcessed undercounts because the batch processor empties the final chunk's list. Two tests fail on the old code (watermark 2000, expected 1000): a failure on the only page, and a 1100-transaction sync failing in its second chunk after a short page; both recover fully on a healthy resync. Wording corrected: the ordinary look-back applies on the first sync of each session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
Staging exercises the artifact path before production does. configVersion goes to 4 so existing staging users pick the flags up (this resets their stored dev-tools config once). Dev and production stay off; the flag rail test now pins staging on and both others off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NRgRqqrM4sg2EeqbGJzff
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The D12 counter-proposal to #2188's D1, built far enough to be measured rather than argued about. Companion to #2196, which proves the same
ATTACHpipeline works in a browser.Same feature, same drive, same tables D2 settled — the drive, both entry tables, all three revision tables, licences;
network_transactionsregenerated on import;arns_records/ant_recordsdo not travel. Only the container changes.Measured, at 41,767 files
Map, each waytest/drive_state_sqlite/artifact_scale_test.dart, run with--run-skipped.Read the timings with care. #2188's 4 s / 9 s are its own reported figures from a different machine; I'm running its measurement here for an apples-to-apples number and will post it as a comment. The size figures are comparable — same file count, same one-level folder depth, both fixtures without
customJsonMetadata/customGQLTags.One caveat I hit and fixed, because #2188 documented it first. My first run reported 1.90 MiB gzipped — a 9.7× ratio — because the fixture interpolated counters into transaction ids. Real ids are 32 bytes of entropy and about a third of the payload. With random 43-char base64url ids the gzipped figure moved to 5.98 MiB, a 3.1× correction. That trap is called out in
artifact_fixture.dartso the next person doesn't re-enter it.My fixture is still slightly lighter than the real drive: one revision per entity against the measured 1.05, and no licence rows. Both push the real number up a few percent, on both sides.
How it works
Folders travel.
folder_entriesandfolder_revisionsare both in the projection, so an empty folder — which nothing else implies, since no file references it — survives, as does a ghost folder and itsisGhostflag.artifact_folder_test.dartexecutes each of those rather than leaving them to be inferred from the snippet above.Import validates everything before writing anything —
integrity_check, thensqlite_masterbyte-identical to the frozen schema, thenmetachecked against the drive being synced — and merges in one transaction, rebuildingnetwork_transactionsfrom the revisions it carried.ArtifactImportRefusalhas ten values and every one is a fallback, never an error.Two decisions worth a reviewer's attention
The schema is frozen, and it is not Drift's. An artifact is a format other clients read, so copying the app's tables would weld the wire format to
schemaVersion— §8's objection. Drift's schema also measured worst of every option tried, 10.14 M gzipped against 6.09 M for a frozen schema, because its indexes are dead weight a reader throws away.Rows are copied in, never deleted out.
artifactProjectionis the only path into an artifact and it names every column;profilesappears nowhere in the file. This is the mechanism, not a filter — an artifact is built by selecting into an empty database, so an unnamed column cannot reach one, and there is no removal step to forget.The obvious alternative — copy everything, drop what must not ship — does not work.
DROP TABLEleaves the dropped bytes on the freelist unlesssecure_deleteis on, and it is off in the wasm build the browser runs (measured in #2196:0on wasm,2on macOS). A test asserts the leak undersecure_delete = 0so the reasoning stays visible.Tests
17 passing, plus the skipped measurement. Notably:
network_transactionsand asserts files are visible. An artifact carrying entries but no revisions imports 41,000 files and shows zero.drivesgrows a column that is neither exported nor withheld, the test fails and someone has to classify it deliberately.One of those guards earned its keep during development: my tamper helper opened an artifact through Drift's
Database, which ranonCreateand wrote the app's whole schema into the file. Thesqlite_mastercheck caught it.What is not here
A prototype for a decision, not a feature.
The browser half is PE-9205: Prototype the drive state artifact ATTACH pipeline in the browser #2196.Done. The web database now runsdrift/wasm.dart, and the artifact sink reads an attached database back through the VFS interface.flutter build webcompiles. See Web below.ArtifactSink/ArtifactSourceare the seam. A file on the VM and in a CLI, a VFS entry in a browser. The exporter doesn't know which — which is what would let a CLI produce an artifact for a drive far too large for a tab.What the probes settled about scale
Running #2196's storage probe on the real origins: both
staging.ardrive.ioandardrive.ar.ioland onsharedIndexedDb, where the whole database is held in RAM. OPFS works in both browsers; the only missing piece isSharedArrayBuffer, i.e. cross-origin isolation — and the permaweb build is served by gateways whose headers nobody controls collectively.So drop the unbounded-drive-size argument for D12; it isn't available where most users are. What survives is what this PR measures: you stop inventing a format, import allocates nothing per row, and producer memory improves by a large constant factor on every deployment rather than only where OPFS is reachable.
🤖 Generated with Claude Code
https://claude.ai/code/session_01T3CbaePfsFZsPqijauuBrD
Summary by CodeRabbit
New Features
Documentation
Tests
Review follow-ups
CodeRabbit's review found two things that were serious, and both were right:
INSERT OR REPLACE, which deletes the conflicting row and reinserts it, so every column the artifact does not carry came back as its default — ondrivesthat isencryptedKey,keyEncryptionIv,driveKeyGenerated,syncCursor,lastBlockHeight. A private drive would have been left permanently unopenable by its own artifact while the import reported success. Withholding key material from the export was only half the guarantee. Now an upsert on the projected columns, with a regression test.metarow.Also fixed:
integrity_checkread withgetSingle()(it returns one row per problem),ATTACHoutside the refusal handler, a duplicated schema gate, the export running without a transaction,maxByteschecked only after the file was materialised, colliding VFS paths for concurrent web exports, and the legacy sql.js migration writing to the real path before validating.Two findings are deliberately not addressed here and are flagged for follow-up:
@TestOn('browser')meansscr testskipsvfs_probe_test.dart, and nothing in CI servessqlite3.wasm. The finding it protects is what the whole web path rests on, so it should be in CI before this merges — but adding a Chrome job is its own change._readDriveStateArtifactruns before the fetch loop's firstcheckCancellation()and does not take the token. Inherited from PE-9205: Implement the drive state artifact #2188 and a sync-behaviour change; belongs in its own commit.Config rollout note: both flags ship
false, soconfigVersionis deliberately not bumped. The rollout that flips either totruemust bump it, or the change reaches nobody with a stored config.Web (done)
lib/models/database/web.dartnow runsWasmSqlite3+IndexedDbFileSystemin the main isolate, replacing drift's deprecated sql.js backend. That swap was not optional:SqlJsDatabase.export()returnsmainand nothing else, so an attached database's bytes could be built and never read back.Deliberately not
WasmDatabase.open(). That probes the browser and picks a storage tier, and on both deployed origins the probe answerssharedIndexedDb— OPFS needs cross-origin isolation, which the permaweb build cannot have because its headers belong to whichever gateway served it. Choosing directly keeps the VFS reachable and the database in the main isolate, which is where sql.js already ran: same concurrency story, supported engine.Reading an artifact back uses
xOpen/xFileSize/xRead/xDelete, public on everyVirtualFileSystem. That matters becauseIndexedDbFileSystemkeeps its file map private and URI filenames are disabled in this build, so neither a direct read nor a per-attach?vfs=works. Both findings are asserted intest/drive_state_sqlite_web/vfs_probe_test.dart.The sql.js database migrates as a byte handoff — its stored bytes already are a SQLite file. Staged at a temporary VFS path, opened,
quick_checked, and only then promoted, because creating the real path is what makes the migration one-shot. The legacy copy is not deleted: it costs some disk and buys a way back.Removed:
web/sql-wasm.wasm(1.14 MB),web/js/sql-wasm.js, its script tag, and the unreferencedweb/worker.js.tool/check_wasm_version.shpinsweb/sqlite3.wasmto the resolvedsqlite3package — the check that would have caught the copy this repo already carried, which was built against a different ABI.Nothing travels that no revision vouches for
The projection originally copied
folder_entries,file_entriesandlicenseswhole, dropping a filter the JSON exporter had. That filter is not cosmetic: afolder_entriesrow with no revision behind it is not something the chain said — it is this client's stand-in for a ghost folder, orDriveDao's root-folder placeholder, and both are stampedDateTime.now(). The merge resolves conflicts by which side is newer, so publishing one would make this client's guess outrank every real row in every client that imported it, permanently.The cost is paid on the other side deliberately: a ghost's files do have revisions and do travel, so the payload names a parent it does not carry, and the importer closes that graph itself with
_ghostFolderStandIn.Tests
629 passing across
test/drive_state,test/drive_state_sqliteandtest/sync;flutter analyze lib testclean. Every other test directory passes too.Folders get their own file because "copy the rows" is least obviously sufficient for them: an empty folder survives (nothing else implies it), a ghost folder does not travel, a file entry with no revision does not travel, folder revisions travel, and a drive whose root folder has no entry row — reachable, since sync declines to ghost a missing root — exports faithfully rather than inventing one.
Known gap
A payload signed with an unreadable version reports
integrityFailed, notunknownVersion.readArtifactAsExporthas no payload-side version gate whereDriveStateExport.fromJsonhad one with three arms, so the tag/payload cross-check catches it first. Nothing unreadable is accepted either way; what is lost is the sentence a sync log gets — the distinction §6.1 and §7 exist to preserve. Worth restoring before merge.Testing on a preview deployment
The flags ship
false; turn them on withCtrl+Shift+Q.Publishing spends real money. The uploader wired at
drive_state_creation_modal.dart:98is the real one, so enablingenableDriveStatePublishingand confirming the dialog posts a permanent Arweave transaction. It is taggedEntity-Type: drive-state-testandState-Version: 0.1, so nothing that later implementsdrive-statewill ever read it.A preview URL is a fresh origin, so the sql.js migration is not exercised there — there is no legacy IndexedDB to migrate. That path only gets tested on an origin that has already run an older build.