Skip to content

fix(core): length-frame file contents in the content digest - #372

Merged
aroff merged 3 commits into
mainfrom
fix/content-digest-framing
Sep 24, 2026
Merged

aroff merged 3 commits into
mainfrom
fix/content-digest-framing

Conversation

@aroff

@aroff aroff commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

The bug

The content digest is what skills.lock and global-skills.lock pin as checksum, what bundle
records and bundle artifacts pin per member and per release, and what install, update, restore
and remove compare with the disk before changing anything. It hashed each file's relative path
with its length but the file's contents without their length. The hashed byte stream therefore
did not decode to a single directory tree, and two different skill directories could produce the
same digest. Content that differs from what a Lock or bundle pinned could pass verification.

The first commit adds content_digest_distinguishes_file_boundaries, which reproduced this on
main (two different trees with one digest) and passes after the fix.

The fix

New module core/content_digest.rs (ADR-0017). A digest is now
sha256-tree-v2:<64 hex>: SHA-256 over a length-prefixed domain tag, then for every regular file,
in byte order of its path, the UTF-8 relative path and the contents, each prefixed with a
big-endian u64 length. Also:

  • Paths are joined from their components, so a Unix file name containing \ no longer hashes like
    a nested path.
  • File names that are not valid UTF-8 are refused, since they have no portable spelling.
  • A file whose length changes while it is hashed is an error.
  • Symbolic links are still refused.

Compatibility and migration

A legacy digest is 64 bare hex characters. The recorded form picks the algorithm used to check it.
Legacy values are still accepted when they match, with a one-time warn_once warning, so
project install --lock keeps working after an upgrade with no edits. fastskill only ever writes
the current form.

Record Legacy value replaced when
skills.lock [[skills]].checksum that entry is re-resolved (skill add, skill update, a newly resolved root)
global-skills.lock checksum that entry is reinstalled or updated
skills.lock [[bundles]] release + member digests the bundle is added, updated, or restored by project install
skills.lock [[overrides]].digest the override is recorded again, or promoted to an individual skill when its last owning bundle is removed
.fastskill/bundle-history.toml that release is recorded again
skills.lock inside a bundle artifact / .fastskill/bundles/*.zip never, because artifacts are immutable. Rebuild with bundle build
  • project install --lock never changes the Lock (documented contract). A locked bundle restore
    keeps the digests it verified, so the Lock stays byte-identical.
  • An artifact whose embedded lock is all-current or all-legacy is accepted. A mixed one is refused,
    because no release produced it. The release-digest algorithm is unchanged, but it hashes the
    member digests, so a rebuilt artifact has a new release digest. Both forms are treated as the
    same release.
  • Two recorded digests of the same form that differ are still a conflict. A legacy and a current
    value are each checked against the content.
  • The Lock format version and fastskill-bundle-lock-v1 are unchanged. Bumping them would make
    older releases refuse the file with a message suggesting that the lockfile be deleted.

Residual risk and what is not fixed

  • Legacy digests are still trusted until their record is rewritten. A tree crafted to collide
    with a legacy digest passes wherever that digest is still recorded, including under
    project install --lock, which deliberately doesn't rewrite. Refusing legacy digests would close
    this. It would also break every committed Lock and every existing bundle, with no way to verify
    what they pinned. This PR takes the compatible path and warns. A later release can refuse legacy
    digests. Until then, re-resolving an entry (skill update) or rebuilding a bundle removes the
    weaker record.
  • Older fastskill cannot verify the new form and reports a checksum mismatch. Every machine and
    CI job that shares a project must upgrade together (noted in the CHANGELOG).
  • Not covered by the digest, unchanged: the executable bit (Windows cannot represent it, and a
    Lock must verify on every platform) and empty directories (ZIP artifacts and git don't preserve
    them consistently).

Docs

  • ADR-0017 (new).
  • The Content digest term in CONTEXT.md.
  • CHANGELOG entries under Breaking Changes and Fixed.
  • manifest-system.mdx, bundle-command.mdx and two-scope-lock-model.md.
  • A format reference in ADR-0016.

Testing

  • bash scripts/run-tests.sh passes: 2010 tests, 0 failures, every modified production file above 90% line coverage, source-size check green.
  • New unit tests cover:
    • the legacy collision vs the v2 difference, framing, path encoding, the non-UTF-8 and symlink
      refusals, and form dispatch;
    • legacy Lock checksums on the offline install path;
    • legacy bundle artifacts, Locks and history: accepted, byte-stable under --lock, upgraded on
      rewrite, and edit protection still enforced;
    • rejection of mixed-form archive locks.

🤖 Generated with Claude Code

A one-file tree and a two-file tree produce the same content digest on
main, because file contents are hashed without their length. This test
fails until the digest frames each file's contents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The content digest pinned by skills.lock, global-skills.lock, bundle
records and bundle archives hashed each file's path with its length but
the file's contents without it. The byte stream did not decode to a
single tree, so two different skill directories could share a digest.

The digest now has a versioned form, `sha256-tree-v2:<hex>`: a domain
tag, then for every file its UTF-8 relative path and its contents, each
prefixed with a big-endian u64 length. Paths are joined from their
components, so a Unix file name containing a backslash no longer reads
as a nested path; non-UTF-8 names are refused; a file whose length
changes while it is hashed is an error.

Digests written by older releases (64 bare hex characters) are still
accepted when they match, with a one-time warning, so existing Locks,
bundle records and bundle artifacts keep installing. Every digest
fastskill writes is in the new form, and a legacy value is replaced
whenever its record is rewritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…R-0017

`project install --lock` restores from the Lock without changing it. A
locked bundle restore now records the digests it verified instead of
upgrading them, so a Lock holding legacy digests stays byte-identical
under --lock; every other rewrite still records the current form.

Documents the versioned content digest: ADR-0017, the Content digest
term in CONTEXT.md, the CHANGELOG, the lockfile and bundle docs, and the
digest reference in ADR-0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aroff
aroff enabled auto-merge (squash) September 24, 2026 21:59
@aroff
aroff merged commit 9f3e86a into main Sep 24, 2026
14 checks passed
@aroff
aroff deleted the fix/content-digest-framing branch September 24, 2026 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants