Skip to content

feat(#417): read a file's raw bytes in bounded windows - #492

Open
YidaWeng wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
YidaWeng:feat/417-read-bytes
Open

YidaWeng wants to merge 1 commit into
Ephemeral-AI-Lab:mainfrom
YidaWeng:feat/417-read-bytes

Conversation

@YidaWeng

Copy link
Copy Markdown
Contributor

What

Adds fs.read_bytes(ref, path, offset = 0, length?) to the SDK, CLI and docs, so a file written as bytes can be read back. fs.read remains text only and still refuses a non-UTF-8 file as Binary.

const { bytes, size, hasMore } = await layerfs.fs.read_bytes("feat", "/logo.png", 0, 65536);

Answer: { kind, snapshot, root, path, bytes, size, hasMore }.

  • One window at the same immutable snapshot as fs.read (Branch, Commit or Layer).
  • length defaults to and is capped at the 48 KiB text window (window::BYTES); page by advancing offset by the returned byte count.
  • offset past EOF → empty bytes, hasMore: false.
  • IsDirectory for a directory, NotFile for a symlink, NoSuchPath for a missing path, InvalidInput for a negative/unsafe offset or length.
  • Engine reads through layerfs_content::file::read_range over the call's read caches (the same byte-window path blame uses); telemetry sdk.bytes. No storage, binding or control-record change — the binding already returns SdkValue::Bytes as a Node Buffer.
  • JS surface: @layerfs/agent wrapper, layerfs fs read_bytes <ref> <path> [--offset <n>] [--length <n>], and the agent docs / architecture text.

Note on the issue's decision gate

#417 is marked "Decision needed (owner)". This PR implements the proposed shape from the issue (same method name, same { bytes, size, hasMore } answer, same refusals and 48 KiB cap). If the owner settles on a different shape before merge, this branch is the thing to adjust; nothing is released by opening it.

Tests

  • New engine/crates/layerfs-api/tests/sdk_read_bytes.rs: 48 KiB + 1000-byte file read in two windows, cap of an oversized length, offset past EOF, zero length; directory / symlink / missing / negative refusals.
  • sdk_method_golden transcript gains six fs.read_bytes lines (whole, offset+length, past EOF, IsDirectory, negative offset, NoSuchPath).
  • JS: fixture cases for SDK and CLI, binary_write.mjs round-trip/paging/EOF/refusals, packaging.mjs fs method count 14 → 15.

Checks run (macOS)

  • cargo +1.85.1 test -p layerfs-api --test sdk_read_bytes --test sdk_method_golden — 3 passed
  • cargo +1.85.1 clippy --locked --all-targets -- -D warnings — clean
  • cargo +1.85.1 fmt --all -- --check — clean
  • cargo +1.85.1 test --locked --all-targets --no-run — builds
  • tools/check_product_boundary.py — PASS (959 files)
  • tools unit tests — 155 passed
  • node api/agent/test/calls.mjs — PASS (41 cases)
  • node --test api/cli/test/cli.test.mjs — 77 passed
  • node api/agent/test/packaging.mjs surface|lifecycle — PASS

The focused Rust tests were run with rusqlite bundled enabled locally (host SQLite 3.54.0 is outside the overlay profile); that manifest change is not in the commit.

Not run locally: api/agent/test/binary_write.mjs and the Linux FUSE targets — they need the real native binding and /dev/fuse on Linux.

Production LOC

102763 → 102848 (delta +85): engine +74, api +11, core 0. Counted with tools/production_loc.py --root on git archive of parent 582bc786f and staged tree 9ae38d1f41.

fs.read is text only: a file with NUL or invalid UTF-8 is refused as
Binary, and nothing else in the SDK returns a file's bytes, so a binary
file cannot be read back after fs.write. This adds the byte read the
issue proposes; the issue's owner decision is still pending, so this is
the proposed shape, not a settled one.

- fs.read_bytes(ref, path, offset = 0, length?) answers { kind,
  snapshot, root, path, bytes, size, hasMore }. It reads one window at
  the same immutable snapshot as fs.read (Branch, Commit or Layer).
  `length` defaults to and is capped at the 48 KiB text window, so a
  whole-file read never holds a large file in memory; a larger file is
  paged by advancing offset by the returned byte count.
- An offset past the end answers empty bytes with hasMore: false. A
  directory is refused IsDirectory, a symlink NotFile, a missing path
  NoSuchPath, and a negative or unsafe offset/length InvalidInput.
- The engine reads through layerfs-content's existing read_range over
  the call's read caches, so it is the same object walk as blame's byte
  windows; telemetry is `sdk.bytes`. No storage, binding or
  control-record change: the binding already returns SdkValue::Bytes as
  a Node Buffer.
- sdk/fs surface: the @layerfs/agent wrapper, the `layerfs fs
  read_bytes` command (--offset, --length), the CLI's JSON bytes output
  (already { base64 }), and the agent/cli docs tables.

Tests: added engine/crates/layerfs-api/tests/sdk_read_bytes.rs (a
48 KiB+1000-byte file read in two windows, cap of a longer request,
offset past EOF, zero length; directory/symlink/missing/negative
refusals). The sdk_method_golden transcript gains six fs.read_bytes
lines. The JS fixture gains SDK and CLI cases; binary_write.mjs checks
a binary round-trip, paging, past-EOF and refusals; packaging.mjs
counts fs methods 14 -> 15. No tests deleted.

Checks (macOS): cargo +1.85.1 test -p layerfs-api --test sdk_read_bytes
--test sdk_method_golden (3 passed); clippy --locked --all-targets -D
warnings; cargo fmt --all -- --check; test --locked --all-targets
--no-run; tools/check_product_boundary.py PASS (959 files); tools
tests 155 passed; node api/agent/test/calls.mjs PASS (41 cases); node
--test api/cli/test/cli.test.mjs (77 passed); packaging surface and
lifecycle pass. The focused Rust tests ran with rusqlite's bundled
SQLite enabled locally (the host SQLite is 3.54.0, outside the overlay
profile); the manifest is unchanged in this commit. Not run here:
api/agent/test/binary_write.mjs and the Linux FUSE targets, which need
the real native binding and /dev/fuse on Linux.

Production LOC: 102763 -> 102848 (delta +85)
  core 38395 -> 38395 (0), engine 61709 -> 61783 (+74), api 2659 ->
  2670 (+11); reference absent (retired R9c; last counted 65,417 at
  tree 498dd19). Method: python3 tools/production_loc.py --root on
  `git archive` of the parent (582bc78) and of the staged tree
  9ae38d1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant