Skip to content

fix(daemon): stream the dirty check past any number of no-op rows - #491

Merged
yifanxuaaa merged 1 commit into
Ephemeral-AI-Lab:mainfrom
YidaWeng:fix/489-dirty-truncation
Oct 28, 2026
Merged

yifanxuaaa merged 1 commit into
Ephemeral-AI-Lab:mainfrom
YidaWeng:fix/489-dirty-truncation

Conversation

@YidaWeng

Copy link
Copy Markdown
Contributor

Fixes #489.

Problem

The checked Unmount read at most 4096 changed names and 4096 changed inodes. Reaching either cap set truncated, so a Workspace holding only no-op rows past the cap was refused Dirty: 0 uncommitted change(s) while workspace.commit answered UpToDate.

Fix

engine/crates/layerfs-daemon/src/control/dirty.rs only. Both indexed windows are now streamed to the end in bounded pages, and only changes are retained:

  • The name scan stops after more than PATHS bindings: each row is a change a Commit keeps and names one path, so more than 256 of them cannot be listed anyway.
  • The inode scan runs to exhaustion. A locally created inode nothing binds, and a base directory whose mode equals the base, remain no change; however many such rows exist, they neither refuse the Unmount nor mark the report truncated.
  • Base-directory modes are compared one window of serials per batch (TreeTable::values, the same lookup_many read locate uses) instead of one tree.value + read_portable per directory.
  • truncated now means only that real changes exist that the report does not list: more than PATHS, or a path the bounded walk could not name. The old row-cap truncation is gone.

A Workspace with real changes still stops early: the inode scan ends once more than PATHS changes are found. WALK is unchanged, and reaching it only leaves a path unnamed.

Resident state stays bounded: at most a page over PATHS bindings, and at most a page over PATHS changed inode serials, are held at once; no rows are collected for no-op windows.

Tests

engine/crates/layerfs-api/tests/unmount_dirty.rs:

  • no_op_rows_past_the_old_cap_do_not_refuse_an_unmount: 4100 create+unlink pairs unmount without discard.
  • a_change_after_many_no_op_rows_is_named: the same plus one real change refuses and names only /kept, not truncated.
  • exactly_the_old_cap_of_no_op_rows_is_not_truncated: 4096 no-op rows plus a real change is named with truncated: false.
  • a_directory_restored_to_its_base_mode_does_not_refuse: chmod .git away and back unmounts cleanly.
  • removing_a_base_directory_refuses_and_names_it: after unlinking its children, rmdir .git refuses and names /.git, /.git/index and /.git/link.
  • more_changes_than_the_report_names_set_truncated: 300 created files name 256 paths and set truncated.

On the parent, no_op_rows_past_the_old_cap_do_not_refuse_an_unmount fails with paths: [], truncated: true.

Checks (macOS, Rust 1.85.1)

  • cargo test -p layerfs-api --test unmount_dirty — 8 passed in 0.8 s (so tools/ci_tests.py's SLOW list is unchanged).
  • cargo test --locked --all-targets --no-run — builds.
  • cargo clippy --locked --all-targets -- -D warnings — clean.
  • cargo fmt --all -- --check — clean.
  • python3 -B tools/check_product_boundary.py — PASS (956 files).
  • python3 -B -m unittest discover -s tools -p 'test_*.py' — 155 passed.

Gap: this host's system SQLite is 3.54.0, which the overlay profile does not qualify, so the fixture panics before any test body. For the local run above I temporarily enabled rusqlite's bundled feature (SQLite 3.53.2, already allowlisted) and did not commit it. Not run: Linux FUSE/mount targets.

Production LOC: 102,763 -> 102,766 (delta +3), layerfs-daemon only; tests and comments are not counted.

…hemeral-AI-Lab#489)

The checked Unmount read at most 4096 changed names and 4096 changed inodes.
Reaching either cap set `truncated`, so a Workspace whose overlay held only
no-op rows past the cap was refused `Dirty: 0 uncommitted change(s)` while
Commit answered `UpToDate`.

Both indexed windows are now streamed to the end, one bounded page at a time,
and only the changes are retained:

- A changed name binding is a change a Commit keeps, so the name scan stops
  after more than `PATHS` rows; those paths cannot all be listed anyway.
- Changed inodes are scanned to exhaustion. A locally created inode nothing
  binds, and a base directory whose mode equals the base, are still no change,
  and however many of them exist they neither refuse the Unmount nor mark the
  report truncated.
- Base-directory modes are compared one window of serials per batch
  (`TreeTable::values`, the same `lookup_many` read the walk uses) instead of
  one lookup per directory.
- `truncated` now means only that real changes exist that the report does not
  list: more than `PATHS` of them, or a path the bounded walk could not name.

A Workspace with real changes still stops early: the inode scan ends once
`PATHS` changes are found. `WALK` is unchanged; reaching it only leaves a path
unnamed.

Tests in engine/crates/layerfs-api/tests/unmount_dirty.rs:
- 4100 create+unlink pairs unmount without `discard`;
- the same plus one real change refuses and names only that change;
- exactly 4096 no-op rows plus a real change is not truncated;
- a directory chmod'd away and back to its base mode unmounts cleanly;
- removing a base directory (after unlinking its children) refuses and names
  `/.git`, `/.git/index` and `/.git/link`;
- 300 real changes name 256 paths and set `truncated`.
The first two fail on the parent with `paths: [], truncated: true`; deleted:
none.

Production LOC: 102763 -> 102766 (delta +3)
  first-parent comparison (upstream/main 582bc78 -> this commit); engine
  layerfs-daemon +3. Scopes: api 2659 -> 2659 (0), core 38395 -> 38395 (0),
  engine 61709 -> 61712 (+3); reference absent (retired R9c; last counted
  65,417 at tree 498dd19). Counted with tools/production_loc.py (blob
  3341e4d) --root on a checkout of the parent commit and on the staged
  tree.
@yifanxuaaa
yifanxuaaa merged commit 26b295c into Ephemeral-AI-Lab:main Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unmount dirty check caps at 4096 changed rows: no-op rows above the cap still refuse "Dirty: 0 uncommitted change(s)"

2 participants