Unmount dirty check caps at 4096 changed rows: no-op rows above the cap still refuse "Dirty: 0 uncommitted change(s)" #489
Description
Version: main at 582bc786f9 (after #488).
Area: engine/crates/layerfs-daemon/src/control/dirty.rs, the check before an Unmount without discard.
Follows: #462, #488.
Summary
Before a checked Unmount without discard, the dirty check reads the Workspace's changed overlay rows and, since #488, skips rows a Commit would drop: a locally created inode that no name binds any more, and a base directory whose mode equals the base. It reads at most ROWS = 4096 changed names and 4096 changed inodes. When it reaches either cap it sets truncated and, since it has not seen every row, cannot prove the Workspace clean, so it refuses Dirty.
So a Workspace with 4096 or more changed-inode rows that are all no-op is still refused, with the #462 symptom at a larger size:
Dirty: 0 uncommitted change(s); unmount with discard to drop them
detail: { changed: [], pendingMerge: false, truncated: true }
while workspace.commit answers UpToDate. Typical causes: a build, install or test run that creates and removes thousands of temporary files in the Workspace, or a command that touches many directories.
This was the behaviour on main before #488 too (main refused any changed inode at all); #488 fixed the case below the cap and left this one unchanged.
Smaller related points
rowscounts every inode row read, including the skipped no-op ones, so the cap is reached by rows that are no change.- When exactly
ROWSrows exist and all were read,>=still marks the report truncated. - Since fix(daemon): name every change a dirty Unmount refusal counts #488 the row counter includes directory rows (main counted only non-directory rows), so with many changed directories fewer file paths are named.
truncatedis set, so the report is honest; only naming is affected. - Each changed base directory costs one
TreeTable::valueplus oneread_portable, one at a time. Commit reads the same records in batches (FilesystemRead::lookup_inodes).
Proposed change
The check needs a bound on memory, not on total rows read:
- Page through
active_inodesandactive_directory_entrieswindow by window until they are exhausted, each window one short overlay job as now (no lock held across Store reads). - Do not collect every row. Keep at most
PATHS = 256named paths and drop each window after it is processed, so resident state stays bounded however many rows exist. - Stop early once
PATHSchanges are found: the Workspace is dirty andtruncated: true. - Reaching the end with no change a Commit would keep means clean: the Unmount proceeds.
- Batch the base-directory mode comparison per window instead of one lookup per directory.
WALK = 4096 (the base walk that names a path) can stay: reaching it only leaves a path unnamed and sets truncated; it does not decide clean or dirty.
Cost: a Workspace made only of no-op rows is scanned to the end before Unmount, the same order of work a Commit does over those rows. A Workspace with real changes stops early.
Expected
- A Workspace whose overlay holds no change a Commit would keep unmounts without
discard, whatever the number of rows. truncatedis set only when changes exist that are not listed.- Memory used by the check stays bounded.
Tests
Extend engine/crates/layerfs-api/tests/unmount_dirty.rs:
- more than 4096 create+unlink pairs unmount without
discard; - the same plus one real change refuses and names that change;
- exactly 4096 no-op rows is not truncated;
chmodon a directory and back to its base mode unmounts cleanly;- removing a base directory refuses and names it.
Workaround until then
workspace.commit before workspace.unmount: it answers UpToDate, settles the overlay, and the Unmount then succeeds.
Activity
fix(daemon): stream the dirty check past any number of no-op rows (#489)
Merge pull request #491 from YidaWeng/fix/489-dirty-truncation