Skip to content

fix(posix): back off retries of files that failed to assimilate - #796

Open
NickWalters wants to merge 1 commit into
opencloud-eu:mainfrom
NickWalters:fix/posix-assimilation-backoff
Open

fix(posix): back off retries of files that failed to assimilate#796
NickWalters wants to merge 1 commit into
opencloud-eu:mainfrom
NickWalters:fix/posix-assimilation-backoff

Conversation

@NickWalters

@NickWalters NickWalters commented Sep 10, 2026

Copy link
Copy Markdown

Fixes opencloud-eu/opencloud#3498

If a file can't be assimilated for a reason that doesn't fix itself (in my case the opencloud user had no write permission on the files, so setting the user.oc.* xattrs failed), nothing remembers that it failed. The next directory listing or scan tries again, and every attempt reads the whole file to compute its checksums before it gets to the part that fails. On my setup that came to about 117 MB/s of reads for 46 hours before I tracked it down.

With this change updateFile remembers files that failed after their checksums were computed, keyed by path, in an expirable LRU (golang-lru is already a dependency). While a file's mtime, size, mode and inode stay the same, it isn't read again until a delay has passed. The delay starts at a minute and doubles with each failure, up to 24h. If the file changes, for example someone chmods it or replaces it, it's retried right away. Entries expire after 48h so paths that no longer exist don't pile up.

Only failures after the checksum step are remembered. Everything before it is cheap (locking, parent lookups, reading attributes), so those still retry on the next scan or listing like before. That way a file that failed because its parent directory couldn't be assimilated is picked up as soon as the directory is fixed, and a short ID cache outage, which mostly fails at the parent lookup, doesn't leave files waiting on a backoff.

The skip still returns the original error, so the logs keep saying what's actually wrong:

item is unchanged since it failed to assimilate, not retrying before 2026-09-10T23:21:10+08:00: failed to set attributes: failed to set 10/10 xattrs: xattr.Set .../readonly user.oc.type: permission denied

One gap: a chown or ACL change doesn't touch mtime, size or mode, so a fix made that way is only picked up at the next retry or after a restart. I thought about comparing ctime instead, but ctime also changes whenever some of the xattr writes succeed, so a file where only part of them fail would loop again.

For tests, there's a new spec in tree_non_watching_test.go that lists a folder containing a read-only file twice and uses the file's access time to check that only the first listing reads it. It fails on main and passes with this change. It skips when running as root, since root can set xattrs on read-only files, and also on file systems that don't update access times. The unit-test pipeline runs as root in the golang image, so I also added whitebox specs for the retry and backoff logic, which do run there. go test -race ./pkg/storage/fs/posix/... passes locally and golangci-lint is clean on the package.

Something related that I didn't change here: updateFile propagates a file's size to its parents before writing the file's attributes, so every failed attempt inflates the tree sizes. With this change that only happens once per retry, but it's still wrong, so I fixed it separately in #797. The two PRs don't depend on each other and merge cleanly in either order.

@NickWalters NickWalters changed the title fix(posix): back off retries of items that failed to assimilate fix(posix): back off retries of files that failed to assimilate Sep 10, 2026
@NickWalters
NickWalters force-pushed the fix/posix-assimilation-backoff branch from ee0cc56 to fcd3901 Compare September 10, 2026 15:40
A file that failed to assimilate after its checksums were computed, e.g.
because the service user can't set extended attributes on it, was
assimilated again on every directory listing and scan, reading the whole
file each time.

Remember such failures per path and don't read the file again while it
is unchanged (same mtime, size, mode and inode) until a retry delay has
passed. The delay starts at a minute and doubles with every failure up
to a day. Failures before the checksum step are cheap and still retried
right away. The skip returns the last error so logs still say why the
file isn't assimilated.

Fixes opencloud-eu/opencloud#3498
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

posix: files that fail to assimilate are re-read in full on every scan, with no backoff

1 participant