I ran into this with the posix driver. When a file can't be assimilated for a reason that isn't going to go away, nothing records that it failed. The file stays uncached, so every scan and every directory listing tries to assimilate it again, and each attempt reads the whole file to compute its sha1, md5 and adler32. There's no backoff and no retry limit, so OpenCloud just keeps re-reading the same files as fast as the disk allows.
In my case it was permissions. opencloud.service runs as User=opencloud, but the tree I pointed it at was owned by root:media, so the opencloud user couldn't write to the files. Setting user.* xattrs needs write permission on the file, so every attempt ended with:
xattr.Set .../00001.m2ts user.oc.id: permission denied
-> failed to assimilate node
and the next listing logged record not found in cache. Assimilate. and read the file again.
I know the docs say the tree should be owned by the OpenCloud user and not shared with anything else, and my setup didn't follow that. I'm not asking for that layout to be supported. The problem is that getting it wrong ends up as a read loop that never stops, instead of an error you'd notice.
What it did on my machine:
- about 117 MB/s of reads for 46 hours, with around 517 assimilation attempts per 24 hours
- everything else on that array was starved for I/O
STORAGE_USERS_POSIX_MAX_CONCURRENCY=2 didn't help, because it limits how many run at once, not how often they repeat- it was hard to trace back to OpenCloud: load was 11 but user CPU was only 4% (47% iowait), so anything looking at CPU showed the service as idle
The permission problem is just what triggered it for me. I'd expect any per-file failure that keeps happening to cause the same loop. #1585 (file name too long) looks like the same thing with a different cause, and that one was fixed by handling long names rather than by changing how failures are retried. #3470 might be this too (constant ~100% I/O pressure, low memory, no requests), but I haven't checked whether that reporter has files that keep failing.
The code is in reva, pkg/storage/fs/posix/tree/assimilation.go:
workScanQueue logs the error and moves on, and nothing about the failure is kept- the
_errSkipAlreadyKnown shortcut compares against the mtime stored in the file's xattrs, and those were never written, so a file that failed can't take that path
updateFile calls node.CalculateChecksums on every attempt, so every retry is a full read
I've opened opencloud-eu/reva#796 with a fix. It remembers files that failed after the checksums were computed and doesn't read them again until they change or a backoff delay has passed. Two other things that might be worth doing: treating errors like EACCES/EPERM on the xattr write or ENAMETOOLONG as permanent, or checking at startup that the service user can actually write xattrs in the tree and logging a clear error if it can't.
Environment:
- OpenCloud 7.3.0 (rolling) when I hit it, and the code above is the same on main as of 7.5.0
- Debian in an unprivileged LXC, native systemd install (no Docker), ZFS storage
STORAGE_USERS_DRIVER=posix, STORAGE_USERS_POSIX_WATCH_FS=true, SCAN_FS=true
I ran into this with the posix driver. When a file can't be assimilated for a reason that isn't going to go away, nothing records that it failed. The file stays uncached, so every scan and every directory listing tries to assimilate it again, and each attempt reads the whole file to compute its sha1, md5 and adler32. There's no backoff and no retry limit, so OpenCloud just keeps re-reading the same files as fast as the disk allows.
In my case it was permissions.
opencloud.serviceruns asUser=opencloud, but the tree I pointed it at was owned byroot:media, so the opencloud user couldn't write to the files. Settinguser.*xattrs needs write permission on the file, so every attempt ended with:and the next listing logged
record not found in cache. Assimilate.and read the file again.I know the docs say the tree should be owned by the OpenCloud user and not shared with anything else, and my setup didn't follow that. I'm not asking for that layout to be supported. The problem is that getting it wrong ends up as a read loop that never stops, instead of an error you'd notice.
What it did on my machine:
STORAGE_USERS_POSIX_MAX_CONCURRENCY=2didn't help, because it limits how many run at once, not how often they repeatThe permission problem is just what triggered it for me. I'd expect any per-file failure that keeps happening to cause the same loop. #1585 (
file name too long) looks like the same thing with a different cause, and that one was fixed by handling long names rather than by changing how failures are retried. #3470 might be this too (constant ~100% I/O pressure, low memory, no requests), but I haven't checked whether that reporter has files that keep failing.The code is in reva,
pkg/storage/fs/posix/tree/assimilation.go:workScanQueuelogs the error and moves on, and nothing about the failure is kept_errSkipAlreadyKnownshortcut compares against the mtime stored in the file's xattrs, and those were never written, so a file that failed can't take that pathupdateFilecallsnode.CalculateChecksumson every attempt, so every retry is a full readI've opened opencloud-eu/reva#796 with a fix. It remembers files that failed after the checksums were computed and doesn't read them again until they change or a backoff delay has passed. Two other things that might be worth doing: treating errors like
EACCES/EPERMon the xattr write orENAMETOOLONGas permanent, or checking at startup that the service user can actually write xattrs in the tree and logging a clear error if it can't.Environment:
STORAGE_USERS_DRIVER=posix,STORAGE_USERS_POSIX_WATCH_FS=true,SCAN_FS=true