perf(extras): batch the post-scan metadata-file cleanup and probe the right base path first - #273
Open
jordanfelle wants to merge 2 commits into
Open
jordanfelle wants to merge 2 commits into
jordanfelle wants to merge 2 commits into
Conversation
… right base path first CleanExtraFileService.Clean ran after every author scan and, for each MetadataFiles row, probed the author's base paths in a fixed audiobook -> ebook -> legacy order and deleted missing rows one by one. DiskProviderBase.FileExists falls back to listing directories when the exact path misses, so an ebook sidecar always paid for a miss under the audiobook folder before its hit, and every missing row cost its own DELETE. - Probe the base path the row's book file lives under first. Sidecars are written under ExtraFilePathHelper.GetPreferredBasePath(author, bookFile), so that is where they are. Rows with no book file keep the existing order, and every base path is still tried before a row counts as missing. New ExtraFilePathHelper.GetAuthorBasePaths(author, preferredBasePath) overload; one IMediaFileService.GetFilesByAuthor query per author (optional constructor dependency, no effect when absent). - Delete all missing rows with one DeleteMany instead of one Delete per row. - Skip rows with a blank RelativePath instead of throwing from Path.Combine. - Return early for an author with no metadata files. Tests: one batched delete with the exact ids; nothing deleted when all files exist; a linked ebook sidecar is found on the first probe and the audiobook folder is never touched; a file that exists only under the other base path is kept; default order for rows without a book file; blank paths ignored; no work for an author with no rows; works without the media file service. The batching and preferred-base tests fail if either change is reverted. Refs Chaptarr#271
… exists Adversarial review of Chaptarr#273: a BookFileId that is not among the author's book files (deleted, or never mapped) must fall back to the default probe order and still keep a file that exists under another base path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #271.
Problem
CleanExtraFileService.Cleanruns after every author scan. For eachMetadataFilesrow it probed the author's base paths in a fixed audiobook, ebook, legacy order, and it deleted missing rows one at a time.DiskProviderBase.FileExistsfalls back to listing directories when the exact path misses, so:DELETE.Change
ExtraFilePathHelper.GetPreferredBasePath(author, bookFile), so for rows that point at a book file that path is tried first. Rows with no book file keep the existing order, and every base path is still tried before a row counts as missing, so which rows get deleted is unchanged. Adds anExtraFilePathHelper.GetAuthorBasePaths(author, preferredBasePath)overload and oneIMediaFileService.GetFilesByAuthorquery per author (optional constructor dependency, no effect when absent).DeleteMany) for all missing rows instead of oneDeleteper row.RelativePathinstead of throwing fromPath.Combine.What this does not change
FileExistsitself, the case-insensitive fallback, and the "list each folder once" idea from #271. That would change matching semantics and belongs in its own change.Tests
CleanExtraFileServiceFixture(9 tests): one batched delete with the exact ids; nothing deleted when all files exist; a linked ebook sidecar is found on the first probe and the audiobook folder is never touched; a file that exists only under the other base path is kept; default order for rows with no book file; blank paths ignored; no work for an author with no rows; a row whose linked book file no longer exists falls back to the default order; works without the media file service. The batching and preferred-base tests fail if either change is reverted (checked). FullChaptarr.Core.Test: 3044 passed.Not measured
I have not measured the wall-clock effect of this on a real library; the claim is fewer probes and fewer statements, not a speedup figure. The cost of a full-library scan on network storage is what #271 describes.
Note on cost
The added
GetFilesByAuthoris one join (BookFile, Edition, Book, Author) per scan, and only when the author has metadata rows. For an author with many book files and very few sidecars it could cost more than the probes it saves; the existing lighterGetMappedFilePathEvidenceByAuthorreturns no book-file id, so it cannot be used here.