Unicode-safe text handling: fix #51, the panic beside it, and one normalization form - #52
Open
vyrti wants to merge 3 commits into
Open
Unicode-safe text handling: fix #51, the panic beside it, and one normalization form#52vyrti wants to merge 3 commits into
vyrti wants to merge 3 commits into
Conversation
…g titles Issue #51 fixed the byte slice that split a `ü`. This covers the rest of the class it belonged to. A `clippy::string_slice` sweep over the app crates enumerated all 28 string index sites. Indices from `find`/`rsplit_once`/`split_at`, and offsets past an ASCII prefix already proven by `starts_with`, are boundary-safe. One site was not: `has_drive_letter` checked characters 1 and 2 but never constrained character 0, so `Ü:\Musik` reached a `path_str[1..]` that splits it in half. The deeper problem is that the same name has more than one spelling. `Füßen` is NFC from a Windows tagger and NFD from macOS, canonically equivalent and sharing no bytes, so `LIKE` and FTS silently said no. Worse, FTS shredded a decomposed term into two tokens, because a combining mark is `Mn` and the tokenizer splits on anything non-alphanumeric — a user searching for their own music found nothing. Stored text is now folded to NFC at the single write funnel and search terms are folded on the way in. Paths are deliberately excluded: a path is a filesystem key, and on ext4 the two spellings are two different files. Rows written before this release are folded once by migration v8, via an `nfc()` SQL scalar, rather than left unfindable until a rescan. Tests pad every sample through eight alignments, because a flat list of awkward strings does not catch an offset bug — the panic needs the slice point to land inside a character, which depends on the string's length. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GA38ynA6ZMQzsogtkAnJpe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue #51 reported a panic on a German umlaut in a FLAC filename. This PR fixes that, then the class it belonged to: every script and symbol, not one report at a time.
The reported panic, and the rest of its class
clippy::string_slicesweep enumerated all 28 string-index sites in the app crates; every remaining one is boundary-safe by construction (indices fromfind/rsplit_once/split_at, or offsets past an ASCII prefix proven bystarts_with)has_drive_letterchecked characters 1 and 2 but never constrained character 0, soÜ:\Musikreached apath_str[1..]that splits it mid-character. Windows-only, so--libcfg's it out locally and CI is the first real check of it.One normalization form for stored and searched text
The same name has more than one spelling.
Füßenis NFC from a Windows tagger and NFD from macOS — canonically equivalent, no bytes in common. Measured with the fold disabled, two of four search directions silently returned nothing:LIKEThe FTS failure is the worse one: a combining mark is
Mn, andfts5_query_from_user_textsplits on anything non-alphanumeric, so a decomposed term was shredded into two tokens. Someone searching for their own music found nothing.LIKEpathscanonical_to_platformhands the stored path straight toPathBuf, and on a byte-exact filesystem the two spellings are two different files — folding it would produce a name the kernel cannot open. Onlyfilename, the display copy, folds; a test asserts the path returns byte-identical.Schema migration (v7 → v8) — worth a reviewer's attention
Rows written before this release would stay unfindable until something happened to rewrite them, so they are folded once by migration v8 through an
nfc()SQL scalar. DIDL object ids are preserved, so renderers do not lose favourites or resume points. TheWHERErestricts the write to rows that actually change, so an ASCII-only library is a scan and no writes.Why the tests pad
A flat list of awkward strings does not catch an offset bug: the panic needs the slice point to land inside a character, which depends on the string's length relative to that character's width.
unicode_corpus::alignment_sweeprepeats every sample at eight paddings so computed offsets land on every byte position, continuation bytes included. Both new guards were verified non-vacuous by reverting the fix and watching them fail — the sweep caught #51 independently, at a different offset from the targeted test.Verification
cargo test -p vuio-core --lib— 595 passed, 1 ignoredcargo clippy -p vuio-core --all-targets --all-features -- -D warningscargo fmt --all -- --checkFixes #51