Skip to content

Unicode-safe text handling: fix #51, the panic beside it, and one normalization form - #52

Open
vyrti wants to merge 3 commits into
mainfrom
fix/issue-51-unicode-filenames
Open

Unicode-safe text handling: fix #51, the panic beside it, and one normalization form#52
vyrti wants to merge 3 commits into
mainfrom
fix/issue-51-unicode-filenames

Conversation

@vyrti

@vyrti vyrti commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Issue #51 reported a panic on a German umlaut in a FLAC filename. This PR fixes that, then the class it belonged to: every script and symbol, not one report at a time.

The reported panic, and the rest of its class

  • parse Samsung title extensions at a Unicode-safe delimiter boundary instead of byte-indexing UTF-8 text
  • enable the Clippy string-slice denial in the XML renderer to prevent unsafe string slicing from returning
  • add regression coverage for the reported German-umlaut FLAC filename
  • exhaustively test every valid Unicode scalar in metadata titles, portable filenames, and XML escaping
  • cover missing, empty, mismatched, case-varied, multi-dot, and Unicode extensions
  • a clippy::string_slice sweep enumerated all 28 string-index sites in the app crates; every remaining one is boundary-safe by construction (indices from find/rsplit_once/split_at, or offsets past an ASCII prefix proven by starts_with)
  • a second panic, not in the report: has_drive_letter checked characters 1 and 2 but never constrained character 0, so Ü:\Musik reached a path_str[1..] that splits it mid-character. Windows-only, so --lib cfg's it out locally and CI is the first real check of it.

One normalization form for stored and searched text

The same name has more than one spelling. Füßen is NFC from a Windows tagger and NFD from macOS — canonically equivalent, no bytes in common. Measured with the fold disabled, two of four search directions silently returned nothing:

NFC query NFD query
FTS found nothing
LIKE nothing found

The FTS failure is the worse one: a combining mark is Mn, and fts5_query_from_user_text splits on anything non-alphanumeric, so a decomposed term was shredded into two tokens. Someone searching for their own music found nothing.

  • stored text folds to NFC at the single DB write funnel, so every writer is covered — tags, filenames, scrapers, playlists
  • search terms fold on the way in, on both the FTS and LIKE paths
  • paths are deliberately excluded. canonical_to_platform hands the stored path straight to PathBuf, and on a byte-exact filesystem the two spellings are two different files — folding it would produce a name the kernel cannot open. Only filename, the display copy, folds; a test asserts the path returns byte-identical.

Schema migration (v7 → v8) — worth a reviewer's attention

Rows written before this release would stay unfindable until something happened to rewrite them, so they are folded once by migration v8 through an nfc() SQL scalar. DIDL object ids are preserved, so renderers do not lose favourites or resume points. The WHERE restricts the write to rows that actually change, so an ASCII-only library is a scan and no writes.

Why the tests pad

A flat list of awkward strings does not catch an offset bug: the panic needs the slice point to land inside a character, which depends on the string's length relative to that character's width. unicode_corpus::alignment_sweep repeats every sample at eight paddings so computed offsets land on every byte position, continuation bytes included. Both new guards were verified non-vacuous by reverting the fix and watching them fail — the sweep caught #51 independently, at a different offset from the targeted test.

Verification

  • cargo test -p vuio-core --lib — 595 passed, 1 ignored
  • cargo clippy -p vuio-core --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check
  • CI green on all 14 checks, including both Windows targets

Fixes #51

vyrti and others added 3 commits September 12, 2026 21:19
…g titles

Issue #51 fixed the byte slice that split a `ü`. This covers the rest of the
class it belonged to.

A `clippy::string_slice` sweep over the app crates enumerated all 28 string
index sites. Indices from `find`/`rsplit_once`/`split_at`, and offsets past an
ASCII prefix already proven by `starts_with`, are boundary-safe. One site was
not: `has_drive_letter` checked characters 1 and 2 but never constrained
character 0, so `Ü:\Musik` reached a `path_str[1..]` that splits it in half.

The deeper problem is that the same name has more than one spelling. `Füßen` is
NFC from a Windows tagger and NFD from macOS, canonically equivalent and sharing
no bytes, so `LIKE` and FTS silently said no. Worse, FTS shredded a decomposed
term into two tokens, because a combining mark is `Mn` and the tokenizer splits
on anything non-alphanumeric — a user searching for their own music found
nothing. Stored text is now folded to NFC at the single write funnel and search
terms are folded on the way in. Paths are deliberately excluded: a path is a
filesystem key, and on ext4 the two spellings are two different files.

Rows written before this release are folded once by migration v8, via an `nfc()`
SQL scalar, rather than left unfindable until a rescan.

Tests pad every sample through eight alignments, because a flat list of awkward
strings does not catch an offset bug — the panic needs the slice point to land
inside a character, which depends on the string's length.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GA38ynA6ZMQzsogtkAnJpe
@vyrti vyrti changed the title fix(xml): handle Unicode Samsung titles Unicode-safe text handling: fix #51, the panic beside it, and one normalization form Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic on Unicode filename (ü, ö, ä)

1 participant