Report a media href that reaches its file only under case folding

CHANGELOG.md

@@ -52,7 +52,7 @@ releases may contain breaking changes.
 - LIFT-folder handling: `RangesFile` (standalone `.lift-ranges` documents,
   same fidelity guarantees), automatic companion discovery/tracking on load
   (`Lexicon.ranges_files`), `save()` writes companions together,
-  `all_ranges()` merged view, `media_refs()` / `missing_media()` helpers,
+  `all_ranges()` merged view, `media_refs()` / `check_media()` helpers,
   build-from-scratch helpers `Lexicon.add_ranges_file()` /
   `RangesFile.add_range()` / `Range.add_element()` (`save()` writes and
   header-references a new companion beside the `.lift`); vendored
@@ -71,7 +71,7 @@ releases may contain breaking changes.
   file, entry, and line it concerns. RELAX NG layer with two documented
   departures from strict validation (invalid `file://` hrefs downgraded to
   `uri-not-rfc` warnings; legal interleaving not falsely flagged); vendored
-  ranges schema over companions; and twelve semantic checks, one `Problem`
+  ranges schema over companions; and thirteen semantic checks, one `Problem`
   code each (with missing-id opt-in via `require_ids`). Every code is
   described in `docs/en/guides/validate.md`.
 - Canonical sort: `Lexicon.sort()` / `RangesFile.sort()` (entries by

CONTRIBUTING.md

@@ -44,10 +44,11 @@ The fidelity tests assert that saving writes back the **exact bytes** of
   `tests/corpus/`, or `tests/tools/xslt/`. Even a trailing-newline tweak breaks
   the suite. `.gitattributes` and `.editorconfig` list exceptions so git and
   editors leave these files alone — don't remove them.
-- Adding a fixture requires an entry in `tests/corpus/PROVENANCE.md`: source
-  URL, commit SHA, fetch date, license. Hand-authored fixtures (e.g. under
-  `tests/corpus/negative/`) carry an XML comment documenting the defect and the
-  expected validator finding.
+- Adding a fetched fixture requires an entry in `tests/corpus/PROVENANCE.md`:
+  source URL, commit SHA, fetch date, license. Hand-authored fixtures under
+  `tests/corpus/negative/` follow the rule in that file's `negative/` section
+  instead of getting an entry each, and carry an XML comment documenting the
+  defect and the expected validator finding.
 - The migrated `spec-examples/0.13/` files are generated by
   `tests/tools/migrate_corpus.py`; regenerate rather than edit.
 

docs/en/guides/cli.md

@@ -9,14 +9,22 @@ sil-lift validate PATH [--format {text,json}] [--strict] [--no-check-media] [--a
 sil-lift stats PATH [--format {text,json}]
                                            entry/sense/language counts (streaming; any size)
 sil-lift sort PATH [-o OUT]               canonically sorted, diff-ready copy (default: in place)
-sil-lift check-media PATH                 missing and orphaned media report; exit 1 if missing
+sil-lift check-media PATH                 missing, misspelled, and orphaned media report; exit 1 if missing or misspelled
 sil-lift export PATH [-o OUT] [--langs L] [--tsv]
                                            one row per leaf sense (subsenses flattened) to CSV/TSV (streaming)
 ```
 
-`--format json` writes a single JSON object to stdout (and nothing else) for CI/automation consumption; see the schema in the example below. `--strict` treats warnings as errors, exiting 1 if any are found — use it to gate a build on no warnings at all rather than on errors alone. `--no-check-media` skips the filesystem media-presence check (suppressing `missing-media` findings), which is useful when validating a freshly generated export whose audio/photo files live elsewhere rather than in the same folder. `--require-ids` additionally fails (a `missing-id` error) on any entry lacking a `guid` or sense lacking an `id` — stricter than LIFT, for workflows that re-import by a stable id. Passing `-` as the path reads the document from stdin (a piped document has no folder, so its companion `.lift-ranges` and media are not resolved). `stats` likewise takes `--format json`, emitting the counts as a single JSON object.
+`validate`'s options:
 
-`--allow CODE[,CODE...]` (repeatable) reports the given codes' findings but leaves them out of the pass/fail decision. It applies to errors and warnings alike, so an allowed warning does not trip `--strict` either. The summary line tallies allowed findings per code; the JSON summary's `allowed` gives their total. A code that never occurs is ignored, so an allow list keeps working after a check is retired.
+- `--format json` writes a single JSON object to stdout (and nothing else) for CI/automation consumption; see the schema in the example below.
+- `--strict` treats warnings as errors, exiting 1 if any are found — use it to gate a build on no warnings at all rather than on errors alone.
+- `--no-check-media` skips the filesystem media-presence check, suppressing `missing-media` and `media-href-mismatch` findings. Useful when validating a freshly generated export whose audio/photo files live elsewhere rather than in the same folder.
+- `--allow CODE[,CODE...]` (repeatable) reports the given codes' findings but leaves them out of the pass/fail decision. It applies to errors and warnings alike, so an allowed warning does not trip `--strict` either. The summary line tallies allowed findings per code; the JSON summary's `allowed` gives their total. A code that never occurs is ignored, so an allow list keeps working after a check is retired.
+- `--require-ids` additionally fails (a `missing-id` error) on any entry lacking a `guid` or sense lacking an `id` — stricter than LIFT, for workflows that re-import by a stable id.
+
+Passing `-` as the path reads the document from stdin. A piped document has no folder, so its companion `.lift-ranges` and media are not resolved.
+
+`stats` likewise takes `--format json`, emitting the counts as a single JSON object.
 
 !!! note
     `validate`'s exit codes and `--format json` schema are a supported automation interface: both are covered by tests and change only under SemVer.
@@ -73,4 +81,4 @@ $ sil-lift export dictionary.lift --langs en,fr -o dictionary.csv
 
 All output is UTF-8, on every platform and whether it goes to a console, a pipe, or a `>` redirect — never the locale encoding (cp1252 on Windows, ASCII under a C/POSIX locale), which cannot represent LIFT content. `sil-lift export dictionary.lift > dictionary.csv` therefore writes exactly the bytes `-o dictionary.csv` writes, CRLF row terminators included.
 
-Exit codes: `0` success (warnings do not fail the run unless `--strict`), `1` findings (validation errors / missing media / warnings under `--strict`, not counting codes given to `--allow`), `2` an I/O failure at either end — input that cannot be read, or output that cannot be written (a reader like `head` closing the pipe, a full disk).
+Exit codes: `0` success (warnings do not fail the run unless `--strict`), `1` findings (validation errors / missing or misspelled media / warnings under `--strict`, not counting codes given to `--allow`), `2` an I/O failure at either end — input that cannot be read, or output that cannot be written (a reader like `head` closing the pipe, a full disk).

docs/en/guides/folder-media.md

@@ -41,11 +41,18 @@ Pass `resolve_ranges=False` to `load()` to skip companion discovery.
 for ref in lex.media_refs():        # every <media> and <illustration>
     print(ref.kind, ref.href, ref.entry_id)
 
-lex.missing_media()                 # refs whose files don't exist
+lex.check_media()                   # refs that don't resolve cleanly
 ```
 
 Resolution follows the conventional layout: a relative href is checked as given (backslashes normalized — WeSay writes `pictures\photo with space.png`) and under `audio/` (for pronunciation media) or `pictures/` (for illustrations). Remote/absolute hrefs can't be checked and are skipped.
 
+`check_media()` reports only the references that did not resolve cleanly, one `MediaResolution` each:
+
+- `status="missing"` — no file answered the href under any spelling.
+- `status="mismatch"` — one did, but only under case folding or NFC; `found` names the file on disk.
+
+A href that spells its file exactly is not reported. Unlike a companion name, which folds silently, a media href is also read by whatever serves the folder afterwards, so the fold is reported as [`media-href-mismatch`](validate.md#problem-codes).
+
 ## Other folder contents
 
 A LIFT folder often holds files sil-lift doesn't model — writing-system LDML under `WritingSystems/`, The Combine's speaker consent audio/image files under `consent/`, and the like; `load()`/`save()` leave these untouched, and [`Lexicon.save_zip()`](lift-export-interop.md) carries them through verbatim when packaging the folder.

docs/en/guides/validate.md

@@ -24,11 +24,11 @@ Each `Problem` carries `level` (`"error"`/`"warning"`), a stable `code`, `messag
 
 1. **RELAX NG** against the LIFT 0.13 grammar (vendored from lift-standard — a byte-identical copy committed into this package).
 2. **Ranges schema** — this project's `lift-ranges-0.13.rng` — over every tracked `.lift-ranges` companion, addressed to the companion rather than the `.lift`.
-3. **Semantic checks** the grammar cannot express — twelve of them, one code each.
+3. **Semantic checks** the grammar cannot express — thirteen of them, one code each.
 
 ## Problem codes
 
-Every finding carries one of these, whichever layer produced it — `schema` and `uri-not-rfc` come from the schema layers, the other twelve are semantic checks. The strings are a supported interface; `--strict` promotes every warning to an error. `sil-lift validate --allow CODE` leaves a code out of the pass/fail decision.
+Every finding carries one of these, whichever layer produced it — `schema` and `uri-not-rfc` come from the schema layers, the other thirteen are semantic checks. The strings are a supported interface; `--strict` promotes every warning to an error. `sil-lift validate --allow CODE` leaves a code out of the pass/fail decision.
 
 | code                     | level   | what it flags                                                              |
 | ------------------------ | ------- | -------------------------------------------------------------------------- |
@@ -38,6 +38,7 @@ Every finding carries one of these, whichever layer produced it — `schema` and
 | `duplicate-form-lang`    | warning | two forms in one multitext sharing a language                              |
 | `duplicate-guid`         | error   | a guid reused among entries, or among one document's ranges/range-elements |
 | `form-missing-lang`      | error   | a `<form>` or `<gloss>` without the `lang` the schema requires             |
+| `media-href-mismatch`    | warning | a media href that reaches its file only under case folding or NFC          |
 | `missing-id`             | error   | opt-in via `require_ids`: an entry without a guid, a sense without an id   |
 | `missing-media`          | warning | a referenced audio or picture file not on disk                             |
 | `normalization-mismatch` | warning | a name that reaches the id it refers to only under NFC                     |
@@ -51,7 +52,14 @@ All three layers work from the document serialized as it stands, so one that can
 
 A companion name matching several files loads none of them: the ranges they define go absent until all but one is renamed or removed.
 
-The three companion-folder codes (`ambiguous-ranges-file`, `dangling-ranges-href`, and `unreadable-ranges-file`) are reported only when companion discovery ran. Loading with `resolve_ranges=False` puts companions out of scope, so none of them is reported; attaching one afterwards with `add_ranges_file()` does not bring them back, since it never reads the folder these codes report on. `missing-media` is unaffected: media is never resolved into the model, so nothing was opted out of.
+The three companion-folder codes (`ambiguous-ranges-file`, `dangling-ranges-href`, and `unreadable-ranges-file`) are reported only when companion discovery ran. Loading with `resolve_ranges=False` puts companions out of scope, so none of them is reported; attaching one afterwards with `add_ranges_file()` does not bring them back, since it never reads the folder these codes report on. `missing-media` and `media-href-mismatch` are unaffected: media is never resolved into the model, so nothing was opted out of.
+
+`missing-media` and `media-href-mismatch` divide the media check between them: the first means no file answered the href under any spelling, the second that one did but the href does not name it exactly. The second is the portability finding — the file is here, and a case-sensitive host serving this folder will not find it.
+
+Two rules govern what `media-href-mismatch` reports:
+
+- A misspelled *folder* is one rename however many references cross it, so it is reported once and carries no entry. A misspelled *filename* is reported per reference, addressed to the entry that wrote it.
+- The conventional `audio/`/`pictures/` subfolder is sil-lift's own guess rather than something the document wrote, so a folder spelling it another way resolves without being reported.
 
 ## Real-world FieldWorks (FLEx) output
 

src/sil_lift/__init__.py

@@ -21,6 +21,7 @@
     GrammaticalInfo,
     Lexicon,
     MediaRef,
+    MediaResolution,
     Note,
     Pronunciation,
     RangesChanges,
@@ -62,6 +63,7 @@
     "LiftWriteError",
     "LiftWriter",
     "MediaRef",
+    "MediaResolution",
     "Multitext",
     "Note",
     "Problem",

src/sil_lift/_cli.py

@@ -25,9 +25,9 @@
 
 from ._canonical import canonicalize
 from ._errors import LiftError
-from ._model import Lexicon, _normalize_href
+from ._model import Lexicon, _folded_entries, _media_matches, _normalize_href
 from ._stream import open_reader
-from ._validate import iter_problems
+from ._validate import iter_problems, media_mismatch_groups
 
 if TYPE_CHECKING:
     from collections.abc import Sequence
@@ -39,6 +39,9 @@
 
 __all__ = ["main"]
 
+#: What --no-check-media suppresses: everything the filesystem media check says.
+_MEDIA_CODES = frozenset({"missing-media", "media-href-mismatch"})
+
 
 def _problem_json(problem: Problem) -> dict[str, object]:
     return {
@@ -68,7 +71,7 @@ def _collect_problems(args: argparse.Namespace) -> list[Problem]:
     return [
         problem
         for problem in problems
-        if not (args.no_check_media and problem.code == "missing-media")
+        if not (args.no_check_media and problem.code in _MEDIA_CODES)
     ]
 
 
@@ -168,25 +171,42 @@ def _cmd_sort(args: argparse.Namespace) -> int:
 
 def _cmd_check_media(args: argparse.Namespace) -> int:
     lexicon = Lexicon.load(args.path)
-    missing = lexicon.missing_media()
-    for ref in missing:
-        owner = ref.entry_id or ref.entry_guid or "?"
-        print(f"missing  {ref.kind:12s} {ref.href!r} (entry {owner})")
-
-    referenced: set[Path] = set()
     base = lexicon.path.parent if lexicon.path is not None else Path(args.path).parent
+    resolutions = lexicon.check_media()
+    missing = [item for item in resolutions if item.status == "missing"]
+    for item in missing:
+        owner = item.ref.entry_id or item.ref.entry_guid or "?"
+        print(f"missing  {item.ref.kind:12s} {item.ref.href!r} (entry {owner})")
+    directories, files = media_mismatch_groups(resolutions, base)
+    # Spellings differing only in normalization render identically.
+    for written, on_disk in directories:
+        print(f"mismatch {'folder':12s} {written!a} is {on_disk!a} on disk")
+    for item, written, on_disk in files:
+        owner = item.ref.entry_id or item.ref.entry_guid or "?"
+        print(f"mismatch {item.ref.kind:12s} {written!a} is {on_disk!a} on disk (entry {owner})")
+
+    # The files the hrefs reach, under the folding check_media() uses, rather
+    # than the paths they spell: a file named inexactly is in use, not orphaned.
+    referenced: set[Path] = set()
+    listings: dict[Path, dict[str, list[Path]]] = {}
     for ref in lexicon.media_refs():
         relative = _normalize_href(ref.href)
-        if relative is None:  # remote/absolute hrefs can't confirm a local file
+        if relative is None or not relative.parts:  # remote/absolute/empty: no local file
             continue
-        referenced.add((base / relative).resolve())
         subfolder = "audio" if ref.kind == "media" else "pictures"
-        referenced.add((base / subfolder / relative).resolve())
+        for candidate in (relative, Path(subfolder) / relative):
+            for match in _media_matches(base, candidate, listings):
+                referenced.add(match.resolve())
+    media_folders = [
+        path
+        for name in ("audio", "pictures")
+        for path in _folded_entries(base, listings).get(name, ())
+        if path.is_dir()
+    ]
     orphans = [
         file
-        for folder in ("audio", "pictures")
-        if (base / folder).is_dir()
-        for file in sorted((base / folder).rglob("*"))
+        for folder in media_folders
+        for file in sorted(folder.rglob("*"))
         if file.is_file() and file.resolve() not in referenced
     ]
     for file in orphans:
@@ -196,8 +216,9 @@ def _cmd_check_media(args: argparse.Namespace) -> int:
             "note: WeSay-style audio writing systems reference files from form "
             "text, which this check does not follow"
         )
-    print(f"{len(missing)} missing, {len(orphans)} orphaned")
-    return 1 if missing else 0
+    mismatched = len(directories) + len(files)
+    print(f"{len(missing)} missing, {mismatched} mismatched, {len(orphans)} orphaned")
+    return 1 if missing or mismatched else 0
 
 
 def _leaf_senses(entry: Entry) -> list[Sense]:
@@ -355,7 +376,10 @@ def main(argv: Sequence[str] | None = None) -> int:
     validate.add_argument(
         "--no-check-media",
         action="store_true",
-        help="skip the filesystem media-presence check (suppresses missing-media findings)",
+        help=(
+            "skip the filesystem media-presence check "
+            "(suppresses missing-media and media-href-mismatch findings)"
+        ),
     )
     validate.add_argument(
         "--allow",