Repository navigation
feat: burn bitmap subtitles into Emby and Jellyfin transcodes - #186
Conversation
…and in the player With Settings > Subtitles on English, Auto and Show Forced Subtitles, every movie and series detail page showed subtitles Off. Settings saves on the device, but the detail pages seeded their subtitle choice from the server's remembered subtitle mode (Silo's settings cascade, or the emulated preferences for Emby and Jellyfin), and a remembered "off" went to the player as an explicit Off, so the player never applied the settings. - Settings > Subtitles, or the device's caption settings with Use Device Settings on, is now the only input to the automatic subtitle pick on Silo, Emby and Jellyfin. The detail pages no longer seed a remembered subtitle or Off. - The detail page's Subtitles control shows what playback will pick, from the player's own resolver and audio choice: "Auto: English (Forced)", "Auto: Off", the track for Always, Off for Off. It shows for every file, including one without subtitles. - Switching audio runs Auto again in every mode, not only with Use Device Settings. - A Silo replan (audio switch, quality change) no longer drops the subtitle: a picked embedded track is re-armed and an automatic one is chosen again. - Show Forced Subtitles off never picks a forced track, even the language's only one. A track titled Forced counts as forced. A forced pick carried to the next episode stays forced. - Use Device Settings: caption settings are read again on returning to the app and at playback start, the caption font chosen in Accessibility is used, and the Apple TV in-player Appearance dialog no longer silently turns the setting off. - Embedded subtitles draw the Shadow style in black at 85%, as external ones do, instead of the unused box colour, and the subtitle style and delay are re-applied after every load. - iPhone and iPad grey out Show Forced Subtitles when Behaviour is Off, as Apple TV does.
Vivid no longer reads a server's remembered subtitle prefs, so the Apple TV detail page's Auto shouldn't delete the per-series pref that Silo's web app may still use. Also notes the Auto row in the plugins guide.
…t be read If reading a file's embedded subtitles failed, the iPhone and iPad sheet dropped Auto and Off, and the Apple TV menu dropped Off, so a viewer couldn't go back to Auto or turn subtitles off for that file.
- A subtitle or Off picked while a Silo replan is in flight is no longer undone: the playing subtitle is read after the wait, not before. - The Silo plan request no longer falls back to the server's remembered subtitle mode and track; without preferences it asks for no subtitles and the player picks from Settings. - With Use Device Settings on, returning to the app applies the device's caption style and choice to an open player. - Removed the now unused subtitle pref writers, their request model and tests, and stale comments on the Apple TV detail views. - Docs: a file without subtitles reads Auto: Off (or Off), and the greyed-out controls and Accessibility caption font are described.
Settings > Subtitles gains Backup Language on iPhone, iPad and Apple TV. When a title has no subtitles in your language, the backup language's full track plays instead, on Silo, Emby and Jellyfin, and the detail pages show it. The backup never counts as your language for Auto's audio check, so an Arabic viewer with an English backup still gets English subtitles on an English film without Arabic ones. It's saved on the device for the profile like the other subtitle settings, and does nothing with Language set to None or Behaviour set to Off.
…et when it matches - With a backup language set, a full track in it now beats a forced (signs-only) track in the main language, so the dialogue can be followed. - Setting Language to the backup's language clears the backup, so its picker never points at a row it doesn't list. - Footers and docs say the backup is used when full subtitles would play, not with Auto and audio already in your language. - The playback prefs header no longer lists the subtitle-prefs endpoints as used.
…utomatic and Forced Only rules - Always with Language set to None no longer shows nothing: it uses the device's language when the file has it, otherwise the file's default track, otherwise its first full track. - Use Device Settings on Automatic now follows Apple's rule: subtitles in the device's language when the audio is in another language, not the caption-language history (which could show Spanish on an English film after one pick in another app). - Use Device Settings on Forced Only shows the forced track in the audio's language, as Apple does.
…th no language With Show Forced Subtitles on, a forced track in the device's language could beat a full track elsewhere. Full-dialogue tracks are now tried first in every step (device language, file default, best track), and a signs-only track plays only when the file has nothing else. The playback guide also notes Forced Only's caption-language fallback for audio that can't be told, and says best full track.
An HLS transcode only carries text subtitles beside it, so PGS, VobSub and DVB tracks (often the forced one) disappeared whenever Emby or Jellyfin transcoded. The player now lists the file's bitmap tracks on a transcode, after its own tracks, and choosing one (by hand or automatically) reloads at the same point with the server drawing that track into the video. Choosing anything else reloads without it. The bridge only asks for a burn-in after a plain PlaybackInfo comes back as a transcode, so direct play is untouched, and plays it only when the transcode URL confirms SubtitleMethod=Encode for that stream. An unconfirmed burn-in, or a burned-in stream that won't open, falls back to playing without subtitles with a notice, and that track isn't offered again for the title. Auto leaves these rows alone while the transcode only copies the video. Silo is unchanged.
…ion the URL fields came from
…erywhere' into feat/emby-jellyfin-subtitle-burn-in # Conflicts: # docs/playback/README.md
There was a problem hiding this comment.
Claude review of the whole PR at 164e9ef. The summary comment lists everything found.
|
Claude review of the new commits 0712b72..5444311 I reviewed the new commits since 0712b72. They add the rule that a confirmed burn-in holds for the item ( P3, I found nothing else. I couldn't check this against live Emby or Jellyfin servers, or on tvOS. I checked the new commits (the hold on a confirmed burn-in, the README paragraph break and the merge of #185, #187 and #188) against the PR description and the changed docs. One P3 doc gap needs fixing. P3
Everything else I checked matches the code:
I did not re-report the existing finding that a confirmed burn-in stops audio switches re-running Auto. Not on a changed line:
claude-sonnet-5-5, high effort, 29 turns, about US$0.95 of API credit |
… failures A burn-in request that got no answer (a timeout, a dropped connection, a server error) was treated like the server refusing the subtitle, so one blip removed the track for the rest of the title. Now only an answer that doesn't confirm the burn-in drops it; a missed request plays without it this time and the track can be chosen again. The fallback after a burned-in stream fails to open now runs only when the failure points at the transcode itself. Network, sign-in, rate-limit and audio session failures take the usual failure path and leave the track alone.
Any URL loading error, and HTTP 401, 403, 408 or 429, now takes the usual failure path whatever failure kind the engine reports it as, so a timeout or lost connection while opening a burned-in stream no longer drops the subtitle for the title. The docs now say a failed burn-in request (a dropped connection, a timeout or an HTTP error) only misses that time.
… a dropped connection
|
On the review of 85de52f: the fallback does still fire for a failed burn-in. Engine load failures reach |
…ries it The failure check returned early for anything under NSURLErrorDomain, so an HTTP 5xx from a failing transcode reported under that domain never triggered the fallback. URL loading errors have negative codes and HTTP statuses positive ones, so only the negative codes now count as the connection, and 401, 403, 408 and 429 are left alone under either domain.
The check compiles the real start path against stubs, so it needs the new burnInSubtitleIndex parameter and ServerSubtitleBurnIn.bitmapStreamIndex.
…btitle-burn-in # Conflicts: # docs/cores/jellyfin.md # iosApp/iosApp/Screens/Player/EmbyPlayback.swift # iosApp/iosApp/Screens/Player/JellyfinPlayback.swift # iosApp/iosApp/Screens/Player/PlayerViewModel+SessionLoad.swift
…btitle-burn-in # Conflicts: # docs/playback/README.md # docs/playback/architecture.md # iosApp/iosApp/Screens/Player/PlayerViewModel.swift
…aph break A quality change or renewal can carry Auto's subtitle pick into a load the server burns it into. Auto then ran again on that load, and on a transcode that only copies the video it hid the burn-in row, switched the subtitle off and reloaded. A confirmed burn-in now counts as the item's pick, so Auto leaves it alone. Also puts back the blank line between two paragraphs of the playback guide that the last merge joined.
|
On the review of d9412c8 (Auto running again after a quality change carries its pick into a burned-in load): fixed in 5444311. |
Stacked on #183 (
fix/subtitles-follow-settings-everywhere). Retarget tomainonce #183 merges.Problem and reproduction
On Emby and Jellyfin, image-based subtitles (PGS, VobSub, DVB) vanished whenever the server transcoded. That's often the forced track, so signs and foreign dialogue went missing on anime and foreign films at lower qualities.
Root cause: Vivid always sent
SubtitleStreamIndex: -1in PlaybackInfo and drew subtitles itself. On direct play the engine draws embedded PGS natively, but an HLS transcode only carries the text tracks Vivid mounts beside it, so bitmap tracks had no way to reach the screen.Implementation and impact
ServerSubtitleBurnIn.negotiate: the bridge sends PlaybackInfo as before. Only when that answer is a transcode, and the load's pick is one of the file's bitmap tracks, it asks once more withSubtitleStreamIndexset. It plays that second answer only when theTranscodingUrlnames the same stream withSubtitleMethod=Encode(or the subtitle'sDeliveryMethodisEncodewhen the URL has no method). Otherwise it plays the first answer without subtitles. Asking only after a transcode means direct play never changes: the engine already draws these tracks, and asking for a DVB or external bitmap subtitle there would make the server transcode just for it. It also lets a detail-page pick or a quality change burn the track in on the first load, with no extra reload.preferredSubtitleTrackIndex, so seek reloads, retries, renewals and quality changes carry it. It's passed to the bridge separately and only the Emby and Jellyfin branches use it, so Silo plans and their subtitle intent are unchanged. Going to Original just direct plays and the engine shows the track natively.TranscodeReasonsare all audio or container reasons, and Vivid allowing a video copy), so an automatic pick doesn't add a full video encode on the server. They can still be picked by hand. TheSubtitleMethodand comma-separatedTranscodeReasonsformats were read from my Emby 4.10 server's own logs.ServerSubtitleBurnIn.swiftandPlayerViewModel+ServerBurnIn.swift, with small hooks in the playback files, inventory adoption and track selection.Deliberately unchanged: direct play, Silo, offline playback, the device profile (no new subtitle profiles, so
legacyBodyis untouched), and the detail page's runtime inventory (burn-in rows are kept out of it).Known limits:
keepsAutomaticSubtitlescarries Auto's pick through a quality change or renewal.Release: this PR merges last and releases the batch. Squash commit body for merging:
The bullets above come from all five PRs, in merge order: #183, #185, #188 (subtitle drawing), #187 (Silo stored subtitles) and this one. I'll check them against each description again right before merging.
Validation
Local, Xcode 27.0, revision 83a818b (code; 1d02d7b only changes docs wording):
ServerSubtitleBurnInTests(new, 15),JellyfinAdapterTests(49),EmbyAdapterTests(59),SubtitleAutoSelectionTests(15),SubtitleAutoResolverTests(38),ServerDeviceProfileTests(3),DetailSubtitlePreviewTests(12),PlaybackProtocolV3Tests(55),PlaybackEndRecoveryTests(18): 264 tests, 0 failures.ServerDeviceProfileTestsstill matcheslegacyBody, so the PlaybackInfo body Vivid sends is unchanged apart from the follow-up burn-in request.ServerSubtitleBurnInTests,SubtitleAutoResolverTests,SubtitleAutoSelectionTests,JellyfinAdapterTests,EmbyAdapterTests: 176 tests, 0 failures.ServerSubtitleBurnInTests(now 18),JellyfinAdapterTests,EmbyAdapterTests,SubtitleAutoSelectionTests: 141 tests, 0 failures.ServerSubtitleBurnInTests(19),SubtitleAutoSelectionTests,PlayerAudioContinuityTests: 41 tests, 0 failures.ServerSubtitleBurnInTests,SiloStoredSubtitleTests,PlayerAudioContinuityTests,SubtitleAutoSelectionTests,JellyfinAdapterTests,EmbyAdapterTests,PlaybackProtocolV3Tests,DetailSubtitlePreviewTests: 223 tests, 0 failures.scripts/tests/jellyfin-replacement-tests.py: 54 checks pass.python3 scripts/ci/check-docs.py --base origin/main: passes.ServerSubtitleBurnInTests), devices, and live playback on either server.New tests:
ServerSubtitleBurnInTests: no burn-in request on direct play; the follow-up request on a transcode and playing the confirmed answer; falling back to the first answer when the server answers with another method, another stream, an error or a failed request;SubtitleMethod/DeliveryMethodconfirmation (case-insensitive); the video-copy rule; which picks count as bitmap; burn-in rows (sidecar ID, no FFmpeg index, failed rows dropped); the reload decision, including no reload when already burned; a text track beating a bitmap row of the same kind; the burned row staying selected and carrying its stream index through reloads; Auto skipping rows while the video is only copied; an unconfirmed burn-in turning subtitles off with a notice; the pick carrying through seek reloads, retries, renewals and quality changes, and never for Silo or offline.JellyfinAdapterTests: a transcode with a PGS pick sends PlaybackInfo twice (-1, then 3), plays the burned session, reports stream 3 and mounts only the text track beside it.EmbyAdapterTests: an unconfirmed burn-in plays the first transcode without subtitles.Performance
No benchmark run. A burn-in makes the server encode video (it usually already is on a transcode), and the burn-in request adds one PlaybackInfo call only when a bitmap pick meets a transcode.
Testing status
Design impact and approval
Documentation impact
docs/cores/emby.mdanddocs/cores/jellyfin.md: transcodes now offer bitmap tracks for the server to burn in, how it's confirmed, the fallback and what's been checked.docs/playback/README.md: the subtitle menus paragraph.docs/playback/architecture.md: how burn-in rows, the PlaybackInfo retry, the fallback and the Auto rule work.docs/cores/player-engine.md: the subtitle inventory sentence.I checked the docs this change affects and updated them in this PR, or explained above why none needed changes.
AI disclosure
I implemented these changes and used Claude Opus as a tool.