Skip to content

feat: burn bitmap subtitles into Emby and Jellyfin transcodes - #186

Merged
blurbery merged 21 commits into
mainfrom
feat/emby-jellyfin-subtitle-burn-in
Oct 19, 2026
Merged

blurbery merged 21 commits into
mainfrom
feat/emby-jellyfin-subtitle-burn-in

Conversation

@blurbery

@blurbery blurbery commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #183 (fix/subtitles-follow-settings-everywhere). Retarget to main once #183 merges.

Problem and reproduction

On Emby and Jellyfin, image-based subtitles (PGS, VobSub, DVB) vanished whenever the server transcoded. That's often the forced track, so signs and foreign dialogue went missing on anime and foreign films at lower qualities.

  1. Play a title with a PGS track from Emby or Jellyfin.
  2. Pick a quality that makes the server transcode (for example 720p), or play somewhere the server already transcodes.
  3. The PGS track isn't in the subtitle list, and Auto can't pick it, even though the detail page names it.

Root cause: Vivid always sent SubtitleStreamIndex: -1 in PlaybackInfo and drew subtitles itself. On direct play the engine draws embedded PGS natively, but an HLS transcode only carries the text tracks Vivid mounts beside it, so bitmap tracks had no way to reach the screen.

Implementation and impact

  • On an Emby or Jellyfin transcode, the file's bitmap tracks (embedded or beside the media) are listed in the subtitle picker after the tracks Vivid plays itself, in the sidecar ID space keyed by the server stream index. They look like any other row. Text comes first, so Auto still prefers a text track to a bitmap one of the same kind.
  • Choosing one, by hand or by Auto, reloads at the current position like a quality change, with the stream index as the load's subtitle pick. Choosing anything else (Off, a text track, an OpenSubtitles file) reloads without it. No reload when the chosen track is already burned in. Cycling and toggling subtitles skip these rows.
  • ServerSubtitleBurnIn.negotiate: the bridge sends PlaybackInfo as before. Only when that answer is a transcode, and the load's pick is one of the file's bitmap tracks, it asks once more with SubtitleStreamIndex set. It plays that second answer only when the TranscodingUrl names the same stream with SubtitleMethod=Encode (or the subtitle's DeliveryMethod is Encode when the URL has no method). Otherwise it plays the first answer without subtitles. Asking only after a transcode means direct play never changes: the engine already draws these tracks, and asking for a DVB or external bitmap subtitle there would make the server transcode just for it. It also lets a detail-page pick or a quality change burn the track in on the first load, with no extra reload.
  • The pick rides on the existing preferredSubtitleTrackIndex, so seek reloads, retries, renewals and quality changes carry it. It's passed to the bridge separately and only the Emby and Jellyfin branches use it, so Silo plans and their subtitle intent are unchanged. Going to Original just direct plays and the engine shows the track natively.
  • Failure handling: when the server answers without confirming the burn-in, or the burned-in stream won't open for a reason that points at the transcode (a server error, a rejected playlist, a pipeline failure), playback carries on without subtitles with a notice (the open failure reloads once without it instead of ending playback), and that row is dropped until another title plays. A burn-in request that fails (a dropped connection, a timeout or an HTTP error) only misses that time: a notice says so, but the row stays so it can be chosen again. Open failures from the connection (any URL loading error), sign-in, timeouts, rate limiting or the audio session take the usual failure path and leave the track alone.
  • Auto leaves burn-in rows alone while the transcode only repackages the video (an H.264 source whose TranscodeReasons are all audio or container reasons, and Vivid allowing a video copy), so an automatic pick doesn't add a full video encode on the server. They can still be picked by hand. The SubtitleMethod and comma-separated TranscodeReasons formats were read from my Emby 4.10 server's own logs.
  • A burned-in track takes no subtitle style, so the style controls hide for it as they already do for bitmap tracks the engine draws. Subtitle delay has no effect on it. No other UI changes.
  • New code is mostly in ServerSubtitleBurnIn.swift and PlayerViewModel+ServerBurnIn.swift, with small hooks in the playback files, inventory adoption and track selection.

Deliberately unchanged: direct play, Silo, offline playback, the device profile (no new subtitle profiles, so legacyBody is untouched), and the detail page's runtime inventory (burn-in rows are kept out of it).

Known limits:

  • An automatic burn-in pick on a fresh load starts the transcode once without it and then reloads with it, since the pick is made after the file opens.
  • A burned-in stream that fails mid-playback, rather than at open, goes through the usual recovery and may ask for the burn-in again.
  • Lucid reports a stream that won't open without saying why, so a connection drop right as a burned-in stream opens still counts against that track until another title plays.
  • External bitmap files beside the media still aren't offered on direct play (unchanged).
  • Merged with fix: make audio follow Settings on every server, on detail pages and in the player [skip release] #185 by a merge commit, no force push. A confirmed burn-in counts as the item's pick, so Auto doesn't run again on that load, including when fix: make audio follow Settings on every server, on detail pages and in the player [skip release] #185's keepsAutomaticSubtitles carries Auto's pick through a quality change or renewal.
  • The open-failure reload and the deferred reload after a pick have no unit tests; the decisions they rely on (which failures count, whether a reload is needed) do.

Release: this PR merges last and releases the batch. Squash commit body for merging:

Release-Notes:
- Subtitles now follow your Subtitles settings on Silo, Emby and Jellyfin alike. A subtitle setting saved on the server could turn subtitles off on movie and series pages and in the player, whatever you had chosen.
- Movie and series pages now show exactly which subtitles will play, such as "Auto: English (Forced)", and the Subtitles box shows for every file, even one without subtitles.
- Added Backup Language in Settings > Subtitles, used when full subtitles would play but a title has none in your main language.
- Switching audio now re-checks Auto, and on Silo an audio or quality change keeps the subtitle you had.
- Show Forced Subtitles is honoured everywhere: with it off a forced track never plays, a track named Forced counts as forced, and a forced pick stays forced into the next episode.
- Always with no subtitle language set now shows subtitles instead of nothing.
- Use Device Settings now follows Apple's rules for Automatic and Forced Only, uses the caption font chosen in Accessibility, and picks up changes when you come back to Vivid.
- Fixed the Shadow subtitle style sometimes being invisible on embedded subtitles, and subtitle delay being lost after a reload.
- Movie, series and season pages now show the audio track that will actually play, chosen from your Audio Language, instead of a track the server remembered, so the page and the player agree on Silo, Emby and Jellyfin.
- Audio picks are no longer saved to the server: your Audio Language decides the audio for every title, and a track you pick lasts for that viewing.
- An audio track you switch to now carries on to the next episodes, instead of going back to your Audio Language while your subtitle choice carried on.
- Fixed audio jumping back to the file's first track after a quality change or reconnect while transcoding, which could turn Japanese audio into English mid-episode.
- Retry after a playback error now keeps the audio track you switched to.
- After a quality change on Emby or Jellyfin, switching audio re-checks Auto subtitles again.
- Downloaded movies and episodes now start with the audio your Audio Language picks, the same as streaming.
- Emby and Jellyfin now see which audio and subtitle tracks are playing, instead of always the starting audio and no subtitles.
- On iPhone and iPad, the Audio menu's Auto option now says it uses your Playback audio preference, as on Apple TV.
- Continue Watching on Apple TV shows the audio format of the track that will play.
- Embedded subtitles now match external subtitle files with the same Settings > Subtitles style: the same size, position, box and outline, and a closer font and weight. Before, embedded text was smaller, thinner and lower on screen.
- External subtitle files keep their italics, bold, underline, strike-through and colours, and a line the file moves to the top now shows there instead of showing the tag as text.
- Overlapping external subtitle lines now stack instead of drawing over each other.
- With Zoom on, external subtitles stay the same size and on screen instead of growing off the bottom on iPhone.
- External subtitles on anamorphic video now sit in the right place, and custom subtitle fonts no longer grow with larger text settings.
- External subtitle files now show in Picture in Picture on iPhone and iPad.
- Subtitles stored on your Silo server (downloaded, uploaded or made by Silo's AI) now show in the player's subtitle menu and the detail page's Subtitles menu on iPhone, iPad and Apple TV. Before, only the file's own tracks and the subtitle files beside it were listed.
- Auto and the rest of Settings > Subtitles now count those stored subtitles too, so a title whose only subtitle in your language is stored on Silo plays it, and the detail page's Auto preview names it.
- A subtitle Silo finishes making while you're watching now joins the player's subtitle menu straight away.
- On Emby and Jellyfin, image-based subtitles (PGS, VobSub and DVB) now show when the server transcodes: the server draws the one you choose, or the one Auto picks, into the picture. Before, they disappeared whenever the server transcoded, which often hid forced subtitles for signs and foreign dialogue.
- If the server can't draw a subtitle in, playback now carries on without subtitles and tells you, instead of failing.
- This release will be available on TestFlight soon.

The bullets above come from all five PRs, in merge order: #183, #185, #188 (subtitle drawing), #187 (Silo stored subtitles) and this one. I'll check them against each description again right before merging.

Validation

Local, Xcode 27.0, revision 83a818b (code; 1d02d7b only changes docs wording):

New tests:

  • ServerSubtitleBurnInTests: no burn-in request on direct play; the follow-up request on a transcode and playing the confirmed answer; falling back to the first answer when the server answers with another method, another stream, an error or a failed request; SubtitleMethod/DeliveryMethod confirmation (case-insensitive); the video-copy rule; which picks count as bitmap; burn-in rows (sidecar ID, no FFmpeg index, failed rows dropped); the reload decision, including no reload when already burned; a text track beating a bitmap row of the same kind; the burned row staying selected and carrying its stream index through reloads; Auto skipping rows while the video is only copied; an unconfirmed burn-in turning subtitles off with a notice; the pick carrying through seek reloads, retries, renewals and quality changes, and never for Silo or offline.
  • JellyfinAdapterTests: a transcode with a PGS pick sends PlaybackInfo twice (-1, then 3), plays the burned session, reports stream 3 and mounts only the text track beside it.
  • EmbyAdapterTests: an unconfirmed burn-in plays the first transcode without subtitles.

Performance

No benchmark run. A burn-in makes the server encode video (it usually already is on a transcode), and the burn-in request adds one PlaybackInfo call only when a bitmap pick meets a transcode.

Testing status

Design impact and approval

Documentation impact

  • docs/cores/emby.md and docs/cores/jellyfin.md: transcodes now offer bitmap tracks for the server to burn in, how it's confirmed, the fallback and what's been checked.

  • docs/playback/README.md: the subtitle menus paragraph.

  • docs/playback/architecture.md: how burn-in rows, the PlaybackInfo retry, the fallback and the Auto rule work.

  • docs/cores/player-engine.md: the subtitle inventory sentence.

  • I checked the docs this change affects and updated them in this PR, or explained above why none needed changes.

AI disclosure

I implemented these changes and used Claude Opus as a tool.

…and in the player

With Settings > Subtitles on English, Auto and Show Forced Subtitles, every
movie and series detail page showed subtitles Off. Settings saves on the
device, but the detail pages seeded their subtitle choice from the server's
remembered subtitle mode (Silo's settings cascade, or the emulated
preferences for Emby and Jellyfin), and a remembered "off" went to the
player as an explicit Off, so the player never applied the settings.

- Settings > Subtitles, or the device's caption settings with Use Device
  Settings on, is now the only input to the automatic subtitle pick on
  Silo, Emby and Jellyfin. The detail pages no longer seed a remembered
  subtitle or Off.
- The detail page's Subtitles control shows what playback will pick, from
  the player's own resolver and audio choice: "Auto: English (Forced)",
  "Auto: Off", the track for Always, Off for Off. It shows for every file,
  including one without subtitles.
- Switching audio runs Auto again in every mode, not only with Use Device
  Settings.
- A Silo replan (audio switch, quality change) no longer drops the
  subtitle: a picked embedded track is re-armed and an automatic one is
  chosen again.
- Show Forced Subtitles off never picks a forced track, even the
  language's only one. A track titled Forced counts as forced. A forced
  pick carried to the next episode stays forced.
- Use Device Settings: caption settings are read again on returning to
  the app and at playback start, the caption font chosen in Accessibility
  is used, and the Apple TV in-player Appearance dialog no longer silently
  turns the setting off.
- Embedded subtitles draw the Shadow style in black at 85%, as external
  ones do, instead of the unused box colour, and the subtitle style and
  delay are re-applied after every load.
- iPhone and iPad grey out Show Forced Subtitles when Behaviour is Off, as
  Apple TV does.
Vivid no longer reads a server's remembered subtitle prefs, so the Apple TV detail page's Auto shouldn't delete the per-series pref that Silo's web app may still use. Also notes the Auto row in the plugins guide.
…t be read

If reading a file's embedded subtitles failed, the iPhone and iPad sheet dropped Auto and Off, and the Apple TV menu dropped Off, so a viewer couldn't go back to Auto or turn subtitles off for that file.
- A subtitle or Off picked while a Silo replan is in flight is no longer undone: the playing subtitle is read after the wait, not before.
- The Silo plan request no longer falls back to the server's remembered subtitle mode and track; without preferences it asks for no subtitles and the player picks from Settings.
- With Use Device Settings on, returning to the app applies the device's caption style and choice to an open player.
- Removed the now unused subtitle pref writers, their request model and tests, and stale comments on the Apple TV detail views.
- Docs: a file without subtitles reads Auto: Off (or Off), and the greyed-out controls and Accessibility caption font are described.
Settings > Subtitles gains Backup Language on iPhone, iPad and Apple TV. When a title has no subtitles in your language, the backup language's full track plays instead, on Silo, Emby and Jellyfin, and the detail pages show it. The backup never counts as your language for Auto's audio check, so an Arabic viewer with an English backup still gets English subtitles on an English film without Arabic ones. It's saved on the device for the profile like the other subtitle settings, and does nothing with Language set to None or Behaviour set to Off.
…et when it matches

- With a backup language set, a full track in it now beats a forced (signs-only) track in the main language, so the dialogue can be followed.
- Setting Language to the backup's language clears the backup, so its picker never points at a row it doesn't list.
- Footers and docs say the backup is used when full subtitles would play, not with Auto and audio already in your language.
- The playback prefs header no longer lists the subtitle-prefs endpoints as used.
…utomatic and Forced Only rules

- Always with Language set to None no longer shows nothing: it uses the device's language when the file has it, otherwise the file's default track, otherwise its first full track.
- Use Device Settings on Automatic now follows Apple's rule: subtitles in the device's language when the audio is in another language, not the caption-language history (which could show Spanish on an English film after one pick in another app).
- Use Device Settings on Forced Only shows the forced track in the audio's language, as Apple does.
…th no language

With Show Forced Subtitles on, a forced track in the device's language could beat a full track elsewhere. Full-dialogue tracks are now tried first in every step (device language, file default, best track), and a signs-only track plays only when the file has nothing else. The playback guide also notes Forced Only's caption-language fallback for audio that can't be told, and says best full track.
An HLS transcode only carries text subtitles beside it, so PGS, VobSub and
DVB tracks (often the forced one) disappeared whenever Emby or Jellyfin
transcoded. The player now lists the file's bitmap tracks on a transcode,
after its own tracks, and choosing one (by hand or automatically) reloads
at the same point with the server drawing that track into the video.
Choosing anything else reloads without it.

The bridge only asks for a burn-in after a plain PlaybackInfo comes back
as a transcode, so direct play is untouched, and plays it only when the
transcode URL confirms SubtitleMethod=Encode for that stream. An
unconfirmed burn-in, or a burned-in stream that won't open, falls back to
playing without subtitles with a notice, and that track isn't offered again
for the title. Auto leaves these rows alone while the transcode only copies
the video. Silo is unchanged.
…erywhere' into feat/emby-jellyfin-subtitle-burn-in

# Conflicts:
#	docs/playback/README.md

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review of the whole PR at 164e9ef. The summary comment lists everything found.

Comment thread iosApp/iosApp/Screens/Player/PlaybackSessionBridge.swift
Comment thread iosApp/iosApp/Screens/Player/ServerSubtitleBurnIn.swift Outdated
Comment thread iosApp/iosApp/Screens/Player/PlayerViewModel+SessionLoad.swift
Comment thread docs/playback/README.md Outdated
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Claude review of the new commits 0712b72..5444311

I reviewed the new commits since 0712b72. They add the rule that a confirmed burn-in holds for the item (adoptServerTranscodeSubtitles), the matching test and the README paragraph break. I also read the merged #185, #187 and #188 changes where they touch burn-in: Emby and Jellyfin progress reports, keepsAutomaticSubtitles and the audio continuity changes. I skimmed the large docs diffs and the project.yml and Python test-stub edits but did not check every sentence in them.

P3, PlayerViewModel+ServerBurnIn.swift: when Auto picks a burn-in row and the server confirms it, an audio switch no longer re-runs Auto. The README and the release notes still say it does.

I found nothing else. I couldn't check this against live Emby or Jellyfin servers, or on tvOS.

I checked the new commits (the hold on a confirmed burn-in, the README paragraph break and the merge of #185, #187 and #188) against the PR description and the changed docs. One P3 doc gap needs fixing.

P3

  • docs/cores/emby.md: the description says the file now covers burn-in on transcodes, how it's confirmed, the fallback and what's been checked. It doesn't mention burn-in anywhere, though docs/cores/jellyfin.md has the full paragraph.

Everything else I checked matches the code:

  • The confirmed-burn-in hold matches adoptServerTranscodeSubtitles and its new test, and the keepsAutomaticSubtitles ordering is correct.
  • The Emby and Jellyfin progress-report text matches MediaServerPlayingTracks.streamIndexes.
  • The audio-rule claims, including downloads, and the Silo stored-subtitle claims match the code.
  • The doc anchors I checked resolve, and the new test files exist.

I did not re-report the existing finding that a confirmed burn-in stops audio switches re-running Auto.

Not on a changed line:

  • P3 docs/cores/emby.md:73: Emby core guide never mentions the burn-in behaviour The description says docs/cores/emby.md now covers bitmap tracks on transcodes, how burn-in is confirmed, the fallback and what's been checked. A search of the file for burn-in, bitmap and PGS finds nothing. The only Emby doc change is the audio and progress-report text on this line. docs/cores/jellyfin.md has the full paragraph, and docs/playback/architecture.md describes the shared behaviour. Add the equivalent Emby text: the SubtitleMethod=Encode confirmation read from the 4.10 logs, the fallback without subtitles, the row dropped for the title, a failed request counting only that attempt, and that it hasn't been checked live.

claude-sonnet-5-5, high effort, 29 turns, about US$0.95 of API credit

… failures

A burn-in request that got no answer (a timeout, a dropped connection, a
server error) was treated like the server refusing the subtitle, so one
blip removed the track for the rest of the title. Now only an answer that
doesn't confirm the burn-in drops it; a missed request plays without it
this time and the track can be chosen again.

The fallback after a burned-in stream fails to open now runs only when the
failure points at the transcode itself. Network, sign-in, rate-limit and
audio session failures take the usual failure path and leave the track
alone.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review of the new commits 164e9ef..1061f7a. The summary comment lists everything found.

Comment thread iosApp/iosApp/Screens/Player/PlayerViewModel+ServerBurnIn.swift Outdated
Comment thread iosApp/iosApp/Screens/Player/PlayerViewModel+ServerBurnIn.swift
Comment thread iosApp/iosApp/Screens/Player/ServerSubtitleBurnIn.swift
Any URL loading error, and HTTP 401, 403, 408 or 429, now takes the usual
failure path whatever failure kind the engine reports it as, so a timeout
or lost connection while opening a burned-in stream no longer drops the
subtitle for the title. The docs now say a failed burn-in request (a
dropped connection, a timeout or an HTTP error) only misses that time.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review of the new commits 1061f7a..85de52f. The summary comment lists everything found.

Comment thread iosApp/iosApp/Screens/Player/PlayerViewModel+ServerBurnIn.swift Outdated
@blurbery

Copy link
Copy Markdown
Owner Author

On the review of 85de52f: the fallback does still fire for a failed burn-in. Engine load failures reach loadFailureBlamesServerBurnIn as LoadFailures, and Lucid reports a stream that won't open as .softwarePipelineFailed or .noPlayableTrackWithinBudget with no error domain (VividMPVPlayer). The only place an HTTP status is put under NSURLErrorDomain is VividAuthenticationRecoveryPolicy.finalFailure on the renewal path, and PlaybackInfo HTTP errors arrive as HTTPError, so neither reaches the check. The real limit is the other way round: Lucid doesn't say why a stream won't open, so a connection drop right as a burned-in stream opens still counts against the track. I've added that to the architecture doc and the PR's known limits rather than guess at it in code.

…ries it

The failure check returned early for anything under NSURLErrorDomain, so
an HTTP 5xx from a failing transcode reported under that domain never
triggered the fallback. URL loading errors have negative codes and HTTP
statuses positive ones, so only the negative codes now count as the
connection, and 401, 403, 408 and 429 are left alone under either domain.
The check compiles the real start path against stubs, so it needs the new
burnInSubtitleIndex parameter and ServerSubtitleBurnIn.bitmapStreamIndex.
Base automatically changed from fix/subtitles-follow-settings-everywhere to main October 19, 2026 11:38
…btitle-burn-in

# Conflicts:
#	docs/cores/jellyfin.md
#	iosApp/iosApp/Screens/Player/EmbyPlayback.swift
#	iosApp/iosApp/Screens/Player/JellyfinPlayback.swift
#	iosApp/iosApp/Screens/Player/PlayerViewModel+SessionLoad.swift
…btitle-burn-in

# Conflicts:
#	docs/playback/README.md
#	docs/playback/architecture.md
#	iosApp/iosApp/Screens/Player/PlayerViewModel.swift

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review of the new commits 0712b72..d9412c8. The summary comment lists everything found.

Comment thread docs/playback/README.md
…aph break

A quality change or renewal can carry Auto's subtitle pick into a load
the server burns it into. Auto then ran again on that load, and on a
transcode that only copies the video it hid the burn-in row, switched
the subtitle off and reloaded. A confirmed burn-in now counts as the
item's pick, so Auto leaves it alone.

Also puts back the blank line between two paragraphs of the playback
guide that the last merge joined.
@blurbery

Copy link
Copy Markdown
Owner Author

On the review of d9412c8 (Auto running again after a quality change carries its pick into a burned-in load): fixed in 5444311. adoptServerTranscodeSubtitles now treats a confirmed burn-in as the item's pick (hasExplicitSubtitleChoice, no automatic prefs), so Auto doesn't run again on that load and can't hide the row and reload on a video-copying transcode. testAConfirmedBurnInHoldsForTheItemEvenWhenAutoChoseIt covers it, and the architecture doc says so.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review of the new commits 0712b72..5444311. The summary comment lists everything found.

Comment thread iosApp/iosApp/Screens/Player/PlayerViewModel+ServerBurnIn.swift
@blurbery
blurbery merged commit 2a6ada1 into main Oct 19, 2026
28 checks passed
@blurbery
blurbery deleted the feat/emby-jellyfin-subtitle-burn-in branch October 19, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant