[DanielNg0729]

Hi maintainers, this PR stops PPTX and ODP conversion from dropping slide video and audio, and adds an opt-in step that transcribes them with the existing audio and video pipelines.

Issue resolved by this Pull Request:
Resolves #4740

What changes

1. Media is kept (bug fix, always on).
   • PPTX: python-pptx reads a video as a Movie, which is not a Picture, so the backend skipped it. A video now becomes a picture of its poster frame, with provenance on its slide. Audio was already a picture of the speaker icon.
   • Each video or audio picture records its media file in the picture meta, as docling__video or docling__audio. The value is the path inside the package for an embedded file (ppt/media/media1.mp4), or the link target for a linked file. In PPTX, the media element in p:nvPr (a:videoFile, a:quickTimeFile, a:audioFile, a:wavAudioFile) gives the kind, and the p14:media relationship, or else the one of the media element, gives the file.
   • ODP: the backend ignored the draw:plugin of a media frame, and only its preview came out. The preview picture now records the same field. A frame without a preview gets an empty picture. When LibreOffice declares its generic application/vnd.sun.star.media type, the manifest or the file extension gives the kind.
2. Backends can load the media. The new DeclarativeDocumentBackend.load_media(location) returns the bytes of a recorded file. The default returns None. PPTX reads the package part, and ODP reads the archive. A linked file follows the ImageResourceLoader rules: a URL needs enable_remote_fetch, and a local file needs enable_local_fetch, a document given as a Path, and a place in the folder of the document or below it.
3. Opt-in conversion in the pipeline. With the new ConvertPipelineOptions.do_media_conversion (default False), SimplePipeline sends each recorded file to AsrPipeline (audio) or VideoPipeline (video), with the Whisper model of media_asr_options. The result goes in a section group right after the picture: the transcript with its TrackSource timestamps and, for a video, the sampled frames. The items take the provenance (slide and box) and the content layer of the picture. A file that cannot be loaded or converted, also for a missing asr extra or ffmpeg, keeps its picture, is listed in ConversionResult.errors, and makes the result PARTIAL_SUCCESS. A linked file that the options do not allow is skipped with a warning.
4. Docs. supported_formats.md, a new "Video and audio in slides" section in processing_audio_media.md, and the Python SDK reference of the usage skill.

Design note: the conversion is a pipeline step, not a backend feature, because docling.backend must not import docling.pipeline (Tach layers). As a pipeline step, the user sets it in the options they already pass, the media pipelines are built only when a document has media, and any declarative format that records docling__video / docling__audio can use it later.

Usage

from pathlib import Path

from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import ConvertPipelineOptions
from docling.document_converter import (
    DocumentConverter,
    OdpFormatOption,
    PowerpointFormatOption,
)

pipeline_options = ConvertPipelineOptions(do_media_conversion=True)
converter = DocumentConverter(
    format_options={
        InputFormat.PPTX: PowerpointFormatOption(pipeline_options=pipeline_options),
        InputFormat.ODP: OdpFormatOption(pipeline_options=pipeline_options),
    }
)
result = converter.convert(Path("slides.pptx"))
print(result.document.export_to_markdown())

Markdown of the first slide of my test deck (Whisper tiny):

Quarterly review

<!-- image -->

ppt/media/media1.mp4

<!-- image -->

Welcome to the Quarterly Review.

Revenue Group by 12% this quarter.

<!-- image -->

ppt/media/media2.mp3

 This is the audio note on the first slide.

 Please read the appendix.

Without do_media_conversion, only the picture and the path line are there.

Notes for reviewers. I am happy to change any of these:

• The docling-core Markdown serializer prints custom meta fields as text, so the media path shows under the <!-- image --> line. I used a plain string for this reason, because a dict prints as a Python repr. A dedicated field in docling-core is another option.
• The new options are on ConvertPipelineOptions, next to do_picture_description and do_chart_extraction. Only SimplePipeline reads them. Frames use the default VideoPipelineOptions, and only the Whisper model can be set.
• There is no CLI flag yet.

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[github-actions]

✅ DCO Check Passed

Thanks @DanielNg0729, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

❌ Patch coverage is 85.63218% with 25 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling/backend/utils/media.py	85.71%	3 Missing and 4 partials ⚠️
docling/backend/opendocument_backend.py	73.91%	3 Missing and 3 partials ⚠️
docling/pipeline/simple_pipeline.py	91.17%	2 Missing and 4 partials ⚠️
docling/backend/mspowerpoint_backend.py	83.33%	2 Missing and 3 partials ⚠️
docling/backend/abstract_backend.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

AI triage (advisory)

Checked commit 6747af53e3. The analysis read the diff only. It did not run code from this pull request.

The PR adds video and audio support to the PPTX and ODP backends: each media object is kept as a picture with its file path in the picture meta. A new opt-in `do_media_conversion` pipeline option sends each media file through the ASR or video pipeline and places the result after its picture.

CI safety: needs maintainer care before you approve CI.

• tach.toml: changes CI test selection
• tach.toml: Already flagged by risk.forced: adds new pipeline module dependencies that change which modules tach checks.

3 related pull request(s) in the same code

PR	State	Signal	Assessment
#3791	open	changes the same code	related: PR 3791 refactors docling_core import paths in the same backend files but does not add any media feature.
#4443	open	changes the same code	related: PR 4443 adds run-level hyperlink and formatting support in the PPTX text-handling code; it changes mspowerpoint_backend.py in a different area from this PR.
#4771	open	changes the same code	related: PR 4771 adds ASR model preset documentation to the same python-sdk.md skill file, but covers a different topic (ASR model selection, not slide media).

_{The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.}

[PeterStaar-IBM]

@DanielNg0729 I love this feature, but I need to understand a bit more how you want to construct it and in particular, I need to see how this will play with the new <track> capabilities that we have in doclang (see https://github.com/doclang-project/doclang/blob/main/spec.md#tracks).

I would love to see:

1. we need to first support the serialization/deserialization from DoclingDocument into DocLang.
2. we need to see where we add the media files (the audio/video) on the slides. Intuitively, I suppose this should be children of the slides

[PeterStaar-IBM]

see #4752 (comment)

let's first do the updates and release it in docling-core.

[DanielNg0729]

Thank you very much Dr @PeterStaar-IBM. I agree, docling-core first. I will put this PR on hold, and I will open the docling-core PR for <track> before I come back here. May I check the shape with you before I start? My plan is this:

1. DoclingDocument ↔ DocLang. I would add a TrackItem that maps 1:1 to <track>. It is a floating item like PictureItem, so it keeps the <location>, the captions and an optional cover image (<cover>). Its children are the cue items that the ASR and video pipelines already make: TextItem / PictureItem with a TrackSource. Children with the same start and end time form one cue block, TrackSource.voice becomes <voice>, and a frame picture becomes <frame>. The deserializer does the reverse. A plain audio or video conversion would then be one TrackItem under the body.

2. Media on the slides. Yes, I would make the track a child of its slide, at the place of the video shape, with the poster frame as <cover>. A sketch for the test deck of this PR:

   <text>Quarterly review</text>
   <track>
     <location value="26"/><location value="93"/><location value="193"/><location value="259"/>
     <cover><src uri="assets/picture_1.png"/></cover>
     <bdiv/>
     <seconds value="0"/>
     <seconds value="2"/><msecs value="120"/>
     Welcome to the quarterly review.
     <bdiv/>
     <seconds value="3"/><msecs value="100"/>
     <seconds value="5"/><msecs value="360"/>
     Revenue grew by 12% this quarter.
   </track>
   <page_break/>

   Without transcription, the track has only its location and cover. The recording itself would go into the .dclx archive as video/{N}.mp4 / audio/{N}.mp3, as in the doclang archive demo. I would do that part in a second, smaller docling-core PR.

And one thing is that i saw is that the <track> and the audio/ / video/ archive parts you mentioned are on doclang main (doclang-project/doclang#175), but not yet in a release (the latest is 0.7.3, and docling-core pins doclang>=0.7,<0.8). Could you please bump and release the doclang version? Then I can open the docling-core PR against it.

Thank you very much for the review!

[PeterStaar-IBM]

Thank you very much Dr @PeterStaar-IBM. I agree, docling-core first. I will put this PR on hold, and I will open the docling-core PR for <track> before I come back here. May I check the shape with you before I start? My plan is this:

1. DoclingDocument ↔ DocLang. I would add a TrackItem that maps 1:1 to <track>. It is a floating item like PictureItem, so it keeps the <location>, the captions and an optional cover image (<cover>). Its children are the cue items that the ASR and video pipelines already make: TextItem / PictureItem with a TrackSource. Children with the same start and end time form one cue block, TrackSource.voice becomes <voice>, and a frame picture becomes <frame>. The deserializer does the reverse. A plain audio or video conversion would then be one TrackItem under the body.

2. Media on the slides. Yes, I would make the track a child of its slide, at the place of the video shape, with the poster frame as <cover>. A sketch for the test deck of this PR:

   <text>Quarterly review</text>
   <track>
     <location value="26"/><location value="93"/><location value="193"/><location value="259"/>
     <cover><src uri="assets/picture_1.png"/></cover>
     <bdiv/>
     <seconds value="0"/>
     <seconds value="2"/><msecs value="120"/>
     Welcome to the quarterly review.
     <bdiv/>
     <seconds value="3"/><msecs value="100"/>
     <seconds value="5"/><msecs value="360"/>
     Revenue grew by 12% this quarter.
   </track>
   <page_break/>
   

   Without transcription, the track has only its location and cover. The recording itself would go into the .dclx archive as video/{N}.mp4 / audio/{N}.mp3, as in the doclang archive demo. I would do that part in a second, smaller docling-core PR.

And one thing is that i saw is that the <track> and the audio/ / video/ archive parts you mentioned are on doclang main (doclang-project/doclang#175), but not yet in a release (the latest is 0.7.3, and docling-core pins doclang>=0.7,<0.8). Could you please bump and release the doclang version? Then I can open the docling-core PR against it.

Thank you very much for the review!

wonderful, please make sure @vagenas is involved in the entire process!

cc: @dolfim-ibm @cau-git

[vagenas]

@DanielNg0729 as you correctly pointed out, first of all this depends on releasing the latest DocLang changes including <track>. We'll try to follow up with that one very soon.