[PeterStaar-IBM]

resolves: #4684

Summary

• Add Transformers and MLX ASR support, with configurable model revisions.
• Add Granite, Nemotron, Parakeet CTC, and Whisper presets. Keep the existing WhisperS2T models.
• Add an optional script that runs models on the same audio fragment, saves DCLX output, and prints a status and timing table.

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

Review evidence:

[github-actions]

✅ DCO Check Passed

Thanks @PeterStaar-IBM, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

[codecov]

Codecov Report

❌ Patch coverage is 87.93970% with 24 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling/pipeline/asr_transcriber.py	70.90%	10 Missing and 6 partials ⚠️
docling/datamodel/asr_presets.py	88.23%	2 Missing and 2 partials ⚠️
docling/backend/md_backend.py	50.00%	1 Missing and 1 partial ⚠️
docling/pipeline/asr_transformers.py	95.74%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

AI triage (advisory)

Checked commit a70a23c8ce. The analysis read the diff only. It did not run code from this pull request.

The PR adds Transformers and MLX ASR presets for Granite, Nemotron, Parakeet CTC and Whisper, with a new --asr-language CLI option and a feat-asr-transformers extra. It also adds a model validation script and changes the Markdown HTML-block restore in md_backend.py.

CI safety: needs maintainer care before you approve CI.

• pyproject.toml: changes build, dependency, or tool configuration
• scripts/validate_asr_models.py: changes maintenance scripts
• tach.toml: changes CI test selection
• uv.lock: changes locked dependencies

4 related pull request(s) in the same code

PR	State	Signal	Assessment
#4898	open	also references issue #4684; changes the same code	related: Both PRs change ASR code in asr_transcriber.py and asr_model_specs.py. #4898 adds offline --artifacts-path loading for WhisperS2T, and this PR adds Transformers and MLX presets. The two changes do different things.
#4985	open	changes the same code	related: Both PRs change how md_backend.py restores HTML blocks that the serializer escapes. Here it is a small side change in an ASR PR. Only one of the two Markdown fixes can be merged in this code.
#4984	open	changes the same code	related: Both PRs fix the escaped HTML block restore in md_backend.py, with different methods. Here the fix is a small part of an ASR PR.
#4848	open	changes the same code	related: Both PRs change the Markdown HTML block handling in md_backend.py. #4848 replaces the marker approach with a direct HTML backend call, and here the fix is a small part of an ASR PR.

Reference data changes

Change	Files
tables	1
text	13
coordinates only	4

• tables tests/data/html/groundtruth/wiki_duck.html.md: +53 / -53 lines; Markdown table rows changed
• text tests/data/latex/groundtruth/1706.03762_main.tex.dclg: +1 / -1 lines
• text tests/data/latex/groundtruth/2305.03393_main.tex.dclg: +1 / -1 lines
• text tests/data/latex/groundtruth/2310.06825_main.tex.dclg: +1 / -1 lines
• text tests/data/latex/groundtruth/2412.19437_main.tex.dclg: +1 / -1 lines
• text tests/data/latex/groundtruth/2501.00089_main.tex.dclg: +1 / -1 lines
• text tests/data/latex/groundtruth/arXiv-2501.01300v2_main.tex.dclg: +1 / -1 lines
• text tests/data/latex/groundtruth/example_01.tex.dclg: +1 / -1 lines
• text tests/data/latex/groundtruth/example_02.tex.dclg: +1 / -1 lines
• text tests/data/md/groundtruth/escaped_characters.md.json: 2 text item(s) changed
• text tests/data/md/groundtruth/escaped_characters.md.md: +2 / -2 lines
• text tests/data/md/groundtruth/inline_and_formatting.md.yaml: +91 / -91 lines
• text tests/data/pdf/groundtruth/elsevier-00.md: +52 / -52 lines
• text tests/data/pptx/groundtruth/powerpoint_with_image.pptx.json: item attributes changed
• coordinates only tests/data/pdf/groundtruth/2203.01017v2.json
• coordinates only tests/data/pdf/groundtruth/2206.01062.json
• coordinates only tests/data/pdf/groundtruth/2305.03393v1.json
• coordinates only tests/data/pdf/groundtruth/amt_handbook_sample.json

Assessment: unclear. The Markdown reference changes may follow from the md_backend.py fix. The ASR code cannot change the LaTeX, PDF, PPTX and HTML reference data. The PR gives no reason for those changes, and the lock file diff is not available to check for a dependency cause.

_{The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.}

[github-actions]

AI first review (advisory): blocker found

This review covers the full diff at 76768ffefe. It read the code and did not run it.

The PR changes how the existing MLX Whisper model uses artifacts_path. This may break current Apple Silicon users who set artifacts_path for other models. Check whether this offline-only MLX behavior is intended and whether it needs a fallback or a release note.

_{A maintainer makes the merge decision. This review never approves the pull request.}

[BBC-Esq]

@PeterStaar-IBM Thanks for adding these presets! I'm extremely familiar with Parakeet, Granite Speech, and the Transformers implementation of Whisper, and I've benchmarked all three, so feel free to delegate future tasks on them to me if you want. The only exception is MLX, which I can't help with.

[dolfim-ibm]

lgtm

[BBC-Esq]

@PeterStaar-IBM thanks for explaining how you plan to implement this. I've done some extensive benchmarking on all of the models you're planning on adding and I can share them with you if you'd like. They include word error rate testing at various chunk sizes as well as "stitching" strategies. To summarize with respect to the models you want to add, here are my recommendations, followed by the results behind them:

Recommendations

• Drop both Nemotron presets. On English, TurboCTC matches the English model's WER (3.53% vs 3.56%) with more than twice the punctuation, 27× the speed and less VRAM, and the 3.5 model is worse on every measure. Nemotron's strength is low-latency streaming, which file transcription doesn't use. The one remaining case for 3.5 is languages Granite doesn't cover, which I haven't tested.
• Replace Parakeet CTC with Parakeet TDT. CTC 1.1B is close to TurboCTC on WER but writes lower case with no punctuation. Via transformers on the same set, TDT v2 (English) scores 3.42% at 90 s chunks and TDT v3 (25 European languages) 2.95% at 300 s chunks, both punctuated (11.9 and 11.6 per 100), at 150× and 140×. v3 needs the longer chunks: at 90 s it skips speech in some windows (4.10%).
• Drop the three Granite 4.x presets. I haven't benchmarked them on this set, but your 5.0 model card calls 5.0 2B the successor to 4.1 2B (same size, more capabilities), they're new in this PR so no users depend on them, and TurboCTC covers the small, fast niche. Plus's timestamp and speaker tags would be the one reason to keep it, but the PR doesn't map them to docling tracks.
• Keep Granite 5.0 2B for quality and TurboCTC for speed. TurboCTC NC has the best WER but is lower case only and non-commercial.

Results

Setup: 15 long English recordings (7 distil-whisper/meanwhile clips over 90 s + 8 distil-whisper/tedlium-long-form talks of 3–22 min; 19,653 words), Whisper's English normalizer, corpus WER, bf16, transformers 5.20.2, the same checkpoints your presets pin. The references have 10.6 punctuated words per 100; normalized WER ignores punctuation, so I count it separately.

Preset	Optimal setting	WER	Punctuated words per 100	Speed (batch 1)	Other results
granite_speech_5_0_2b	90 s chunks, 5 s overlap, text-alignment stitch	3.42%	10.7	14× (197× at batch 32)	60 s: 3.87%, 120 s: 4.21%. One pass over ~2 min silently drops 25–50% of the speech.
granite_speech_5_0_470m_turboctc	45 s chunks, 15 s overlap, word-timing stitch	3.53%	11.1	1,949×	30–90 s all within 0.12 points. One pass: 3.73% (memory grows linearly).
granite_speech_5_0_470m_turboctc_nc	45 s chunks, 15 s overlap, word-timing stitch	3.28%	none (lower case)	1,705×	One pass: 3.44%. English only, CC-BY-NC-SA.
nemotron_speech_streaming_en_0_6b	90 s chunks, 15 s overlap, word-timing stitch, lookahead 13	3.56%	4.8	73×	Lookahead 6: 3.66%, 0: 4.64%.
nemotron_3_5_asr_streaming	90 s chunks, 15 s overlap, word-timing stitch, lookahead 13	4.80%	5.3	72×	Lookahead 3 (its default): 5.11%. Forcing en-US instead of auto-detect: 4.83%.
parakeet_ctc_1_1b	120 s chunks, 15 s overlap, word-timing stitch	3.58%	none (lower case)	1,019×	60–180 s within 0.03 points; 90 s: 3.77%, 30 s: 3.90%.

Stitching

• Models with word timestamps (TurboCTC, Parakeet, Nemotron): split each overlap at its midpoint and keep the words whose centre falls on that chunk's side.
• Granite 5.0 2B gives no timestamps, so I align the words in the overlap (difflib, lower-cased, punctuation removed) and cut both texts at a matched word. Its weak spot is repeated words in the overlap ("sometimes, sometimes"): it can match the wrong occurrence and drop a passage. That's why 30 s and 60 s chunks (more seams) lost 5 and 2 passages of 8+ words, while 90 s lost none.
• In all cases I chunk outside the pipeline and call generate() per chunk (code in my earlier comment).

Other settings

• Granite 5.0 2B: scale max_new_tokens with chunk length. I use 16 per second of audio (speech is about 4 tokens/s); a flat 256 covers only about a minute of speech. Batch the chunks too (see the speeds above).
• If you keep Nemotron: set the lookahead with processor.set_num_lookahead_tokens(13). Nemotron 3.5 also writes a language tag (<en-US>) after each sentence; it's a special token, so decode with skip_special_tokens=True.

I haven't tested the Whisper presets. Happy to share the scripts and per-recording results, or go into more depth on any of these.

[PeterStaar-IBM]

@PeterStaar-IBM thanks for explaining how you plan to implement this. I've done some extensive benchmarking on all of the models you're planning on adding and I can share them with you if you'd like. They include word error rate testing at various chunk sizes as well as "stitching" strategies. To summarize with respect to the models you want to add, here are my recommendations, followed by the results behind them:

Recommendations

• Drop both Nemotron presets. On English, TurboCTC matches the English model's WER (3.53% vs 3.56%) with more than twice the punctuation, 27× the speed and less VRAM, and the 3.5 model is worse on every measure. Nemotron's strength is low-latency streaming, which file transcription doesn't use. The one remaining case for 3.5 is languages Granite doesn't cover, which I haven't tested.
• Replace Parakeet CTC with Parakeet TDT. CTC 1.1B is close to TurboCTC on WER but writes lower case with no punctuation. Via transformers on the same set, TDT v2 (English) scores 3.42% at 90 s chunks and TDT v3 (25 European languages) 2.95% at 300 s chunks, both punctuated (11.9 and 11.6 per 100), at 150× and 140×. v3 needs the longer chunks: at 90 s it skips speech in some windows (4.10%).
• Drop the three Granite 4.x presets. I haven't benchmarked them on this set, but your 5.0 model card calls 5.0 2B the successor to 4.1 2B (same size, more capabilities), they're new in this PR so no users depend on them, and TurboCTC covers the small, fast niche. Plus's timestamp and speaker tags would be the one reason to keep it, but the PR doesn't map them to docling tracks.
• Keep Granite 5.0 2B for quality and TurboCTC for speed. TurboCTC NC has the best WER but is lower case only and non-commercial.

Results

Setup: 15 long English recordings (7 distil-whisper/meanwhile clips over 90 s + 8 distil-whisper/tedlium-long-form talks of 3–22 min; 19,653 words), Whisper's English normalizer, corpus WER, bf16, transformers 5.20.2, the same checkpoints your presets pin. The references have 10.6 punctuated words per 100; normalized WER ignores punctuation, so I count it separately.

Preset Optimal setting WER Punctuated words per 100 Speed (batch 1) Other results
granite_speech_5_0_2b 90 s chunks, 5 s overlap, text-alignment stitch 3.42% 10.7 14× (197× at batch 32) 60 s: 3.87%, 120 s: 4.21%. One pass over ~2 min silently drops 25–50% of the speech.
granite_speech_5_0_470m_turboctc 45 s chunks, 15 s overlap, word-timing stitch 3.53% 11.1 1,949× 30–90 s all within 0.12 points. One pass: 3.73% (memory grows linearly).
granite_speech_5_0_470m_turboctc_nc 45 s chunks, 15 s overlap, word-timing stitch 3.28% none (lower case) 1,705× One pass: 3.44%. English only, CC-BY-NC-SA.
nemotron_speech_streaming_en_0_6b 90 s chunks, 15 s overlap, word-timing stitch, lookahead 13 3.56% 4.8 73× Lookahead 6: 3.66%, 0: 4.64%.
nemotron_3_5_asr_streaming 90 s chunks, 15 s overlap, word-timing stitch, lookahead 13 4.80% 5.3 72× Lookahead 3 (its default): 5.11%. Forcing en-US instead of auto-detect: 4.83%.
parakeet_ctc_1_1b 120 s chunks, 15 s overlap, word-timing stitch 3.58% none (lower case) 1,019× 60–180 s within 0.03 points; 90 s: 3.77%, 30 s: 3.90%.
Stitching

• Models with word timestamps (TurboCTC, Parakeet, Nemotron): split each overlap at its midpoint and keep the words whose centre falls on that chunk's side.
• Granite 5.0 2B gives no timestamps, so I align the words in the overlap (difflib, lower-cased, punctuation removed) and cut both texts at a matched word. Its weak spot is repeated words in the overlap ("sometimes, sometimes"): it can match the wrong occurrence and drop a passage. That's why 30 s and 60 s chunks (more seams) lost 5 and 2 passages of 8+ words, while 90 s lost none.
• In all cases I chunk outside the pipeline and call generate() per chunk (code in my earlier comment).

Other settings

• Granite 5.0 2B: scale max_new_tokens with chunk length. I use 16 per second of audio (speech is about 4 tokens/s); a flat 256 covers only about a minute of speech. Batch the chunks too (see the speeds above).
• If you keep Nemotron: set the lookahead with processor.set_num_lookahead_tokens(13). Nemotron 3.5 also writes a language tag (<en-US>) after each sentence; it's a special token, so decode with skip_special_tokens=True.

I haven't tested the Whisper presets. Happy to share the scripts and per-recording results, or go into more depth on any of these.

I like this evaluation, would it be possible (once merged) that we work together and:

1. provide a way to run these tests through docling
2. add a clean documentation page on quality, performance and features for each model (in the direction you have done above)

[BBC-Esq]

Any way I can convince you to at least NOT include the nemotron models since they don't fill any "niche" that I can tell? ;-) And possibly my other recommended exclusions? But either way, yes, it'd be a fun project. Would it be within docling or I was perhaps looking at docling-eval as a possibility?

[maxmnemonic]

LGTM!