Repository navigation
feat: add nvidia/Nemotron-3.5-Parse as supported VLM model - #4897
Conversation
The inline Markdown parser removes the SetextHeading block element, but the paragraph parser still looked it up when a line such as "=" follows text. That raised KeyError for bodies like "a *\n=\nb". Keep such lines as paragraph text. Also expose the Markdown-to-inline-HTML step as _markdown_inline_html so other VLM mappers can reuse it for table cells. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Add trust_remote_code to AutoInlineVlmEngineOptions and pass options.trust_remote_code or model_spec.trust_remote_code to the selected engine, so existing presets keep their behavior. A Transformers engine config can set requires_trust_remote_code in its extra_config. Without the opt-in, the engine then raises a clear error that names the repository and revision, before it downloads the model. Add a --trust-remote-code CLI flag that sets the opt-in on the local engine options of the --vlm-model preset. Revision pinning is covered by #4227. This commit adds the user consent gate. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Add the nemotron_parse_v3_5 preset for nvidia/Nemotron-3.5-Parse, pinned to revision e7ea79242a87ea0ad3919b672bfedb429a683206. The model needs code from its repository, so the preset does not trust it by default: the user must set trust_remote_code=True or use --trust-remote-code. Engines are Transformers and the OpenAI-compatible API for NVIDIA's vLLM server. In-process vLLM and MLX are not supported. Add a mapper for the native block output with content-normalized coordinates: - LaTeX tabular tables with multirow, multicolumn and thead map to table spans and column headers, with formatted rich cells. - Charts map to pictures with a chart class and data table, one picture for each panel that has its own table. - Markdown emphasis and inline HTML map to formatting runs. - Captions link to an adjacent table or picture, list items keep their markers, page headers and footers go to the furniture layer, and consecutive TOC blocks share one group. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
TestShippedSpecsArePinned skipped presets whose model_spec keeps trust_remote_code=False but whose engine overrides set extra_config["requires_trust_remote_code"]. These presets run remote code once the user opts in, so also check that their revision and every engine-override revision is a full commit SHA. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
|
✅ DCO Check Passed Thanks @cau-git, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Map heatmap, stacked bar, bubble and donut charts to the classification labels that DocLang serializes chart data for, and use SCATTER_CHART for scatter plots. Accept chart panels without the optional label, which the model usually omits. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Shared DOTS / Nemotron Parse / Chandra mapping fixes: - Unescape `\*` and `\_` in text without emphasis, outside math, code and backtick spans, and `\$` in Nemotron LaTeX table cells. Other escapes stay, so LaTeX such as `$\$58$` is not damaged. - Fold a single formatted run into its empty owner item. Headings such as `## **bold**` no longer export as an empty heading plus a paragraph, and single-run text keeps its provenance on one item. - Keep soft line breaks in text with emphasis, and keep a `<br>` between runs when a neighbor run is plain. A break between two formatted runs, or in a heading, becomes a space until docling-core serializes run edge whitespace outside the emphasis markers. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
AI triage (advisory)Checked commit The PR adds the CI safety: no concern found. A maintainer must still approve CI. 2 related pull request(s) in the same code
The |
nvidia/Nemotron-3.5-Parse as supported VLM model
|
Some feedback on the way formulas are stored:
This is the same bug as #4903 for Nemotron v2. Please strip the Evidence: I run docling locally on a sample from OmniDocBench. Here is a zip with the input image and the |
Nemotron 3.5 wraps each formula segment in \(...\), often with text between segments, and leaves unpaired delimiters in truncated output. The serializer adds its own delimiters, so formulas exported as $$\(...\)$$. Remove every \( and \) from formula text, then any outer \[...\], $$...$$ or $...$ pair. orig keeps the raw response. Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Add nvidia/Nemotron-3.5-Parse as the VLM preset
nemotron_parse_v3_5, with a mapper from its native output toDoclingDocument.Preset
e7ea79242a87ea0ad3919b672bfedb429a683206.AutoModel, raw prompt, greedy,repetition_penalty=1.1,max_new_tokens=8744) and the OpenAI-compatible API engine, for NVIDIA's vLLM server (top_k=1,skip_special_tokens=False).Remote code consent
model_typenemotron_parseis not native in transformers). The preset does not trust it by default.trust_remote_codeonAutoInlineVlmEngineOptionsand the--trust-remote-codeCLI flag. Without the opt-in, the Transformers engine raises a clear error that names the repository and revision, before it downloads the model.TestShippedSpecsArePinnedto presets that require the opt-in.Mapper (
docling/utils/nemotron_parse_v3_5_utils.py)tabulartables with\multirow,\multicolumn,\theadand\mbox{}map to cell spans, column headers and rich cells. The LaTeX backend used for 2.0 does not handle these.<chart type=...>blocks map to pictures with a chart classification and aTabularChartMetaField(title and data); multi-panel charts give one picture per panel.<sup>/<sub>/<u>map to formatting runs.Also included: a fix for a
KeyErrorin the dots inline Markdown parser (setext underline inside emphasis), which the new mapper reuses.Checklist:
Review evidence:
--vlm-model nemotron_parse_v3_5 --trust-remote-codeconverts pages with the Transformers engine.tests/test_nemotron_parse_v3_5.py,tests/test_hf_model_download_security.py,tests/test_vlm_presets_and_runtime_options.py,tests/test_nemotron_parse_v2.py,tests/test_dots_parser.py: 133 passed, 1 skipped (existing marker). CLI and auto-inline related test files: 370 passed. Pre-commit hooks pass on all changed files.Open points for review:
**Table 1.**) have child runs, andlink_adjacent_captionsdoes not link them (same limit as dots).