Skip to content

feat: add nvidia/Nemotron-3.5-Parse as supported VLM model - #4897

Merged
dolfim-ibm merged 7 commits into
mainfrom
cau/nemotron-parse-3.5
Oct 20, 2026
Merged

dolfim-ibm merged 7 commits into
mainfrom
cau/nemotron-parse-3.5

Conversation

@cau-git

@cau-git cau-git commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Add nvidia/Nemotron-3.5-Parse as the VLM preset nemotron_parse_v3_5, with a mapper from its native output to DoclingDocument.

Preset

  • Pinned to revision e7ea79242a87ea0ad3919b672bfedb429a683206.
  • Engines: Transformers (AutoModel, raw prompt, greedy, repetition_penalty=1.1, max_new_tokens=8744) and the OpenAI-compatible API engine, for NVIDIA's vLLM server (top_k=1, skip_special_tokens=False).
  • In-process vLLM is not supported: the model needs NVIDIA's out-of-tree vLLM plugin with exact version pins. MLX is not supported: no MLX port exists.
  • Single-pass parsing only (no crop re-parsing).

Remote code consent

  • The model needs code from its repository (model_type nemotron_parse is not native in transformers). The preset does not trust it by default.
  • New opt-in: trust_remote_code on AutoInlineVlmEngineOptions and the --trust-remote-code CLI flag. Without the opt-in, the Transformers engine raises a clear error that names the repository and revision, before it downloads the model.
  • Revision pinning comes from fix(models): warn on unpinned revisions for trust_remote_code model specs #4227. This PR adds the consent gate and extends TestShippedSpecsArePinned to presets that require the opt-in.
  • Other presets keep their current behavior; moving them to the gate is a follow-up.

Mapper (docling/utils/nemotron_parse_v3_5_utils.py)

  • Content-normalized coordinates map directly to the page (no padding inverse as in 2.0).
  • LaTeX tabular tables with \multirow, \multicolumn, \thead and \mbox{} map to cell spans, column headers and rich cells. The LaTeX backend used for 2.0 does not handle these.
  • <chart type=...> blocks map to pictures with a chart classification and a TabularChartMetaField (title and data); multi-panel charts give one picture per panel.
  • Markdown emphasis and inline <sup>/<sub>/<u> map to formatting runs.
  • Captions link to an adjacent table or picture; list items keep their markers; page headers and footers go to the furniture layer; consecutive TOC blocks share one group; text in pictures becomes picture children; Code and Formula use their own item types.

Also included: a fix for a KeyError in the dots inline Markdown parser (setext underline inside emphasis), which the new mapper reuses.

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

Review evidence:

  • Before: no preset for Nemotron 3.5 Parse. After: --vlm-model nemotron_parse_v3_5 --trust-remote-code converts pages with the Transformers engine.
  • Tests: tests/test_nemotron_parse_v3_5.py, tests/test_hf_model_download_security.py, tests/test_vlm_presets_and_runtime_options.py, tests/test_nemotron_parse_v2.py, tests/test_dots_parser.py: 133 passed, 1 skipped (existing marker). CLI and auto-inline related test files: 370 passed. Pre-commit hooks pass on all changed files.
  • Mapper check against NVIDIA's 500 published reference predictions: 10,143 blocks, 375 tables (none empty), 141 charts, 0 errors.

Open points for review:

  • Captions with inline formatting (e.g. **Table 1.**) have child runs, and link_adjacent_captions does not link them (same limit as dots).
  • The API engine sends the image before the text; NVIDIA's client sends the text first. The chat template only concatenates content, so no effect is expected, but this is not tested.

The inline Markdown parser removes the SetextHeading block element, but
the paragraph parser still looked it up when a line such as "=" follows
text. That raised KeyError for bodies like "a *\n=\nb". Keep such lines
as paragraph text.

Also expose the Markdown-to-inline-HTML step as _markdown_inline_html so
other VLM mappers can reuse it for table cells.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Add trust_remote_code to AutoInlineVlmEngineOptions and pass
options.trust_remote_code or model_spec.trust_remote_code to the
selected engine, so existing presets keep their behavior.

A Transformers engine config can set requires_trust_remote_code in its
extra_config. Without the opt-in, the engine then raises a clear error
that names the repository and revision, before it downloads the model.

Add a --trust-remote-code CLI flag that sets the opt-in on the local
engine options of the --vlm-model preset.

Revision pinning is covered by #4227. This commit adds the user consent
gate.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Add the nemotron_parse_v3_5 preset for nvidia/Nemotron-3.5-Parse,
pinned to revision e7ea79242a87ea0ad3919b672bfedb429a683206. The model
needs code from its repository, so the preset does not trust it by
default: the user must set trust_remote_code=True or use
--trust-remote-code. Engines are Transformers and the OpenAI-compatible
API for NVIDIA's vLLM server. In-process vLLM and MLX are not supported.

Add a mapper for the native block output with content-normalized
coordinates:
- LaTeX tabular tables with multirow, multicolumn and thead map to
  table spans and column headers, with formatted rich cells.
- Charts map to pictures with a chart class and data table, one picture
  for each panel that has its own table.
- Markdown emphasis and inline HTML map to formatting runs.
- Captions link to an adjacent table or picture, list items keep their
  markers, page headers and footers go to the furniture layer, and
  consecutive TOC blocks share one group.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
TestShippedSpecsArePinned skipped presets whose model_spec keeps
trust_remote_code=False but whose engine overrides set
extra_config["requires_trust_remote_code"]. These presets run remote
code once the user opts in, so also check that their revision and every
engine-override revision is a full commit SHA.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @cau-git, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.25620% with 39 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling/utils/nemotron_parse_v3_5_utils.py 88.66% 18 Missing and 16 partials ⚠️
docling/pipeline/vlm_pipeline.py 0.00% 2 Missing and 1 partial ⚠️
docling/cli/main.py 50.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Map heatmap, stacked bar, bubble and donut charts to the classification
labels that DocLang serializes chart data for, and use SCATTER_CHART for
scatter plots. Accept chart panels without the optional label, which the
model usually omits.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Shared DOTS / Nemotron Parse / Chandra mapping fixes:

- Unescape `\*` and `\_` in text without emphasis, outside math, code and
  backtick spans, and `\$` in Nemotron LaTeX table cells. Other escapes stay,
  so LaTeX such as `$\$58$` is not damaged.
- Fold a single formatted run into its empty owner item. Headings such as
  `## **bold**` no longer export as an empty heading plus a paragraph, and
  single-run text keeps its provenance on one item.
- Keep soft line breaks in text with emphasis, and keep a `<br>` between runs
  when a neighbor run is plain. A break between two formatted runs, or in a
  heading, becomes a space until docling-core serializes run edge whitespace
  outside the emphasis markers.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
cau-git marked this pull request as ready for review October 20, 2026 08:33
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) CLI issues related to the command line interface (CLI) docling-document enhancement New feature or request ocr table structure vlm-pipeline labels Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit 1d090d3cb2. The analysis read the diff only. It did not run code from this pull request.

The PR adds the nemotron_parse_v3_5 VLM preset for nvidia/Nemotron-3.5-Parse, with a mapper from its native output to DoclingDocument. It also adds an opt-in trust_remote_code option and a --trust-remote-code CLI flag. Small changes to the chandra_utils.py and dots_utils.py helpers support the mapper.

CI safety: no concern found. A maintainer must still approve CI.

2 related pull request(s) in the same code
PR State Signal Assessment
#4789 open changes the same code related: This PR adds LightOnOCR-3 presets. It touches the same preset registration and ResponseFormat code, but it adds a different model.
#4908 open changes the same code related: Both PRs change _add_inline_text and the Markdown handling in dots_utils.py, with different behavior. They will likely conflict when merged.

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@cau-git cau-git changed the title feat: add nvidia/Nemotron-3.5-Parse as vlm model feat: add nvidia/Nemotron-3.5-Parse as supported VLM model Oct 20, 2026
@nikos-livathinos

Copy link
Copy Markdown
Member

Some feedback on the way formulas are stored:

  • The mapper stores formulas as is (nemotron_parse_v3_5_utils.py:544).
  • The model usually writes LaTeX formulas as \(…\).
  • The Markdown serializer then adds $$, so we end up with something like:
$$\(f(x)=\sum_{n\in S_x}\frac{1}{2^n},\)$$

This is the same bug as #4903 for Nemotron v2. Please strip the \(…\) of each segment.

Evidence: I run docling locally on a sample from OmniDocBench. Here is a zip with the input image and the
predictions.
nemotron_v3_5_formula_pages.zip

Nemotron 3.5 wraps each formula segment in \(...\), often with text between
segments, and leaves unpaired delimiters in truncated output. The serializer
adds its own delimiters, so formulas exported as $$\(...\)$$. Remove every
\( and \) from formula text, then any outer \[...\], $$...$$ or $...$ pair.
orig keeps the raw response.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) and removed ai:ci-safe AI triage found no CI-safety concern (advisory) labels Oct 20, 2026

@dolfim-ibm dolfim-ibm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dolfim-ibm
dolfim-ibm merged commit 2d1d814 into main Oct 20, 2026
33 checks passed
@dolfim-ibm
dolfim-ibm deleted the cau/nemotron-parse-3.5 branch October 20, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-safe AI triage found no CI-safety concern (advisory) CLI issues related to the command line interface (CLI) docling-document enhancement New feature or request ocr table structure vlm-pipeline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants