Skip to content

feat(vlm): multi-pass (double/triple) parsing for Nemotron-3.5-Parse #4931

Open

Description

@geoHeil

Requested feature

Support the double-pass and triple-pass modes of nvidia/Nemotron-3.5-Parse in the nemotron_parse_v3_5 VLM preset. Docling 2.141.0 added the preset in #4897, and it does single-pass only ("Single-pass parsing only (no crop re-parsing)").

NVIDIA defines three modes (model card, docs/postprocessing.md and reference/parsebench/pipeline/README.md in the model repository):

Mode Stages ParseBench Pages/s (1× H100)
Single Full-page parse 72.74 15.42
Double Full-page parse → re-parse Table/Chart crops from the predicted boxes 74.92 11.65
Triple Full-page parse → separate detection request → re-parse Table/Chart crops (detector regions, or union with parser regions) 75.20 10.62

Double-pass gives +2.18 points for about 25% less throughput, and most of the gain is in tables. For table-heavy documents this is the most useful mode.

NVIDIA's reference implementation is nemotron_parse.inference.DocumentParser (parse_double, parse_triple(triple_regions="union" | "detector")). Its main parameters:

  • Crops: 4% + 12 px margin, JPEG quality 95, tiny crops skipped, at most 80 regions per page. Crop coordinates map back to the page.
  • Sampling: full-page and detection at temperature 0.2 / repetition penalty 1.05; crops at temperature 0.1 / repetition penalty 1.0.
  • Merge: merge_policy="alternatives" (default) keeps differing overlapping full-page and crop predictions. merge_policy="guarded" replaces a table only if the crop grid is valid and the column counts are compatible, and rejects overlapping same-class additions.

Proposal:

  1. Add a pass-mode option to the preset or its options, for example passes: Literal["single", "double", "triple"] = "single" and triple_regions: Literal["detector", "union"]. Keep the default at single-pass so current behavior does not change.
  2. Implement the crop stage in the VLM pipeline: after the page pass, take the Table/Chart boxes, crop the page image, run the model on each crop (batched, for the Transformers and API engines), and map crop coordinates back to the page.
  3. Merge with a guarded policy only, because a DoclingDocument must not contain overlapping alternative tables. Keep the page-pass result when the crop result is not valid.
  4. Triple-pass also needs the detection prompt and parser.
  5. Record the pass mode and request counts in the document provenance or metadata, so users can tell outputs apart.

Alternatives

  • Call NVIDIA's DocumentParser outside Docling, then map its output with docling/utils/nemotron_parse_v3_5_utils.py. This works with NVIDIA's vLLM server only, it duplicates the crop and merge logic, and every user must write the glue code.
  • A generic "crop re-parse" stage for all VLM presets that predict table boxes. This is more general, but it needs more design. The Nemotron-specific option can come first.

Activity

  1. self-assigned this
    on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions