Repository navigation
feat(vlm): multi-pass (double/triple) parsing for Nemotron-3.5-Parse #4931
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Requested feature
Support the double-pass and triple-pass modes of nvidia/Nemotron-3.5-Parse in the nemotron_parse_v3_5 VLM preset. Docling 2.141.0 added the preset in #4897, and it does single-pass only ("Single-pass parsing only (no crop re-parsing)").
NVIDIA defines three modes (model card, docs/postprocessing.md and reference/parsebench/pipeline/README.md in the model repository):
| Mode | Stages | ParseBench | Pages/s (1× H100) |
|---|---|---|---|
| Single | Full-page parse | 72.74 | 15.42 |
| Double | Full-page parse → re-parse Table/Chart crops from the predicted boxes | 74.92 | 11.65 |
| Triple | Full-page parse → separate detection request → re-parse Table/Chart crops (detector regions, or union with parser regions) |
75.20 | 10.62 |
Double-pass gives +2.18 points for about 25% less throughput, and most of the gain is in tables. For table-heavy documents this is the most useful mode.
NVIDIA's reference implementation is nemotron_parse.inference.DocumentParser (parse_double, parse_triple(triple_regions="union" | "detector")). Its main parameters:
- Crops: 4% + 12 px margin, JPEG quality 95, tiny crops skipped, at most 80 regions per page. Crop coordinates map back to the page.
- Sampling: full-page and detection at temperature 0.2 / repetition penalty 1.05; crops at temperature 0.1 / repetition penalty 1.0.
- Merge:
merge_policy="alternatives"(default) keeps differing overlapping full-page and crop predictions.merge_policy="guarded"replaces a table only if the crop grid is valid and the column counts are compatible, and rejects overlapping same-class additions.
Proposal:
- Add a pass-mode option to the preset or its options, for example
passes: Literal["single", "double", "triple"] = "single"andtriple_regions: Literal["detector", "union"]. Keep the default at single-pass so current behavior does not change. - Implement the crop stage in the VLM pipeline: after the page pass, take the Table/Chart boxes, crop the page image, run the model on each crop (batched, for the Transformers and API engines), and map crop coordinates back to the page.
- Merge with a guarded policy only, because a
DoclingDocumentmust not contain overlapping alternative tables. Keep the page-pass result when the crop result is not valid. - Triple-pass also needs the detection prompt and parser.
- Record the pass mode and request counts in the document provenance or metadata, so users can tell outputs apart.
Alternatives
- Call NVIDIA's
DocumentParseroutside Docling, then map its output withdocling/utils/nemotron_parse_v3_5_utils.py. This works with NVIDIA's vLLM server only, it duplicates the crop and merge logic, and every user must write the glue code. - A generic "crop re-parse" stage for all VLM presets that predict table boxes. This is more general, but it needs more design. The Nemotron-specific option can come first.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity