Skip to content

Path input with an upper-case extension (.VTT, .ADOC, .EML, .HTML, ...) is not recognised, while the same DocumentStream is #4429

Closed

Description

@JoeyTan21

Bug

_DocumentConversionInput._guess_format lower-cases the file extension when the input is a DocumentStream (added in #1609), but not when the input is a Path. For formats that filetype cannot identify from magic bytes and that are therefore identified by their extension alone (vtt, asciidoc, tex, eml, html without a doctype, boxnote, json, mhtml), a path with an upper-case suffix resolves to no format and the conversion fails with File format not allowed. The identical bytes passed as a DocumentStream with the same file name convert fine.

The CLI collects files from a directory case-insensitively (_name_matches_format in docling/cli/main.py), so docling ./folder picks up NOTES.VTT and then rejects it for the same reason.

Steps to reproduce

import io, tempfile
from pathlib import Path
from docling.datamodel.base_models import DocumentStream, InputFormat
from docling.datamodel.document import _DocumentConversionInput
from docling.document_converter import DocumentConverter

data = b"WEBVTT\n\n00:00:01.000 --> 00:00:02.000\nHello world\n"
path = Path(tempfile.mkdtemp()) / "doc.VTT"
path.write_bytes(data)

dci = _DocumentConversionInput(path_or_stream_iterator=[])
print(dci._guess_format(path))                                              # None
print(dci._guess_format(DocumentStream(name="doc.VTT", stream=io.BytesIO(data))))  # InputFormat.VTT

DocumentConverter(allowed_formats=[InputFormat.VTT]).convert(path)
# docling.exceptions.ConversionError: Conversion failed for: .../doc.VTT with status: skipped.
# Errors: File format not allowed: .../doc.VTT

Renaming the file to doc.vtt makes the conversion succeed. The same happens for .ADOC, .TEX, .EML, .HTML (no doctype), .BOXNOTE, .JSON and .MHTML; .MD/.TXT/.CSV are unaffected because they have a content-based fallback.

Cause: docling/datamodel/document.py, Path branch of _guess_format, obj_ext = obj.suffix[1:] is passed to _mime_from_extension without .lower(), whereas the DocumentStream branch calls _mime_from_extension(obj_ext.lower()). Lower-casing the suffix on the path branch fixes it; I can open a PR with a regression test.

Docling version

2.130.0 (main @ eb17f57)

Python version

3.12

Disclosure: found, reproduced and the draft fix tested locally with the help of an AI coding agent (Claude Code); reviewed before filing.

Fixes #N

For text formats that are identified by their extension alone (vtt, adoc, tex, eml, html without a doctype, boxnote, json, mhtml), _guess_format returned None for a Path whose suffix is upper-case, so DocumentConverter.convert(Path("notes.VTT")) failed with File format not allowed. The same bytes passed as a DocumentStream named notes.VTT converted fine, because #1609 lower-cased the extension on the stream branch only. The CLI also matches directory entries case-insensitively, so such files were collected and then rejected.

This lower-cases the suffix on the path branch as well, and adds a parametrized test that checks a path and a stream with the same upper-case name resolve to the same format.

Checklist:

  • Documentation has been updated, if necessary. (no user-facing docs change)
  • Examples have been added, if necessary. (n/a)
  • Tests have been added, if necessary.

Disclosure: found, reproduced and the draft fix tested locally with the help of an AI coding agent (Claude Code); reviewed before filing.

🤖 Generated with Claude Code

面试 5 句话

  1. Bug:docling 转换 notes.VTT 这种大写扩展名的文件路径会报 "File format not allowed",但同一份内容用 DocumentStream 传进去就没问题。
  2. 根因:格式推断 _guess_format 有 Path 和 Stream 两条分支,一年前有人修过 Stream 分支加了 .lower(),Path 分支漏了;靠 magic bytes 认不出的纯文本格式只能靠扩展名,大写查表就落空。
  3. 修法:在 Path 分支把 obj.suffix[1:] 小写化,和 Stream 分支对齐,一行改动;加了一个参数化测试同时断言 Path 和 Stream 结果一致。
  4. 验证:写了 8 种格式的对比脚本,修前 8 处不一致、端到端抛异常,修后全一致;把修复 stash 掉新测试 4 个 case 全红,跑 prek 全套钩子(ruff/ty/tach)全绿。
  5. 教训:两条并行分支干同一件事时要么抽成一个函数要么两边同步改;review 别人补丁时要问"另一条路径也修了吗";另外 CLI 已经大小写不敏感地收文件,下游却拒绝,说明跨模块契约没对齐。

下一步命令

# 1. 提 issue
gh issue create -R docling-project/docling --title "Path input with an upper-case extension (.VTT, .ADOC, .EML, .HTML, ...) is not recognised, while the same DocumentStream is" --label bug --body-file ISSUE_BODY.md
# 2. fork + push
gh repo fork docling-project/docling --clone=false
cd <scratch>/docling/docling && git remote add fork https://github.com/JoeyTan21/docling.git
git push fork fix/uppercase-extension-path
# 3. PR(先把 Fixes #N 填上)
gh pr create -R docling-project/docling --base main --head JoeyTan21:fix/uppercase-extension-path \
  --title "fix: detect the format of path inputs with an upper-case extension" --body-file PR_BODY.md

如果 scratchpad 丢了:重新 clone,git checkout -b fix/uppercase-extension-path && git am 0001-*.patch。

落选候选

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions