Skip to content

fix: resolve /CIDToGIDMap streams for CIDFontType2 glyph lookup - #426

Merged
cau-git merged 3 commits into
mainfrom
fix/cid-to-gid-map-stream
Oct 27, 2026
Merged

cau-git merged 3 commits into
mainfrom
fix/cid-to-gid-map-stream

Conversation

@cau-git

@cau-git cau-git commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Problem

CIDFontType2 (TrueType) subsets that address their glyphs only through a /CIDToGIDMap stream could not be drawn from the embedded program. This is valid per ISO 32000-1 §9.7.4.2 and common in PDFs from Office-like producers.

  • Blend2D refuses these faces at load: face_res=65613 = BL_ERROR_FONT_MISSING_IMPORTANT_TABLE. The subsets have no name table, so NameImpl::init() fails. They also lack cmap and post.
  • The FreeType fallback loads them, but resolved glyphs only for an /Identity map. Without cmap/post, the name, builtin-encoding and Unicode lookups all fail.
  • Result: every glyph went to the system font resolver. On hosts with Times-like fonts the page looked fine but used a substitute face. On hosts without system fonts (e.g. distroless containers) every glyph was stroked as a blue outline box, and downstream layout models missed or fragmented tables.

Example: Amazon 2024 annual report p.36 and Q4 2024 earnings p.8. pdfium renders them correctly.

Changes

  • page_font.h: resolve_code_to_gid_map() decodes the /CIDToGIDMap stream (/CIDFontType2 only) and composes it with the /Encoding CMap, so the renderer can map the content-stream character code straight to a glyph: GID = map[CMap(code)].
    • Identity-H/V: code = CID, so the map is used as is.
    • Embedded CMap streams: the code-to-CID table init_embedded_cmap() already parses is now kept (left empty if the CMap uses usecmap).
    • Predefined CMaps (e.g. /UniJIS-UCS2-H): via the existing CMap resources (font_cid::get_code_to_cid()).
    • No complete code-to-CID table: the map stays empty and text renders through Unicode shaping, as before.
  • embedded_font_blob.h: stores the composed map; has_code_to_gid() / code_to_gid(code).
  • freetype_font_cache.h, blend2d_renderer.h: glyph resolution and char-code recovery use it; GID 0 from an explicit map counts as unmapped.
  • tests/test_unit_cid_to_gid_map.py:
    • the Identity-H case;
    • a 2 × 2 matrix of {embedded CMap, predefined /UniJIS-UCS2-H} × {FreeType path, Blend2D path}, with distinct code, CID and GID (code → CID → GID differs from code → GID).

Identity/absent maps, CFF CIDFonts and simple fonts behave as before.

Testing

  • Full suite: uv run pytest ./tests gives 410 passed, 2 skipped. Unit tests: 300 passed, 2 skipped.
  • New tests: the four code ≠ CID cases fail on the first commit of this PR (it drew the CID-indexed glyph) and pass now. The independent review probe (code 1 → CID 2 → GID 1) passes too.
  • Corpus vs pdfium, mean abs error on page 1 (lower is better). Only these 7 corpus files have CIDFontType2 fonts with a map stream; every other page of the 940 is pixel-identical to before:
file before this PR
3af8995b53546b34_0010 6.69 2.03
4796728975040539044-1 12.62 6.52
94e3fe046fa21a9c_0001 4.54 2.20
9574724313327355313-1 7.84 6.51
font_03 3.15 2.85
3040977140767198439-1 (embedded CMap + font cmap) 1.74 1.74 (unchanged)
6515533261975788184-1 (embedded CMap + font cmap) 2.23 2.23 (unchanged)
  • No system fonts: in a distroless Linux container with no fonts installed (arm64 wheel built from this branch), the Amazon annual report (91 pages) and earnings (13 pages) render with 0 outline-only text cells (277,134 and 34,128 before). Docling then finds the expected tables: 22 × 3 on p.36, 22 × 4 on p.78, and 33 × 5 on earnings p.8 (before: none; 22 × 7; 7 × 5 + 1 × 5).

Reference images

The renderer groundtruth of the 5 improved pages is refreshed, since they now draw the embedded glyphs instead of a substitute face: regression-dataset-for-docling-parse@21374211. The commit updates groundtruth/render/pages/{3af8995b53546b34_0010,4796728975040539044-1,94e3fe046fa21a9c_0001,9574724313327355313-1,font_03}.pdf.page_no_1.full_page.png and nothing else. HF_DATASET_REVISION is bumped accordingly. With the new pin, all 940 reference pages match within a mean abs error of 0.05.

Known limits / follow-ups

  • An identity /CIDToGIDMap combined with a non-identity encoding still uses the code as the GID. This is pre-existing and left unchanged so other references do not move.
  • Codes above 0xFFFF and embedded CMaps with usecmap skip the map (Unicode shaping, as before).
  • CI is unchanged: the Ubuntu test jobs already install Noto fonts, and the new tests do not depend on system fonts.

CIDFontType2 subsets that carry no cmap table address their glyphs only
through a /CIDToGIDMap stream (ISO 32000-1, 9.7.4.2). Blend2D refuses
such a face (no character mapping), and the FreeType fallback honoured
only an /Identity map, so every glyph fell through to a system font, or
to a stroked outline box where none is installed.

Decode the stream (2 bytes per CID, big-endian GID; CIDs past its end
map to GID 0) into the embedded font blob, and use it wherever the CID
identity was used: FreeType glyph resolution and Blend2D's char-code
glyph recovery. Identity and absent maps behave as before.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @cau-git, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

PeterStaar-IBM
PeterStaar-IBM previously approved these changes Oct 27, 2026
The renderer indexed the /CIDToGIDMap stream with the content-stream
character code, which is the CID only under Identity-H/V. Fonts whose
/Encoding is an embedded or predefined CMap drew the wrong glyphs:
"50-99" came out as "xso" and two greater-or-equal signs.

Compose the map with the encoding's code -> CID when the font blob is
built. Identity-H/V use the map as is; an embedded CMap reuses the
code -> CID table that the codespace scan already parses; a predefined
CMap takes it from its cmap resource. When no complete table is known
(an embedded CMap with usecmap, or any other encoding), the stream is
not applied and Unicode shaping draws the text as before.

Rename the blob helpers to has_code_to_gid()/code_to_gid(), and drop
the constructor default that no caller used. Correct the test's account
of why Blend2D rejects these subsets: they lack a name table
(BL_ERROR_FONT_MISSING_IMPORTANT_TABLE), not a character mapping.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
The five pages whose /CIDToGIDMap stream fonts now render with their
embedded glyphs get new reference images in
docling-project/regression-dataset-for-docling-parse@21374211.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
cau-git marked this pull request as ready for review October 27, 2026 18:38
@cau-git
cau-git requested a review from dolfim-ibm October 27, 2026 20:10
@cau-git
cau-git merged commit 8099e94 into main Oct 27, 2026
47 checks passed
@cau-git
cau-git deleted the fix/cid-to-gid-map-stream branch October 27, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants