Page renderer ignores a soft mask set in the graphics state (/SMask in ExtGState)

Bug

When an object is painted under a graphics state whose ExtGState sets an /SMask, the page image rendered by docling-parse paints the object fully opaque, as if the mask were not there.

This matters for docling's PDF pipeline, since docling 2.128 takes its page images from this renderer: layout detection, OCR and every figure crop work on that image. InDesign builds photo effects (drop shadows, feathered and faded edges) from exactly this kind of soft mask. On a 2025 textbook, photos came back near-black under solid grey boxes, two figures on one page were detected as one, and a text box's words sat under a grey fill.

In our earlier checks (hand-built pages compared with pdfium), constant alpha (/ca), blend modes and an image's own /SMask entry rendered correctly; the graphics-state soft mask was the one we found ignored.

Steps to reproduce

A hand-built one-page PDF: a red page, then a blue box drawn under /G1, whose soft mask is a 50% grey luminosity group, so half of the blue should come through.

from pathlib import Path

import pypdfium2 as pdfium
from docling_parse.pdf_parser import DoclingThreadedPdfParser, RenderConfig, ThreadedPdfParserConfig

content = b"1 0 0 rg 0 0 200 100 re f /G1 gs 0 0 1 rg 50 25 100 50 re f"
objects = [
    b"<< /Type /Catalog /Pages 2 0 R >>",
    b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
    b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 100] /Contents 4 0 R /Resources"
    b" << /ExtGState << /G1 << /SMask << /S /Luminosity /G 5 0 R >> >> >> >> >>",
    b"<< /Length %d >>\nstream\n" % len(content) + content + b"\nendstream",
    b"<< /Type /XObject /Subtype /Form /BBox [0 0 200 100]"
    b" /Group << /S /Transparency /CS /DeviceGray >> /Length 22 >>\n"
    b"stream\n0.5 g 0 0 200 100 re f\nendstream",
]
pdf = bytearray(b"%PDF-1.7\n")
offsets = []
for number, obj in enumerate(objects, 1):
    offsets.append(len(pdf))
    pdf += b"%d 0 obj\n" % number + obj + b"\nendobj\n"
xref = len(pdf)
pdf += b"xref\n0 %d\n0000000000 65535 f \n" % (len(objects) + 1)
pdf += b"".join(b"%010d 00000 n \n" % o for o in offsets)
pdf += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (len(objects) + 1, xref)
Path("soft-mask.pdf").write_bytes(pdf)

parser = DoclingThreadedPdfParser(ThreadedPdfParserConfig(render_config=RenderConfig()))
parser.load("soft-mask.pdf")
result = next(iter(parser.iterate_results()))
print("docling-parse:", result.get_image(scale=1).convert("RGB").getpixel((100, 50)))

page = pdfium.PdfDocument("soft-mask.pdf")[0]
print("pdfium:       ", page.render(scale=1).to_pil().convert("RGB").getpixel((100, 50)))

Output:

docling-parse: (0, 0, 255)
pdfium:        (127, 0, 128)

Expected

The masked box is composited through its soft mask, as pdfium draws it: (127, 0, 128) in the middle of the box, not opaque blue.

Environment

docling-parse 7.22.2 (also 7.22.0), pypdfium2 5.14.0
Python 3.14, Linux x86_64

Confirmed on 7.22.1 with your script, and the relevant code is unchanged on main (7.22.3). Two independent renderers agree on the expected pixel: pdfium gives (127, 0, 128) and MuPDF gives (128, 0, 126).

Root cause

pdf_resource<PAGE_GRPH>::set_transparency_parameters (src/parse/pdf_resources/page_grph.h) reduces /SMask to the soft_mask_state enum that #306 introduced and logs "soft mask is not applied". pdf_state<GRPH>::gs carries that enum through q/Q, but no render instruction has a soft-mask field, so the renderer has nothing to composite. /ca, /BM and an image's own /SMask take a different path, which is why those rendered correctly in your checks.

How common it is

In the regression corpus, 12 of 482 files carry a graphics-state soft mask. All are /S /Luminosity; /TR appears in one file and /BC in one. In those files the masks draw drop shadows, so the per-page difference against pypdfium2 is small. The InDesign case you describe, a shadow or feather object over a photo, is where the gap does real damage.

Proposed fix

The renderer already paints non-rectangular clips by filling a layer, multiplying it by an A8 coverage plane and blitting it (build_clip_mask, multiply_prgb32_by_a8). A soft mask is the same operation with a different A8 source, so the change stays inside that structure.

Parse. Keep the /SMask dictionary on pdf_resource<PAGE_GRPH> (/G, /S, /BC, /TR) with the same presence-flag convention as the other entries. pdf_state<GRPH>::gs snapshots the CTM at that point, because 11.6.5 fixes the mask's coordinate system when gs executes, not when the masked object paints.
Decode. When an instruction is emitted under a present mask, the stream decoder sub-decodes the /G form into its own pdf_render_instructions, the way widget appearance streams are handled: seeded with set_base_matrix(ctm_at_gs × /Matrix), with the group's /Resources as child resources, clipped to /BBox. One decode per ExtGState entry, shared by reference from each instruction like clip_state.
Render. Paint the mask instructions onto an offscreen PRGB32 canvas over the /BC backdrop (black by default), convert luminosity to an A8 plane through /TR (pdf_function.h already evaluates the function types), with the area outside /BBox taking the backdrop luminosity. The masked object then goes through the existing layer path: fill, multiply by the soft-mask plane and by the clip mask if there is one, blit. The A8 plane is cached per ExtGState entry for the page render, since one corpus file carries 138 masks, and the area is bounded the way build_clip_mask bounds it. Gated by a render_soft_masks flag next to render_non_rect_clip_masks, with a timing key under the render phase tree.

Scope limits, each reported with a warning as #306 does:

Transparency groups stay inlined. A soft mask inside the mask group itself is not applied.
/S /Alpha takes the layer's alpha instead of its luminosity, same path.
Knockout and isolated group attributes stay ignored, as today.

Tests: extend tests/test_unit_transparency.py with your page, a gradient mask (feathered edge), a mask whose /BBox is smaller than the object, and /SMask /None inside q/Q. The 12 corpus groundtruth renders will shift slightly, and the pypdfium2 quality benchmark should show the delta.

I can open the PR if this direction works for you.

@wittjeff

Hi @wittjeff,

Thanks for looking into this! Yes, please go ahead with the PR. Once it's up, I'm happy to test it on the documents where I ran into the problem.

The PR is up: #413. On your page it now gives (127, 0, 128), the pdfium value, and the regression corpus moves closer to pdfium on every page that changed.

Your InDesign pages are the real test, since the corpus masks are only drop shadows and small gradients. If you can build the branch, RenderConfig.render_soft_masks lets you switch the compositing on and off for a side-by-side. If building is not practical, a page or two of the textbook (or a hand-reduced copy) that I can run here would do as well.

Hi Jeff,

Thanks! I built the branch and ran it on the textbook pages. Most of the photos now look right.

What remains on my pages comes from the "transparency groups stay inlined" limit you listed: a soft mask is set inside a form XObject that is itself painted under a soft mask, and only the inner mask takes effect. I think that is why grey boxes remain behind the photos. Unlike the mask-inside-the-mask-group case,
nothing is logged for it.

On the left is pypdfium2: the statue stands on the plain page background. On the right is this PR: a grey box is drawn behind the statue.

A small page with the same structure, built with the repo's tests/pdf_builder.py:

from tests.pdf_builder import simple_page_pdf, stream_object

box = "[0 0 200 100]"
mask = stream_object(
  f"/Type /XObject /Subtype /Form /BBox {box} /Group << /S /Transparency /CS /DeviceGray >>",
  b"0.5 g 0 0 200 100 re f",
)
group = stream_object(
  f"/Type /XObject /Subtype /Form /BBox {box} /Group << /S /Transparency /CS /DeviceRGB >>",
  b"/Masked gs\n0 0 1 rg 0 0 200 100 re f\n",
)
pdf = simple_page_pdf(
  "1 0 0 rg 0 0 200 100 re f\n/Masked gs /Fm6 Do\n",
  media_box=box,
  resources="/ExtGState << /Masked << /SMask << /S /Luminosity /G 5 0 R >> >> >>"
  " /XObject << /Fm6 6 0 R >>",
  extra_objects=[mask, group],
)
open("nested-soft-mask.pdf", "wb").write(pdf)

The page is red. A group is painted on it under a 50% mask, and inside the group the same 50% mask is set again before it paints blue. pdfium shows (191, 0, 64) in the centre, a quarter of the blue; the PR shows (127, 0, 128), half of the blue.

fix(render): keep the outer soft mask inside a masked transparency group

Thank you for the test and for the small page. Your diagnosis is correct.

Cause. When a transparency group starts, the decoder moved the alpha and the blend mode in force at the Do onto the paints inside the group, but it did not move the soft mask. Thus a gs inside the group replaced the outer mask, and only the inner mask had an effect. Also, /SMask /None inside a group removed the outer mask.

Fix. Commit a2722e1 on #413:

At the Do of a transparency group, the graphics state keeps the mask in force as the group mask, and starts the group's content with no mask of its own (11.6.6).
A paint inside the group carries the intersection of the group mask and its own mask. The renderer makes the plane of the intersection as the product of the two planes. Intersections are cached per pair, the same as decoded masks.

Result.

Your page now gives (191, 0, 64) in the centre, the same as pdfium. It is now a test with three variants: no inner mask, /SMask /None inside, and a second mask inside.
On the regression corpus, 8 of 23 masked pages are now closer to pdfium than in 7.22.1, and no page is further away. The largest change is a magazine cover with the same structure as your pages (complex_invisible_fonts_01.pdf): 17.19 to 3.55 mean absolute difference.

Limit. The group is still not painted into a buffer of its own. Its mask is applied to each paint inside it. This is exact when the paints inside the group do not overlap. Where they overlap under a partial outer mask, the result is a little different from pdfium. The group alpha has the same limit. I did not add a warning for this case, because it is not a gap: the mask is applied.

Could you rebuild the branch and run the textbook pages again? If grey boxes remain, a page or a crop of the content stream will help me find the next cause.

Hi Jeff,

Thanks, that fixed it. I rebuilt the branch at a2722e1 and ran my test cases again. They all improved: the grey boxes behind the photos are gone, and the small nested page now gives (191, 0, 64) in the centre, the same as pdfium.

Thanks again for the quick fix.

fix(render): composite ExtGState soft masks (/SMask)

fix(render): keep the outer soft mask inside a masked transparency group

Navigation Menu

Repository navigation

Page renderer ignores a soft mask set in the graphics state (/SMask in ExtGState) #384

Bug

Steps to reproduce

Expected

Environment

Root cause

How common it is

Proposed fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Page renderer ignores a soft mask set in the graphics state (/SMask in ExtGState) #384

Description

Bug

Steps to reproduce

Expected

Environment

Activity

wittjeff commented on Oct 16, 2026

Root cause

How common it is

Proposed fix

dlaszlo commented on Oct 16, 2026

wittjeff commented on Oct 16, 2026

dlaszlo commented on Oct 17, 2026

wittjeff commented on Oct 17, 2026

dlaszlo commented on Oct 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions