Skip to content

feat(forms): map form fields with a geometric solver when AcroForm widgets are available - #4496

Merged
PeterStaar-IBM merged 53 commits into
mainfrom
pr-4364
Oct 19, 2026
Merged

PeterStaar-IBM merged 53 commits into
mainfrom
pr-4364

Conversation

@cau-git

@cau-git cau-git commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Adds opt-in extraction of native PDF AcroForm widgets as format-neutral, fillable fields in DoclingDocument. Enable it with PdfPipelineOptions(extract_form_fields=True) or --extract-form-fields. It is supported by the standard PDF pipeline and its deprecated legacy alias with the docling-parse backend. The option is off by default.

A FieldItem holds an optional field key and one or more fillable FieldValueItem children. A field key names values using printed text beside a widget, in its cell, or in a paragraph that contains it. Checkbox and radio option text is an option key; text that introduces several options is a shared prompt. Additional context is a field hint, stored as hint_text in predictions and emitted after the values. Here, layout label means an element class (DocItemLabel), and caption means a document caption associated with a picture, table, code block, or form.

As set out in the scope decisions, this PR makes a best-effort match between native widgets and field keys in the printed form. The solver balances matching quality with computational cost. AI inference and language analysis are outside this PR's scope. Inferring widget locations or order in PDFs without AcroForm metadata is follow-up work.

The geometric solver scores candidate key-value associations using alignment, distance, intervening content, and repeated field structures. It favors consistent key positions without making them mandatory. It supports individual options, composite fields, and paragraphs that contain several widgets. Widget geometry, type, and order come from native AcroForm metadata. Native widget order determines value order within field items. Native text supplies field values. Checkbox and radio selection uses native appearance state when available; selected or unselected checkbox items are nested inside the fillable values. Push buttons and zero-size widgets are excluded. PDF tooltips and field names do not supply key text: they can be empty, unreliable, or machine identifiers. Shared PDF field names can support option grouping, but printed text supplies the field keys.

Field assignment runs after table structure. Widgets with an established table-cell assignment are emitted inside rich table cells, including grid positions omitted by table detection. Their keys come only from text in the same cell; the table's row and column headers carry the wider association. Printed strokes and thin shape boxes can refine cell ownership within detected tables. A widget inside a detected table that cannot be assigned to a cell remains outside the table without a key. Outside detected tables, aligned headers across the other axis of a value grid can supply field hints.

Text cells used as keys are removed from body text when ownership is unambiguous; unused text in partly consumed blocks remains. Shared-prompt associations leave the prompt as ordinary text. Matching printed copies of filled values are suppressed with geometry checks. Solver limits and timeouts retain free-form values without keys while retaining established table assignments. Keying exceptions fall back to unkeyed values; failed page edits restore the original layout.

This requires PDFs with native widget metadata. Scanned or flattened forms without that metadata are unsupported. Full-page OCR skips native widget extraction. Table-cell placement requires do_table_structure=True. Raw PDF widget metadata is available on parsed pages retained with generate_parsed_pages=True, rather than in the document model. The service client does not expose this option, and DocTags does not support field items; the CLI rejects --extract-form-fields --to doctags. The dependency floors include docling-core 2.99.0 for field types, SciPy 1.9.0 for the solver, and docling-parse 7.22.2 for widget coordinates aligned with cropped-page text.

Validation at 4b4e1feb: all 78 tests in the command below passed, including field-hint materialization. The real acroform_sample.pdf also matched committed JSON, Markdown, and page metadata. make validate passed in an isolated checkout. The sample is included in the PDF end-to-end test; only its unsupported DocTags comparison is skipped.

DOCLING_DEVICE=cpu uv run --frozen --extra format-pdf-docling pytest \
  tests/test_acroform_keying.py tests/test_form_extraction.py \
  tests/test_cli_form_fields.py tests/test_backend_docling_parse.py

docling form.pdf --extract-form-fields --to json

cau-git and others added 30 commits September 2, 2026 16:34
Extract native interactive PDF widgets from docling-parse parsed pages and
materialize them as keyless, format-neutral fillable field values.

A small PDF form-field stage runs after OCR and before layout postprocessing —
the only point where both the raw layout FORM clusters and the parsed-page
widgets are available. It normalizes each widget into a format-neutral field
prediction, binds it to a raw FORM cluster (with a page-local fallback region
for unmatched widgets), and never mutates the layout prediction or infers
labels. Page assembly turns matched FORM clusters into FieldRegionElements and
emits fallback regions; reading order materializes them as
FieldRegionItem/FieldItem/FieldValueItem in native widget order.

Raw AcroForm state (/Ff, /AS, field names) stays on SegmentedPdfPage.widgets;
nothing PDF-specific enters DoclingDocument or add_field_value(). The legacy
late-pypdfium2 FormItem+GraphData path (form_utils.py) is removed.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Document experiment findings on native widget state vs. visual layout
extraction colliding for filled form fields, alongside form field
model and pipeline updates.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Materialize a non-pushbutton /Btn widget as an empty-text FieldValueItem
that nests a CHECKBOX_SELECTED/UNSELECTED child, instead of encoding state
as a synthesized "checked"/"unchecked" value string. The value text stays
empty so the doclang serializer inlines the <checkbox> token into <value>.
Keeps /AS as the authoritative state and preserves the raw value in orig.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…abel

Reconcile the native /Btn widget with the layout CHECKBOX_* cluster it
overlaps: match on widget-rect-in-cluster overlap, lift the cluster's
option label (mark glyph stripped) onto the nested checkbox child, and
promote the cluster out of the plain-text stream so it is no longer a
disjoint duplicate. /AS stays authoritative for the selection state,
overriding the glyph classifier. Unselected boxes with no overlapping
cluster keep the widget-only path.

Verified on f1040s1_filled: the checked option serializes as
<value class="fillable"><checkbox class="selected"/>4797</value> with the
duplicate cluster removed.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…ts bbox as prov

Drop the mark-glyph sanitation; the checkbox cluster's text is now lifted as-is
onto the value, and the matched cluster bbox becomes the value's prov so the box
wraps the whole checkbox+label rather than the tiny widget rect.
…ield_item

Introduce an item layer between field region and value: FieldRegionPrediction
now holds FieldItemPrediction items, each a keyless single value (today's flat
shape, unchanged) or a keyed group. A widget with no smaller enclosing container
than a text paragraph is matched to that paragraph -- the smallest enclosing
cluster wins over the page-wide FORM -- and widgets sharing one paragraph
accumulate under a single key (the paragraph text/bbox), with the paragraph
promoted out of the plain-text body. Reading order materializes the key via
add_field_key.

IRS Schedule 1 line 7 ("check here [] and enter amount repaid: 1221.00") now
emits one field_item: key + checkbox-unselected value + fillable "1221.00",
instead of two disjoint value-only fields.
…ld_region

An inlined-widget paragraph was pulled out into its own field_region, detaching
it from the list (and enclosing form region) it belonged to. Keep the paragraph
cluster in the body and attach the keyed item onto its TextElement; reading order
now emits a field_item in the paragraph's own place -- inside its list when it is
a list item, else under the current parent -- with its text as the key. No
separate field_region is created for the inline case.

On IRS Sch.1, line 7's field_item now sits inside its list within the form's
existing field_region instead of a standalone region ahead of it.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
A widget of zero height or width is an artifact (Well-Tagged PDF 1.0,
8.9.2.4.13) and a push button holds no field value; both were being
emitted as empty fillable values. Skip them before normalization.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Pure geometry for the widget-to-key binding: _gap (rectangle edge-gap,
direction logic for free), _precedes (no-crossing reading-order guard,
line-height row band), and _match_labels (monotonic forward pass, 1:1).
Not yet wired into the form-field model -- this is the isolated,
fixture-free core so the minimization property is unit-tested on its own.

Tests cover the label-steal and crossing cases that the reframing exists
to remove, plus 1:1 shared-header exhaustion and the gap cap.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Wire the order-preserving binding into the form-field model: after the
per-widget triage, run _match_labels over the keyless FORM-matched and
unmatched widgets (in widget.index order) against the body label pool
(text clusters minus consumed paragraph containers). A bound label becomes
the field item's key_text/key_bbox and is promoted out of the body via the
existing _drop_clusters machinery, so no new plumbing on the assembly side.

Matched checkboxes already carry their option label and stay untouched.
Adds the two calibration knobs (gap cap, row band, in line-heights) and a
median-line-height text scale; the 15-form harness sets their values.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…der gate

Two bugs found by inspecting gst494 output against the rendered page:

- The keyless guard tested `checkbox_label is None`, but the field defaults
  to "" -- so no widget ever reached the binding pass and the feature was
  dead. Gate on falsy instead.
- `_precedes` gated the no-crossing guard on left/right order within a fuzzy
  row band. Binding a right-side label (a trailing "RT") poisoned the frontier
  so every left-side label in the next row was wrongly ruled "before" it,
  blocking Contact person / Title / Telephone number (all gap 0). The guard is
  now purely vertical: block only a strictly-higher row. No left-to-right
  assumption; 1:1 pairing is handled by per-label consumption alone.

On gst494 this takes binding from 0 real keys to 11/12 correct (the one miss
is a garbage layout text cluster, not a binding error). Updates the pipeline
test: the text field now correctly binds its detached "Full name:" left label.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Capture what the fixture inspection taught: the two bugs that made the
binding pass dead, the _precedes correction (row-only, no left-order gate),
and the accepted table-form column-crossing ceiling (rf-1084s). Add a
ponytail note at the binding site pointing at the handoff doc.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Add §7: what was actually built (files, functions, knobs, the keyless
predicate and _precedes correction), the 11-fixture results table, the
gst494 deep check (11/12, the ULUL garbage-cluster miss), and every
known-broken case tagged with its fixing phase (rf-1084s column crossing
= Phase 4, layout-noise filter, un-swept cap/row-band). Note the 15-vs-11
corpus-count discrepancy to reconcile.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
§8: resume state (branch/PR target, test command, corpus-count reconcile),
the headless judge recipe (unzip .dclx, parse document.xml locations, Read
the page png -- replacing the viewer-only instruction the agent cannot use),
the two debugging patterns that found the bugs, and the guardrails not to
re-introduce (no left tiebreak, no premature DP/RO fence).

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Sweep _LABEL_GAP_CAP_LINES / _LABEL_ROW_BAND_LINES against the 11-form
corpus by replaying the pure _match_labels offline over a cap×band grid
(capture once, no re-conversion). Cap is the correctness knob: 2.0 is the
lower edge of the [2.0, 3.0] plateau that scores 12/12/0 on gst494; below
it under-binds, at 2.5+ the corpus flips to wrong labels. Row-band is
forgiving over [1.0, 2.0]. Keep both constants, no code change.

Add §7.6 with the scored ground-truth method (frozen GST494_GT map, read
off the raster since AcroForm has no widget→label oracle) so the verdict
is rerunnable, and flip the §7.4 un-swept bullet. Reconcile the corpus
count: 11 PDFs in the dir, not 15 — stale name, no convert failures.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
At form_field time (which runs before the table model) a table's row/column
keys are absorbed as TABLE children, never free-standing labels in the binding
pool -- probed to be 0 in-table standalone labels on every widget-containing
table across the 11-form corpus. So a widget inside a table reached OUTSIDE it
for the nearest free-text and mis-bound a section header (italy SEZIONE/QUADRO,
rf-1177s) or a wrong-column neighbour onto a body cell.

Rule: partition the keyless widgets and the label pool by containing table
(pure _table_of: smallest enclosing region by center-in-rect) and run the
unchanged _match_labels per group, so binding never crosses a table boundary.
A cell that carries its own key+value still binds (label in the same table);
a body cell whose only nearby label is an out-of-table header stays keyless.

This reverses the original §7.4 sketch (feed table cells INTO the pool): the
cell grid does not exist yet at this stage, so the correct move is a boundary,
not a merge. gst494 stays byte-identical (all 12 ground-truth binds intact);
italy drops 11 header->body mis-binds; rf-1177s drops its header mis-bind;
rf-1125s loses 3 option-label binds (accepted). Details in §9.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Preserve the reviewed local labels for all 946 retained native widgets,
including explicit no-label and ambiguous dispositions and 503 table
exclusions. Include 19 frozen page images/layout snapshots, native identities,
visual review overlays, the evaluator and the design handoff.

This is a reconstructed checkpoint from saved working files, not a claim
of contemporaneous historical commits. The evaluator imports the optimizer
introduced by the next checkpoint; the stdlib identity validator runs here.
Original PDFs remain external and hash-verified; replay needs no conversion.
Exclude only frozen generated snapshot JSON from the source line-count
limit so recorded byte hashes remain intact.

Validation: make validate; reference validator verified 11 PDF hashes,
19 pages, 970 native identities and 128 explicit table-cell memberships.
Production code is unchanged.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Preserve the accepted offline MILP prototype, focused tests, baseline
reports, coverage ablation and failure audit. Keep the pre-coverage source
and change.diff alongside the accepted implementation for direct comparison.

Use intersection divided by widget area for inline-clause containment at
80 percent coverage; keep detected table/cell ownership strict and retain
exact page-wide native value order. The reviewed replay improves from
256 to 276 correct local labels with no previous correct labels lost.
Caption fragments and missed/truncated tables remain upstream defects.

Validation: 13 focused tests; all 19 pages replay optimally with 276 correct,
85 wrong and 73 unassigned labels; make validate. Generated report/diagnostic
JSON is exempted from the source line-count limit to preserve saved evidence.
This is an offline checkpoint, not production integration.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Preserve conditional question/caption configurations, structural tests,
complete-assignment uncertainty checks and the raw/filtered replay reports.
Include the linear-reward control and the subsequent role investigation.

Reject this version for promotion: it fixes RC7190 but displaces correct
GST111/F1120 captions and replaces the 21-option language group with groups
seeded by caption fragments. A linear reward restores that group but merges
RC7190's two questions. The README records the tradeoffs and failure evidence.

Validation: 10 structural tests; frozen comparison verifies native values,
order, all 503 table exclusions and reference/input hashes across 19 pages;
make validate. Keep the accepted optimizer unchanged.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Keep multiple compatible captions available during optimization and use
geometric option-row context for question proposals. Let nested groups
reference their parent option's caption without consuming its text twice.
Retain competing boundaries and check caption alternatives even when the
question and membership agree. Conditional rewards can retain primary
associations that would otherwise be pruned as costlier than abstention.

Across 19 pages, fix one wrong local association without losing any previous
correct labels. Recover the full language group and keep RC7190's questions
separate. Sami caption selection and the outer Yes/No boundary remain
unresolved; retain this version as an experiment, not production code.

Validation: 30 focused tests; all 19 raw and uncertainty replays optimal;
unchanged native values/order and 503 exclusions; text/name replacement
leaves all group proposals unchanged; source hashes and make validate.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
It was never intended to be committed.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…o codex/acroform-keying-snapshots

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…ield stage

Replace the greedy label binding with a solver that assigns each AcroForm
widget its visible caption from page geometry (MILP over candidate
captions). Symmetry rules make repeated rows and columns of fields read
their captions from the same side, and an inline clause no longer claims
checkboxes that carry their own option caption. The form_field stage now
runs after table structure, and the unused checkbox_label is replaced by
context_text (the aligned header of a value grid), emitted as a hint in
reading order.

A widget inside a detected table goes into its table cell as a rich
table cell holding the field, keyed only by text printed in that same
cell. Printed rules (thin painted shapes, read through the new
PdfPageBackend.get_thin_shape_boxes) bound the printed cell when the
detected grid does not match it; cells TableFormer left out are inserted.

Field extraction is opt-in via --extract-form-fields (off by default), so
default conversions are unchanged. Also place AcroForm widgets in the
crop-box frame of their text, which misplaced widgets on pages with a
non-zero CropBox origin, and remove the experimental keying prototype,
replay script, snapshot docs and annotation data from the tree.

Signed-off-by: Daniel DeAlcala <danieldealcalavalcarcel@gmail.com>
…leftovers

Pass the page's widgets, layout clusters and table cells to the keying
directly instead of through a snapshot copy, and clean up what the
experiments left behind:

- remove Assignment.objective, the null_cost/time_limit parameters and
  their validation, the duplicate-index guard and the unused re-exports
- share the FORM-region placement between the keyed and the keyless plan,
  and the value/hint materialization in the reading-order model
- define the rule thickness once; the backend query takes it as argument
- type the keep parameter as frozenset[int]
- cover acroform_sample.pdf in the e2e conversion test with reference
  output (DocTags excluded: its export does not support field items)

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

# Conflicts:
#	docling/backend/docling_parse_backend.py
#	docling/cli/main.py
#	pyproject.toml
#	tests/test_backend_docling_parse.py
#	tests/test_e2e_conversion.py
#	uv.lock
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
A checkbox block's text is now a strongly preferred candidate in the
search instead of a key fixed beforehand, so its text can still key
another field of the row and the checkbox keeps its group prompt.
Blocks without letters are skipped, a checkbox counts as inside a block
at the host coverage, and a hosted checkbox's key is located by its text.

Signed-off-by: Daniel DeAlcala <danieldealcalavalcarcel@gmail.com>
@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI and removed ai:ci-needs-care AI triage: check CI-safety findings before you approve CI labels Oct 16, 2026
A checkbox in its checkbox block is again keyed by the block alone: it
joins no other candidate, group prompts included, and has no blank
option, so its block text is its only key and needs no prior. The text
itself stays in the search under the sharing rules, so it can still key a
field of the same row. This stops a group of boxed checkboxes from taking
a nearby field's label as its prompt, and a boxed checkbox from being
keyed by a group while its paragraph prints it a second time.

A group of one boxed and one plain option now has no prompt, so the
question keys the plain option; the test records this limit.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI and removed ai:ci-needs-care AI triage: check CI-safety findings before you approve CI labels Oct 16, 2026
A boxed checkbox takes part in no association but its block's, so its
other candidates are left out of the search; they also fed add_hints,
and a boxed checkbox in a grid lost its column header as hint. Hints are
chosen after the search and never move a key, so they now come from all
candidates.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI and removed ai:ci-needs-care AI triage: check CI-safety findings before you approve CI labels Oct 16, 2026
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

# Conflicts:
#	docling/models/stages/reading_order/readingorder_model.py
@github-actions github-actions Bot removed the ai:ci-needs-care AI triage: check CI-safety findings before you approve CI label Oct 19, 2026
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions github-actions Bot added the ai:ci-needs-care AI triage: check CI-safety findings before you approve CI label Oct 19, 2026
Replace the page-wide field_regions and table_fields lists with one
FormFieldPrediction on PagePredictions, with items by placement route:
inline host paragraph, FORM cluster, and page-wide. Write table cell
items once, on the Table in table_map. Store the checkbox state as its
DocItemLabel, and use an isinstance check in place of a getattr probe.
Conversion output does not change.

Tests read the new structures. The FORM region test runs assembly on the
layout the form stage leaves, and two tests no longer use monkeypatch.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI and removed ai:ci-needs-care AI triage: check CI-safety findings before you approve CI labels Oct 19, 2026
The --extract-form-fields and --to doctags check reads only options, so
run it with the other option checks, before the inputs are resolved.
Move its test into test_cli.py; it uses a missing source, so it never
reads a file or starts a conversion.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Move the checkbox-block tests that check only keys into
test_acroform_keying.py, where they call assign() directly, and the
widget-contract test into test_backend_docling_parse.py. Merge two pairs
of tests that check one rule with different inputs. Remove two tests
that only locked in known wrong output. Covered form code is unchanged.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI and removed ai:ci-needs-care AI triage: check CI-safety findings before you approve CI labels Oct 19, 2026

@PeterStaar-IBM PeterStaar-IBM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@dolfim-ibm dolfim-ibm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@PeterStaar-IBM
PeterStaar-IBM merged commit 2f13b3d into main Oct 19, 2026
53 checks passed
@PeterStaar-IBM
PeterStaar-IBM deleted the pr-4364 branch October 19, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-needs-care AI triage: check CI-safety findings before you approve CI CLI issues related to the command line interface (CLI) enhancement New feature or request pdf parsing PDF issue related to docling-parse pdf PDF issue (except docling-parse) reading_order table structure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants